olla

High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local an

What is olla?

High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends.

Overview of olla

What it is

Olla is a high-performance, low-overhead, low-latency proxy and load balancer for managing LLM infrastructure. It intelligently routes LLM requests across self-hosted inference nodes with a wide variety of natively supported endpoints and is extensible to support others. Olla provides model discovery and unified model catalogues within each provider, enabling seamless routing to available models on compatible endpoints. It works alongside API gateways like LiteLLM or orchestration platforms like GPUStack, focusing on making existing LLM infrastructure reliable through intelligent routing and failover. Users can choose between two proxy engines: Sherpa for simplicity and maintainability, or Olla for maximum performance with advanced features like circuit breakers and connection pooling. A single CLI application and config file is all that is needed to run Olla.

Key features

  • Smart Load Balancing: Priority-based routing with automatic failover and connection retry
  • Sticky Sessions: KV-cache-aware affinity routing that pins multi-turn conversations to the same backend
  • Smart Model Unification: Per-provider unification plus OpenAI-compatible cross-provider routing
  • Dual Proxy Engines: Sherpa (simple) and Olla (high-performance)
  • Advanced Filtering: Profile and model filtering with glob patterns for precise control
  • Health Monitoring: Continuous endpoint health checks with circuit breakers and automatic recovery
  • Intelligent Retry: Automatic retry on connection failures with immediate transparent endpoint failover
  • Self-Healing: Automatic model discovery refresh when endpoints recover
  • Request Tracking: Detailed response headers and statistics
  • Anthropic Messages API: Passthrough for backends with native support; automatic translation for others
  • Backend Authentication: Outbound auth for local backends requiring API keys, bearer tokens, or basic auth
  • Production Ready: Rate limiting, request size limits, optional CORS for browser clients, configurable proxy/server timeouts, graceful shutdown
  • High Performance: Sub-millisecond endpoint selection with lock-free atomic stats
  • LLM-Optimised: Streaming-first design with optimised timeouts for long inference
  • Admin Dashboard: Read-only fleet overview embedded in the binary at /internal/ui/, on by default and loopback-only

Use cases

Olla is designed for managing LLM infrastructure that involves multiple self-hosted inference backends. It can be used to intelligently route requests across different backends, automatically failover when endpoints become unhealthy, and unify model discovery across providers. It is suitable for environments where reliability, low latency, and efficient resource utilization are critical, such as production deployments of LLM services. Olla works alongside existing API gateways or orchestration platforms to enhance their capabilities.

Getting started

bash# Download latest release (auto-detects your platform)bash

License and pricing

Olla is open source under the Apache-2.0 license. Pricing is not stated beyond being open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/thushanolla.svg)](https://yunodb.com/tools/thushanolla?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/thushanolla?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/thushanolla.svg" alt="Featured on YunoDB" height="20"></a>