ai-gateway
Unified AI Gateway for 30+ LLMs (OpenAI, Anthropic, Bedrock, Azure etc) with Caching, Guardrails, A/B test & cost controls. Go-native Fastest & Scalable AI Gate
What is ai-gateway?
Unified AI Gateway for 30+ LLMs (OpenAI, Anthropic, Bedrock, Azure etc) with Caching, Guardrails, A/B test & cost controls. Go-native Fastest & Scalable AI Gateway LiteLLM & Kong AI Gateway alternative.
Overview of ai-gateway
What it is
Ferro Labs AI Gateway is an open-source, OpenAI-compatible LLM gateway written in Go. It routes LLM requests across 30 providers (including OpenAI, Anthropic, Bedrock, Azure) via a single API. It is designed for high throughput and low latency, with a single static binary that requires no external services and has a base memory of 32 MB. The project is licensed under Apache 2.0.
Key features
- Supports 30 providers and 2,500+ models through one OpenAI-compatible API.
- 8 routing strategies: single, fallback, load balance, least latency, cost-optimized, content-based, A/B test, conditional.
- Built-in guardrails and plugins: word filter, regex guard, PII redaction, secret scanning, prompt-injection shield, response schema validation, token/message limits, response cache, rate limiting, per-key budgets, request logging.
- MCP (Model Context Protocol) support: connects to stdio and Streamable HTTP tool servers, injects tools into chat completions.
- Observability: OpenTelemetry tracing and Prometheus metrics.
- Built-in operations dashboard served at
/for traffic, spend, provider health, request logs, audit trail. - Achieved 13,925 RPS at 1,000 concurrent users in v1.0.0 benchmark.
Use cases
- Centralizing access to multiple LLM providers behind a single API endpoint.
- Implementing fallback, load balancing, or A/B testing across different models and providers.
- Adding guardrails such as content filtering, PII redaction, and rate limiting to LLM requests.
- Using MCP to connect to external tool servers and enable agentic tool calls.
- Monitoring and managing LLM usage, costs, and performance through the built-in dashboard.
Getting started
Install via one of the following methods: bash # macOS, Linux curl -fsSL https://get.ferrolabs.ai | sh # Homebrew brew install ferro-labs/tap/ferrogw # Docker docker run -p 8080:8080 ghcr.io/ferro-labs/ai-gateway:latest Then configure and run: bash export OPENAI_API_KEY=sk-your-key ferrogw init # writes config.yaml, prints master key export GATEWAY_CONFIG=./config.yaml export MASTER_KEY=fgw_your-master-key ferrogw serve # starts server on :8080 For Docker, set environment variables and run the container.
License and pricing
The project is open source under the Apache 2.0 license. Pricing is not stated beyond being open source; the managed cloud is listed as 'Coming Soon'.
Based on the project's public README and repository metadata.