lm-proxy
OpenAI-compatible HTTP LLM proxy / gateway for multi-provider inference (Google, Anthropic, OpenAI, PyTorch). Lightweight, extensible Python/FastAPI—use as libr
What is lm-proxy?
OpenAI-compatible HTTP LLM proxy / gateway for multi-provider inference (Google, Anthropic, OpenAI, PyTorch). Lightweight, extensible Python/FastAPI—use as library or standalone service.
Overview of lm-proxy
What it is
LM-Proxy is a lightweight, OpenAI-compatible HTTP proxy server / gateway that unifies access to multiple Large Language Model providers and local inference through a single, standardized API endpoint. It is built with Python, FastAPI and MicroCore. It works as a drop-in replacement for OpenAI's API, allowing switching between cloud providers and local models without modifying existing client code. It supports real-time token streaming, secure Virtual API key management, and can be used both as an importable Python library and as a standalone HTTP service.
Key features
- Provider Agnostic: Connect to OpenAI, Anthropic, Google AI, local models, and more using a single API
- Unified Interface: Access all models through the standard OpenAI API format
- Dynamic Routing: Route requests to different LLM providers based on model name patterns
- Stream Support: Full streaming support for real-time responses
- API Key Management: Configurable API key validation and access control
- Easy Configuration: Simple TOML/YAML/JSON/Python configuration files for setup
- Extensible by Design: Minimal core with clearly defined extension points, enabling seamless customization and expansion without modifying the core system
Use cases
LM-Proxy can be used in production applications or for experimenting with different models. It eliminates integration complexity and keeps codebase provider-agnostic. It can be used as a library or standalone service. It supports proxying to Anthropic API, Google Gemini via Vertex AI or Google AI Studio, and local PyTorch-based inference.
Getting started
Requirements: Python 3.11, 3.12, or 3.13.Installation:pip install lm-proxyFor optional dependencies (Anthropic, Google):pip install lm-proxy[anthropic,google]orpip install lm-proxy[all]Quick Start:1. Create a config.toml file (see README for example).2. Start the server:lm-proxyAlternatively:python -m lm_proxy3. Use with any OpenAI-compatible client (see README for Python example).Docker:docker run -d --name lm-proxy -p 8000:8000 -v ./config.toml:/app/config.toml:ro --env-file .env lmproxy/lmproxy:edge
License and pricing
License: MIT. Pricing: open source (not stated otherwise).
Based on the project's public README and repository metadata.