lm-proxy

OpenAI-compatible HTTP LLM proxy / gateway for multi-provider inference (Google, Anthropic, OpenAI, PyTorch). Lightweight, extensible Python/FastAPI—use as libr

What is lm-proxy?

OpenAI-compatible HTTP LLM proxy / gateway for multi-provider inference (Google, Anthropic, OpenAI, PyTorch). Lightweight, extensible Python/FastAPI—use as library or standalone service.

Overview of lm-proxy

What it is

LM-Proxy is a lightweight, OpenAI-compatible HTTP proxy server / gateway that unifies access to multiple Large Language Model providers and local inference through a single, standardized API endpoint. It is built with Python, FastAPI and MicroCore. It works as a drop-in replacement for OpenAI's API, allowing switching between cloud providers and local models without modifying existing client code. It supports real-time token streaming, secure Virtual API key management, and can be used both as an importable Python library and as a standalone HTTP service.

Key features

  • Provider Agnostic: Connect to OpenAI, Anthropic, Google AI, local models, and more using a single API
  • Unified Interface: Access all models through the standard OpenAI API format
  • Dynamic Routing: Route requests to different LLM providers based on model name patterns
  • Stream Support: Full streaming support for real-time responses
  • API Key Management: Configurable API key validation and access control
  • Easy Configuration: Simple TOML/YAML/JSON/Python configuration files for setup
  • Extensible by Design: Minimal core with clearly defined extension points, enabling seamless customization and expansion without modifying the core system

Use cases

LM-Proxy can be used in production applications or for experimenting with different models. It eliminates integration complexity and keeps codebase provider-agnostic. It can be used as a library or standalone service. It supports proxying to Anthropic API, Google Gemini via Vertex AI or Google AI Studio, and local PyTorch-based inference.

Getting started

Requirements: Python 3.11, 3.12, or 3.13.Installation:pip install lm-proxyFor optional dependencies (Anthropic, Google):pip install lm-proxy[anthropic,google]orpip install lm-proxy[all]Quick Start:1. Create a config.toml file (see README for example).2. Start the server:lm-proxyAlternatively:python -m lm_proxy3. Use with any OpenAI-compatible client (see README for Python example).Docker:docker run -d --name lm-proxy -p 8000:8000 -v ./config.toml:/app/config.toml:ro --env-file .env lmproxy/lmproxy:edge

License and pricing

License: MIT. Pricing: open source (not stated otherwise).

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/nayjestlm-proxy.svg)](https://yunodb.com/tools/nayjestlm-proxy?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/nayjestlm-proxy?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/nayjestlm-proxy.svg" alt="Featured on YunoDB" height="20"></a>