vllm.cpp
a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-aware scheduling, ...)
What is vllm.cpp?
a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-aware scheduling, ...)
Overview of vllm.cpp
What it is
vllm.cpp is a community-oriented inference engine written in C++20. It is designed as a 1:1 port of vLLM, providing continuous batching and paged KV cache, while adding features such as GGUF support, RadixAttention, and cache-aware scheduling. The engine runs on CUDA, CPU, Metal, and Vulkan, and does not require Python or PyTorch at inference time. It is developed by the LocalAI team and is independent and unofficial, not affiliated with the vLLM project.
Key features
- Continuous batching and block-paged KV cache - Supports CUDA, CPU, Metal, and Vulkan backends - GGUF model format support with on-the-fly quantization - RadixAttention and LPM cache-aware admission scheduling - Speculative decoding and jump-forward decoding - C ABI for embedding (versioned, stable interface)
- No Python or PyTorch dependencies at runtime
Use cases
- Deploying LLM inference servers with high throughput and low latency - Running large language models on diverse hardware (GPU, CPU, Apple Silicon)
- Embedding inference into C++ applications via the C ABI - Experimenting with advanced scheduling and caching strategies
Getting started
See the repository README for installation and quick start instructions.
License and pricing
Licensed under Apache-2.0. The project is open source and free to use.
Based on the project's public README and repository metadata.