vllm.cpp

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-aware scheduling, ...)

What is vllm.cpp?

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-aware scheduling, ...)

Overview of vllm.cpp

What it is

vllm.cpp is a community-oriented inference engine written in C++20. It is designed as a 1:1 port of vLLM, providing continuous batching and paged KV cache, while adding features such as GGUF support, RadixAttention, and cache-aware scheduling. The engine runs on CUDA, CPU, Metal, and Vulkan, and does not require Python or PyTorch at inference time. It is developed by the LocalAI team and is independent and unofficial, not affiliated with the vLLM project.

Key features

  • Continuous batching and block-paged KV cache - Supports CUDA, CPU, Metal, and Vulkan backends - GGUF model format support with on-the-fly quantization - RadixAttention and LPM cache-aware admission scheduling - Speculative decoding and jump-forward decoding - C ABI for embedding (versioned, stable interface)
  • No Python or PyTorch dependencies at runtime

Use cases

  • Deploying LLM inference servers with high throughput and low latency - Running large language models on diverse hardware (GPU, CPU, Apple Silicon)
  • Embedding inference into C++ applications via the C ABI - Experimenting with advanced scheduling and caching strategies

Getting started

See the repository README for installation and quick start instructions.

License and pricing

Licensed under Apache-2.0. The project is open source and free to use.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/localai-orgvllmcpp.svg)](https://yunodb.com/tools/localai-orgvllmcpp?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/localai-orgvllmcpp?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/localai-orgvllmcpp.svg" alt="Featured on YunoDB" height="20"></a>