kvcached

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

What is kvcached?

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

Overview of kvcached

What it is

kvcached (KV cache daemon) is a KV cache library for LLM serving/training on shared GPUs. It brings OS-style virtual memory abstraction to LLM systems, enabling elastic and demand-driven KV cache allocation to improve GPU utilization under dynamic workloads. It decouples GPU virtual addressing from physical memory allocation for KV caches, allowing serving engines to initially reserve virtual memory only and later back it with physical GPU memory when the cache is actively used.

Key features

  • Elastic KV cache: allocate and reclaim KV memory dynamically to match live load.
  • GPU virtual memory: decouple logical KV from physical GPU memory via runtime mapping.
  • Memory control: enforce memory limits with the kvcached CLI or the revisioned production-control API.
  • Frontend router and sleep mode: route requests to target models and put models to sleep when idle.
  • Support mainstream serving engines: integrate with SGLang and vLLM.
  • Prefix caching: support automatic prefix caching (APC) with a configurable memory bound.

Use cases

  • Multi-LLM serving: multiple LLMs share a GPU's memory elastically, improving GPU utilization and saving serving costs.
  • Serverless LLM: allocate KV cache only when needed, supporting serverless deployments where models spin up and down on demand.
  • Compound AI systems: elastically allocate memory across specialized models in a pipeline (e.g., retrieval, reasoning, and summarization).
  • GPU workload colocation: allow LLM inference to coexist with other GPU workloads such as training jobs, fine-tuning, or vision models.

Getting started

pip install kvcached --no-build-isolation

Or use Docker:

docker pull ghcr.io/ovg-project/kvcached-sglang:latest
docker pull ghcr.io/ovg-project/kvcached-vllm:latest

License and pricing

Apache-2.0 license, open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/ovg-projectkvcached.svg)](https://yunodb.com/tools/ovg-projectkvcached?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/ovg-projectkvcached?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/ovg-projectkvcached.svg" alt="Featured on YunoDB" height="20"></a>