vllm-metal
Community maintained hardware plugin for vLLM on Apple Silicon
What is vllm-metal?
Community maintained hardware plugin for vLLM on Apple Silicon
Overview of vllm-metal
What it is
vLLM Metal is a community maintained hardware plugin that enables vLLM to run on Apple Silicon Macs. It uses MLX as the primary compute backend and unifies MLX and PyTorch under a single lowering path. The plugin is hosted under the vLLM project on GitHub and is licensed under Apache-2.0. It is categorized as an AI tool and is open source with no stated pricing.
Key features
- Runs vLLM on Apple Silicon using MLX as the compute backend.
- Unifies MLX and PyTorch under a single lowering path.
- Provides a paged varlen kernel for request-aware attention.
- Supports M5 NAX tensor units to accelerate MHA, GQA, and MQA prefill.
- Includes speculative decoding support.
- Offers a sleep mode that lets an idle model server return memory to other applications.
Use cases
vLLM Metal is intended for running large language model inference on Apple Silicon Macs. It can be used for local model serving, concurrent duplex audio/video conversations (via vLLM-Omni), text-to-image generation (via vLLM-Omni), and structured decisions (e.g., text-only choice, yes/no, score requests). It is also used in the SiliconBench evaluation of serving engines on speed, memory use, and output fidelity.
Getting started
Stable release via Homebrew: bash brew tap vllm-project/vllm-metal https://github.com/vllm-project/vllm-metal brew install vllm-project/vllm-metal/vllm-metal Latest development build via curl installer: bash curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash source ~/.venv-vllm-metal/bin/activate For contributing Python or kernel code, see the repository README for source setup instructions.
License and pricing
The license is Apache-2.0. The project is open source with no stated pricing.
Based on the project's public README and repository metadata.