memra
memra is a Rust + CUDA inference engine built for NVIDIA RTX PRO 6000 Blackwell and RTX 5090. It serves GGUF models over an OpenAI-compatible API, and every def
What is memra?
memra is a Rust + CUDA inference engine built for NVIDIA RTX PRO 6000 Blackwell and RTX 5090. It serves GGUF models over an OpenAI-compatible API, and every default it ships was measured on those two cards — both Blackwell sm_120a, with a separately compile-gated Hopper/H100 (sm_90a) lane alongside.
Overview of memra
What it is
memra is a Rust + CUDA inference engine built for NVIDIA RTX PRO 6000 Blackwell and RTX 5090. It serves GGUF models over an OpenAI-compatible API. Every default it ships was measured on those two cards — both Blackwell sm_120a, with a separately compile-gated Hopper/H100 (sm_90a) lane alongside. The engine is developed by the tiyuvta lab and is narrow on purpose: it measures model, checkpoint, hardware, and workload combinations separately instead of promising universal support.
Key features
- From-scratch Rust host runtime with hand-written CUDA kernels, compiled ahead of time into fatbins embedded in the binary; no Python or framework in the serving path.
- OpenAI-compatible HTTP serving (chat, completions, Messages, Responses, embeddings, rerank) with admission and prefix cache.
- Support states are model-, quantization-, and drafter-specific: NativeReference (plan compiles and runs in reference executor), NativeQualified (checkpoint and serving gates pass), NativeTuned (qualified plus optimized rewrites).
- Correctness gates: kernel-level bit identity (
kernel-check), argmax parity of generation against reference path, speculative-decode self-consistency, and served-path byte identity. - Prebuilt binaries for Linux x86_64 with CUDA 13 runtime support; installer verifies checksum and does not require
nvcc. - Supports multiple crates including engine, server, GGUF loading, tokenizer, sampling, KV-cache format policy, and more.
Use cases
- Interactive agents - Long-context and prefix reuse - Batch throughput - Multimodal workloads - Large models on multiple GPUs
Getting started
Prebuilt binaries require Linux x86_64, an NVIDIA driver with CUDA 13 runtime support, and the CUDA runtime libraries. bash curl -fsSL https://raw.githubusercontent.com/avifenesh/memra/main/tools/install.sh | sh export PATH=$HOME/.local/bin:$PATH kernel-check # expect: ALL GREEN Start a server with a supported public checkpoint: bash MEMRA_MODELS=q38=hf:tiyuvta/Qwen3.8-27B-NVFP4-MTP-GGUF:Q5K-mtp memra-server The server listens on 127.0.0.1:8080. Send a streaming chat completion: bash curl -sS -N http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{ model: q38, messages: [{role: user, content: Explain KV caching in one sentence.}], max_tokens: 128, stream: true }'
License and pricing
The project is licensed under MIT and is open source.
Based on the project's public README and repository metadata.