memra

memra is a Rust + CUDA inference engine built for NVIDIA RTX PRO 6000 Blackwell and RTX 5090. It serves GGUF models over an OpenAI-compatible API, and every def

What is memra?

memra is a Rust + CUDA inference engine built for NVIDIA RTX PRO 6000 Blackwell and RTX 5090. It serves GGUF models over an OpenAI-compatible API, and every default it ships was measured on those two cards — both Blackwell sm_120a, with a separately compile-gated Hopper/H100 (sm_90a) lane alongside.

Overview of memra

What it is

memra is a Rust + CUDA inference engine built for NVIDIA RTX PRO 6000 Blackwell and RTX 5090. It serves GGUF models over an OpenAI-compatible API. Every default it ships was measured on those two cards — both Blackwell sm_120a, with a separately compile-gated Hopper/H100 (sm_90a) lane alongside. The engine is developed by the tiyuvta lab and is narrow on purpose: it measures model, checkpoint, hardware, and workload combinations separately instead of promising universal support.

Key features

  • From-scratch Rust host runtime with hand-written CUDA kernels, compiled ahead of time into fatbins embedded in the binary; no Python or framework in the serving path.
  • OpenAI-compatible HTTP serving (chat, completions, Messages, Responses, embeddings, rerank) with admission and prefix cache.
  • Support states are model-, quantization-, and drafter-specific: NativeReference (plan compiles and runs in reference executor), NativeQualified (checkpoint and serving gates pass), NativeTuned (qualified plus optimized rewrites).
  • Correctness gates: kernel-level bit identity (kernel-check), argmax parity of generation against reference path, speculative-decode self-consistency, and served-path byte identity.
  • Prebuilt binaries for Linux x86_64 with CUDA 13 runtime support; installer verifies checksum and does not require nvcc.
  • Supports multiple crates including engine, server, GGUF loading, tokenizer, sampling, KV-cache format policy, and more.

Use cases

  • Interactive agents - Long-context and prefix reuse - Batch throughput - Multimodal workloads - Large models on multiple GPUs

Getting started

Prebuilt binaries require Linux x86_64, an NVIDIA driver with CUDA 13 runtime support, and the CUDA runtime libraries. bash curl -fsSL https://raw.githubusercontent.com/avifenesh/memra/main/tools/install.sh | sh export PATH=$HOME/.local/bin:$PATH kernel-check # expect: ALL GREEN Start a server with a supported public checkpoint: bash MEMRA_MODELS=q38=hf:tiyuvta/Qwen3.8-27B-NVFP4-MTP-GGUF:Q5K-mtp memra-server The server listens on 127.0.0.1:8080. Send a streaming chat completion: bash curl -sS -N http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{ model: q38, messages: [{role: user, content: Explain KV caching in one sentence.}], max_tokens: 128, stream: true }'

License and pricing

The project is licensed under MIT and is open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/avifeneshmemra.svg)](https://yunodb.com/tools/avifeneshmemra?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/avifeneshmemra?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/avifeneshmemra.svg" alt="Featured on YunoDB" height="20"></a>