Qwenox

An inference framework customized for AI MAX 395, compatible with both Linux and Windows, and supports deploying only qwen3.8-flash-next. This project is for re

What is Qwenox?

An inference framework customized for AI MAX 395, compatible with both Linux and Windows, and supports deploying only qwen3.8-flash-next. This project is for reference and learning purposes only. 为AI MAX 395定制的推理框架,Linux和Windows通用,只能部署qwen3.8-flash-next。本项目仅供参考学习使用。

Overview of Qwenox

What it is

Qwenox is a local inference engine for the Qwen3.8-Flash-Next (177B MoE, qwen4_exp architecture) model, specifically written for AMD Strix Halo APUs (Ryzen AI Max+ 395, gfx1151) with 128 GB unified memory. It runs on both Linux and Windows. The engine provides an OpenAI-compatible API, an Anthropic Messages API (streaming, tool calling, image input), and a web console. Key design choices include storing routing experts in 4-bit format read directly by GPU kernels, keeping the PLE n-gram table (27–48 GiB) on disk for on-demand loading to save memory, and using speculative decoding with ngram drafting prioritized and MTP draft head as fallback. It also implements a paged KV pool with two-level prompt caching (memory checkpoints at message boundaries and SSD snapshots recoverable across restarts) to avoid redundant prefill.

Key features

  • Weight optimization: routing experts at 4-bit, GPU kernel direct read; PLE n-gram table on disk, on-demand loading, no memory consumption.
  • Speculative decoding: ngram drafting first, MTP draft head fallback; greedy token-by-token verification, output distribution identical to non-speculative mode; enabled by default with no parameters.
  • No redundant prefill: paged KV pool with two-level prompt caching (memory checkpoints and SSD snapshots) for reuse across edited messages, multi-turn continuation, and conversation switching.
  • Multiple weight formats supported: hgn v2 (recommended), GGUF UD-Q4_K_XL, GGUF UD-IQ4_XS, hgn w4b.
  • API endpoints: OpenAI-compatible API and Anthropic Messages API with streaming, tool calling, and image input; web console at http://:8731/.
  • Cross-platform: same source code, model directory, and configuration file for Linux and Windows.

Use cases

Local deployment of the Qwen3.8-Flash-Next model for inference on AMD Strix Halo hardware. Suitable for development, testing, research, and any scenario requiring high-performance local inference with up to 256K context length. The engine is designed for reference and learning purposes only.

Getting started

Download weights using Hugging Face CLI (install with pip install -U huggingface_hub). Example for hgn v2:

hf download peonist-ai/halogen-qwen3.8-flash-next --local-dir ./models --include \
  qwen38-flash-next-v2.hgn qwen38-flash-next-ngram.hgn \
  qwen38-flash-next-mtp.hgn qwen38-flash-next-vision.hgn "tokenizer/*"

On Linux, compile and start:

bash build.sh
bash start_hgn.sh

On Windows, use the same source and configuration; see the repository README for details.

License and pricing

License: AGPL-3.0. Open source and free of charge.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/iiiiilllliiiiilllllqwenox.svg)](https://yunodb.com/tools/iiiiilllliiiiilllllqwenox?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/iiiiilllliiiiilllllqwenox?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/iiiiilllliiiiilllllqwenox.svg" alt="Featured on YunoDB" height="20"></a>