HyperQwen

Today: Qwen3.8-27B on a single 24 GB consumer GPU with vLLM — 150k token context and an OpenAI-compatible API with key auth, in two ready-made modes. On an RTX 3090 at 250 W: 127 tok/s single-stream in single-user mode, ~1,035 tok/s aggregate decode at 64 concurrent in batch mode, and 381 tok/s when the answer quotes its own prompt.

Where it is going: more Qwen models, more classes of card. What this repo really is, under the serving setup, is a patch series against a pinned vLLM plus a model-preparation pipeline — requantized heads, a calibrated draft vocabulary, speculation that drafts out of the prompt. Most of that is not specific to one checkpoint or one card; it is just that one checkpoint and one card are what has been measured to death. See Roadmap for what is and is not portable yet, and Field tests wanted if you have half an hour on a GPU we have never touched.

git clone https://github.com/syv-ai/HyperQwen && cd HyperQwen
cp .env.example .env                      # PowerShell: Copy-Item .env.example .env

docker compose --profile single up -d     # one or a few people chatting
docker compose --profile batch  up -d     # API backend, many concurrent requests

First start pulls the image (9.5 GB) and requantizes the model (~20 GB, once, into ./models), then serves on :18020. One GPU runs one mode at a time.

• Before exposing it — compose publishes the port on 127.0.0.1 only (BIND in .env, default 127.0.0.1), so another machine cannot reach it until you set a key and BIND=0.0.0.0; outside a container, no key means the server binds 127.0.0.1 only: echo "VLLM_API_KEY=$(openssl rand -hex 24)" >> .env && echo BIND=0.0.0.0 >> .env
• Docker Desktop on WSL2 — keep VLLM_WSL2_ENABLE_PIN_MEMORY=1 in .env, or the V2 runner aborts with RuntimeError: UVA is not available
• No compose, or no Docker at all — docs/docker.md · docs/install.md

Three questions, and the answers are the whole of your .env.

flowchart TD
  Q1{"Who is sending<br/>the requests?"}
  Q1 -->|"an API backend, a pipeline,<br/>8+ at once"| A["<b>A — batch</b>"]
  Q1 -->|"one or a few<br/>people chatting"| Q2{"Longest prompt<br/>you will send?"}
  Q2 -->|"under 64k tokens"| Q3{"Do the answers quote<br/>the prompt back?"}
  Q2 -->|"up to 150k"| D["<b>D — long context</b>"]
  Q2 -->|"200k and beyond"| E["<b>E — huge context</b>"]
  Q3 -->|"no — normal chat"| B["<b>B — single, default</b>"]
  Q3 -->|"yes — code edits, RAG,<br/>rewrites, translation"| C["<b>C — reproduction</b>"]

Starting from the .env you copied in Quick start, which already ships B:

Torn between B and C: B is the safe default, and C only pays off when the output really does repeat the input — it trades request slots and context for it. D drops back to MTP speculation on purpose, because DFlash2 past 64k is worth it only for reproduction and loses to SPEC=mtp CTX=long about 2:1 on everything else; at E the KVarN cache buys the context instead, so DFlash2 stays on. A needs no edit at all — batch ignores SPEC and says so on startup. Every knob, and what each costs: single-user/README.md · batch/README.md.

On a compute capability 8.6 GPU (RTX 3090-class), SPEC=mtp CTX=long and SPEC=dflash2 CTX=long run on an int8 KV cache read by the pth-flash-attn kernel by default (PTH_FA=1, source vendored in pthfa/): the target and the drafter on --attention-backend CUSTOM --kv-cache-dtype int8_per_token_head. That needs the package installed (the Docker image has it; bash pthfa/install.sh in a venv) and the measured setup (docs/long-context.md). Otherwise the arm keeps its own default, fp8 on FlashInfer on mtp and the FA2 arm below on dflash2, and PTH_FA=0 restores it. Any other GPU, CTX=fast, CTX=huge and batch mode are unchanged. On the native RTX 3090 (TP=1, 250 W, vLLM 0.31.0, two boots per arm, #345), SPEC=mtp CTX=long decodes 84.7-84.8 tok/s at ~66k and 78.5 at ~100k against 75.5-75.6 and 68.8-68.9 with PTH_FA=0 (+12.2% and +14.0%), with the same cold TTFT at ~66k and a 1.9% smaller KV pool; the ~95-100 tok/s of row D above is the fp8 + FlashInfer arm. Only compute capability 8.6 has been run on a GPU (3090 and 3080 here), on vLLM 0.31.0, and the kernel is forward only.

Without the kernel (PTH_FA=0, or a launch outside its measured setup), SPEC=dflash2 CTX=long runs on fp8 KV with the FlashAttention-2 plugin in fa2/ by default (FA2_FP8KV=1) on an RTX 3090-class GPU (compute capability 8.6); on any other GPU the default stays on int8 + Triton with a note, and FA2_FP8KV=1 opts in on a 4090; FA2_FP8KV=0 restores the int8 + Triton arm. Native installs need bash fa2/install.sh once, and the launcher refuses to start without it rather than fall back (the Docker image has it). KV_OFFLOAD_GB turns the default off with a note; an explicit FA2_FP8KV=1 refuses it, and refuses any other SPEC/CTX. The evidence is the T4 battery on the native 3090 (#153): on 2026-10-18 (bare metal, 250 W, vLLM 0.31.0) the fp8 + FA2 arm passed every bar against the shipped int8 + Triton arm: GSM8K 0.950 vs 0.955 (n=200), perplexity +0.04%, 0 of 128 residues broken, a 20-minute soak with no restart, decode 54.0 vs 48.2 tok/s at ~66k and 48.2 vs 37.9 tok/s at ~100k (+27%), cold TTFT 75.5 vs 132.3 s and 54.7 vs 130.5 s, and a 3.3% larger pool (140,902 vs 136,429 tokens).

Speculation wins below ~8 concurrent users, plain batching above — earlier on long sessions, where a speculating request reserves recurrent-state pages the pool has few of (measurement). Both modes share one install. Full prefill matrix · how each number was won.

The name changed because the scope did. This started as "Qwen3.8-27B on one RTX 3090" and the README still reflects that, because that is the pair which has been measured properly. The plan is to widen both axes. Being specific about what transfers, since "supports N models" is easy to claim and expensive to be wrong about:

Portable already — nothing model- or card-specific in it. The vLLM patch series (patches/, one line each in PATCHES.md), the KVarN long-context backend, int8 Marlin GEMM layer selection, the SSE keep-alive, the engine stall sentinel. These are vLLM fixes that happen to have been written here.

Model-specific, and this is the real work. The int8-QK prefill attention kernel is gated on this checkpoint's exact geometry — num_heads == 24, num_kv_heads == 4, head_size == 256 — and falls back to FA2 for anything else, so a new model gets correctness for free and none of the prefill win until the gate is generalized. The 40k-token draft vocabulary is calibrated on this model's own output distribution. The DFlash2 drafter is a per-model checkpoint. prepare/ assumes a head layout that only just learned to handle heads in different shards (#120).

Card-specific. The Marlin tune tables are measured on sm86. sm120 (RTX 5090) reproduces (#35); sm80 runs but has an open speculation fault at any k (#98, #72). Multi-GPU works at TP=2 and TP=4; TP=3 and PP=3 are invalid for this model's head count, which is a property of the checkpoint rather than a missing feature.

What we want next. Smaller Qwen checkpoints so 16 GB and 12 GB cards get the same treatment rather than a shrug; a larger one for 48 GB; the EXL3 route in #103 as a second quantization path. If you want a specific model or card prioritized, say so in an issue — the order is mostly driven by who turns up with a reproduction.

Every number above is one RTX 3090 at 250 W, shared with training jobs. Every row for any other card came from someone else running the harness — so the ask here is a protocol rather than "try it and let us know".

About 30 minutes on one GPU, most of it unattended. The harness lives in the container, so it runs through compose exec — on a venv install drop that prefix and use python instead of venv/bin/python:

sudo nvidia-smi -pl 250                 # your card's reference wattage; say which
docker compose --profile single up -d   # first start also prepares the model

docker compose exec single bash bench/run_benchmarks.sh single   # discard: a first run after a start reads 30-50% low
docker compose exec single bash bench/run_benchmarks.sh single   # keep this one — its ROW lines are the report

docker compose exec single venv/bin/hf download openai/gsm8k --repo-type dataset \
  --include "main/test-*" --local-dir bench/quality-data/gsm8k
docker compose exec single venv/bin/pip install pyarrow   # only on images published before 2026-09-21
docker compose exec single venv/bin/python bench/quality_battery.py mycard --gsm-only

mycard is a label you choose for this run (for example rtx3090-250w), not a model name. Results are saved to bench/quality-data/result_mycard.json. Use your service name in place of single when testing another Compose profile.

Then open a field report. The form asks for the six things that make two runs comparable — card, power cap, driver, OS or container, commit, which setup letter — because without them a number cannot honestly be put next to the row above it. Ran something narrower, or with your own client? Post it anyway and say what you skipped; it gets listed as an independent report instead of a table row, which is what happened to most of docs/reproductions/.

Most wanted, in order. Several of these already have a thread with someone's hardware in it — check before you duplicate, and add to theirs if it matches:

What you get: the run published in docs/reproductions/ with your raw output and credit, the gotchas your platform exposes written into docs/gotchas.md, and an honest account of what your hardware does badly — historically the more useful half.

Rather give hardware or money than time? A box we can reach, or credits: open an issue titled "compute offer". Otherwise ko-fi.com/mhenrichsen buys hours on cards this project does not own, and those runs get written up here like any other reproduction.

Everything that used to be in this README, in the folder it belongs to.

Running it

How it works, and how fast

Building it

Apache-2.0, same as the model.