Qwen3.8-27B on one RTX 4090

Everything in this part runs the released container unchanged: target and drafter weights byte for byte as published. It covers the first two arrows in Figure 3.

More tokens per round: verify trees

The cheapest win was already in upstream's code. Instead of one chain of drafts, the drafter's per-position candidates are grown best-first into a 16-node tree, and the target verifies the whole tree in one pass with a tree attention mask. A 16-node tree accepts 20% more tokens per round than a 16-token chain at the same cost. Against the 7-token chain of Cinference-4090, our defaults accept 29% more tokens per round at the same round time, even though each round now verifies 16 positions instead of 8.

The drafter was trained to propose blocks of 8, and we ask it for 15. A tree temperature of 1.25 on the drafter's scores spreads the 15 nodes a little wider than its raw probabilities would. Wider trees do not pay on this card. Replaying recorded trees offline, a 32-node tree accepts 11% more tokens, but 32-column GEMMs cost 12–39% more per shape, because the activations no longer fit the 16-column tiles.

Where a round goes

A round has a fixed shape (Figure 5). It commits the previous round's recurrent state, runs the drafter (five layers) and its proposal head, grows the tree, verifies it through the target's 64 layers, scores the result with the LM head, and accepts. In the Oct 3 trace, 20.4 of 22.9 ms went to weight streaming: target and drafter GEMMs and the two LM heads. The rest went to hundreds of small kernels: norms, the recurrent GDN update, attention. Those kernels are latency-bound and leave DRAM idle while they run. Hopper can overlap one kernel's tail with the next kernel's start (programmatic dependent launch); Ada cannot, so every kernel boundary is a small bubble.

Streaming kernels

Small-T GEMM schedules tuned for Ada, single launches that cover both input projections of a layer, and evict-first hints on weight loads bring the weight GEMMs to 930–1,000 GB/s. The tuning alone took the median round from 25.3 to 22.9 ms (Oct 2 → Oct 3 in Figure 5). The largest GEMM, the Q4 gate/up projection, streams 100 MB in 98 µs. The 248k × 5,120 LM head is read through a 3-bit copy that keeps 32 candidates per position, and only those are rescored exactly. That screen runs at 927 GB/s, the card's measured streaming rate, and its outputs were identical on all 480 prompts. The drafter's proposal head uses the same trick.

L2 fills

A latency-bound kernel is DRAM-idle time, and the obvious ways to use it fail on Ada. prefetch.global.L2 does not populate L2, and prefetching from a side stream slows whatever kernel it lands next to. What works is giving each latency-bound kernel a few extra CTAs. After the kernel's own loads are issued, they pull the first megabytes of the next GEMM's weights into L2 with ordinary cached loads (Figure 6).

Fills on five kernel types cut round time by about 0.5%. The accounting was subtle in two ways. First, size matters: doubling the fills made rounds slower, because the stretches they ride on were already full. Second, deleting a small kernel also deletes its fill window. Our faster attention kernel at first showed no gain, because the old kernel had been carrying the fill for the next projection. Folding the RMSNorms into neighbouring GEMMs removed 80 launches per round and saved no time at all, for the same reason.

Small kernels, and one CMake property

The GDN convolution moved into the input projection's epilogue (−0.4%, bit-exact). Verify attention was rewritten to split rows across warps and CTAs: 17 µs became 7 µs at the median Spec-Bench context, and it reproduces the old kernel's split partials bit for bit. A chunked (WY-form) recurrent update takes 6 µs instead of 12–20. That one is not bit-exact: it misrounds 0.02% of outputs by one BF16 ulp, against 0.01% for the staged kernel it replaced.

The engine compiled every CUDA file as relocatable device code (-rdc=true), although nothing calls across translation units. Compiled whole-program, 541 of 2,801 kernels use fewer registers. The gate/up GEMM drops from 84 registers to 72, which lets three 256-thread CTAs share an SM instead of two. That is −0.5% round time with identical outputs. It also explained an earlier negative result: an epilogue fusion that had lost 1.5% to the same register cliff.

A cheaper cache and state

KV-cache values move from NVFP4 to INT8. Keys stay FP8, after a Hadamard rotation. Against a BF16 cache, INT8 values measure as lossless, where NVFP4 had cost about 0.0015 KL and 0.002 top-1, and verify attention gets faster too. The recurrent GDN state is stored in FP16 between rounds. In the final round (Figure 5, Oct 6) the GPU is idle for 0.29 ms of 17.0, mostly while the host prepares the next round.

Part I left a round that is almost all weight streaming. From there, the only big lever left is fewer bytes. But a lower-bit model that is worse than the official one would not count. So we first pinned down what "no worse" means.

The reference is FP32 compute over the BF16 checkpoint, streamed layer by layer on the same card. We score 46,059 next-token predictions in 44 windows of 2,048 tokens, half Wikipedia prose and half CPython source, none of it overlapping any calibration data. We track three numbers: KL₆₄ (KL from the reference to the candidate on the reference's top-64 tokens, which carry 98% of its mass), top-1 agreement and perplexity. A change ships only if every number, on prose and on code, is at least as good as the release's.

GPTQ, then spend the headroom

The release quantizes the target with round-to-nearest at 4 and 5 bits, and that leaves a lot on the table. GPTQ with act-order and a per-group clip search, at the release's own bit allocation, cuts KL₆₄ by more than a third. Everything after that spends this headroom on bytes:

1. All projections at Q4, with 8-bit group scales. This saves 1.8 GB per round. Larger calibration sets (96, then 384 windows) won back about a third of the KL that all-Q4 cost.
2. 3 bits where they are cheap. Sensitivity is very uneven (Figure 7). Late MLP down projections cost 6× more KL per byte saved than early ones, and code is 4× more sensitive than prose. MLP gate/up in layers 0–31 and down in layers 0–15 now run at Q3. They stay in the existing Q4 container as codes restricted to [−4, 3] and are repacked into 3-bit tiles at load, so no new file format was needed. The Q4 copy stays resident for prefill (see Limitations).
3. End-to-end scale tuning. With the integer codes frozen, we train only the group scales, a per-row offset and the RMSNorm gains (4.6 M parameters) to match the BF16 model's output distribution on held-out text, in the spirit of EfficientQAT's E2E-QP. The engine's exact KV-cache codec runs inside the training loop. Held-out KL drops 10.5%, most of it on code.

Step by step, the final model reads 19% fewer target bytes per round than the release, at lower KL and perplexity:

Quality is also a speed variable. A noisier target agrees less often with a drafter distilled from the BF16 model. An RTN all-Q4 target accepted 5–7% fewer tokens per round than the release, which cancelled most of its byte savings. So every quantization step was judged on round time and tokens per round together, never on bytes alone.

A drafter for the new target

A LoRA fine-tune of the drafter on 8,000 sequences generated by the quantized target gave +2.3% tok/s. Higher rank, more epochs, soft labels and new data all landed at the same place. Measured teacher-forced on a chain, that fine-tune raised acceptance by 6.8%. Measured in the engine, tokens per round rose only 1.4%. The tree recovers most of what a weak chain loses, so chain metrics overstate drafter changes about threefold. Every drafter decision after that was measured in the engine.

The drafter scores only a 131,072-token shortlist of the 248,320-token vocabulary, and a target token outside the shortlist ends the draft. Re-ranking the shortlist with the target's own outputs cut those misses from 0.96% to 0.35% of tokens, worth 2% more tokens per round. The drafter's own precision is a free parameter, since it can only change acceptance. Q4 and Q3 drafter projections paid for themselves once they ran on kernels that stream at full bandwidth.

Checking the path that actually runs

The fidelity numbers above score text through the prefill path. A speculative decoder spends its life elsewhere, in 16-token verify passes that read and write recurrent state between rounds, and some changes, such as FP16 state storage, exist only there. So we also measured the decode path directly. We run forced decoding on Spec-Bench prompts and dump the final hidden state at every accepted position. We then apply the BF16 head offline, exactly as the reference does, and score against an FP32 reference over the same text. Over about 40,000 positions, FP16 state moved KL₆₄ from 0.02125 to 0.02079 and perplexity from 1.1942 to 1.1930.

The same tool exposed a bias in quality bars. Any perturbation that is not bit-exact loses 10–25 top-1 agreements out of about 46,000, whatever its size: rounding the state to 5 mantissa bits cost fewer than rounding it to FP16. Near a tie, the engine agrees with the reference more often than chance, because the two computations are close, and added noise of any size re-randomizes those ties. A rule like "top-1 must not drop" therefore rejects every harmless change. We used non-inferiority against the release instead, and treated the top-1 headroom as a budget that each non-bit-exact change spends.

Late in the project most wins were 0.3–1% of round time, and speculative decoding makes effects that small hard to see. A change that is not bit-exact flips a few near-tie tokens. From the first flip on, the generated text differs, and different text has different acceptance. A perturbation we knew to be null read as +0.44% tokens per round. Four practices made small effects measurable:

1. Forced text. The engine can replace its greedy choice with a reference token at each position. Every configuration then walks the same 480 texts, and each round costs exactly what it would anyway. Validate the instrument first: forcing a configuration onto its own reference must reproduce its outputs. Our first version reproduced 0 of 482 because of an off-by-one, which would have passed for a plausible 3% drafter regression.
2. Alternate configurations on one GPU, in a drift-cancelling order. Repeats of the same configuration drift by 0.05–0.6% within a job, as large as the effects. The SM clock steps from 2,715 to 2,685 MHz as the card warms, and a run that starts on an idle GPU reads fast. We kept builds off the box (a niced rebuild once cost 0.9%), held the other GPU's load constant and discarded a warm-up run. Then we ordered runs A B B A B A A B, which is orthogonal to linear and quadratic drift, and fitted the drift away (Figure 8).
3. Paired per-prompt bootstrap. Comparing the same prompt across runs cancels prompt difficulty, giving intervals 5–13× tighter than unpaired ones. Next to it we report the run-level interval from the drift fit, which is the honest one.
4. Explain every result. Each end-to-end number was checked against nsys per-call kernel times. Host-side costs came from unprofiled logs, because tracing inflates them several-fold.

NINFER_TWO_LEVEL_HEAD=1 NINFER_HEAD_SCREEN=3 NINFER_SCREEN_TILED=1 \
NINFER_PROPOSAL_SCREEN=3 NINFER_TREE_TEMPERATURE=1.25 \
NINFER_LS8=1 NINFER_Q3_SHADOW=1 NINFER_Q3_TILED=1 NINFER_DRAFT_CONV_Q4=1 \
NINFER_KV_VI8=1 NINFER_GDN_STATE_F16=1 NINFER_GDN_CHUNKED_RECORD=1 \
NINFER_CONV_EPILOGUE=1 NINFER_ATTN_ROWSPLIT=3 \
NINFER_L2_FILL=121 NINFER_L2_FILL_NORM_KB=1536 NINFER_L2_FILL_POST_KB=768 \
NINFER_L2_FILL_GATED_KB=768 NINFER_L2_FILL_RECORD_KB=2048 \
NINFER_L2_FILL_ATTN_KB=4096 \
ninfer-serve MODEL.ninfer --spec dflash2 --draft-tokens 15 --lm-head-draft \
  --verify-tree --kv-dtype k8v4 --max-context 32768 --kv-capacity 32768 \
  --max-concurrency 1 --prefill-chunk 1024 --temperature 0 --no-thinking \
  --no-prefix-reuse

@misc{peng2026qwen38rtx4090,
  author = {Zedong Peng},
  title  = {Qwen3.8-27B on One RTX 4090},
  year   = {2026},
  month  = {oct},
  note   = {Blog post}
}