Speed, with the run behind every number, TandemLLM v0.14.2 documentation

All figures are one request at a time on one DGX Spark with greedy decoding. Each caption names the weights and the build or date, and gives the shape of the run. Parallel requests are not in them.

Qwen3.8-27B, the paper's benchmark row

StairCut 49.89

fixed block 8, the narrow drafter alone 47.13

fixed block 16, the wide drafter alone 44.80

plain greedy decoding 13.69

tok/s, mean over 50 requests

Published NVFP4 weights; 50 requests of 256 prompt tokens and 256 new tokens after 3 warm-ups; thinking off, temperature 0; the lookup store off, so none of the benchmark's own text can help. From the StairCut paper, commit 431ddee. We measured plain greedy decoding on 28 September on an earlier build; its path with speculation off has not changed since.

StairCut writes 3.64 times as many tokens a second as plain decoding on the same weights. It is 5.9 % ahead of the narrow drafter alone and 11.4 % ahead of the wide one. Both of those draft at one fixed width; StairCut picks the size of its guess tree every round.

More models, by kind of text

tok/s with drafting on, one request. Edit: a file sent back with one name changed. Copy: a text sent back unchanged.

model and pack	plain	prose	chat	code	edit	copy
Qwen3.8-Flash-Nextturboderp's EXL3 3.05, MTP layer + lookup, 2026-10-12, up to 512 tokens	41.5 to 45.7	51	64	83	132	133
LFM2.5-24B-A2Bour EXL3 3.5, lookup, 2026-10-14, 512 tokens	146.9 to 151.3	147.5	143.7	147.2	268.7	286.6
Nemotron 3.5 Bolt 74B-A9Bour EXL3 2.0, LiLiCorr + lookup, 2026-10-16, 512 tokens, mean of 3	43.3	65.6	71.1	102.1	163.5	247.2
Nemotron 3.5 Super VL 120B-A12Bour EXL3 4.0, LiLiCorr + lookup, 2026-10-16, 512 tokens, mean of 3	31.1	33.1	43.9	53.6	101.0	108.3
Ornith-1.5-122Bour EXL3 4.0, MTP layer + lookup, 2026-10-20, 512 tokens, one run	37.6 to 38.4	85.3	110.0	114.2	115.9	124.8
Recursive Flintour EXL3 4.0, lookup, 2026-10-22, 512 tokens, one run	128.2 to 136.6	135.4	134.0	139.8	142.3	480.0

Edits and copies run fastest because the lookup drafter can offer whole lines the model then accepts. Fresh prose gains least. On LFM2.5, already fast without help, the lookup costs up to 5 % in prose and chat. Flint has no draft layer, so the lookup drafts alone, and a copy ask runs at 480 tok/s against 128 plain. In every run above, the drafted text equals the plain run's text.

A long prompt sent twice

32k tokens 44.4 s cold0.3 s resumed

64k tokens 93.7 s cold0.6 s resumed

128k tokens 208.3 s cold1.1 s resumed

Time to the first token for a prose prompt, Qwen3.8-27B on the published NVFP4 set, one request each (the paper). The second time, the engine resumes from the state it kept and reads only the tail.

Bar charts. Plain decode of Nemotron 3.5 Bolt: llama.cpp on NVIDIA's IQ2 GGUF 38.2 tok/s, TandemLLM on the IQ2 GGUF 27.7, and TandemLLM on the EXL3 packs 43.3 at 2.0 bits, 42.1 at 2.5, 41.8 at 3.0, 40.8 at 3.5, 40.5 at 4.0, 34.0 at 5.0 and 29.2 at 6.0. With LiLiCorr and the lookup, the EXL3 2.0 pack writes 66 tok/s on prose, 71 on chat, 102 on code, 163 on edits and 247 on copies; the 4.0 pack 52, 62, 98, 137 and 210; the IQ2 GGUF 28, 28, 32, 98 and 132. — Nemotron 3.5 Bolt at every EXL3 size and on NVIDIA's 2-bit GGUF. One request, 512 greedy tokens, thinking off, mean of 3 runs, release 0.11.1, 2026-10-16. The llama.cpp bar is from the earlier 2026-10-15 run.

Bar charts for Nemotron 3.5 Super VL. Size on disk: EXL3 2.0 36.8 GB, EXL3 4.0 65.0 GB, NVIDIA NVFP4 75.1 GB. KL against BF16: 0.325, 0.049 and 0.041. Top-1 agreement with BF16: 82.1, 92.9 and 93.4 percent. Plain decode: 36.5, 31.1 and 19.6 tok/s in TandemLLM, and 19.4 for NVIDIA's NVFP4 in vLLM. With the LiLiCorr drafter, the EXL3 packs lead on prose, chat, edit and copy asks, and vLLM is level with the 4.0 pack on code. — Nemotron 3.5 Super VL: our EXL3 packs and NVIDIA's NVFP4 release in TandemLLM, and NVIDIA's NVFP4 in vLLM. Quality against the BF16 checkpoint's logits on 8 wikitext-2 rows of 1,024 tokens; speed as for Bolt, 2026-10-16. vLLM reads prompts 4 to 5 times as fast as TandemLLM on this model.

What is different

The model's own greedy text

Under greedy decoding, TandemLLM writes the text its target model writes one token at a time, with one exception: at a near tie between the two best tokens, a batched check can round the other way. The drafters, the lookup store and the caches change how fast the text arrives, not what it says.

We counted how often the exception happens. On 30 held-out prompts with StairCut on the published NVFP4 set, 17 answers match plain greedy decoding to the end. 13 follow it up to a near tie of 0.04 to 0.76 ulp, take the other token and go on from there. None differs at a larger gap. On Flash-Next the check rows equal the one-token rows bit for bit, so there is no near-tie exception at all.

The exactness page states the promise and lists the checks that hold the engine to it.

Thirty dots in four rows. 13 answers identical over all 256 tokens. 4 identical to the end of the answer, differing only past it. 13 took the other token at a near tie of 0.04 to 0.76 BF16 ulp, then continued. 0 differ at a larger gap. — StairCut against plain greedy decoding, 30 prompts, 256 new tokens each, published NVFP4 weights (the paper).

StairCut sizes the guess tree every round

On this board a check of 16 rows costs the same as one of 8: 74.4 ms against 75.5 ms at 1k tokens of context. Row 17 opens a new tile in the 4-bit kernel, and the cost jumps. StairCut prices every round on that measured staircase and keeps the tree size with the most expected tokens per millisecond.

On a teacher-forced bench of 25 workloads, where every setup writes the same reference text, StairCut averages 90.45 tok/s against 81.95 for the fixed block 16 and 59.31 for the fixed block 8. The paper has the method and every run.

Line chart of check-plus-commit time against rows checked in one pass, at 1k and 32k tokens of context. At 1k: 75.5 ms at 8 rows, 74.4 ms at 16, then a step at row 17 to about 80 ms, 83.1 ms at 24 and 87.8 ms at 32. The 32k line runs about 10 ms higher with a taller step. — The cost of one check by rows, median of 8, published NVFP4 weights, tree shape. These are the tables StairCut prices on.

Our own EXL3 packs

We publish EXL3 packs of Qwen3.8-27B, LFM2.5, the Nemotron models, Ornith-1.5-122B and Flint, made with turboderp's exllamav3. Each card states its KL and top-1 against the BF16 checkpoint. Bolt's 2.0 pack and NVIDIA's own 2-bit GGUF are about the same download, 23.48 and 22.04 GB, and the pack's KL against BF16 is 0.170 against the GGUF's 0.259. Qwen3.8-Flash-Next runs from turboderp's own EXL3 packs: community quants, not ours.

Qwen3.8-27B also comes as an NVFP4 set, quantised once from Qwen's BF16 release with GPTQ. Against BF16 it costs +0.0221 nats on prose and +0.0426 on code.

Many models in tandem

The big models share one server, one set of caches, one way of drafting and one setup guide. Small models work beside the big one: drafters guess tokens, Wren (a 2B model) answers questions in the setup guide, and a decision model and a narrator can run next to the served model.

The dashboard can also stand in front of any OpenAI-compatible engine or cloud API, from vLLM and llama.cpp to OpenAI and DeepSeek, and keep one ledger of tokens and cost for every backend.