TandemLLM

TandemLLM

Your private AI environment

TandemLLM runs open models on one NVIDIA DGX Spark. You get an engine with an OpenAI-compatible API, a chat with its own sandbox, and a dashboard that shows what each request is doing right now. Small drafters guess the next tokens and the big model checks every guess in one pass. The answer arrives faster and is still the model's own greedy text, with one documented exception.

Install 0.11.1 Source on GitHub Weights on Hugging Face
Qwen3.8-27B with StairCut, one request
49.89 tok/s
plain greedy decoding on the same box: 13.69
LFM2.5-24B-A2B, chat, one request
150.9 tok/s
plain decode on the default EXL3 3.5 pack, 14 October 2026
first token of a 128k-token prompt sent again
1.1 s
the same prompt from cold: 208.3 s

What happens to a request

A chat request passes seven stations before its first token streams back. Each one is code in the repository and has its own chapter in the manual.

The path below is the 27B's. Every other model shares it; only the drafters differ.

  1. Client

    POST /v1/chat/completions

    Open WebUI, opencode, pi, the OpenAI SDK or plain curl. Anything that speaks the OpenAI chat API works without a plugin.

  2. Proxy

    routes by model id

    A request for a model on another backend, such as vLLM, llama.cpp or a cloud API, goes there and still lands in the same usage ledger. A request for TandemLLM stays on the box.

  3. Conductor

    task class, effort, packages

    The router decides what kind of task the turn is and how hard the model should think. It adds the instructions and skill packages for that kind of task, and memory notes from earlier sessions when you turn them on. None of it changes the model's arithmetic; it changes what goes into the prompt.

  4. Admission

    a queue of 8

    A request that finds the engine busy waits in a short queue. The running request serves a small one, such as a chat title, between two of its own steps. A memory gate checks that the prompt fits before the prefill starts.

  5. Caches

    an anchor every 1,024 tokens

    Reading a long prompt is the most expensive thing the engine does. The next turn of a conversation resumes from the state the last one left, in memory or on the NVMe, and reads only what is new. Each restore gives the stored state back bit for bit.

  6. Drafters and the model, round after round

    up to 15 guesses a round

    A block drafter guesses up to 15 tokens and a lookup drafter offers text it has seen before. StairCut merges them into one tree and cuts it to the size that pays on the measured cost of a check. The big model checks the whole tree in one pass and keeps the longest path it agrees with, plus one token of its own.

    One round, schematic A tree of drafted tokens grows from the last committed token. StairCut cuts the tree at 16 rows. The model checks the kept tree in one pass, accepts four drafted tokens along one path and adds one token of its own. Guesses past the cut are never checked. cut: 16 rows last token drafted guesses kept: the model agrees not checked +1 own
    One round, schematic. Kept guesses are free tokens. At the first wrong guess the model puts in its own token, so nothing a drafter writes reaches the answer unchecked.

    The drafters depend on the model: fine-tuned DFlash2 drafters for Qwen3.8-27B, the checkpoint's own draft layer for Flash-Next, NVIDIA's LiLiCorr for the Nemotron models, and the lookup for every model.

  7. Monitor and stream

    four safe points

    While the model writes, the conductor reads the output at fixed points. It can close a reasoning loop, continue a turn that ended empty, or repair a broken tool call. Each round's verified tokens stream back at once, and tool calls come back typed by their JSON schema.

Speed, with the run behind every number

All figures are one request at a time on one DGX Spark with greedy decoding. Each caption names the weights and the build or date, and gives the shape of the run. Parallel requests are not in them.

Qwen3.8-27B, the paper's benchmark row

StairCut 49.89
fixed block 8, the narrow drafter alone 47.13
fixed block 16, the wide drafter alone 44.80
plain greedy decoding 13.69

tok/s, mean over 50 requests

Published NVFP4 weights; 50 requests of 256 prompt tokens and 256 new tokens after 3 warm-ups; thinking off, temperature 0; the lookup store off, so none of the benchmark's own text can help. From the StairCut paper, commit 431ddee. We measured plain greedy decoding on 28 September on an earlier build; its path with speculation off has not changed since.

StairCut writes 3.64 times as many tokens a second as plain decoding on the same weights. It is 5.9 % ahead of the narrow drafter alone and 11.4 % ahead of the wide one. Both of those draft at one fixed width; StairCut picks the size of its guess tree every round.

More models, by kind of text

tok/s with drafting on, one request. Edit: a file sent back with one name changed. Copy: a text sent back unchanged.

model and packplainprosechatcodeeditcopy
Qwen3.8-Flash-Nextturboderp's EXL3 3.05, MTP layer + lookup, 2026-10-12, up to 512 tokens 41.5 to 45.7 516483132133
LFM2.5-24B-A2Bour EXL3 3.5, lookup, 2026-10-14, 512 tokens 146.9 to 151.3 147.5143.7147.2268.7286.6
Nemotron 3.5 Bolt 74B-A9Bour EXL3 2.0, LiLiCorr + lookup, 2026-10-16, 512 tokens, mean of 3 43.3 65.671.1102.1163.5247.2
Nemotron 3.5 Super VL 120B-A12Bour EXL3 4.0, LiLiCorr + lookup, 2026-10-16, 512 tokens, mean of 3 31.1 33.143.953.6101.0108.3

Edits and copies run fastest because the lookup drafter can offer whole lines the model then accepts. Fresh prose gains least. On LFM2.5, already fast without help, the lookup costs up to 5 % in prose and chat. In every run above, the drafted text equals the plain run's text.

A long prompt sent twice

32k tokens 44.4 s cold0.3 s resumed
64k tokens 93.7 s cold0.6 s resumed
128k tokens 208.3 s cold1.1 s resumed
Time to the first token for a prose prompt, Qwen3.8-27B on the published NVFP4 set, one request each (the paper). The second time, the engine resumes from the state it kept and reads only the tail.
Bar charts. Plain decode of Nemotron 3.5 Bolt: llama.cpp on NVIDIA's IQ2 GGUF 38.2 tok/s, TandemLLM on the IQ2 GGUF 27.7, and TandemLLM on the EXL3 packs 43.3 at 2.0 bits, 42.1 at 2.5, 41.8 at 3.0, 40.8 at 3.5, 40.5 at 4.0, 34.0 at 5.0 and 29.2 at 6.0. With LiLiCorr and the lookup, the EXL3 2.0 pack writes 66 tok/s on prose, 71 on chat, 102 on code, 163 on edits and 247 on copies; the 4.0 pack 52, 62, 98, 137 and 210; the IQ2 GGUF 28, 28, 32, 98 and 132.
Nemotron 3.5 Bolt at every EXL3 size and on NVIDIA's 2-bit GGUF. One request, 512 greedy tokens, thinking off, mean of 3 runs, release 0.11.1, 2026-10-16. The llama.cpp bar is from the earlier 2026-10-15 run.
Bar charts for Nemotron 3.5 Super VL. Size on disk: EXL3 2.0 36.8 GB, EXL3 4.0 65.0 GB, NVIDIA NVFP4 75.1 GB. KL against BF16: 0.325, 0.049 and 0.041. Top-1 agreement with BF16: 82.1, 92.9 and 93.4 percent. Plain decode: 36.5, 31.1 and 19.6 tok/s in TandemLLM, and 19.4 for NVIDIA's NVFP4 in vLLM. With the LiLiCorr drafter, the EXL3 packs lead on prose, chat, edit and copy asks, and vLLM is level with the 4.0 pack on code.
Nemotron 3.5 Super VL: our EXL3 packs and NVIDIA's NVFP4 release in TandemLLM, and NVIDIA's NVFP4 in vLLM. Quality against the BF16 checkpoint's logits on 8 wikitext-2 rows of 1,024 tokens; speed as for Bolt, 2026-10-16. vLLM reads prompts 4 to 5 times as fast as TandemLLM on this model.

Models

The box serves one model at a time. This list comes from the model registry in the repository, the same files the Setup Guide reads, so a new model shows up here when it lands on GitHub. Every weight pack names its Hugging Face repository and who made it. Every file is public; you need no Hugging Face token.

Loading the model list…

The model list loads from the repository. Without JavaScript, see models/ on GitHub.

TandemLLM quant: a pack we made and publish under 0xBakeer. Vendor release: the model maker's own files. Community quant: a pack someone else published. Sizes are downloads in GB; the speed is the default pack's measured row in the registry.

Community weights

Weight packs other people published on Hugging Face that TandemLLM loads. The Setup Guide lists them next to ours. A maintainer runs each pack on a DGX Spark before its entry says verified.

Loading the community list…

The list loads from models/community-index.json on GitHub.

Add your pack

A pack is one JSON file in a pull request: the Hugging Face repository at a pinned commit, its format and size, and your GitHub handle. Or open the issue form and a maintainer writes the file for you.

Open the issue form How to add a pack

A chat that runs on the box

The dashboard's Chat tab feels like the chat products you know from the cloud, and it runs on your own box. A run lives on the server. Close the browser in the middle of an answer, open the chat on your phone, and it shows the answer at the token the engine is writing now.

The Chat tab. On the left, the chat list with pinned chats, folders and a running chat. In the middle, a run that calls a file-editing tool, a follow-up message queued to go in after the current step, and a terminal inside the chat's sandbox with no network. On the right, the context panel: the window in use by part, the live decode rate, and which model and weights serve the chat.
A run in progress, with a follow-up queued behind it and the chat's sandbox terminal open below. Rendered from the dashboard's mock data; the figures in it are not measurements.
  • Each chat gets its own sandbox container: no capabilities, a read-only root, 1 GiB of memory, 4 CPUs and no GPU. Its internet goes through a per-chat proxy that refuses your local network, and you can switch it off per chat.
  • The model gets a shell, file tools, terminals and page rendering inside that sandbox. Web search goes through DuckDuckGo or your own SearXNG. MCP servers plug in; stdio servers run inside the sandbox, and their secrets never come back in an answer.
  • You choose how much it may do alone: Ask, Auto (the default) or YOLO. A tool call that needs your answer waits, and gives its slot back while it waits.
  • Images go to a model that reads images. A text file goes into the prompt within a token budget, and a larger one goes to the chat's workspace for the tools to read.
A chat waiting for permission. The model read a file, and its next tool call, an edit of that file, shows the change it wants to make with three buttons: Allow once, Always allow edit_file, and Deny.
A tool call waiting for your answer, with the exact change it wants to make. Mock data.

The Chat tab is in beta, and its page in the docs lists what it does not do yet. Your own client works too: Open WebUI, opencode and pi talk to the engine through the OpenAI-compatible API, and every client shows up in the same dashboard, down to a year of usage.

What is different

The model's own greedy text

Under greedy decoding, TandemLLM writes the text its target model writes one token at a time, with one exception: at a near tie between the two best tokens, a batched check can round the other way. The drafters, the lookup store and the caches change how fast the text arrives, not what it says.

We counted how often the exception happens. On 30 held-out prompts with StairCut on the published NVFP4 set, 17 answers match plain greedy decoding to the end. 13 follow it up to a near tie of 0.04 to 0.76 ulp, take the other token and go on from there. None differs at a larger gap. On Flash-Next the check rows equal the one-token rows bit for bit, so there is no near-tie exception at all.

The exactness page states the promise and lists the checks that hold the engine to it.

Thirty dots in four rows. 13 answers identical over all 256 tokens. 4 identical to the end of the answer, differing only past it. 13 took the other token at a near tie of 0.04 to 0.76 BF16 ulp, then continued. 0 differ at a larger gap.
StairCut against plain greedy decoding, 30 prompts, 256 new tokens each, published NVFP4 weights (the paper).

StairCut sizes the guess tree every round

On this board a check of 16 rows costs the same as one of 8: 74.4 ms against 75.5 ms at 1k tokens of context. Row 17 opens a new tile in the 4-bit kernel, and the cost jumps. StairCut prices every round on that measured staircase and keeps the tree size with the most expected tokens per millisecond.

On a teacher-forced bench of 25 workloads, where every setup writes the same reference text, StairCut averages 90.45 tok/s against 81.95 for the fixed block 16 and 59.31 for the fixed block 8. The paper has the method and every run.

Line chart of check-plus-commit time against rows checked in one pass, at 1k and 32k tokens of context. At 1k: 75.5 ms at 8 rows, 74.4 ms at 16, then a step at row 17 to about 80 ms, 83.1 ms at 24 and 87.8 ms at 32. The 32k line runs about 10 ms higher with a taller step.
The cost of one check by rows, median of 8, published NVFP4 weights, tree shape. These are the tables StairCut prices on.

Our own EXL3 packs

We publish EXL3 packs of Qwen3.8-27B, LFM2.5 and the Nemotron models, made with turboderp's exllamav3. Each card states its KL and top-1 against the BF16 checkpoint. Bolt's 2.0 pack and NVIDIA's own 2-bit GGUF are about the same download, 23.48 and 22.04 GB, and the pack's KL against BF16 is 0.170 against the GGUF's 0.259. Qwen3.8-Flash-Next runs from turboderp's own EXL3 packs: community quants, not ours.

Qwen3.8-27B also comes as an NVFP4 set, quantised once from Qwen's BF16 release with GPTQ. Against BF16 it costs +0.0221 nats on prose and +0.0426 on code.

Many models in tandem

The big models share one server, one set of caches, one way of drafting and one setup guide. Small models work beside the big one: drafters guess tokens, Wren (a 2B model) answers questions in the setup guide, and a decision model and a narrator can run next to the served model.

The dashboard can also stand in front of any OpenAI-compatible engine or cloud API, from vLLM and llama.cpp to OpenAI and DeepSeek, and keep one ledger of tokens and cost for every backend.

Install

You need a DGX Spark: a GB10 GPU with 128 GB of memory shared by CPU and GPU. Other CUDA GPUs are best effort. Two commands install TandemLLM; neither needs a Hugging Face token.

bash, release 0.11.1
curl -fsSLO https://raw.githubusercontent.com/0xBakeer/TandemLLM/v0.11.1/install.sh
bash install.sh --setup --yes
  1. The installer checks the board, builds a pinned Python environment and downloads only Wren's model, 4.3 GB. Then it starts setup mode. No model is loaded yet.
  2. Open the Setup Guide in your browser and pick a model. On the box, the address is http://127.0.0.1:8000/dashboard/#/setup.
  3. The guide checks that your choice fits in memory, downloads the weights, checks every file and starts the engine on port 8000.

From another computer

Setup mode listens on the box itself only. Open an SSH tunnel to the box, then open the same address on your computer:

ssh -L 8000:127.0.0.1:8000 <user>@<box>

Or add --host 0.0.0.0 to the install command, and open http://<box>:8000/dashboard/#/setup from any computer on your network.

The dashboard asks for no password until you set one. Until then, anyone who can reach the port can run the setup and change what it shows, so keep the engine on a trusted network.

The Setup Guide walks through the model, its weights, the tools beside it and the parallel requests in eight steps, shows every change and then installs it. Wren, a small model on the box, answers questions on the side.

Advanced: from the command line

The Setup Guide runs use-profile.sh for you. To pick the weights from a shell instead, after the two commands:

~/TandemLLM/run/ops/use-profile.sh --list                # every profile
~/TandemLLM/run/ops/use-profile.sh <profile> --dry-run   # each file's URL and size
~/TandemLLM/run/ops/use-profile.sh <profile>             # download, check, start

Setup guide The manual, one PDF Changelog

Open source, with a paper

License

The engine is AGPL-3.0-only for everyone, or under a commercial license for use without the AGPL's obligations. The docs are CC BY 4.0. The weights derived from Qwen are Apache-2.0; the Nemotron packs carry NVIDIA's OpenMDW-1.1.

Commercial license

Paper

Khaled Bakeer. StairCut: sizing speculative draft trees on a measured verification-cost staircase. 2026. Zenodo, 10.5281/zenodo.23045275.

The paper measured commit 431ddee; release v0.2.0-staircut is its code.

Community weights

Packs other people publish sit in the Setup Guide next to ours, each with its author and its Hugging Face repository. Adding one takes a pull request with one JSON file, and a maintainer runs it on a DGX Spark before the entry says verified.

The community packs Add your pack

TandemLLM builds on Qwen's models, turboderp's exllamav3, Inco AI's DFlash2 drafter, NVIDIA's Nemotron models and LiLiCorr drafters, LiquidAI's LFM2.5, PyTorch and Triton.