Contents TandemLLM Docs
v0.14.2
Star58

TandemLLM in brief, from the website

What happens to a request

A chat request passes seven stations before its first token streams back. Each one is code in the repository and has its own chapter in the manual.

The path below is the 27B's. Every other model shares it; only the drafters differ.

  1. Client

    POST /v1/chat/completions

    Open WebUI, opencode, pi, the OpenAI SDK or plain curl. Anything that speaks the OpenAI chat API works without a plugin.

  2. Proxy

    routes by model id

    A request for a model on another backend, such as vLLM, llama.cpp or a cloud API, goes there and still lands in the same usage ledger. A request for TandemLLM stays on the box.

  3. Conductor

    task class, effort, packages

    The router decides what kind of task the turn is and how hard the model should think. It adds the instructions and skill packages for that kind of task, and memory notes from earlier sessions when you turn them on. None of it changes the model's arithmetic; it changes what goes into the prompt.

  4. Admission

    a queue of 8

    A request that finds the engine busy waits in a short queue. The running request serves a small one, such as a chat title, between two of its own steps. A memory gate checks that the prompt fits before the prefill starts.

  5. Caches

    an anchor every 1,024 tokens

    Reading a long prompt is the most expensive thing the engine does. The next turn of a conversation resumes from the state the last one left, in memory or on the NVMe, and reads only what is new. Each restore gives the stored state back bit for bit.

  6. Drafters and the model, round after round

    up to 15 guesses a round

    A block drafter guesses up to 15 tokens and a lookup drafter offers text it has seen before. StairCut merges them into one tree and cuts it to the size that pays on the measured cost of a check. The big model checks the whole tree in one pass and keeps the longest path it agrees with, plus one token of its own.

    One round, schematic A tree of drafted tokens grows from the last committed token. StairCut cuts the tree at 16 rows. The model checks the kept tree in one pass, accepts four drafted tokens along one path and adds one token of its own. Guesses past the cut are never checked. cut: 16 rows last token drafted guesses kept: the model agrees not checked +1 own
    One round, schematic. Kept guesses are free tokens. At the first wrong guess the model puts in its own token, so nothing a drafter writes reaches the answer unchecked.

    The drafters depend on the model: fine-tuned DFlash2 drafters for Qwen3.8-27B, the checkpoint's own draft layer for Flash-Next and Ornith, NVIDIA's LiLiCorr for the Nemotron models, and the lookup for every model. Flint drafts with the lookup alone.

  7. Monitor and stream

    four safe points

    While the model writes, the conductor reads the output at fixed points. It can close a reasoning loop, continue a turn that ended empty, or repair a broken tool call. Each round's verified tokens stream back at once, and tool calls come back typed by their JSON schema.

TandemLLM 0.14.2, manual of October 2026. Built 2026-10-23. The text is CC BY 4.0.