Architecture

A decode step of Qwen3.8-27B reads about 27 GB of weights in the vendor's FP8 format. The DGX Spark reads memory at 273 GB/s on paper and about 240 GB/s in practice. Those two numbers put a floor of about 100 ms under every step, or 10 tok/s, before any code runs. TandemLLM exists to get under that floor while writing the model's own greedy text, with one documented exception at near ties (exactness.md).

It does two things. It cuts the bytes a step reads, from 27 GB to 15 GB. And it makes one step produce about 4.6 tokens: cheap drafters guess, and one pass of the big model checks all the guesses.

The engine serves a second model the same way, Qwen3.8-Flash-Next, a mixture of experts of the same family that reads about 4.23 GB a plain token from turboderp's 3.05 EXL3 pack. The box runs one of the two at a time. Most of this page describes the 27B. The second model says what differs, and Methods written once shows how one server, one set of caches and one speculation method serve both.

The model

engine/config.py reads config.json and nothing else. Its text model has 64 decoder layers, hidden size 5120, a dense SiLU MLP of width 17,408 in every layer, and a vocabulary of 248,320 tokens. The model has no mixture of experts. A 0.92 GB vision tower sits in the same checkpoint. The language-model loader skips it by name and engine/vision.py loads it on its own for image input (server.md).

Every fourth layer (3, 7, 11 and so on up to 63) is full attention, 16 layers in all. Gated DeltaNet fills the other 48. It is a recurrent layer that keeps a fixed-size state instead of a growing KV cache.

That split shapes most of the code. Published work on speculation and caching assumes attention, where undoing a token means moving a pointer back by one row. A recurrent state has absorbed every token and has no inverse, and the engine needs exact ways to undo, branch and restore it. speculative-decoding.md shows the undo and the branch, and caches.md shows the restore.

The byte budget

A decode step reads every weight once. This table sums the bytes from the checkpoint's safetensors headers.

The three MLP projections are 63.6 % of that. Quantising them first, then the rest, moved the budget down in steps, and each step had to pass the quality gate (quantisation.md):

State is small beside the weights. The 48 recurrent states are 151 MB together. KV costs 64 KB a token across the 16 attention layers, so 2.1 GB at 32k tokens and about 17 GB at the full 262,144-token context.

The decode loop

Speed is one fraction:

tok/s = tokens committed a round / (draft ms + verify ms + commit ms)

A round starts from the last committed token. Next, the drafters propose a tree of guesses after it. One forward pass then runs over the whole tree through all 64 layers. The engine reads the model's own choice at every node and keeps the longest path where the model agrees with the guesses. It always keeps one more token, the model's own choice after the last accepted guess, so a round yields at least one token. Then it commits the state of that path and starts again.

On the served profile (the published NVFP4 set, StairCut on) a round on the benchmark row takes 93.4 ms and commits 4.58 tokens, and the row's rate is 49.89 tok/s. The rate is the mean of the 50 requests' own rates (measurement.md), not the quotient of the two means (4.58 tokens in 93.4 ms is 49.0 tok/s). Without speculation, the same box writes 13.69 tok/s at 72.8 ms a token (measured on 28 September on an earlier build, whose path with speculation off is unchanged).

sequenceDiagram
    participant C as Client
    participant S as Server
    participant D as Drafters and router
    participant M as Target model
    C->>S: POST /v1/chat/completions
    S->>M: prefill the prompt (or restore it from a cache)
    loop every round, about 93 ms
        S->>D: last committed token and context
        D->>S: a draft tree (up to 32 nodes)
        S->>M: verify the whole tree in one pass
        M->>S: the model's own token at every node
        S->>M: commit the accepted path (KV rows and recurrent state)
        S->>C: stream the committed tokens
    end

The second model, Qwen3.8-Flash-Next

Flash-Next keeps the 27B's hybrid, Gated DeltaNet layers with an attention layer every fourth, and changes most of what sits around it. engine/models/flashnext/config.py reads its config and refuses any field it does not know by name.

A plain decode token on the 3.05 pack reads about 4.23 GB (arithmetic from the pack's headers, with the mixers as int8). The engine decodes 45.8 tok/s at 1k of context without speculation, about 21.8 ms a token (measured 2026-10-11, one run a class). Its verify rows equal its decode rows bit for bit, so greedy speculation on Flash-Next writes the plain greedy text token for token, with no near-tie exception (exactness.md). The one known difference is where a think budget closes: at the first block boundary past the cap (ENG-235). Through the server on 2026-10-12, one greedy request of up to 512 tokens a class with the MTP drafter and the lookup: prose 51, chat 64, code 83, an edit 132 and a copy 133 tok/s, against 41.5 to 45.7 plain. quantisation.md covers the packs and speculative-decoding.md the drafter.

Methods written once, models plugged in

A second model could have meant a second server, a second set of caches and a second StairCut. The engine avoids that with five layers that depend only downward. A method (speculation, a cache tier, the memory governor, the server) reads a model through its adapter and through data files. It never branches on a model's name, never asks hasattr of an engine and never touches a private field. tests/test_load_shared.py registers a third, made-up architecture on the CPU and gets an answer on /v1/chat/completions with no edit under server/.

flowchart TB
    subgraph M[methods, written once]
        SPEC[StairCut, PriceModel, the MTP drafter, the walk]
        CACHE[every cache tier, the KV pool]
        SRV[server, memory governor, conductor, dashboard, setup guide]
    end
    subgraph A[model adapters, one per architecture]
        Q35[qwen3_5: the 27B]
        FN[qwen4_exp: Flash-Next]
    end
    subgraph F[weight formats]
        FMT[EXL3, NVFP4, FP8, int8 rows, BF16]
    end
    subgraph H[hardware]
        KREG[kernel registry by op, format and device]
        DEV[device profile]
    end
    subgraph T[topology]
        MESH[Mesh, world 1]
    end
    M --> A --> F --> H --> T
    A --> H

Half of it is built. Since 2026-10-12 the adapters, the state, the shared methods and the server's one loader serve every model. Four parts are still design, each with its ticket: format plugins, a kernel registry, device profiles and the mesh.

Model adapters

engine/models/__init__.py keeps a registry of architectures. An entry names the architectures strings and model_types that select it, its config reader, its engine class and, when the model brings one, its server builder, all by import path. architecture(config) returns the entry, or fails naming every registered one.

An adapter is the engine object with a fixed set of verbs. The hot verbs run every round, so they are the engine's own bound methods and never a wrapper: a decode and a verify call the same kernels through the same Python frames as before the layering, and CUDA graphs capture the same launches.

The cold verbs run between requests or around a cache copy. The 27B's are installed on its class by engine/models/qwen3_5/adapter.py, so engine/model.py keeps its bytes and its disk-tier fingerprint.

A model may offer more, and the server checks for each before it uses it: draft_heads() (Flash-Next's MTP layer), side_slot() (a slot over the same weights), forward_tree_multi() (several slots' blocks in one pass), idle_captures() (graphs to capture while the server is idle). Without them the server takes the 27B's path or leaves the feature off.

ModelState: sections and views

The caches copy a sequence through eng.cache_state() (engine/models/state.py), and they know no model. A state has two parts:

Every tier keeps sections in the state's order and views by length. A signature over names, shapes and dtypes refuses a restore across models. Derived buffers are not stored: Flash-Next's pooled indexer keys are rebuilt from the keys in resume_at. tests/test_cache_any_model.py runs a made-up state, whose sections and views have dtypes and shapes neither model uses, through four tiers with no edit in the cache code.

Weight formats

Every weight the engine multiplies by implements Linear (engine/linear.py): a shape, the bytes a use reads, and matmul. The model calls w.matmul(x) and never asks the format. Today's classes: FP8Block, NVFP4Block, FP8Head, BF16Block, EXL3Block (tools/exl3_linear.py), and Flash-Next's Int8Rows for its mixers. Flash-Next's routed experts are not Linear objects: a layer's experts and its shared expert sit in one MoEBank (tools/fn_kernels.py), which the MoE kernels read directly. Each object holds its kernel, so a call costs nothing extra.

The planned format plugin adds two verbs to that: read (one reader for every checkpoint: headers, byte ranges, row and column slices, expert ranges) and bind (set the object's kernel once from the registry below). Until then the 27B reads through engine/loader.py and Flash-Next through engine/models/flashnext/pack.py.

Kernels and devices, then the mesh

The planned registry will describe each kernel as a KernelImpl: the op (gemv, moe_experts, attention_rows and so on), the weight format, the row regime, the device features it needs, whether it is row-invariant, and its entry point with its tile table and torch reference. A format will resolve its op once at load, keyed by (op, fmt, device). A device without a kernel will get the reference, and an exact model there will serve plain decode until a row-invariant twin exists, because the load-time self-check gates speculation (kernels.md).

A device profile will hold what the memory governor and the Setup Guide read about a board (memory, reserve, floor, whether page cache counts as available) and what the kernel build reads (bandwidth, compute capability, features). Today the DGX Spark's numbers sit in each model.yaml and in the profiles, and engine/methods/device.py turns the GPU's name into the id price files are keyed by (dgx-spark for the GB10).

The mesh marks where two boxes will split a model. At world 1, the only size built, it is absent from the code path: no collective runs and the graphs hold the same kernels. Its seams are ENG-259's and come before any work across boxes: weights read by slice, a shard tag on each state view, and the mesh in cache keys and price file names.

The parts

flowchart TB
    subgraph server[server/]
        API[OpenAI API, streaming, tool calls, structured outputs]
        LIVE[Live activity, metrics, usage ledger]
        DASH[Dashboard files and API]
    end
    subgraph engine[engine/]
        LOOP[Decode loop and sampling]
        METHODS[engine/methods: StairCut, MTP drafter, walk]
        ROUTER[the 27B's StairCut, length router, tree router]
        DRAFT[DFlash2 drafters and the lookup drafter]
        MODELS[engine/models: registry, adapters, state]
        MODEL[the 27B: forward, tree verify, commit]
        FNX[Flash-Next: forward, verify, commit]
        CACHE[Resident prefix, state store, disk tier, spill, suffix store]
    end
    subgraph kernels[tools/, CUDA and Triton kernels]
        W4[NVFP4, FP8 and EXL3 matrix products]
        GDN[Gated delta rule for one token, a block, a tree]
        FNK[Flash-Next: MoE experts, row steps, prefill]
    end
    API --> LOOP
    LOOP --> ROUTER --> DRAFT
    LOOP --> METHODS
    LOOP --> MODELS
    MODELS --> MODEL
    MODELS --> FNX
    LOOP --> CACHE
    MODEL --> W4
    MODEL --> GDN
    FNX --> FNK
    LOOP --> LIVE

One request, one engine

By default the engine holds one sequence at a time. Requests wait for the engine lock in a bounded queue (8 by default), and a request that waits longer than the queue timeout gets a 503. The numbers on this page describe that single-request loop. With --parallel N the engine holds N sequences over one copy of the weights and verifies them in one joint pass (parallel-requests.md); it is off in every shipped profile, the Flash-Next ones included.