8Architecture

This part explains how the engine works, from the model’s layers to the measurement method. This first chapter is the tour: the model, the byte budget that sets the speed limit, the decode loop, and a map of the code.

8.1 The model

engine/config.py reads the checkpoint’s config.json and nothing else. The text model has 64 decoder layers, a hidden size of 5,120, a dense SiLU MLP of width 17,408 in every layer, and a vocabulary of 248,320 tokens. It has no mixture of experts. A 0.92 GB vision tower sits in the same checkpoint; the language-model loader skips it by name, and engine/vision.py loads it on its own for image input.

Every fourth layer (3, 7, 11 and so on up to 63) is full attention, 16 layers in all. Gated DeltaNet fills the other 48. It is a recurrent layer: it keeps a fixed-size state per head instead of a KV cache that grows with every token.

layer kind	count	state per sequence
Gated DeltaNet	48	an fp32 state of 48 heads, each 128 by 128 (3.15 MB), and a 4-wide convolution tail (80 KB), per layer
full attention	16	BF16 keys and values, 4 KV heads of 256: 4 KB a token per layer

That split shapes most of the code. Published work on speculation and caching assumes attention, where undoing a token means moving a pointer back by one row. A recurrent state has absorbed every token and has no inverse. The engine needs exact ways to undo it, to branch it for a tree of guesses, and to restore it from a cache. Chapter 11 shows the undo and the branch, and Chapter 13 the restore.

8.2 The byte budget

A decode step reads every weight once. Table 8.2 sums the bytes from the checkpoint’s safetensors headers.

group	bytes a step
48 Gated DeltaNet layers	18.43 GB
16 attention layers	5.96 GB
the output head, BF16, 248,320 rows by 5,120	2.54 GB
total, FP8 as shipped	26.93 GB

The three MLP projections are 63.6 % of that. Quantising them first, then the rest, moved the budget down in steps, and each step had to pass the quality gate (Chapter 9). Figure 8.2 shows the steps and the floor each puts under a step at the board’s paper bandwidth of 273 GB/s.

State is small beside the weights. The 48 recurrent states are 151 MB together. The KV rows cost 64 KB a token across the 16 attention layers: 2.1 GB at 32k tokens and about 17 GB at the full 262,144-token context.

8.3 The decode loop

Speed is one fraction:

A round starts from the last committed token. The drafters propose a tree of guesses after it. One forward pass runs over the whole tree through all 64 layers. The engine reads the model’s own choice at every node and keeps the longest path where the model agrees with the guesses, plus one more token, the model’s own choice after the last accepted guess, so a round yields at least one token. Then it commits the state of that path and starts again.

On the served profile a round on the benchmark row takes 93.4 ms and commits 4.58 tokens, and the row’s rate is 49.89 tok/s (measured, the paper). The rate is the mean of the 50 requests’ own rates, not the quotient of the two means (4.58 tokens in 93.4 ms is 49.0 tok/s). Without speculation the same box writes 13.69 tok/s at 72.8 ms a token. Chapter 11 follows a round step by step.

8.4 The parts

Table 8.3 maps each part of the engine to the files that implement it and the chapter that explains it.

part	where	chapter
checkpoint layout and loading	engine/loader.py, engine/config.py	16
the quantiser and its gate	tools/quant_nvfp4.py, tools/quant_head.py, tools/quality_gate.py	9
matrix-product kernels	tools/nvfp4_*.py, tools/fp8_linear.py, tools/head_gemv.py	10
recurrent-layer kernels	tools/gdn_*_kernels.py, engine/gdn.py	10
target model, verify and commit	engine/model.py, engine/tree.py	11
drafters	engine/drafters/	11
routers: StairCut, length router, tree	engine/lenrouter.py, engine/router.py, engine/prices.py	11
sampling and penalties	engine/sample.py, engine/penalty.py	12
structured outputs	engine/grammar.py	14
caches	engine/cache.py, engine/disk_cache.py, engine/spill.py	13
settings registry	engine/settings.py	6
the server	server/app.py, server/stream.py, server/toolcall.py	14
the conductor	server/conductor/	19
live activity, metrics, ledger	server/live.py, server/activity.py, server/metrics.py, server/ledger.py	25
the dashboard	dashboard/, docs/contract/dashboard-v1/	25
the proxy and the collector	proxy/	5
service scripts and the gate	ops/	6
measurement tools	tools/row3.py, tools/block_ab.py, tools/verify_spec.py	15

8.5 One sequence, or several

The numbers in this part describe the loop of one sequence, which is what the shipped profile runs. With --parallel N the engine holds N sequences over the one copy of the weights, and requests that decode at the same time verify their trees in one forward pass, so the weights are read once for all of them (Section 14.13). With one sequence, requests wait for the engine’s lock in a bounded queue, the side lane serves a short request inside the running one (Section 14.11), and the proxy relays other backends’ requests without touching the engine’s lock (Chapter 4).