21Flint

Recursive’s Flint is a mixture-of-experts model for coding agents. About 2.5B of its 39.8B parameters run for each token, and a decode token reads 1.34 GB of our 4.0-bit EXL3 pack and 2.00 GB of the 6.0-bit one (arithmetic, from the headers). On one DGX Spark the engine decodes the 4.0 pack at 128.2 to 136.6 tok/s plain and up to 480.0 tok/s with the lookup (measured, 2026-10-22). Its support ships in release 0.14.0. This chapter says what is different about it and what is measured; Chapter 16 describes the adapter path it came in through, and docs/flint.md has every row.

21.1 The model

part	Flint
layers	49: 2 dense, 47 mixture-of-experts
attention	32 query heads, 4 KV heads of 128, q and k RMS-normed, a learned sink logit per query head
sliding layers	35, a window of 513 positions with rotary; a ring of 640 rows a layer
full layers	14, every position, no positional encoding; KV 28,672 bytes a token
MLP	256 experts a layer, 6 routed by sigmoid score with a bias, and 1 shared; dense SwiGLU in layers 0 and 1
vocabulary	150,144 tokens, four end tokens
context	262,144 tokens
draft layer	none

engine/models/flint holds the family. It reads the configuration strictly, the BF16 checkpoint or our EXL3 packs. The reference forward (ref.py) is written from the published model description and checked against a plain transcription of it. Flint’s router is sensitive to rounding: one BF16 rounding of the residual stream changes the chosen experts for 2 % of tokens at layer 2 and for 46 % at layer 46, and two BF16 forwards of the same weights differ by KL 0.034 (measured, 8 rows of 1,024 tokens).

21.2 The packs

Every projection of a pack is at one rate: 4 bits with the head at 5, or 6 bits with the head at 8. exllamav3 1.6.2 converted them from the BF16 checkpoint at revision 2a247be0, with a patch that adds the Flint architecture. The routed experts’ SwiGLU activations reach 260,000 in the top layers, past fp16, so the packs store each routed expert’s up projection divided by 128 (quantization_config.expert_interm_div), and the engine multiplies the routing weights back. The repositories 0xBakeer/TandemLLM-Flint-EXL3-4.0 (20.6 GB) and -6.0 (30.5 GB) are public and pinned by commit; the BF16 profile reads RecursiveSI/flint.

profile	docs KL	docs top-1	engine KL	engine top-1
flint-exl3-4.0	0.194	79.7 %	0.160	80.5 %
flint-exl3-6.0	0.070	90.8 %	0.052	91.6 %
flint-bf16 (two BF16 forwards)			0.034	92.9 %

21.3 Drafting

Flint has no draft layer. Every profile drafts with the lookup, cut by StairCut from the profile’s price file (ops/prices/flint/), which holds the measured cost of a verify block by size and the plain step’s cost for each context class. After a round with no draft, the next 1, 2, 4, then 8 rounds take the plain step without looking. A verify block reads the KV once, so the lookup pays more at long context.

		prose	code	chat	edit	copy	game
4.0	plain	136.6	133.2	133.3	128.2	128.2	129.3
	lookup	135.4	139.8	134.0	142.3	480.0	149.5
6.0	plain	98.4	97.2	97.1	94.3	94.3	95.0
	lookup	97.1	103.2	98.5	107.3	353.4	114.1
BF16	plain	42.6	42.4	42.4	41.9	41.9	41.9
	lookup	42.2	50.9	42.9	48.4	144.1	42.5

On 4.0, decode falls with the context: 101.7 tok/s at 16k, 61.0 at 64k, 39.5 at 128k and 23.0 at 256k. At 128k a step spends 17.8 of its 25.3 ms reading 3.76 GB of keys and values at about 211 GB/s. Behind a 124,000-token prompt of code the lookup ran a copy ask at 79.5 tok/s against 41.3 plain and an edit ask at 129.7 against 42.4 (measured).

21.4 Serving

Two slots, a window up to 262,144 tokens (131,072 for BF16). The model opens its own <think> block as its first token; thinking off puts an empty block after the generation prompt, and an effort level or a budget closes the block at its cap. Tools are called in Flint’s own form, <tool_call>NAME<arg_key>K</arg_key><arg_value>V</arg_value></tool_call>, and written back the same way. Temperature 1.0, top_k 20 and top_p 0.95 come from its generation config. The decode step runs the shared expert on a second stream and its Triton kernels as programmatic dependents, with the same bits as one stream. A 4,096-token prompt reads at 5,375 tok/s on 4.0.

21.5 The e4m3 KV cache

FLINT_KV_FP8=1 keeps the 14 full layers’ keys and values as e4m3 codes with one fp32 scale a (head, token), 14,784 bytes a token instead of 28,672. The output is no longer the exact model’s, so the knob is off by default and the exactness checks hold only with it off. On 4.0, against the exact cache, the last 1,024 tokens of a 32,768-token source file read at KL 0.070 (median 0.005), 93.8 % top-1 agreement and perplexity 7.165 against 7.465; at 122,880 tokens, KL 0.050, 95.7 % and 4.053 against 4.132. Plain decode runs at 79.4 tok/s against 60.6 at 64k, 57.1 against 39.3 at 128k and 36.6 against 23.05 at 256k; prompt reading is 5 to 12 % slower (measured, one run a row).

To turn it on in the Setup Guide, pick Flint, go to the Tools step, switch on Long-context cache (FP8) under Features, and finish Review and Install, which restarts the engine. From the command line, add the line QWEN38_FLINT_KV_FP8=1 at the end of ~/TandemLLM/run/ops/serve.env and run ~/TandemLLM/bin/tandem restart. /health then shows "kv_cache": "e4m3" under weights. To turn it off, switch it off in the Setup Guide, or delete the line and restart; /health shows "kv_cache": "bf16" again. A line in serve.env wins over the Setup Guide’s switch.

21.6 What is not measured

• Sampled decoding, images (the model reads text) and repeat runs of the speed rows.
• tool-eval-bench beyond three scenarios and the held-out pack.
• A draft head. One we trained on the model’s own outputs reached 21 % depth-1 acceptance on held-out text; 1.4 times plain needs about 80 % at 2k and 55 % at 128k.