19Nemotron 3.5 Bolt and Super VL

NVIDIA’s Nemotron 3.5 Bolt 74B-A9B is a NemotronH hybrid: 88 layers, 40 of them Mamba2, 40 mixture-of-experts and 8 attention layers without positional encoding. A token runs 9 of up to 496 experts a layer. It is the fourth model TandemLLM serves; its support ships in release 0.10.0. On one DGX Spark the engine decodes it at 43.3 tok/s from our 2.0 EXL3 pack and at 27.8 tok/s from NVIDIA’s IQ2 GGUF, and with NVIDIA’s LiLiCorr drafter and the lookup at 65.7 tok/s on prose and 102.8 on code (measured, 2026-10-16, the mean of 3 runs). TandemLLM serves Bolt’s text; its RADIO image tower is not served yet. Release 0.11.0 adds the larger Nemotron 3.5 Super VL 120B-A12B (Section 19.5). docs/nemotron-h.md is the same material for the repository, with every check and run.

19.1 The model

part	Bolt 74B-A9B	Lightning 30B-A3B
layers	88: 40 Mamba2, 40 MoE, 8 attention	52: 23 Mamba2, 23 MoE, 6 attention
hidden	4,096	2,688
Mamba2	128 heads of 64, state 128, 8 groups	64 heads of 64, state 128, 8 groups
attention	32 query and 2 KV heads of 128, no positional encoding	the same
MoE	240 to 496 experts a layer, 9 a token, relu$^2$, a 1,024-wide latent, one shared expert of 5,376	128 experts, 6 a token, a shared expert of 3,712
draft layer	none (NVIDIA’s LiLiCorr drafter instead)	one MTP block pair
context	262,144 tokens	262,144 tokens

engine/models/nemotron_h/config.py builds one layer table from hybrid_override_pattern, layers_block_type or block_configs (per-block expert count, top k and width) and from the Omni wrapper’s llm_config; it stops on a field it does not know. Bolt’s KV is 8,192 bytes a token (8 attention layers), 2 GiB at the whole context; each Mamba2 layer keeps its SSM state (fp32) and its convolution’s last three inputs.

19.2 Profiles

profile	disk	weights	KL	top-1	plain
nemotron-bolt-exl3-2.0	23.48 GB	20.22 GiB	0.170	84.1 %	43.3
nemotron-bolt-exl3-2.5	27.61 GB	24.07 GiB	0.094	87.3 %	42.1
nemotron-bolt-exl3-3.0	31.75 GB	27.92 GiB	0.061	89.7 %	41.7
nemotron-bolt-exl3-3.5	35.88 GB	32.01 GiB	0.044	91.1 %	40.6
nemotron-bolt-exl3-4.0	40.01 GB	36.18 GiB	0.034	92.7 %	40.5
nemotron-bolt-exl3-5.0	49.13 GB	44.42 GiB	0.016	94.8 %	34.1
nemotron-bolt-exl3-6.0	58.37 GB	53.00 GiB	0.0087	96.2 %	29.2
nemotron-bolt-iq2	22.04 GB	22.19 GiB	0.259	79.6 %	27.8

The EXL3 packs are converted from NVIDIA’s BF16 checkpoint by exllamav3 1.6.2 with a patch for Bolt’s layer table, head at 6 bits; each is a public repository 0xBakeer/TandemLLM-Nemotron-3.5-Bolt-74B-A9B-EXL3-<bpw>, pinned by commit. The 2.0 pack keeps its Mamba2, attention, latent and shared-expert projections at 4 bits and its routed experts at 2; it is the default, better and faster than NVIDIA’s 2-bit GGUF. nemotron-bolt-iq2 reads NVIDIA’s GGUF as stored: IQ2_XXS experts, NVFP4 Mamba2 projections with their per-tensor scales, Q8_0 head and attention; against llama.cpp on the same file its top token agrees at 97.75 % of 1,024 chat positions, every difference a near tie. The 2.5 to 6.0 profiles have no StairCut price file yet, so the cut learns its prices online.

19.3 One row path and exact rollback

The decode step and every row of a verify block run the same kernels a row at a time: every projection on 16-row tiles, the Mamba2 convolution over the row’s own window, the SSM update walked in fp32 from the committed state along the row’s path, attention over fixed key chunks, the router and the relu$^2$ experts with an ordered combine. A verify block walks each SSM once a head, so a layer’s 4 MB state is read once a block. A verify block writes nothing but its KV rows; the commit walks each state along the kept path with the decode step’s own kernel. So a verify row has the decode step’s bits, and a rollback is exact. The load-time self-check verifies a recorded 24-token decode as one block, every row and the committed states bit for bit, before speculation turns on.

The EXL3 projections run on tools/fn_kernels.dense_rows: each row’s input is rotated once, then one CTA decodes the trellis tiles of a 128-column block once for a group of 16 rows, with each row’s K slices, warp order and output rotation those of the one-row dense call, so every output byte is its byte. An 8-row verify block of the 2.0 pack takes 33.8 ms instead of 74.7 with one CTA a row. The expert bank groups a block’s pairs of one expert, so a group reads that expert once.

19.4 LiLiCorr

Bolt has no MTP head. NVIDIA’s drafter for it is LiLiCorr (0.96 GB, NVFP4), ported from vLLM’s and llama.cpp’s implementations to engine/drafters/lilicorr.py. The engine keeps the residual rows entering six of Bolt’s layers in a ring a sequence; the drafter reads them, runs a block of 8 positions through five layers of sliding-window attention with sink logits and a dynamic convolution, and walks a candidate lattice to 7 tokens, all as one CUDA graph. It drafts beside the lookup in the EXL3 profiles, and StairCut cuts every block; the IQ2 profile drafts with the lookup alone, since with LiLiCorr beside it StairCut kept plain steps on this build (cause not isolated). Every drafted token is verified, so the text is the plain decode’s. On 8 chat prompts its accepted length is 4.55 tokens a block, against 4.57 for llama.cpp with the same drafter (measured).

ask	plain	LiLiCorr + lookup	llama.cpp + LiLiCorr
prose	43.4	65.7 (3.0)	49.3
code	43.4	102.8 (4.9)	92.2
chat	43.3	69.4 (3.1)	62.0
edit	43.2	180.5 (13.6)	92.5
copy	43.0	244.7 (15.8)	132.0

Three requests at once with LiLiCorr and the lookup run at 72.1 tok/s in total, every text its request’s alone. Of 20 server requests, 16 equal the plain answers and 4 equal them up to the reasoning budget’s forced close, which lands at the end of a verified block (Chapter 12).

19.5 Super VL 120B-A12B

Nemotron 3.5 Super VL 120B-A12B has Bolt’s layout at a larger size: 88 layers (40 Mamba2, 40 mixture-of-experts, 8 attention), 512 experts a layer with 22 a token and one shared expert, and one MTP block. The checkpoint reads 1,048,576 tokens of context; the profiles serve 262,144. It is the fifth model TandemLLM serves, from release 0.11.0, text only.

Three profiles run it. Two are our EXL3 packs, converted from NVIDIA’s BF16 checkpoint (revision 97e00975) by exllamav3 1.6.2 with a patch for Super VL’s layout; each is a public repository 0xBakeer/TandemLLM-Nemotron-3.5-Super-VL-120B-A12B-EXL3-<bpw>, pinned by commit. The third reads NVIDIA’s NVFP4 release as stored: ModelOpt NVFP4 for the routed and shared experts, FP8 with a tensor scale for the Mamba2 and attention projections and the head (tools/nvfp4_relu2.py). A decode token reads 10.1 GB of that release.

profile	disk	weights	KL	top-1	plain	LiLiCorr
nemotron-super-vl-exl3-4.0	65.0 GB	58.35 GiB	0.049	92.9 %	31.1	33.1 / 53.6 / 101.0
nemotron-super-vl-exl3-2.0	36.84 GB	31.24 GiB	0.325	82.1 %	36.5	35.6 / 46.3 / 125.1
nemotron-super-vl-nvfp4	75.06 GB	66.63 GiB	0.041	93.4 %	19.6	23.9 / 39.9 / 73.5

The 4.0 pack has every projection at 4 bits, head 6; it is the default, within 0.008 KL of NVIDIA’s NVFP4 and 1.6 times as fast plain. The 2.0 pack keeps the routed experts at 2 bits and every other projection at 4. It is 17 % faster plain, but its KL is more than six times the 4.0 pack’s, nearly twice what the same recipe gave on Bolt; it is for a box where 4.0 does not fit.

Super VL’s LiLiCorr drafter carries its own NVFP4 head over the target’s vocabulary and reads five taps with a window of 8,192. Its accepted length is 4.65 tokens a block on the 4.0 pack (4 prompts of 192 tokens, thinking on). The price files carry a spec prior, the 8-row chain plus the draft over 0.75 of that length, so StairCut drafts from the first request. Every drafted text equals the plain run’s.

vLLM (nightly-3f1bcfd8, NVIDIA’s flags for the Spark) on NVIDIA’s NVFP4 release decodes at 19.4 tok/s plain and at 29.9 / 52.7 / 72.4 on prose / code / edit with LiLiCorr; its greedy text changes from run to run, plain included, since those flags round the Mamba cache at random. It reads prompts 4 to 5 times as fast (2,158 to 2,521 tok/s against 476 to 589), and it batches requests at once, where TandemLLM’s drafted slots take turns.

19.6 What is not done

• Bolt’s image tower: the GGUF’s mmproj file and the packs’ BF16 tower are not loaded.
• A CUDA GEMV for the IQ2_XXS shared experts: measured at 32.7 tok/s against 27.9, not in this release, so the IQ2 profile decodes and drafts slower than it could.
• StairCut price files for the 2.5 to 6.0 packs. On the 4.0 pack StairCut keeps plain steps with LiLiCorr in this release (40.3 to 40.5 tok/s on every ask against 40.5 plain); the cause is not isolated, and the other packs have no LiLiCorr rows.
• Super VL: grouped expert prefill (prompts read 4 to 5 times slower than vLLM), a joint verify for requests at once, the vision tower and the MTP head as a draft head.
• Any task benchmark, and the benchmark row of Chapter 15.