19Nemotron 3.5 Bolt 74B-A9B

NVIDIA’s Nemotron 3.5 Bolt 74B-A9B is a NemotronH hybrid: 88 layers, 40 of them Mamba2, 40 mixture-of-experts and 8 attention layers without positional encoding. A token runs 9 of up to 496 experts a layer. It is the fourth model TandemLLM serves; its support ships in release 0.10.0. On one DGX Spark the engine decodes it at 43.3 tok/s from our 2.0 EXL3 pack and at 27.8 tok/s from NVIDIA’s IQ2 GGUF, and with NVIDIA’s LiLiCorr drafter and the lookup at 65.7 tok/s on prose and 102.8 on code (measured, 2026-10-16, the mean of 3 runs). TandemLLM serves Bolt’s text; its RADIO image tower is not served yet. docs/nemotron-h.md is the same material for the repository, with every check and run.

19.1 The model

part	Bolt 74B-A9B	Lightning 30B-A3B
layers	88: 40 Mamba2, 40 MoE, 8 attention	52: 23 Mamba2, 23 MoE, 6 attention
hidden	4,096	2,688
Mamba2	128 heads of 64, state 128, 8 groups	64 heads of 64, state 128, 8 groups
attention	32 query and 2 KV heads of 128, no positional encoding	the same
MoE	240 to 496 experts a layer, 9 a token, relu$^2$, a 1,024-wide latent, one shared expert of 5,376	128 experts, 6 a token, a shared expert of 3,712
draft layer	none (NVIDIA’s LiLiCorr drafter instead)	one MTP block pair
context	262,144 tokens	262,144 tokens

engine/models/nemotron_h/config.py builds one layer table from hybrid_override_pattern, layers_block_type or block_configs (per-block expert count, top k and width) and from the Omni wrapper’s llm_config; it stops on a field it does not know. Bolt’s KV is 8,192 bytes a token (8 attention layers), 2 GiB at the whole context; each Mamba2 layer keeps its SSM state (fp32) and its convolution’s last three inputs.

19.2 Profiles

profile	disk	weights	KL	top-1	plain
nemotron-bolt-exl3-2.0	23.48 GB	20.22 GiB	0.170	84.1 %	43.3
nemotron-bolt-exl3-2.5	27.61 GB	24.07 GiB	0.094	87.3 %	42.1
nemotron-bolt-exl3-3.0	31.75 GB	27.92 GiB	0.061	89.7 %	41.7
nemotron-bolt-exl3-3.5	35.88 GB	32.01 GiB	0.044	91.1 %	40.6
nemotron-bolt-exl3-4.0	40.01 GB	36.18 GiB	0.034	92.7 %	40.5
nemotron-bolt-exl3-5.0	49.13 GB	44.42 GiB	0.016	94.8 %	34.1
nemotron-bolt-exl3-6.0	58.37 GB	53.00 GiB	0.0087	96.2 %	29.2
nemotron-bolt-iq2	22.04 GB	22.19 GiB	0.259	79.6 %	27.8

The EXL3 packs are converted from NVIDIA’s BF16 checkpoint by exllamav3 1.6.2 with a patch for Bolt’s layer table, head at 6 bits; each is a public repository 0xBakeer/TandemLLM-Nemotron-3.5-Bolt-74B-A9B-EXL3-<bpw>, pinned by commit. The 2.0 pack keeps its Mamba2, attention, latent and shared-expert projections at 4 bits and its routed experts at 2; it is the default, better and faster than NVIDIA’s 2-bit GGUF. nemotron-bolt-iq2 reads NVIDIA’s GGUF as stored: IQ2_XXS experts, NVFP4 Mamba2 projections with their per-tensor scales, Q8_0 head and attention; against llama.cpp on the same file its top token agrees at 97.75 % of 1,024 chat positions, every difference a near tie. The 2.5 to 6.0 profiles have no StairCut price file yet, so the cut learns its prices online.

19.3 One row path and exact rollback

The decode step and every row of a verify block run the same kernels a row at a time: every projection on 16-row tiles, the Mamba2 convolution over the row’s own window, the SSM update walked in fp32 from the committed state along the row’s path, attention over fixed key chunks, the router and the relu$^2$ experts with an ordered combine. A verify block walks each SSM once a head, so a layer’s 4 MB state is read once a block. A verify block writes nothing but its KV rows; the commit walks each state along the kept path with the decode step’s own kernel. So a verify row has the decode step’s bits, and a rollback is exact. The load-time self-check verifies a recorded 24-token decode as one block, every row and the committed states bit for bit, before speculation turns on.

The EXL3 projections run on tools/fn_kernels.dense_rows: each row’s input is rotated once, then one CTA decodes the trellis tiles of a 128-column block once for a group of 16 rows, with each row’s K slices, warp order and output rotation those of the one-row dense call, so every output byte is its byte. An 8-row verify block of the 2.0 pack takes 33.8 ms instead of 74.7 with one CTA a row. The expert bank groups a block’s pairs of one expert, so a group reads that expert once.

19.4 LiLiCorr

Bolt has no MTP head. NVIDIA’s drafter for it is LiLiCorr (0.96 GB, NVFP4), ported from vLLM’s and llama.cpp’s implementations to engine/drafters/lilicorr.py. The engine keeps the residual rows entering six of Bolt’s layers in a ring a sequence; the drafter reads them, runs a block of 8 positions through five layers of sliding-window attention with sink logits and a dynamic convolution, and walks a candidate lattice to 7 tokens, all as one CUDA graph. It drafts beside the lookup in the EXL3 profiles, and StairCut cuts every block; the IQ2 profile drafts with the lookup alone, since with LiLiCorr beside it StairCut kept plain steps on this build (cause not isolated). Every drafted token is verified, so the text is the plain decode’s. On 8 chat prompts its accepted length is 4.55 tokens a block, against 4.57 for llama.cpp with the same drafter (measured).

ask	plain	LiLiCorr + lookup	llama.cpp + LiLiCorr
prose	43.4	65.7 (3.0)	49.3
code	43.4	102.8 (4.9)	92.2
chat	43.3	69.4 (3.1)	62.0
edit	43.2	180.5 (13.6)	92.5
copy	43.0	244.7 (15.8)	132.0

Three requests at once with LiLiCorr and the lookup run at 72.1 tok/s in total, every text its request’s alone. Of 20 server requests, 16 equal the plain answers and 4 equal them up to the reasoning budget’s forced close, which lands at the end of a verified block (Chapter 12).

19.5 What is not done

• Bolt’s image tower: the GGUF’s mmproj file and the packs’ BF16 tower are not loaded.
• A CUDA GEMV for the IQ2_XXS shared experts: measured at 32.7 tok/s against 27.9, not in this release, so the IQ2 profile decodes and drafts slower than it could.
• StairCut price files for the 2.5 to 6.0 packs. On the 4.0 pack StairCut keeps plain steps with LiLiCorr in this release (40.3 to 40.5 tok/s on every ask against 40.5 plain); the cause is not isolated, and the other packs have no LiLiCorr rows.
• Any task benchmark, and the benchmark row of Chapter 15.