Release notes, v0.10.0
TandemLLM 0.10.0 adds a fourth model, NVIDIA's Nemotron 3.5 Bolt 74B-A9B. It runs from our EXL3 packs at 2.0 to 6.0 bits a weight or from NVIDIA's own 2-bit GGUF, and the EXL3 profiles draft with NVIDIA's LiLiCorr drafter. The Qwen and LFM2.5 models write the same text at the same speed as in 0.9.2, and their saved chats carry over. Every change is in CHANGELOG.md, and the PDF below is the manual.
Nemotron 3.5 Bolt 74B-A9B
Bolt is a hybrid of 40 Mamba2 layers, 40 mixture-of-experts layers and 8 attention layers, with 9 of up to 496 experts per token. The engine runs a verify row with the same bits as the decode step, so a drafted answer is the plain answer. Where a reasoning budget forces a thinking block closed, the close can land a few tokens later (docs/exactness.md).
| profile | download | KL vs BF16 | top-1 vs BF16 | plain decode, tok/s |
|---|---|---|---|---|
nemotron-bolt-exl3-2.0 (default) |
23.48 GB | 0.170 | 84.1 % | 43.3 |
nemotron-bolt-exl3-3.0 |
31.75 GB | 0.061 | 89.7 % | 41.7 |
nemotron-bolt-exl3-4.0 |
40.01 GB | 0.034 | 92.7 % | 40.5 |
nemotron-bolt-exl3-6.0 |
58.37 GB | 0.0087 | 96.2 % | 29.2 |
nemotron-bolt-iq2 (NVIDIA's GGUF) |
22.04 GB | 0.259 | 79.6 % | 27.8 |
The 2.5, 3.5 and 5.0 packs have profiles too. One DGX Spark, 2026-10-15 and 16. Quality: 8 wikitext-2 rows of 1,024 tokens against the logits of NVIDIA's BF16 checkpoint. Decode: one request through the server, 512 greedy tokens, the mean of 3 runs.
The 2.0 pack keeps its routed experts at 2 bits and its dense projections at 4. It scores better and runs faster than NVIDIA's GGUF. With LiLiCorr and the lookup it writes 65.7 tok/s on prose and 102.8 on code. Edit and copy asks reach 180.5 and 244.7 tok/s (512 greedy tokens, thinking off, 3 runs). On the 4.0 pack LiLiCorr gives no speed in this release: StairCut keeps plain steps, and we have not found why. Three requests at once run at 72.1 tok/s in total.
Each pack is a public repository on Hugging Face, 0xBakeer/TandemLLM-Nemotron-3.5-Bolt-74B-A9B-EXL3-<bpw>, pinned by commit, and all seven are in one collection. The GGUF and the drafter come from NVIDIA's repositories. docs/nemotron-h.md has the checks against transformers, llama.cpp and BF16.
Chat tab
- Each chat's sandbox has 1 GiB of memory, and a call that hits the limit fails with "out of memory".
- Attachments are never inlined past the window. A file over a quarter of the free window goes in as one line with its path under
/workspace/uploads/. - A tool call no longer waits without end for its sandbox. After 60 s it fails with the reason.
Upgrade
Run the 0.10.0 installer again, then switch to Bolt. On a box you installed with --setup:
curl -fsSLO https://raw.githubusercontent.com/0xBakeer/TandemLLM/v0.10.0/install.sh
bash install.sh --setup --force-setup --yes
~/TandemLLM/run/ops/use-profile.sh nemotron-bolt-exl3-2.0 --dry-run # each file's URL and size
~/TandemLLM/run/ops/use-profile.sh nemotron-bolt-exl3-2.0
A box installed without --setup upgrades with bash install.sh --yes and the same --dir.
Known limits
- Bolt is text only. Its image tower is not served yet.
- The IQ2 profile runs without a CUDA kernel for its shared experts, so it decodes at 27.9 tok/s, not 32.7.
- The IQ2 profile drafts with the lookup alone. With LiLiCorr beside it, StairCut kept plain steps on this build, and the cause is not known yet. Edit and copy asks still run at 97.9 and 131.2 tok/s.
- The 2.5 to 6.0 packs have no StairCut price file yet. The cut learns their prices while it serves.
Files of the release
- tandemllm-manual-0.10.0.pdf (4460 KiB)