A model named 24GB invites a tidy mental calculation: buy that much memory, download the weights, and start an agent. NVIDIA’s Nemotron 3.5 Bolt 74B-A9B 24GB GGUF is a useful case study in why that calculation is wrong without making the model itself a villain. The number describes a carefully compressed model, not a universal budget for the server, its context, its vision stack, its speculative decoder, or the other work on the machine.

I checked NVIDIA’s published model card and the actual sizes of the downloadable files. I did not run the model or independently measure throughput, memory use, quality, or fit on a particular device. This is a deployment-budget audit, not a benchmark disguised as one.

Start with what is really in the download

Bolt has 74 billion total parameters, about 9 billion active per token, and a hybrid Mamba-2/Transformer mixture-of-experts design. Its native IQ2 GGUF compresses the weights aggressively. The main downloadable GGUF is 22,018,326,528 bytes. The separate vision projector is 1,733,692,736 bytes. Those two files alone occupy about 23.75 GB in decimal units on disk. NVIDIA also publishes a compatible LiLiCorr speculative-decoding drafter whose GGUF is 998,242,560 bytes; including that optional file takes the three-file download to about 24.75 GB. I obtained those byte counts from the official Hugging Face repository file metadata, not from a local inference run. Main model and projector · LiLiCorr drafter.

Disk bytes are not peak GPU or system-memory bytes. File mapping, offloading, and the backend change what is resident where; conversion from decimal GB to binary GiB changes the printed number, not the underlying bytes. A file-size sum cannot tell me whether a particular deployment will fit. But it immediately tells me what the model’s label does not include.

The projector matters only if you want the advertised image/video path; the drafter is optional and is intended to accelerate generation through speculative decoding, not to add model intelligence. Neither is a mysterious mandatory tax on a text-only session. Conversely, omitting them and announcing that every advertised capability fits in the weight file would be sleight of hand.

Context is a capacity promise, not free memory

NVIDIA advertises up to 256K tokens of context and says its architecture search targeted a sub-24 GB footprint at batch size one and 128K context, including selected runtime states. That is a meaningful engineering target, not merely a file-name trick. But its own published local recipes do not run maximum context with unlimited parallelism: the cited 28 GB recipe specifies a 65,536-token context and parallelism of one, with quantized key/value cache and a separate draft model. The card also lists a 32 GB recipe for full 262,144-token context. Those are vendor recipes, not my measured minimums, and they should not be read as proof that a 24 GB GPU cannot run a shorter, differently configured text-only workload. They are a warning against treating the model name as a plug-and-play VRAM requirement. See NVIDIA’s deployment instructions.

Every additional live sequence needs runtime state. More context, more concurrent requests, larger batches, a vision path, and speculative decoding change the memory and latency picture. The architecture and quantization of the weights do not erase those costs. This is especially important for agents: a short chat response and a long tool-using investigation may share the same model name while stressing completely different parts of the runtime.

There is a second number trick in the A9B suffix. Active parameters describe approximate per-token compute, not the total stored weights. A mixture-of-experts model can activate only a fraction of its experts while still needing a way to store and access the larger parameter set. Comparing a 9B-active MoE with a dense 9B model solely on that suffix is not a memory comparison.

A benchmark table is not your benchmark

NVIDIA’s card reports 50.98 on Terminal-Bench 2.1 for its IQ2 Bolt and 44.38 for a Qwen3.6-35B-A3B BF16 baseline. That is a vendor-run comparison between different models and different precisions, not a matched local test, and not evidence that Bolt will outperform your existing agent stack at your workload or latency budget. Other reported task scores likewise belong to the publisher’s evaluation setup. I would use them to choose what to test, not as a purchase order. Benchmark table and setup.

The model is downloadable under OpenMDW-1.1, not Apache-2.0 or MIT. The license permits broad use subject to its terms, including notice-retention requirements on distribution and a litigation-related termination clause. Calling it “open weights” describes availability; it is not a substitute for reading the license before redistribution. Nor does a downloadable GGUF guarantee that your inference backend handles this particular hybrid architecture, chat template, projector, and draft path correctly.

The test I would actually run

Before replacing an agent model, I would fix a task set and record four things side by side: successful final outcomes, end-to-end latency (including tool calls), peak total memory under the intended context/concurrency, and exact runtime configuration. I would include text-only and multimodal tasks separately; test the same context and permission boundary; record failures and retries rather than only the best answer. I would run the baseline and Bolt through the same harness. Until that exists, there is no measured replacement verdict.

This is not a takedown of compression. Getting a 74B-total-parameter hybrid into a roughly 22 GB weight file is interesting engineering. It makes large models more accessible. It just doesn’t repeal runtime accounting. When a model page says “24GB,” ask: 24 GB of which resource, for which files, at which context, on which backend, with how many simultaneous requests? The answer is the deployment. The label is only the beginning.