Nvidia has published its answer to the AI industry’s memory crunch: design the whole machine around the memory, not the other way round. In a technical post on 5 November, the company said its Vera Rubin NVL72 system delivers up to 30 times the agentic throughput per dollar of its Blackwell Ultra predecessor, measured on the SemiAnalysis AgentX benchmark, while high-bandwidth memory per GPU rose only about 3%. The claim lands as memory supply tightens and prices climb across the industry.
The memory wall, in Nvidia’s telling
The problem, as Nvidia frames it, is that high-bandwidth memory — HBM — is the fastest, most expensive and most contested tier in an AI factory, and it is where model weights and the KV cache compete for space. Agentic workloads make it worse: each session carries context across model and tool calls, sometimes processing up to 3 million tokens.
Adding more HBM looks like the obvious fix, but Nvidia argues capacity alone cannot help once bandwidth becomes the bottleneck. Its analogy is a warehouse: more floor space holds more stock, yet the loading docks set how fast goods move, and once they saturate, extra space adds cost and power without speeding anything up.
Five levers for the same bytes
Nvidia’s answer is what it calls full-stack codesign: designing GPUs, CPUs, networking, storage and software together so every byte does more work. The first lever is precision. Its NVFP4 format stores weights in 4 bits, shrinking a model to roughly a third of its FP16 size and freeing HBM for the KV cache, larger batches and more concurrent users.
The second lever spreads model weights across the NVL72 rack, so each GPU holds a smaller share and keeps more HBM free. The third is software: Nvidia Dynamo treats the KV cache as a fleet-wide resource, routing requests to where their context already lives and moving idle context off HBM to cheaper storage instead of recomputing it later.
Two more levers handle context that is waiting rather than working. The Vera CPU parks idle KV cache in up to 1.5 terabytes of LPDDR5X memory per chip, linked to Rubin GPUs at 1.8 terabytes per second, while a CMX storage tier — shared flash built for KV cache — holds context that must survive across turns, agents and nodes.
The numbers behind the claim
The headline figures come from SemiAnalysis AgentX, a benchmark that replays recorded coding-agent sessions with real tool-call gaps and growing context. On a DeepSeek V4 Pro workload, Nvidia says GB300 NVL72 delivers up to 10 times the throughput per dollar of the H200 platform at 100 tokens per second per user, with HBM per GPU roughly doubling.
Vera Rubin NVL72 delivers up to 30 times the throughput per dollar of GB300 NVL72 at 125 tokens per second per user, while HBM capacity per GPU rose only about 3% as bandwidth climbed to 19.2 terabytes per second. The gains come from using memory better, not from adding much more of it.
Six model builders — Black Forest Labs, Thinking Machines Lab, Reflection, Meta and StepFun, plus Nvidia itself — have shipped models with native NVFP4 checkpoints, including FLUX.2 [klein], Inkling, Beam 1.0, Meta Muse Spark and Step 5 Preview. Nine storage vendors, from Dell and HPE to NetApp and VAST, are named as CMX partners.
Our Read
The 30x figure is striking, but it is a vendor benchmark on a vendor-chosen workload at a specific interactivity level — 125 tokens per second per user — and it has not been independently reproduced. Read it as the ceiling of what codesign can deliver when everything lines up, not as a blanket speedup every deployment will see.
Even discounted, the direction matters. Memory, not raw compute, is now the binding constraint on AI factories: HBM is scarce, its price is rising, and Nvidia itself has warned that memory costs will pressure its margins. A platform that serves more agents from the same memory is worth more than one that simply adds capacity nobody can buy.
The competitive read is pointed. AMD, Google’s TPUs and the hyperscalers’ custom silicon all face the same memory wall, and the winner will be whoever wastes the fewest bytes. Nvidia’s answer — shrink, share and tier the memory — is also a moat: it only works this well because the GPU, CPU, networking and software all come from one vendor.
Why It Matters
For anyone buying or building AI infrastructure, the metric that matters is shifting from how much memory a system has to how much useful work each byte produces. If Nvidia’s codesign numbers hold up in the field, the same power and memory budgets serve far more agents — and the chip shortfall becomes a problem of efficiency, not just supply.
It reframes SpaceX’s $40 billion Rubin lease. Rubin is not only a faster chip; it is the first platform where the memory system — NVFP4, Vera’s LPDDR5X, Dynamo and CMX — was designed as one stack. Rivals that match Rubin on raw speed will still have to answer the memory question, and that is where this post aims its argument.
Background
Nvidia began production shipments of Vera Rubin in August 2026, with purchase orders from every major hyperscaler, and the platform is expected to make up about a fifth of data centre revenue the following quarter. Memory prices have climbed through the year as HBM demand outruns supply.
Source: NVIDIA blog on memory codesign and the technical deep dive
0 Comments