Nvidia Proposes Full-Stack Co-Designed AI Memory Approach as Vera Rubin Platform Grows Per-GPU Memory by Only ~3%

Nvidia proposed a full-stack co-designed approach to AI memory efficiency, emphasizing optimization across GPU, CPU, networking, storage, and software rather than simply increasing GPU memory capacity. The strategy addresses rising memory pressure from larger AI models, longer context windows, and widespread agentic applications amid tightening HBM supply and climbing prices. Key techniques include using the NVFP4 low-precision format to compress model weights and KV cache, migrating temporarily inactive context to lower-cost storage tiers, and freeing up HBM space through multi-GPU distributed deployment. According to SemiAnalysis's AgentX tests, Nvidia's GB300 NVL72 achieved up to 10x higher inference throughput per unit cost compared to the H200 platform under specific workloads. The Vera Rubin NVL72 delivers up to 30x higher inference throughput per unit cost versus the GB300 NVL72, while HBM capacity per GPU increases by only about 3%.