The Hidden Hardware Wall: Memory Bandwidth
When you look at GPU spec sheets, everyone obsesses over TFLOPS: how many trillions of floating-point operations the silicon can execute per second. But in production autoregressive generation, compute is rarely the bottleneck. The real villain is Memory Bandwidth.
Every single time an LLM decodes a new token, it must stream the entire accumulated Key and Value (KV) tensors for all prior tokens across the memory bus. For a 70B model with a 128k context window, the uncompressed FP16 KV-cache occupies a mind-boggling 32 gigabytes of VRAM per concurrent user.
The OOM Nightmare in Multi-User Production
If you serve just four simultaneous users querying 64k tokens on an 80GB GPU, the KV-cache consumes 64GB of memory, instantly triggering Out-Of-Memory (OOM) kernel panics.
The Breakthrough: SnapKV, Sparse Eviction, and 4-Bit Tensors
Over the last few months, a combination of three algorithmic breakthroughs has revolutionized KV-cache density:
- 1. SnapKV Attention Eviction: Analyzes attention weight matrices during prefill to identify tokens with negligible causal influence, evicting 80% of historical tokens without degrading retrieval accuracy.
- 2. FP4/INT4 KV Quantization: Quantizes key and value tensors from 16-bit floats down to 4-bit integers with per-channel outlier scaling.
- 3. Cross-Layer KV Sharing: Shares intermediate key-value projections across adjacent decoder layers, cutting memory footprint by half.
| Cache Compression Strategy |
|---|