BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design
The key-value (KV) cache dominates memory capacity and bandwidth in long-context LLM serving. While 4-bit microscaling formats (NVFP4, MXFP4) shrink the cache and run attention on FP4 Tensor Cores, a
- Upvotes3