While prefill is bottlenecked by GPU compute time, decode might be bottlenecked by GPU memory bandwidth, as you basically need to go through entire KV cache for

While prefill is bottlenecked by GPU compute time, decode might be bottlenecked by GPU memory bandwidth, as you basically need to go through entire KV cache for each new token. So compression can make it faster - you will use more GPU compute but less memory bandwidth for attention calculation