4 ms·
> Decode is memory bound. > FlashAttention ... such that you can remain compute bound at lower batch sizes during decode. So, which one is it then?
by menaerus 2y ago
> Decode is memory bound.
> FlashAttention ... such that you can remain compute bound at lower batch sizes during decode.
So, which one is it then?
- FL33TW00D 2y agoIt depends on the batch size and the accelerator you're running on! Decode is *typically* memory bound unless you can hit high batch sizes (in the hundreds), which is hard during serving due to the contention between batch size and low TTFT. https://jax-ml.github.io/scaling-book/inference/ https://jax-ml.github.io/scaling-book/inference/ - good read!