3 ms·
Materializing the attention matrix has you slower than FlashAttention much of the time, just because you get severely memory bandwidth bound and are worse off t
by namibj 3y ago
Materializing the attention matrix has you slower than FlashAttention much of the time, just because you get severely memory bandwidth bound and are worse off than mild extra computations resulting from streaming it (and especially re-computing the attention for the backwards pass, though that won't need extra memory bandwidth, just minor compute.).
- hansvm 3y agoYes, but the relevance of that is escaping me. Would you mind elaborating?
- namibj 3y ago> Current performant (high accuracy) LLMs have a quadratic cost (space and time) in sequence length, FlashAttention only takes polylogarithmic space overhead beyond the linear needed for attention. Yes, quadratic compute, but that's much more tame.