3 ms·
Wouldn’t the softmax typically be “fused” with the matmul though?
by bjornsing 2y ago
Wouldn’t the softmax typically be “fused” with the matmul though?
- anewhnaccount2 2y agoYes but as far as I understand this is only really usefully possible with FlashAttention. (The main idea is that you have to use the log-sum-exp trick when computing the softmax, but can't compute the max activation incrementally so have to rescale everything.)