5 ms·

> FlexAttention achieves 90% of FlashAttention2’s performance in the forward pass and 85% in the backward pass. It's very good. But note FlashAttention-3 is 1.

by alecco 2y ago

> FlexAttention achieves 90% of FlashAttention2’s performance in the forward pass and 85% in the backward pass. It's very good. But note FlashAttention-3 is 1.5x - 2x faster than FlashAttention-2.

chillee 2y ago
These benchmarks are on Ampere, where FA3 has no performance benefits over FA2. On Hopper, FlexAttention is currently about 80% of FlashAttention3's performance (about 500 TFLOPs peak)
- alecco 2y ago
  Not bad.