3 ms·
Has anybody used FlashAttention in their model? Are there any benchmark numbers on the quality impact?
by 1024core 3y ago
Has anybody used FlashAttention in their model? Are there any benchmark numbers on the quality impact?
- pama 3y agoThe result is identical to regular attention in transformers but training can be about four times faster, so there is almost no reason to not use it.
- NLPaep 3y agoNot quite. There can be non-deterministic race conditions, and strange head size and sequence length requirements.
- pama 3y agoYes. For a model within the limits of the head requirements, however, you wouldn’t be able to see a quality difference from regular attention. Non determinism is a performance price; regular transformers may also suffer from it depending on the implementation.