3 ms·
FlashAttention: Fast Transformer training with long sequences
- sebzim4500 3y agoIt's insane that FlashAttention was released 16 months ago. It feels like a decade.
- thawab 3y agoThe same author Tri Dao, released FlashAttention 2 in July. https://together.ai/blog/tri-dao-flash-attention https://together.ai/blog/tri-dao-flash-attention
- kken 3y agoHere is a recent interview with the author of FlashAttention, Tri Dao: https://www.youtube.com/watch?v=J4-qZ6KBalk https://www.youtube.com/watch?v=J4-qZ6KBalk
- deleted 3y ago[deleted]
- ttul 3y agoIt’s basically a way of more efficiently making use of memory transfers during the calculation of the attention blocks in a transformer. You transfer a block at a time, increasing inference throughout because less time is spent overall fetching things from slow memory.
- 1024core 3y agoHas anybody used FlashAttention in their model? Are there any benchmark numbers on the quality impact?
- pama 3y agoThe result is identical to regular attention in transformers but training can be about four times faster, so there is almost no reason to not use it.
- NLPaep 3y agoNot quite. There can be non-deterministic race conditions, and strange head size and sequence length requirements.
- pama 3y agoYes. For a model within the limits of the head requirements, however, you wouldn’t be able to see a quality difference from regular attention. Non determinism is a performance price; regular transformers may also suffer from it depending on the implementation.
- dang 3y agoRelated: FlashAttention-2, 2x faster than FlashAttention - https://news.ycombinator.com/item?id=36761988 https://news.ycombinator.com/item?id=36761988 - July 2023 (18 comments) FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness - https://news.ycombinator.com/item?id=31568090 https://news.ycombinator.com/item?id=31568090 - May 2022 (3 comments)
- fzysingularity 3y agoAlso, isn't the author Tri Dao at Together AI now as their chief scientist?