4 ms·

hoping an expert can answer a few Qs I have :) Is FlashAttention simply a drop-in replacement for the attention operation in an LLM? Can it be used anywhere th

hoping an expert can answer a few Qs I have :) Is FlashAttention simply a drop-in replacement for the attention operation in an LLM? Can it be used anywhere that an "attention" operation is used? Or does a LLM need to be trained specially to use FA? How does FA relate to attention strategies like GQA (grouped query attention) or sliding-window attention? Are they orthogonal concepts? Or you need a specific FA implementation for each strategy? Recently llama.cpp added flash attention support - does this just mean they started consuming a flash attention-provided CUDA kernel or something? lastly, in this post, they compare FlashAttention to Triton. I thought Triton was like an abstraction layer? Couldn't FA be implemented in Triton? I just don't really get what it means to say "FlashAttention vs. Triton".

zaptrem 2y ago
> Is FlashAttention simply a drop-in replacement for the attention operation in an LLM? Can it be used anywhere that an "attention" operation is used? Or does a LLM need to be trained specially to use FA? Yes > How does FA relate to attention strategies like GQA (grouped query attention) or sliding-window attention? Are they orthogonal concepts? Or you need a specific FA implementation for each strategy? Flash Attention is a way of calculating the Softmax(QK^T)V part of attention, whereas GQA is a way of calculating the Q, K, and V matricies. Sliding window attention (less sure about this, there are a bunch of windowed attention techniques) change the attention mask (the thing that controls which queries can attend to which keys). > Recently llama.cpp added flash attention support - does this just mean they started consuming a flash attention-provided CUDA kernel or something? I don't use llama.cpp but that sounds about right. > lastly, in this post, they compare FlashAttention to Triton. I thought Triton was like an abstraction layer? Couldn't FA be implemented in Triton? I just don't really get what it means to say "FlashAttention vs. Triton". They're talking about a previous Flash Attention implementation written in Triton.
apsec112 2y ago
1) Pretty much, it's mathematically equivalent. The only software issues are things like managing dependency versions and data formats in-memory, but Flash Attention 2 is already built into HuggingFace and other popular libraries. Flash Attention 3 probably will be soon, although it requires an H100 GPU to run 2) Flash Attention 2 added support for GQA in past version updates: https://github.com/Dao-AILab/flash-attention https://github.com/Dao-AILab/flash-attention 3) They're comparing this implementation of Flash Attention (which is written in raw CUDA C++) to the Triton implementation of a similar algorithm (which is written in Triton): https://triton-lang.org/main/getting-started/tutorials/06-fused-attention.html https://triton-lang.org/main/getting-started/tutorials/06-fu...

zaptrem 2y ago

> Is FlashAttention simply a drop-in replacement for the attention operation in an LLM? Can it be used anywhere that an "attention" operation is used? Or does a LLM need to be trained specially to use FA? Yes > How does FA relate to attention strategies like GQA (grouped query attention) or sliding-window attention? Are they orthogonal concepts? Or you need a specific FA implementation for each strategy? Flash Attention is a way of calculating the Softmax(QK^T)V part of attention, whereas GQA is a way of calculating the Q, K, and V matricies. Sliding window attention (less sure about this, there are a bunch of windowed attention techniques) change the attention mask (the thing that controls which queries can attend to which keys). > Recently llama.cpp added flash attention support - does this just mean they started consuming a flash attention-provided CUDA kernel or something? I don't use llama.cpp but that sounds about right. > lastly, in this post, they compare FlashAttention to Triton. I thought Triton was like an abstraction layer? Couldn't FA be implemented in Triton? I just don't really get what it means to say "FlashAttention vs. Triton". They're talking about a previous Flash Attention implementation written in Triton.

apsec112 2y ago

1) Pretty much, it's mathematically equivalent. The only software issues are things like managing dependency versions and data formats in-memory, but Flash Attention 2 is already built into HuggingFace and other popular libraries. Flash Attention 3 probably will be soon, although it requires an H100 GPU to run 2) Flash Attention 2 added support for GQA in past version updates: https://github.com/Dao-AILab/flash-attention https://github.com/Dao-AILab/flash-attention 3) They're comparing this implementation of Flash Attention (which is written in raw CUDA C++) to the Triton implementation of a similar algorithm (which is written in Triton): https://triton-lang.org/main/getting-started/tutorials/06-fused-attention.html https://triton-lang.org/main/getting-started/tutorials/06-fu...