3 msΒ·
According to the paper Flash Attention also needs quadratic memory: Let π be the sequence length, π be the head dimension, and π be size of SRAM with π <=
by WithinReason 3y ago
According to the paper Flash Attention also needs quadratic memory:
Let π be the sequence length, π be the head dimension, and π be size of SRAM with π <= π <= ππ. Standard attention (Algorithm 0) requires Ξ(ππ+πΒ²) HBM accesses, while FlashAttention (Algorithm 1) requires Ξ(πΒ²πΒ²Mβ»ΒΉ) HBM accesses.
- GaggiX 3y agohttps://github.com/HazyResearch/flash-attention#memory https://github.com/HazyResearch/flash-attention#memory "standard attention has memory quadratic in sequence length, whereas FlashAttention has memory linear in sequence length." I guess you have just reported how many times the layer will need to access the memory, not how much memory usage scales with sequence length.