Aaryan44

4 ms·

1. 1.▲
   by Aaryan44 2y ago

   Kayvon and Kunle are amazing - I took CS149 Parallel Programming two quarters ago and loved it :)
2. 2.▲
   by Aaryan44 2y ago

   @pama, if useful - here are utilization numbers for our attention backwards kernels (causal and non-causal, head dim = 64): https://github.com/HazyResearch/ThunderKittens/blob/main/att...
3. 3.▲
   by Aaryan44 2y ago

   Thanks @lucidrains :)
4. 4.▲
   by Aaryan44 2y ago

   good news - we've actually included optimized causal and non-causal versions of the flash attention backwards pass with TK - would love for you to check them out! causal: https://github.com/HazyResearch/ThunderKitt