2 ms·
Awesome. Do you happen to have a benchmark against the latest (v9.1) cuDNN implementation?
by pama 2y ago
Awesome. Do you happen to have a benchmark against the latest (v9.1) cuDNN implementation?
- Aaryan44 2y ago@pama, if useful - here are utilization numbers for our attention backwards kernels (causal and non-causal, head dim = 64): https://github.com/HazyResearch/ThunderKittens/blob/main/attn.png https://github.com/HazyResearch/ThunderKittens/blob/main/att...