This is pretty great. PyTorch uses triton as the backend for torch.compile (the big feature of PyTorch 2.0, and the necessary part for making Flex Attention in

This is pretty great. PyTorch uses triton as the backend for torch.compile (the big feature of PyTorch 2.0, and the necessary part for making Flex Attention in the about to be released 2.5 usably fast). Triton's team doesn't support Windows, and, worse yet, does not accept community PRs to enable any sort of support. Here's the github issue: https://github.com/triton-lang/triton/issues/1640 https://github.com/triton-lang/triton/issues/1640 And here's the performance comparison of Flex Attention with and without torch.compile (tldr it's 3x slower than a standard MHA when not compiled): https://github.com/rasbt/LLMs-from-scratch/blob/76e9a9ec02a1a060aac61608598fdd50cc7d52bd/ch03/02_bonus_efficient-multihead-attention/mha-implementations.ipynb https://github.com/rasbt/LLMs-from-scratch/blob/76e9a9ec02a1... EDIT: after taking a look at the repo, the only thing changed in the "46 commits ahead of [official triton]" is the README. Somewhat sketchy.