5 ms·

Triton Fork for Windows Support

21 points by lnyan 2y ago

yjftsjthsd-h 2y ago
"Triton" here is apparently a programming language, which upstream describes as > This is the development repository of Triton, a language and compiler for writing highly efficient custom Deep-Learning primitives. The aim of Triton is to provide an open-source environment to write fast code at higher productivity than CUDA, but also with higher flexibility than other existing DSLs. So if you clicked in expecting the illumos-based virtualization platform, this isn't that. Though > This is the basis for torchao, which crucially changes some large models from "can't run" to "can run" on consumer GPUs. That's easier than supporting them in other quantization frameworks, or letting the consumers use Linux or WSL Does sound neat on its own merits.
- deleted 2y ago
  [deleted]
Scene_Cast2 2y ago
This is pretty great. PyTorch uses triton as the backend for torch.compile (the big feature of PyTorch 2.0, and the necessary part for making Flex Attention in the about to be released 2.5 usably fast). Triton's team doesn't support Windows, and, worse yet, does not accept community PRs to enable any sort of support. Here's the github issue: https://github.com/triton-lang/triton/issues/1640 https://github.com/triton-lang/triton/issues/1640 And here's the performance comparison of Flex Attention with and without torch.compile (tldr it's 3x slower than a standard MHA when not compiled): https://github.com/rasbt/LLMs-from-scratch/blob/76e9a9ec02a1a060aac61608598fdd50cc7d52bd/ch03/02_bonus_efficient-multihead-attention/mha-implementations.ipynb https://github.com/rasbt/LLMs-from-scratch/blob/76e9a9ec02a1... EDIT: after taking a look at the repo, the only thing changed in the "46 commits ahead of [official triton]" is the README. Somewhat sketchy.
- zorgmonkey 2y ago
  It is mentioned at the top of the readme, but the actual code changes appear to be on the branch v3.1.x-windows (also a few other branches with -windows in the name). Also the triton seem willing to collaborate as long as the patches sent are reasonably minimal and high quality https://github.com/triton-lang/triton/pull/4045#issuecomment-2409922915 https://github.com/triton-lang/triton/pull/4045#issuecomment...
  - lostmsu 2y ago
    Things might have changed since then, but I personally contributed a few Windows support-related changes back in 2021 as an independent contributor: https://github.com/triton-lang/triton/pulls?q=is%3Apr+is%3Aclosed+author%3Alostmsu https://github.com/triton-lang/triton/pulls?q=is%3Apr+is%3Ac...
lostmsu 2y ago
Microsoft should help this project setup CI infra with necessary GPUs.

21 points by lnyan 2y ago

"Triton" here is apparently a programming language, which upstream describes as > This is the development repository of Triton, a language and compiler for writing highly efficient custom Deep-Learning primitives. The aim of Triton is to provide an open-source environment to write fast code at higher productivity than CUDA, but also with higher flexibility than other existing DSLs. So if you clicked in expecting the illumos-based virtualization platform, this isn't that. Though > This is the basis for torchao, which crucially changes some large models from "can't run" to "can run" on consumer GPUs. That's easier than supporting them in other quantization frameworks, or letting the consumers use Linux or WSL Does sound neat on its own merits.

deleted 2y ago
[deleted]

Scene_Cast2 2y ago

This is pretty great. PyTorch uses triton as the backend for torch.compile (the big feature of PyTorch 2.0, and the necessary part for making Flex Attention in the about to be released 2.5 usably fast). Triton's team doesn't support Windows, and, worse yet, does not accept community PRs to enable any sort of support. Here's the github issue: https://github.com/triton-lang/triton/issues/1640 https://github.com/triton-lang/triton/issues/1640 And here's the performance comparison of Flex Attention with and without torch.compile (tldr it's 3x slower than a standard MHA when not compiled): https://github.com/rasbt/LLMs-from-scratch/blob/76e9a9ec02a1a060aac61608598fdd50cc7d52bd/ch03/02_bonus_efficient-multihead-attention/mha-implementations.ipynb https://github.com/rasbt/LLMs-from-scratch/blob/76e9a9ec02a1... EDIT: after taking a look at the repo, the only thing changed in the "46 commits ahead of [official triton]" is the README. Somewhat sketchy.

zorgmonkey 2y ago
It is mentioned at the top of the readme, but the actual code changes appear to be on the branch v3.1.x-windows (also a few other branches with -windows in the name). Also the triton seem willing to collaborate as long as the patches sent are reasonably minimal and high quality https://github.com/triton-lang/triton/pull/4045#issuecomment-2409922915 https://github.com/triton-lang/triton/pull/4045#issuecomment...
- lostmsu 2y ago
  Things might have changed since then, but I personally contributed a few Windows support-related changes back in 2021 as an independent contributor: https://github.com/triton-lang/triton/pulls?q=is%3Apr+is%3Aclosed+author%3Alostmsu https://github.com/triton-lang/triton/pulls?q=is%3Apr+is%3Ac...

lostmsu 2y ago

Microsoft should help this project setup CI infra with necessary GPUs.