2 ms·
This tweet appears to be taking the original material out of context to misrepresent it: > Rewrite the attention kernel to be persistent. This gives better per
by zahlman 1y ago
This tweet appears to be taking the original material out of context to misrepresent it:
> Rewrite the attention kernel to be persistent. This gives better performance at low-contexts. However, fp16 at large context has suffered a bit due to a ptxas instruction scheduling issue in the softmax partition. fp8 is ~100 tflops faster when the kernel name has "cutlass" in it.
The charitable reading is that, on certain kernels, using fp8 rather than fp16 values gives better performance. (Although I can't even see how the numbers relate to a "~100 tflops faster" claim in any respect, nor does it even list any kernel names or suggest a control kernel!) But this is being presented as if someone has uncovered evidence of cheating on benchmarks.
- saagarjha 1y agoI think you're the one doing that to the tweet, actually.
- zahlman 1y agoWhat are you talking about? When I view the tweet, the only text I see is: > > fp8 is 100 tflops faster when the kernel name has "cutlass" in it > kms
- saagarjha 1y agoAnd it includes a link to show that this is the context it came from.
- zahlman 1y agoAnd when I look at the link, the part I quoted is the relevant text I see. In order to get to the part that you're trying to hold me accountable for, I would furthermore have to click onto the commits tab and search through a 93-commit PR. I thought today I was using a site where trying to think the best of people and propose that someone had taken something out of context, based on the immediately available context having a simpler explanation, would not get me treated like a corporate shill (for a company I don't even care about). Apparently I was wrong.
- saagarjha 1y agoI don't think you are a corporate shill. I do think that you immediately going "clearly the tweet is wrong" without doing any research whatsoever was unwarranted, though. You also keep bringing up that it's 93 commits but all getting squashed you have to do is search for "cutlass" to find out what is going on. I think you're obligated to do at least that when you call it out for being wrong.
- zahlman 1y agoHow did you get "clearly" out of "appears to be"? How did you get "without doing any research whatsoever" out of me demonstrably following the link and reading and quoting what appeared on the facing page?
- saagarjha 1y agoI think if you spent the amount of effort you’ve used to reply to me here to skim through the PR you would’ve changed your mind. I understand that the code requires some domain knowledge to fully understand but I think even a cursory skim would be enough to disabuse me of the idea that the tweet was cherry-picking.
- zettabomb 1y agoNo, that sentence is separate from the rest. Take a look at the pull request: # Up to 150 TFLOPS faster for fp8! if specialization.constants["dtype"] == gl.float8e5: name = "cutlass_" + name
- zahlman 1y agoThe tweet is quoting from the first message in the "conversation" on the PR. There are 93 commits in the PR and GitHub doesn't even default to that tab. I looked at the obvious text and drew the conclusion that was obvious to me.
- imtringued 1y agohttps://github.com/triton-lang/triton/pull/7298/commits/a5e23d8e7e64b8a11af3edc1705407d91084b01d#r2194894641 https://github.com/triton-lang/triton/pull/7298/commits/a5e2... It's literally in the code.
- zahlman 1y agoI already had to deal with Twitter and a link shortening service just to get to GitHub and then it still only pointed to the facing page of a 93-commit PR.