7 ms·
Fp8 runs ~100 tflops faster when the kernel name has "cutlass" in it
- vasco 1y agoThis was discussed before at the time the PR was created and there's nothing new that I can see. https://news.ycombinator.com/item?id=44530581 https://news.ycombinator.com/item?id=44530581
- rob_c 1y agoThought this looked familiar...
- nulld3v 1y agohttps://github.com/triton-lang/triton/pull/7298#discussion_r2202281596 https://github.com/triton-lang/triton/pull/7298#discussion_r... > By disassembly of ptxas, it is indeed hard-coded that they have logic like: strstr(kernel_name, "cutlass"). > it is likely that, this is an unstable, experimental, aggressive optimization by NVIDIA, and blindly always enabling it may produce some elusive bugs.
- temp0826 1y agoThanks for a little context, this is not my wheelhouse at all (never even heard of this project) and I could not make heads or tails of the title or the linked PR.
- frogblast 1y agoOften not elusive bugs, but elusive performance. GPU compilers are hard: Once you've done the basics, trying to do further transforms in a mature compiler will almost always produced mixed results. Some kernels will go faster, some will go slower, and you're hoping to move the balance and not hit any critical kernel too hard in your efforts to make another go faster. An optimization with a universal >=0 speedup across your entire suite of tests is a really hard thing to come by. Something is always going to have a negative speedup. My experience is with non-Nvidia GPU systems, but this feels like a familiar situation. They probably found something that has great outcomes for one set of kernels, terrible outcomes for another, and no known reliable heuristic or modeling they could use to automatically choose.
- Eridrus 1y agoA saner design would turn this optimization into a documented flag that anyone can opt into.
- rcoveson 1y agoSpeaking from a place of long-term frustration with Java, some compiler authors just absolutely hate exposing the ability to hint/force optimizations. Never mind that it might improve performance for N-5 and N+5 major releases, it might be meaningless or unhelpful or difficult to maintain in a release ten years from now, so it must not be exposed today.
- MichaelZuo 1y agoThat seems valid for customers expecting a warranty or support. But they should allow it if customers waive all such in writing.
- Dylan16807 1y agoWarranty and support specifically for that flag? Because I don't see how general warranty and support requires keeping any hint flags forever.
- shadowpho 1y agoIf you remove the hint flag peoples build will break
- Dylan16807 1y agoDoesn't need to, it can acknowledge and ignore the hints.
- shadowpho 1y agoTrue, but there might be more problems — like if you drop support their run time will be slow because they rely on this flag and they are unhappy
- fooker 1y agoWhen intel did it, the pitchforks came out. Nvidia seems to get a pass. Whys that?
- whatevaa 1y agoIntel did this to consumers, nvidia does this to enterprises.
- m00x 1y agoThis isn't the same thing
- kimixa 1y agoIt really depends on details. If intentionally slowing non CUTLASS shaders, sure pitchfork time. If it's an option that /technically/ breaks the CUDA shader compatibility contract, then enabling it in specific "known good" situations is just business as usual for GPU drivers. That can be for all kinds of reasons - straightforward bugs or incomplete paths in the optimization implementation, the app not actually needing the stricter parts of the contract so can have a faster path, or even bugs in apps that need workarounds. Though piggybacking into these without understanding can be extremely fragile - you don't know why they've limited it, and you run the risk of tripping over some situation that will simply fail, either with incorrect results or something like a crash. And possibly in rather unexpected, unpredictable situations.
- fancyfredbot 1y agoIntel disabled optimisations when they detected they were running on their competitors hardware. The motivation was to make competitors compare badly in benchmarks. Nvidia are disabling optimisations on their own hardware. The motivation appears to be related to these optimisations being unsafe to apply to general code.
- flamedoge 1y agoLet's be clear here. Intel searched for "GenuineIntel" and ran optimized code on their hardware.
- 1y ago
- kilpikaarna 1y agoKeeping it real with the commit msgs
- RestartKernel 1y agoI much prefer this over those AI generated commit messages that just say "refactored X" every single commit.
- rkomorn 1y agoSo AI really does learn from humans...
- spullara 1y agowhat kind of AI are you using that generates shitty commit messages? This a common kind of message from Claude / Augment: Fix dynamic channel list by passing auth via metadata - Pass userId and userEmail in metadata when calling HTTP transport - AuthenticatedToolsProviderFactory now reads from context.metadata - Each tools/list request creates a fresh ToolsProvider with authentication - Execute command description now correctly shows currently online machines - Tested locally and working correctly
- xnorswap 1y ago> - Tested locally and working correctly This is completely meaningless and just pollutes the log.
- lambdaone 1y ago"ready for production", "fully working" and other Claude-isms come to mind
- withinboredom 1y agoYou're totally right!
- globular-toast 1y agoSomeone really needs to learn to use `git commit --amend`. Almost 100 commits with pointless commit messages like "wip" or "x"? Be kinder to your reviewers...
- Zardoz84 1y agoand git commit --fixup git rebase -i --autosquash
- lemonlearnings 1y agoClaude, look at this git history, analyse diffs and create an intelligent commit message to replace each commit message. Do a rebase to fix it all up.
- kristopolous 1y agoWould you actually do that? It's information destruction. You can machine generate at any time, but you can only delete the human input once
- trenchpilgrim 1y agoMy human inputs are usually commit messages like "awdjhwahdwadga" until I do the rebase at the end
- lloeki 1y ago> you can only delete the human input once git branch "backup/$(git branch --show-current)/$(date +%s)" # do whatever you fancy git reset --hard "backup/$(git branch --show-current)/${thattimestampabove}" You can't lose anything as long as you have a pointer to it (which doubles as making it easy to find)
- globular-toast 1y agoNo need to make a "backup" branch. Learn to trust the reflog.
- haunter 1y agoHeh. Does anyone remember when almost 25 years ago ATI (AMD) caught manipulating the Quake III benchmarks by renaming the executables to ‘quack’? https://web.archive.org/web/20230929180112/https://techreport.com/review/how-atis-drivers-optimize-quake-iii/ https://web.archive.org/web/20230929180112/https://techrepor... https://web.archive.org/web/20011108190056/https://hardocp.com/reviews/vidcards/ati/8500/index5.html https://web.archive.org/web/20011108190056/https://hardocp.c... https://web.archive.org/web/20011118183932/www.3dcenter.de/artikel/2001/10-24_a.php https://web.archive.org/web/20011118183932/www.3dcenter.de/a...
- a-french-anon 1y agoOr Intel checking for "GenuineIntel" in ICC's output: https://en.wikipedia.org/wiki/Intel_C%2B%2B_Compiler#Support_for_non-Intel_processors https://en.wikipedia.org/wiki/Intel_C%2B%2B_Compiler#Support...
- cratermoon 1y agoOr Win 3.1 looking for whatever shibboleth was in MS-DOS and popping up a scary-looking message if it found another DOS? https://en.wikipedia.org/wiki/AARD_code https://en.wikipedia.org/wiki/AARD_code
- keanb 1y agoI don’t think anybody remembers this since that code never shipped in retail.
- GeekyBear 1y agoIt didn't ship (in the final retail version) only after the tech press of the day exposed what Microsoft had done.
- cratermoon 1y agoIt did ship in the final retail version, in way. It was disabled, but the code was still there, and a flag was all that was needed to enable it.
- toolslive 1y agonil novi sub soli. - Intel faced a "cheating compiler" controversy when SPEC, the benchmark standard-setter, invalidated over 2,600 benchmark results for Intel Xeon processors in early 2024. ( https://www.tomshardware.com/pc-components/cpus/spec-invalidates-2600-intel-cpu-benchmarks-says-companys-compiler-used-unfair-optimizations-that-boosted-performance https://www.tomshardware.com/pc-components/cpus/spec-invalid... ) - microsoft doing similar things (java benchmarks, C compiler benchmarks) - and everybody cheating on AI benchmarks (https://www.thestack.technology/ai-benchmarking-scandal-were-top-models-caught-gaming-the-system/ https://www.thestack.technology/ai-benchmarking-scandal-were...)
- high_na_euv 1y agoI work with compilers And despite it not being nice, some optimizations rely on type or function names schemas/substrings/etc It sucks, but thats how it works It doesnt have to be malicious just sometimes it is safer to deploy optimization only for your libs than risk breaking stuff Or your frontend is not giving you more data which you can rely on
- tliltocatl 1y agoIt is probably not malicious, but it certainly does create new barriers, which is not a good thing.
- Hizonner 1y ago> than risk breaking stuff ... until somebody randomly chooses the same name for some reason and gets hosed. You're not helping.
- wiz21c 1y agoOn function types or schema, I can understand that. But names ?
- mhh__ 1y agoSomething like: if(AskLLM("Does function signature+name look like error handling code")) { TurnOffInliner(); } is actually probably a lot more effective than you'd think (generating PGO traces with a machine learning tool is apparently a thing that sort of works)
- colejohnson66 1y agoNot exactly the same, but intrinsics in some languages are purely name+signature matched.
- high_na_euv 1y agoSure, but will anyone name a method like "__nvidia_experimental_feature_xyz_v1"?
- trumbitta2 1y agoReminds me of when ~10 years ago with a particular version of Webpack the build would fail if I had one SVG called add.svg so I had to rename it plus.svg
- compass_copium 1y agoor breaking numpy by importing your api key from a little file you wrote called "secret.py" :P
- muhiyadiin 1y ago[flagged]
- user3939382 1y agoIt would be nice if we could find economics that allowed us to share code instead of all the bullshit with the binary blob drivers. Same for basebands and everything else. How many collective hours and months of our society’s finest minds has been wasted reverse engineering binary blobs, controllers through IO pins, trying to reverse engineer circuit schematics —- when all of this is already sitting on someone’s computer somewhere and they could just GIVE you the docs. CUDA and NVIDIA can go to hell.
- imtringued 1y agoThe problem is as follows: You have a fixed cost investment to produce a software code base, then you have fixed ongoing maintenance costs, at a minimum one developer who knows the codebase. Preferably two for a commercial product. On top of that you have small distribution costs over time. E.g. servers that host the software downloads. The marginal costs per user are very small or even zero for desktop applications. This means that software needs a funding structure with periodic payments, but at the same time the payments shouldn't grow with the number of users. There also needs to be a way for the initial investors who pay for the creation of new features or entire code bases to get their money back as the product becomes popular. This in itself is not problematic, but it is not covered by traditional crowdfunding. The problem is that the funding goal needs to be met no matter what, and the contribution per user shrinks as more users contribute. You can't expect everyone to chip in 100%, 10% or even 1% of the funding cost, since that could be thousands of dollars even at the minimum. You need some sort of auctioning process where people can pledge a fixed quantity and if the user count is low enough, their pledge counts, otherwise it doesn't. This has one problem though. What's problematic is the transition from the exclusive to non-exclusive mode. There will be freeloaders who might pitch in five dollars, but they know five big corporations have chipped in and this covered the full development cost, leading to open sourcing the entire codebase. Everyone else is a freeloader. Including cheapskate corporations.
- user3939382 1y agoThat’s a problem but it’s not the irreducible problem. Which is that the computing stack is constructed wrong (obvious and fixable) so you have these ridiculous requirements in the first place. The capital issue is a second order effect.
- david-gpu 1y agoSometimes you write some heavily tuned code in a high level language like C++ that you know could be translated into very specific GPU assembly, then find that the compiler isn't producing the exact assembly that you had in mind. When you talk to the computer team about it they may offer a range of solutions, some of which may not be applicable to open source code. Picture proprietary #pragmas, intrinsics, or whatnot. What do you do? You can't ship a high performance library that doesn't deliver high performance. It is then when you rely on things like function names to enable specific code transformations that can't be used in general because they would sometimes break third party code. I never worked on Cutlass, but this is the sort of thing that is done in the real world. There is nothing nefarious about this sort of optimizatkon. People comparing this to cheating on benchmarks by rendering lower quality images are not on the right track.
- School-Cotton 1y agoWhy wouldn’t you just use inline assembly in that case?
- krapht 1y agoThere is no way to write inline SASS (assembly equivalent) for CUDA code. You can inline PTX, but PTX is a high level bytecode designed to be portable.
- dahart 1y agoPTX is sometimes referred to as assembly, and it is an ISA, much lower level than C++. When people talk about writing inline assembly for CUDA, they mean PTX, and the C++ compiler’s inline assembly “asm” statement assumes PTX. For the most part you have much more control and ability to produce exactly the SASS you want when using PTX compared to when using C/C++. https://developer.nvidia.com/blog/understanding-ptx-the-assembly-language-of-cuda-gpu-computing/ https://developer.nvidia.com/blog/understanding-ptx-the-asse... https://eunomia.dev/others/cuda-tutorial/02-ptx-assembly/ https://eunomia.dev/others/cuda-tutorial/02-ptx-assembly/
- luc_ 1y agoReminds me of when I was working on NVIDIA Jetson systems, and learning how to use them, that you can run 1 command to make everything go faster... (https://jetsonhacks.com/2019/04/10/jetson-nano-use-more-power/ https://jetsonhacks.com/2019/04/10/jetson-nano-use-more-powe...)
- huflungdung 1y ago[dead]
- dahart 1y agoDo you mean the power mode? The article says there are 2 power modes, 5W and 10W, and that 10W is the default. From that I would assume that you can make everything go faster by using 10W mode, but only if you had already made things go slower by switching away from the default to the 5W mode? Did I miss something?