11 ms·
Don’t most orgs that are deep enough to run custom cuda kernels have dedicated engineers for this stuff. I can’t imagine a person who can write raw cuda not bei
by bootsmann 2y ago
Don’t most orgs that are deep enough to run custom cuda kernels have dedicated engineers for this stuff. I can’t imagine a person who can write raw cuda not being able to handle things more difficult than pip install.
- gaogao 2y agoEngineers who are really, really good at CUDA are worth their weight in gold, so there's more projects for them than they have time. Worth their weight in gold isn't figurative here – the one I know has a ski house more expensive than 180 lbs of gold (~$5,320,814).
- bbkane 2y agoWould you (or your friend) be able to drop any good CUDA learning resources? I'd like to be worth my weight in gold...
- throwaway81523 2y agoA working knowledge of C++, plus a bit of online reading about CUDA and the NVidia GPU architecture, plus studying the LCZero chess engine source code (the CUDA neural net part, I mean) seems like enough to get started. I did that and felt like I could contribute to that code, at least at a newbie level, given the hardware and build tools. At least in the pre-NNUE era, the code was pretty readable. I didn't pursue it though. Of course becoming "really good" is a lot different and like anything else, it presumably takes a lot of callused fingertips (from typing) to get there.
- 8n4vidtmkvmk 2y agoDoes this pay more than $500k/yr? I already know C++, could be tempted to learn CUDA.
- throwaway81523 2y agoI kinda doubt it. Nobody paid me to do that though. I was just interested in LCZero. To get that $500k/year, I think you need up to date ML understanding and not just CUDA. CUDA is just another programming language while ML is a big area of active research. You could watch some of the fast.ai ML videos and then enter some Kaggle competitions if you want to go that route.
- almostgotcaught 2y agoYou're wrong. The people building the models don't write CUDA kernels. The people optimizing the models write CUDA kernels. And you don't need to know a bunch of ML bs to optimize kernels. Source: I optimize GPU kernels. I don't make 500k but I'm not that far from.
- throwaway81523 2y agoHeh I'm in the wrong business then. Interesting. Used to be that game programmers spent lots of time optimizing non-ML CUDA code. They didn't make anything like 500k at that time. I wonder what the ML industry has done to game development, or for that matter to scientific programming. Wow.
- HarHarVeryFunny 2y agoHow much performance difference is there between writing a kernel in a high level language/framework like PyTorch (torch.compile) or Triton, and hand optimizing? Are you writing kernels in PTX? What's your opinion on the future of writing optimized GPU code/kernels - how long before compilers are as good or better than (most) humans writing hand-optimized PTX?
- throwaway81523 2y agoThe CUDA version of LCZero was around 2x or 3x faster than the Tensorflow(?) version iirc.
- deleted 2y ago
- mosselman 2y agoThe real challenge is probably getting your hands on a 4090 for a price you can pay before you are worth your weight in gold. Because an arm and a limb in gold is quite a lot.
- throwaway81523 2y agoYou don't really need a 4090. An older board is plenty. The software is basically the same. I fooled around with what I think was a 1080 on Paperspace for something like 50 cents an hour, but it was mostly with some Pytorch models rather than CUDA directly.
- saagarjha 2y agoModern GPU architectures are quite different than what comes before them if you truly want to push them to their limits.
- wing-_-nuts 2y agoEveryone and I mean everyone I know doing AI / ML work values VRAM above all. The absolute bang for buck are buying used p40's and if you actually want to have those cards be usable for other stuff, used 3090's are the best deal around and they should be ~ $700 right now.
- saagarjha 2y agoWhat they really value is bandwidth. More VRAM is just more bandwidth.
- wing-_-nuts 2y agoWell, to give an example, 32GB of vram would be vastly more preferable to 24GB of higher bandwidth vram. You really need to be able to put the entire LLM in memory for best results, because otherwise you're bottlenecking on the speed of transfer between regular old system ram and the gpu. You'll also note that M1/2 macs with large amounts of system memory are good at inference because of the fact that the gpu has a very high speed interconnect between the soldiered on ram modules and the on die gpu. It's all about avoiding bottlenecks whereever possible.
- robotnikman 2y agoAre there any certifications or other ways to prove your knowledge to employers in order to get your foot in the door?
- suresk 2y agoHaving dabbled in CUDA, but not worked on it professionally, it feels like a lot of the complexity isn't really in CUDA/C++, but in the algorithms you have to come up with to really take advantage of the hardware. Optimizing something for SIMD execution isn't often straightforward and it isn't something a lot of developers encounter outside a few small areas. There are also a lot of hardware architecture considerations you have to work with (memory transfer speed is a big one) to even come close to saturating the compute units.
- iftheshoefitss 2y ago[flagged]
- eigenvalue 2y agoThat’s pretty funny. Good test of value across the millennia. I wonder if the best aqueduct engineers during the peak of Ancient Rome’s power had villas worth their body weight in gold.
- Willish42 2y agoThe fact that "worth their weight in cold" typically means in the single-digit millions is fascinating to me (though I doubt I'll be able to get there myself, maybe someday). I looked it up though and I think this is undercounting the current value of gold per ounce/lb/etc. 5320814 / 180 / 16 = ~1847.5 Per https://www.apmex.com/gold-price https://www.apmex.com/gold-price and https://goldprice.org/ https://goldprice.org/, current value is north of $2400 / oz. It was around $1800 in 2020. That growth for _gold_ of all things (up 71% in the last 5 years) is crazy to me. It's worth noting that anyone with a ski house that expensive probably has a net worth well over twice the price of that ski house. I guess it's time to start learning CUDA!
- boulos 2y agoNote: gold uses troy ounces, so adjust by ~10%. It's easier to just use grams or kilograms :).
- Willish42 2y agoThanks, I'm a bit new to this entire concept. Do troy lbs also exist, or is that just a term when measuring ounces?
- someguydave 2y agoyes, there are troy pounds, they are 12 troy ounces (not 16 ounces, like normal (avoirdupois) pounds) https://en.wikipedia.org/wiki/Troy_weight https://en.wikipedia.org/wiki/Troy_weight 180 avoirdupois pounds is 2,625 ounces troy. The gold price is around $2470/ounce troy today, so $2470*2625 ~= $6.483 million
- atwrk 2y ago> That growth for _gold_ of all things (up 71% in the last 5 years) is crazy to me. For comparison: S&P500 grew about the same during that period (more than 100% from Jan 2019, about 70 from Dec 2019), so the higher price of gold did not outperform the growth of the general (financial) economy.
- iftheshoefitss 2y agoWhat do people study to figure out CUDA? I’m studying to get me GED and hope to go to school one day
- paulmd 2y agoComputer science. This is a grad level topic probably. Nvidia literally wrote most of the textbooks in this field and you’d probably be taught using one of these anyway: https://developer.nvidia.com/cuda-books-archive https://developer.nvidia.com/cuda-books-archive “GPGPU Gems” is another “cookbook” sort of textbook that might be helpful starting out but you’ll want a good understanding of the SIMT model etc.
- amelius 2y agoJust wait until someone trains an ML model that can translate any CUDA code into something more portable like HIP. GP says it is just some #ifdefs in most cases, so an LLM should be able to do it, right?
- FuriouslyAdrift 2y agoOpenAI Triton? Pytorch 2.0 already uses it. https://openai.com/index/triton/ https://openai.com/index/triton/
- varelse 2y ago[dead]
- radarsat1 2y agoSelection bias. I'm sure there are lots of people who are really good at CUDA and don't have those kind of assets. Not everyone knows how to sell their skills.
- smallnamespace 2y agoUnfortunately it's also hard to buy (find) people who don't know how to sell.
- Der_Einzige 2y agoRight now, nvidias valuations have made a lot of people realize that their CUDA skills were being undervalued. Anyone with GPU or ML skills who hasn’t tried to get a pay raise in this market deserves exactly the life that they are living.
- phkahler 2y ago>> Don’t most orgs that are deep enough to run custom cuda kernels have dedicated engineers for this stuff. I can’t imagine a person who can write raw cuda not being able to handle things more difficult than pip install. This seems to be fairly common problem with software. The people who create software regularly deal with complex tool chains, dependency management, configuration files, and so on. As a result they think that if a solutions "exists" everything is fine. Need to edit a config file for your particular setup? No problem. The thing is, I have been programming stuff for decades and I really hate having to do that stuff and will avoid tools that make me do it. I have my own problems to solve, and don't want to deal with figuring out tools no matter how "simple" the author thinks that is to do. A huge part of the reason commercial software exists today is probably because open source projects don't take things to this extreme. I look at some things that qualify as products and think they're really simplistic, but they take care of some minutia that regular people are will to pay so they don't have to learn or deal with it. The same can be true for developers and ML researchers or whatever.