4 ms·
DeepSeek's AI breakthrough bypasses industry-standard CUDA, uses PTX
- almostgotcaught 2y agolol this is the wackiest non-news; every serious project has at least some parts of their kernels implemented in PTX/AMDGCN.
- snovv_crash 2y agoML engineers are finally finding out that their abstraction layers are actually slowing them down.
- almostgotcaught 2y ago> finally finding out there's no finally - perf people have always written kernels in asm.
- kristjansson 2y agoThis is ridiculous. Since the actual training code for DeepSeek is _not_ public, this is a based only on the technical report, which mentions PTX one (1) time in §3.2.2 Efficient Implementation of Cross-Node All-to-All Communication: > Specifically, we employ customized PTX (Parallel Thread Execution) instructions and auto-tune the communication chunk size, which significantly reduces the use of the L2 cache and the interference to other SMs. So they have some intrinsic in some part of their training framework. That's it.
- pillefitz 2y agoIt all feels like an attempt to prevent replication to further tank the market. Not necessarily the technical details, but the reporting thereo, which was spammed all over Reddit and HN
- kristjansson 2y agoThis whole episode is weird. I can’t tell how much of the popular reporting is misinformed and how much has been disinformed. R1 (and sorta V3) are clearly progress, but are definitely not step-function improvements to prior SOTAs.
- WiSaGaN 2y agoThey are definitely step function improvements in regard to inferrance cost of the same level of reasoning intelligence.
- kristjansson 2y agoOnly because OpenAI overpriced o1, like they did with GPT-4.
- black_13 2y ago[dead]
- lousken 2y agoi would like to assume that low level optimization is crucial part of AI training and openai, meta and others aren't wasting billions on this
- lvl155 2y agoYou know what IS ridiculous is that people are willing to go assembly but still do not use AMD gpus despite the price difference.
- jiggawatts 2y agoWhat is really ridiculous is the astonishing inability of AMD to pivot and start fixing the software stack. It’s been years, and they’re doing… what exactly to catch up?
- jitl 2y agoI still think it’s a good investment.
- danpalmer 2y agoThere are already open stacks out there that help. The problem is that Nvidia provide a full stack option: chips, networking, and software, whereas AMD only provide the chips, you then need another company's networking, and then you need to plug together open source software. I'd also bet that if you buy enough of the Nvidia chips, they'll probably send in a bunch of engineers to get everything working with their full stack. AMD won't be able to do that in the same way because they're not vertically integrated.
- icelancer 2y ago> networking Yep. The interconnect is the most overlooked part of the stack.
- kristjansson 2y agoWriting a few intrinsics where necessary is not really comparable to the work required do reimplement something like CUDA on AMD (or equivalently deal with ROCm)
- whimsicalism 2y agoamd has revealed itself to be surprisingly poorly run. aren’t APIs no longer copyrightable? they should just have reimplemented cuda with their backend
- hintymad 2y agoWhen I grew up, the media in my country kept telling us how great the geek culture in the US was, how deep down the stack the geeks were willing to go, and both adults and us kids were left in awe. The entire nation, for what I could tell, routinely reflected why we couldn't be like the US: educating and nurturing generations of geeks to be the best engineers and scientists in the world. Well, it was quite a reverse culture shock after I moved to the US. I definitely didn't know that "teacher's pet" was a thing, or my coworker, a brilliant engineer who went to a highly reputed public school, was chased off his school bus simply because he used some poetic words, or geeks were not that respected in schools, or a mile wide and an inch deep with great leadership is what the US people revered. In the meantime, I guess other countries more or less picked up the baton of the US culture, and grew their own geeks.
- benreesman 2y agoIt comes in waves. In the 90s there was a monopoly so rapacious that the Department of Justice had to very nearly break it up before it strangled the Web in its crib. Grift and nepotism and cronyism and distorted markets were the norm then too. This is the first time it’s been so bad since, but I’m optimistic: we’re actually in a better position now because serious foreign tech dynamos don’t give a fuck about American mafia nepotism, we can’t just keep this in the family. I don’t know if this DeepSeek App Store thing will be the match that lights this thing up, but the grass is very tall and very, very dry.
- int_19h 2y agoDid you not see American movies? I'm also from a different country, but this part of the American culture is very much front and center in its exports, IMO.
- hintymad 2y agoNot really. Our English education was so bad that I could barely understand spoken English and I knew fewer than 5000 words when graduating high school. Ironically, I did fabulously well in English exams, which said a lot about our English education back then.
- sdedovic 2y agoMaybe I’m misunderstanding but CUDA compiles to PTX? Is the implication they wrote in a different language than CUDA and to generate the PTX?
- lostmsu 2y agoYou can code in PTX assembly (or just inline in C++) or generate binary PTX if you need to.
- nialse 2y agoWhat it does show is that CUDA leaves serious performance optimization on the table despite its gigantic code base. Using compression to reduce memory bandwidth is a well known trick in quantization, and in other scenarios since forever. There has been little competitive pressure on Nvidia to go further since their software stack leaves the competition in the dust. This time, they may actually need to step up their efforts, due to customer pressure. Good times!
- mrbungie 2y agoI see almost no incentives for NVDA to do so as long as they have no real competitors in that space.
- frontalier 2y ago> I see almost no incentives for NVDA to do so > as long as they have no real competitors in that space. ?! isn't _avoiding competition_ a good enough incentive for them to improve their offering?
- kristjansson 2y agoA careful implementation adapted to a specific workload is always going to beat a more generic implementation.
- DiabloD3 2y agoHaha, what a shoddy headline. "Bypasses" and "industry-standard" have no place here. CUDA is not an industry standard. Vulcan is an industry standard. They did not bypass CUDA... that's like saying if I use Vulcan I'm bypassing OpenGL. PTX is an alternative low level API provided by Nvidia because of how awful CUDA is for high performance code. What DeepSeek wrote could only have either been written in PTX or Vulcan. Any other company could have done this, and low latency traders on Wall Street that use Nvidia write their stuff in PTX for obvious reasons. OpenAI, was, is, and always will be, absolutely incompetent when it comes to using their hardware effectively... and they're no different than any other company. Reading is not a goddamned super power! Just read the docs!
- shihab 2y agoI had the impression that gpu isn’t a good fit for ultra low latency usecases. Can you please elaborate on what sort of work hft firms do with gpu?
- jakewins 2y agoNot in HFT, but I guess maybe for being very fast running optimization solvers and forecast models etc? Essentially compute models for ultimately driving market decisions based on lots of input data We do a lot of forecasting and solvers where I am, just run them on CPUs though.. but maybe if you’re wanting to compete on speed you would?
- sterlind 2y agoOptimization solvers usually don't benefit from GPUs. I think it's because it's sparse matrices and a sequential series of pivots.
- aleph_minus_one 2y ago> Optimization solvers usually don't benefit from GPUs. I think it's because it's sparse matrices and a sequential series of pivots. This depends a lot on the problem and the algorithm that is used. For example interior point methods are clearly better suited to be running on GPUs than the primal or dual simplex algorithm.
- gamblor956 2y agotldr: they wrote in low level code instead of using a higher level framework like their competitors have been doing so they were able to hand tune the performance. This gives them a few months head start before meta and Google start doing the same thing.
- t2oi4h324jl234 2y agoThis is strange. PTX is still at the IR level ? IIRC this is still relatively hardware agnostic. Can you actually get very far by doing this ? From a quick perusal, DeepSeek also uses Triton in the codebase.
- konradha 2y agoNext article: What's SASS?
- westurner 2y agoParallel Thread Execution: https://en.wikipedia.org/wiki/Parallel_Thread_Execution https://en.wikipedia.org/wiki/Parallel_Thread_Execution
- hulitu 2y ago> DeepSeek's AI breakthrough bypasses industry-standard CUDA, uses PTX Isn't CUDA an Nvidia child ? This sounds like "Microsoft = industry standard".