10 ms·
Jim Keller criticizes Nvidia's CUDA, x86
- rerx 3y ago> "Basically nobody writes CUDA," wrote Keller in a follow-up post. "If you do write CUDA, it is probably not fast. […] There is a good reason there is Triton, Tensor RT, Neon, and Mojo." > Even Nvidia itself has tools that do not exclusively rely on CUDA. For example, Triton Inference Server is an open-source tool by Nvidia that simplifies deploying AI models at scale, supporting frameworks like TensorFlow, PyTorch, and ONNX. Triton also provides features like model versioning, multi-model serving, and concurrent model execution to optimize the utilization of GPU and CPU resources. > Nvidia's TensorRT is a high-performance deep learning inference optimizer and runtime library that accelerates deep learning inference on Nvidia GPUs. [...] Keller was speaking of OpenAI's Triton (https://openai.com/research/triton https://openai.com/research/triton), a Python-like language that is compiled to code for Nvidia GPUs, but Tom's Hardware mixed this up with Nvidia's Triton Inference Server, a higher level tool that's really not a replacement for CUDA and not directly related to the Triton language. Easy to confuse these if you are a writer in a hurry.
- londons_explore 3y agoIndeed - Keller is a low level hardware guy, and isn't going to have much interest in model versioning...
- 95014_refugee 3y agoKeller is a manager. He was a gateware engineer.
- gumby 3y agoEven so his long and influential experience influences his managing and the direction of the company.
- chrisjc 3y agoIsn't low-level hardware really at the heart of a lot of this? Hasn't a lot of the criticism of Cuda been that it's incredibly difficult for others to implement on other hardware bc of the low-level interactions and Nvidia's usage of dark-APIs (can't recall the term i've heard used). Wasn't this one of the reasons AMD abandoned/deprioritized their efforts on such a project?
- p1esk 3y agoJim Keller works for Tenstorrent - direct Nvidia competitor.
- diggan 3y agoWow, that's some omission in the article. Mentioned in the very bottom, but with no disclaimer that it might influence his opinion as they're a competitor: > His statements also imply that even though he has worked stints at some of the largest chipmakers in the world, including the likes of Apple, Intel, AMD, Broadcom (and now Tenstorrent), we might not see his name on the Nvidia roster any time soon.
- shash 3y agoSomehow, I don't see him going to Nvidia in any case. Their designs are very different from his philosophy, and he can be quite opinionated. I'd characterise Tenstorrent as his attempt to build the opposite of an Nvidia style accelerator.
- Kuinox 3y agoWhat is his philosophy ?
- mcbishop 3y agoHe gets into that in his talk with Lex Fridman.
- andy_ppp 3y agoIf only there was a service that could watch a 2 hour video and give you a detailed answer to this question in a few seconds just by asking it… soon enough I guess.
- kekebo 3y agoFeeding the output of a transcription service (e.g. https://youtubetranscript.com/?v=G4hL5Om4IJ4 https://youtubetranscript.com/?v=G4hL5Om4IJ4 ) into an llm for specific requests should be possible, likely in multiple steps to handle context limitations
- Vvector 3y agoSwamps keep users trapped in, by making it difficult for a competitor to make a compatible alternative. Had Nvidia (or Intel or Microsoft) kept their product lean by aggressively removing backwards compatibility, they would have lost users AND made it easier for another company to compete.
- blackoil 3y agoYeah for some reason people deny to understand that this messy swamp is the reason things are successful. Be it x86, Windows, PDF...
- mschuster91 3y agoPDF hasn't been a vendor-locked swamp for many years now. It got successful because it was better than TIFF for scanned images and better than raw Word/Postscript for print products.
- busterarm 3y agoBut it became successful long before it stopped being a vendor-locked swamp.
- mpweiher 3y agoI can't recall a time when the PDF spec wasn't open, or at least openly available.
- busterarm 3y agoFrom 1993 to 2008?
- mpweiher 3y agoHmm...it wasn't an ISO standard, but the standard was freely available both as a download from the web-site and as a book. My printed copy (version 1.4) is from 2001, and that's a later version, I definitely had earlier, much slimmer books. The 1.0 PDF reference on Adobe's web-site has a Copyright of 1993 and says "First Printing, June 1993". https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/pdfreference1.0.pdf https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandard... And of course there were multiple unlicensed implementations. Famously, Apple switched from Display Postscript in Rhapsody and Mac OS X Server to Quartz and PDF in Mac OS X to remove the DPS licensing costs.
- RcouF1uZ4gsC 3y agoMost developers would rather write code for a “swamp” that has long term backwards compatibility, than for a perfect ivory tower that will get replaced in 2 years by another more perfect ivory tower.
- frfl 3y agoInteresting that the web ecosystem basically works on the ivory tower model. Roughly every 2 years there's some new ivory tower and a lot of greenfield projects move to that. Brownfield projects become legacy and go into maintenance mode or get rewritten after some number of years. I've seen something like this: - Delphi desktop app (some time in the 2000s) - Adobe Flash web app (in late 2000s to mid 2010s) - 2015-era JS framework web app (mid 2010s to present, considered legacy codebase now, no feature development) - 2020-era JS framework web app (late 2010s to present) I expect another major rewrite or move to a new project in the next ~5 years
- spenczar5 3y agoThe web frontend works that way because little depends on the frontend. But the infrastructure one layer down is obsessive about backwards compatibility, both in the browser and in HTTP servers. JavaScript frameworks change all the time, but nginx and apache and friends have lasted for many decades, for good reason.
- pjmlp 3y agoDepends where you look into, I have been pretty much Java and .NET since 2005 in what concerns Web development and backend infrastructure in distributed systems.
- belval 3y ago> "Basically nobody writes CUDA," wrote Keller in a follow-up post. "If you do write CUDA, it is probably not fast. I am a big fan of Jim Keller, but this is semantics that argues CUDA the language is not a moat, when most people refer to CUDA the libraries. CUDA has first party support in most of the libraries in use today, if you were to stray from that happy path you must have deep pocket to work through the hurdles of working with something like XLA. There's a reason why Microsoft, Google, Amazon and Meta are still buying Nvidia accelerators even when they have their own in-house accelerators.
- dist-epoch 3y agoThe impossibility of buying NVIDIA GPUs in quantity today is a giant incentive to port the most important libraries to AMD/....
- photonbeam 3y agoIm surprised we dont hear more about this, it seems like a huge motivator
- cjbgkagh 3y agoI think you’re assuming these big companies are serious about AI itself and not just joining in on a trend. They would much rather spend huge amounts of money on chips than hire a competent researcher who might tell them that they didn’t really need to waste so much money.
- hackerlight 3y agoAndrew Ng says ROCm is a lot better today than early 2023
- pjmlp 3y agoThe computing shops on my city shopping malls have enough laptops with NVIDIA GPUs, while they aren't at the same level as desktop or server GPUs, there are available and usable for people to learn CUDA, quite the contrary with most AMD offerings.
- roenxi 3y agoCUDA seems to have a lot of mind-share as a strategic asset, but all the evidence I've seen is that AMD defeated themselves. I've never seen them fail due to lacking features or capability, and I haven't heard stories of it either. The complaint seems to be, consistently, that the drivers crash. If CUDA is so good, it'd be great to know what it does that AMD cards can't. I've never gotten that far because I hit what seems to be some sort of kernel panic or driver lockup.
- Vvector 3y ago> If CUDA is so good, it'd be great to know what it does that AMD cards can't. You are comparing Nvidia software with AMD hardware. It's the AMD software that has been lacking historically. EDIT: here is a discussion from 2 months ago on CUDA versus ROCm. https://news.ycombinator.com/item?id=38700060 https://news.ycombinator.com/item?id=38700060
- declaredapple 3y ago> I've never seen them fail due to lacking features or capability, and I haven't heard stories of it either. I've seen a mix of both for years. For the last 6 years the story has usually been some combination of "X program technically supports it, but you'll get 20% of the performance you'd expect given the hardware and/or you'll need to go through a bunch of hoops, and even then you'll have to troubleshoot tons of random errors and/or it'll crash the program or system" That's on top of the "nobody supports ROCm, or if they do it's because of a single person who amd the PR - you're on your own though because none of the core contributers have amd hardware" which I'll admit is a chick-egg problem. But given these two factors, it's always meant "if you want buy hardware to do data science, you have to go nvidia if you don't want to write the support yourself, and/or an insane headache that often resulting in switching to nvidia hardware anyway"
- slices 3y agoSupporting anecdote: I built a PC a few years ago, and first tried multiple AMD cards. All of them worked fine while using the default windows drivers, but as soon as I installed AMD's drivers, they stopped working. Spent weeks troubleshooting, no luck. Gave up, bought an Nvidia card, zero problems.
- throw0101c 3y agoGiven the rise in ARM in many areas of computing, I'm not sure how much of a moat x86 is / will be long-term. Is there anything 'inherit' to the instruction set that cannot be done anywhere else? I know that many Intel/AMD systems now have a bazillion cores, which is very handy for some things. Also, AMD seems to have a large numbers PCIe lanes, which is great for I/O between (e.g.) the network and on-system stuff.
- blinkingled 3y agoOTOH I have been hearing about ARM's rise since a long time now. It's not just performance per watt and even that's just Apple beating everyone. See Windows on ARM. x64 is here to stay with its incredible ecosystem and OSS friendliness - Coreboot, Intel and AMD GPU drivers, great choice of operating systems and apps with a fairly long usable lifespan even with Windows updates.
- frfl 3y agoI came across an interesting fact about X86, > currently there are actually no modern x86 CPUs on the market. Both Intel and AMD don't actually use x86 cores, but instead proprietary RISC cores, with microcode that translates the x86 code to RISC code on the fly at execution time. https://cs.stackexchange.com/questions/132211/what-are-the-advantages-of-x86-cpus-over-arm https://cs.stackexchange.com/questions/132211/what-are-the-a... This cs.stackexchange link is a good read. Wikipedia also states something similar, > In the P6 and later microarchitectures, x86 instructions are internally converted into simpler RISC-style micro-operations that are specific to a particular processor and stepping level https://en.wikipedia.org/wiki/Intel_Microcode https://en.wikipedia.org/wiki/Intel_Microcode
- jerf 3y agoAlso, the more transistors we stuff on to a chip, a process that is still continuing, the less are proportionally dedicated to these decoding pipelines. My impression is that they are already relatively insignificant parts of the chips, though corrections welcome. I wonder if there's any ISA that could be written against the x86 cores that would be more efficient. In theory the chip could use that as a mode, so the chip itself wouldn't require an entire OS to shift before things could use it. I don't know anywhere near enough about the inside of an x86 core to have even a clue if such a thing would be possible. But it would be an interesting escape hatch from x86. Anyone who can flesh this idea out with knowledge of the x86 core internals is welcome to explain to me why my idea is bad and I should feel bad.
- boringuser2 3y agoOut of curiosity: Is it legal for China to hire Jim with an obscene pay package? Why wouldn't they?
- fsmv 3y agoChina is a country not a company
- ThrowawayTestr 3y agoHard to tell sometimes
- deleted 3y ago[deleted]
- azinman2 3y agoIt’s blurred when the state owns so many companies
- espadrine 3y agoIt is disallowed by the executive order “on Addressing United States Investments in Certain National Security Technologies and Products in Countries of Concern”: https://www.whitehouse.gov/briefing-room/presidential-actions/2023/08/09/executive-order-on-addressing-united-states-investments-in-certain-national-security-technologies-and-products-in-countries-of-concern/ https://www.whitehouse.gov/briefing-room/presidential-action...
- dfsgdfg 3y agoI think Tenstorrent is based in Canada. At least Jim and his team are all from Canada, and their headquarters seems to be there. Probably more complicated than that but it isn't so clear if this would apply.
- boringuser2 3y agoThat's a 2023 executive order. It's hard to imagine that this gaping hole was left wide open until the deft hand of Joseph Biden took the wheel.
- reubenmorais 3y agoEvery successful platform is a swamp. Human lives are short, so we do things hastily; communication and coordination is hard; zero sum views of competition are widespread; we don't know how to do sustainable fast iteration of ecosystems (or even slow iteration); the swampiness helps concentrate wealth, which is even more highly rewarded. Cities are a swamp. The Web is a swamp. Linux is a swamp. Capitalism is a swamp. Democracy is a swamp.
- the_panopticon 3y agowell said. Reminds me of a comment I received from a manager years ago: "This component is great, I haven't heard of any bugs." My retort was: "It's because no one is using it." Fast forward 10 years. Many bugs, CVE's, complaints of back-compatibility, challenges in porting/maintenance. Show me a technology w/o complains or bugs or technical debut & I suspect you'll find a similarly rarely or not used one. Swampiness is an emergent property of 'success at scale' in usage in a sense.
- deleted 3y ago[deleted]
- brucethemoose2 3y ago> "Basically nobody writes CUDA," Quite a thing to say as head of an AI hardware startup. Most of the machine learning libraries we use/test in production (and that I use personally on my desktop) have hand written, extensively optimized CUDA kernels. It would be problem #1 switching to Tenstorrent hardware. Not that I like that one bit, or that I really understand the optimization of these libraries. I do see triton code as well, but it still seems to be the suboptimal path.
- qwertox 3y agoBut what kind of CUDA kernels have you written? Kernels for AI training and inferencing? Because I think that's what he's trying to say. I doubt he's saying that in the context of CFD or other similar ones.
- brucethemoose2 3y agoIf you are implying we are at the mercy of a few CUDA experts because we can't efficiently write it ourselves, you would be correct. Triton does seem way more approachable as something I'd like to pick up.
- pjmlp 3y agoHence why NVidia is also investing into GPUs JITs for Python. One advantage of CUDA, that Khronos/AMD/Intel realized too late, was the polyglot support that NVidia started to push and support since CUDA 3.0.
- imtringued 3y agoI hear you say this over and over again, but honestly AMDs OpenCL implementations used to be unusable garbage.
- pjmlp 3y agoThat in top of everything else, with OpenCL stuck in 1990's tooling.
- slashdev 3y agoHot take: x86 was a swamp, but overtime we stopped using legacy features, and the instructions got better, and the amount of chip resources needed to implement the CISC architecture decreased to a small percentage of the total, such that it sort of just turned into a wierd compression algorithm for encoding RISC instructions and didn't matter much anymore.
- qwertox 3y agoI think it's more of a subtle publicity stunt for Tenstorrent's Grayskull [0] cards, which are highly optimized for inference. With CUDA you get a lot of flexibility which you don't get with Grayskull, at the expense of having a more layered/patched API which has been evolving for many years while R&D came up with new solutions to new problems. I wonder what Nvidia's capabilities are in creating new, highly optimized cards, similar to Grayskull, but I wouldn't be surprised if they don't have any interest in creating them since their current products are already consuming all their resources. It's great to see that companies like Tenstorrent are offering more optimized and pricewise more accessible products. It would also ease the situation with how hard it is to get Nvidia cards, because even companies doing only inferencing are buying chips which are developed to be able to do much more than that. [0] https://tenstorrent.com/cards/ https://tenstorrent.com/cards/
- camel-cdr 3y ago> which are highly optimized for inference. My understanding is that grayskull is a devboard for inference, Wormhole is supposed to do both, because you can scale it out with multiple cards. > With CUDA you get a lot of flexibility which you don't get with Grayskull Do you? From what I can tel tt-metalium gives you quite low level access. Grayskull/Wormhole/... basically have a bunch of tensix cores, which each have 5 small rv32 cores, where some are used for data movement and some for computation. IIRC two (could be one) drive a SIMT compute unit. From what I can tell there is no "matrix in matrix out" style fixed accelerator, the SIMT unit looks decently flexible, the grayskull one is a bit limited isa wise, wormhole is supposed to improve on that. tt-buda is supposed to be the high level API with pytorch, tensorflow, ... support, but idk how much compatibility there is at the moment. I assume there is a lot of software work left.
- qwertox 3y agoWormhole is an accelerated NIC, like what Nvidia got access to when it bought Mellanox. It's meant for high-speed routing. Yet you got me to reading the following: > t-Series Workstations > Our t-series workstations are turnkey solutions for running training and inference on our processors, from a single-user desktop workstation in the t1000 up to the t7000 rack-mounted system designed specifically to function as a host with our Galaxy 4U Server > 8 Grayskull Cards > 16 Grayskull Chips So these cards can also do training. https://tenstorrent.com/systems/aicloud/ https://tenstorrent.com/systems/aicloud/
- thoi324o23434 3y agoIsn't RoCm just AMD's version of "CUDA" (whatever that means at this point) ? IIRC, it was tied to HIP which was AMDs IR compiler from ptx.
- HarHarVeryFunny 3y agoAMD vs NVIDIA RoCm = CUDA+cuBLAS+cuRAND MiOpen = cuDNN HIP is an AMD portability abstraction layer over the NVIDIA APIs that is pass-thru to cuDNN/cuBLAS/cuRAND on NVIDIA hardware, and pass-thru to AMD's equivalent APIs (that are basically drop-in replacements for NVIDIA's) on AMD hardware.
- zokier 3y agoAMD doesn't have any equivalent of CUDA, i.e. something you can run on commodity GPUs. ROCm is only for specific "pro" devices.
- hackerlight 3y agoROCm officialy supports 7900 series retail cards and works fine on unsupported older AMD retail cards
- atq2119 3y agoI don't think this is a deliberate strategy but rather the result of focusing resources in where there are currently the largest margins, which is in the big server chips. It is somewhat short-sighted though, since having support on all their GPUs would surely do wonders for adoption via more organic channels.
- raggi 3y agoInvest in webgpu compute. It's not just for web, but it is portable.
- cjbgkagh 3y agoI write fast CUDA so I guess I'm one of the lucky few - which I already knew. I spent a lot of time benchmarking, instruction counting the PTX, function approximations, interleaving etc. I don't think it would cost that much money to write new ops for a new hardware for existing frameworks but for some reason very few do it - I figure it must be cheaper than making the hardware. I assume it's "penny wise, pound foolish" or maybe that it's very hard for hardware companies to do software as they try to pay their software guys like hardware guys. More and more I'm doing more branchy algorithms so the CPU is now my new bottleneck - thank you fast GPUs - but I don't think the market will move to this way doing things before the current AI wave has crested. So it is probably not good business sense to target my use cases. I have quite a lot of concurrency so I think my ideal hardware is a whole lot of little CPU cores with decent cache and Half Matrix Multiply Accumulate (HMMA) instructions. Once my CPU bottleneck becomes too painful I'll look at what options are on the market. I think, but am not sure, the early Tesla AI chips focused too much on optimizing ResNet and then later they went with CPUs with the HMMAs. Instead of using a variety of instructions I can instead approximate most functions with more HMMAs. EDIT: changed my reference of matmul intrinsics to the more precise HMMA instruction
- blopp99 3y ago> "or maybe that it's very hard for hardware companies to do software as they try to pay their software guys like hardware guys." Very interesting point.
- abstractcontrol 3y ago> I have quite a lot of concurrency so I think my ideal hardware is a whole lot of little CPU cores with decent cache and matmul intrinsics Back in 2015 I thought this would be the dominant model in 2022. I thought that the AI startups challenging Nvidia would be about that. Instead, they all targetted inference instead of programmability. I thought that a Tenstorrent hardware would be about what you are talking about - lots of tiny cores, local memory, message passing between them, AI/matmult intrinsics. I've been hyped about Tenstorrent for a long time, but now that it is finally coming out with something, I can see that the Grayskulls are very overpriced. And if you look at the docs for their low-level kernel programming, you will see that Tensix cores can only have four registers, have no register spilling, and also don't support function calls. What would one be able to program with that? It would have been interesting had the Grayskull cards been released in 2018. But in 2024 I have no idea what the company wants to do with them. It's over five years behind what I was expecting. My expectations for how the AI hardware wave would unfold were fit for another world entirely. If this is the best the challengers can do, the most we can hope for is that they depress Nvidia's margins somewhat so we can buy its cards cheaper in the future. As we go towards the Singularity, I've gone from expecting revolutionary new hardware from AI startups to hoping Nvidia can keep making GPUs faster and more programmable. Ironically, that latter thing is one trend that I missed, and going from Maxwell cards to the last generation, the GPUs have gained a lot in terms of how general purpose they are. The range of domains they can be used for is definitely going up as time goes on. I thought that AI chips would be necessary for this, and that GPUs would remain as toys, but it has been the other way around.
- danielmarkbruce 3y agoIt's a swamp and a moat. Lots of moats are swamps. Look at a lot of large enterprise software products. Awful to work with, sticky as hell, swamp as it gets. And very much a moat.
- parentheses 3y agoJK is always going to be critical of any winners in HPC chips. They are competitors of his.
- StressedDev 3y agoI think we could summarize this article as Jim Keller calls CUDA and x85 names. Name calling is not an argument and whenever I see someone resort to name calling, it tells me they probably don't have better arguments. In engineering and software development, we have to accept trade-offs and imperfection. Every decision has pluses and minuses. Instead of name calling, Jim Keller should explain what his approach is and why he thinks it is an improvement. He should also explain its limitations and weaknesses. Finally, he should also explain how he will handle the inevitable compromises and imperfections which arise in successful systems.