8 ms·
AMD now has more compute on the top 500 than Nvidia
- pie420 2y agolayperson with no industry knowledge, but it seems like nvidia's CUDA moat will fall in the next 2-5 years. It seems impossible to sustain those margins without competition coming in and getting a decent slice of the pie
- metadat 2y agoBut how will AMD or anyone else push in? CUDA is actually a whole virtualization layer on top of the hardware and isn't easily replicable, Nvidia has been at it for 17 years. You are right, eventually something's gotta give. The path for this next leg isn't yet apparent to me. P.s. how much is an exaflop or petaflop, and how significant is it? The numbers thrown around in this article don't mean anything to me. Is this new cluster way more powerful than the last top?
- bryanlarsen 2y agoAnybody spending tens of billions annually on Nvidia hardware is going to be willing to spend millions to port their software away from CUDA.
- echelon 2y agoFor the average non-FAANG company, there's nothing to port to yet. We don't all have the luxury of custom TPUs.
- talldayo 2y agoTo slower hardware? What are they supposed to port to, ASICs?
- adgjlsfhk1 2y agoif the hardware is 30% slower and 2x cheaper, that's a pretty great deal.
- selectodude 2y agoPower density tends to be the limiting factor for this stuff, not money. If it's 30 percent slower per watt, it's useless.
- Wytwwww 2y agoThe ratio between power usage and GPU cost is very, very different than with CPUs, though. If you could save e.g. 20-30% of the purchase price that might make it worth it. e.g. you could run a H100 at 100% utilization 24/7 for 1 years at $0.4 per kWh (so assuming significant overhead for infrastructure etc.) and that would only cost ~10% of the purchase price of the GPU itself.
- wbl 2y agoPower usage cost isn't the money but the capacity and cooling.
- Wytwwww 2y agoYes, I know that. Hence I quadrupled the price of electricity or are you saying that the cost of capacity and cooling doesn't scale directly with power usage? We can increase that another 2x and the cost would still be relatively low compared to the price/deprecation of the GPU itself.
- pjmlp 2y agoFirst they need to support everything that CUDA is capable of in programing language portfolio, tooling and libraries.
- bryanlarsen 2y agoA typical LLM might use about 0.1% of CUDA. That's all that would have to be ported to get that LLM to work.
- pjmlp 2y agoWhich is missing the point why CUDA has won. Then again, maybe the goal is getting 0.1% of CUDA market share. /s
- its_down_again 2y agoIn the words of Gilfoyle-- I'll bite. Why has CUDA won?
- pjmlp 2y agoCUDA C++, CUDA Fortran, CUDA Anything PTX, plus libraries, IDE integration, GPU graphical debugging. Coupled with Khronos, Intel, AMD never delivering anything comparable with OpenCL, Apple losing interest after Khronos didn't took OpenCL into the direction they wanted, Google never adopting it favouring their Renderscript dialect.
- imtringued 2y agoNvidia has won because their compute drivers don't crash people's systems when they run e.g. Vulkan Compute. You are mostly listing irrelevant nice to have things that aren't deal breakers. AMD's consumer GPUs have a long history of being abandoned a year or two after release.
- pjmlp 2y agoCUDA C++, CUDA Fortran, CUDA Anything PTX, plus libraries, IDE integration, GPU graphical debugging, aren't only nice to have things.
- vlovich123 2y agoThe API part isn't thaaat hard. Indeed HIP already works pretty well at getting existing CUDA code to work unmodified on AMD HW. The bigger challenge is that the AMD and Nvidia architectures are so different that the optimization choices for what the kernels would look like are more different between Nvidia and AMD than they would be between Intel and AMD in CPU land even including SIMD.
- pjmlp 2y agoOnly if the only thing one cares about is CUDA C++, and not CUDA C, CUDA C++, CUDA Fortran, CUDA Anything PTX, plus libraries, IDE integration, GPU graphical debugging.
- vlovich123 2y agoCUDA C works fine with HIP not sure what you're referring to. As for the other pieces, GPU graphical debugging isn't relevant for CUDA and I don't know what IDE integration is special / relevant for CUDA but AMD does have a ROCm debugger which I would imagine would be sufficient for simultaneous debugging of CPU & GPU. You won't get developer tools like nsight systems but I'm pretty sure AMD has equivalent tooling. As for Fortran, that doesn't come up much in modern AI stuff. I haven't observed PTX / GCN assembly within AI codebases but maybe you have extra insight there.
- okdood64 2y agoMaybe the DOJ will come in and call it anti-trust shenanigans. Not that I would want this...
- sangnoir 2y agoCUDA is the assembly to Torch's high-level language; for most, it's a very good intermediary, but an intermediary nonetheless, as it is between the actual code they are interested in, and the hardware that runs it. Most customers care about cost-effectiveness more than best-in-class raw-performance, a fact that AMD has ruthlessly exploited over the past 8 years. It helps that AMD products are occasionally both.
- pjmlp 2y agoCUDA is much more than that, and missing that out is exactly why NVidia keeps winning.
- imtringued 2y agoAgain, I have AMD hardware and can't use it.
- pjmlp 2y agoAMD is to blame for where they stand.
- LeanderK 2y agoits possible. Just look at Apples GPU, its mostly supported by torch, what's left are mostly edge-cases. Apple should make a datacenter GPU :D that would be insanely funny. It's actually somewhat well positioned as, due to the MacBooks, the support is already there. I assume here that most things translate to linux, as I don't think you can sell MacOS in the cloud :D I know a lot developing on apples silicon and just pushing it to clusters for bigger runs. So why not run it on an apple GPU there?
- talldayo 2y ago> what's left are mostly edge-cases. For everything that isn't machine learning, I frankly feel like it's the other way around. Apple's "solution" to these edge cases is telling people to write compute shaders that you could write in Vulkan or DirectX instead. What sets CUDA apart is an integration with a complex acceleration pipeline that Apple gave up trying to replicate years ago. When cryptocurrency mining was king-for-a-day, everyone rushed out to buy Nvidia hardware because it supported accelerated crypto well from the start. The same thing happened with the AI and machine learning boom. Apple and AMD were both late to the party and wrongly assumed that NPU hardware would provide a comparable solution. Without a CUDA competitor, Apple would struggle more than AMD to find market fit.
- LeanderK 2y agowell, but machine learning is the major reason we use GPUs in the datacenter (not talking about consumer GPUs here). The others are edge-cases for data-centre applications! Apple is uniquely positioned exactly because it is already solved due to a significant part of the ML-engineers using MacBooks to develop locally. The code to run these things on apples GPUs exist and is used every day! I don't know anyone using AMD GPUs, but pretty often its nvidia on the cluster and Apple on the laptop. So if nvidia is making these juicy profits, i think apple could seriously think about moving to the cluster if it wants to.
- talldayo 2y agoSoftware developers using Macbooks doesn't mean Apple solved the ML problem. The past 10 years of MacOS removing features has somewhat proved that software developers will keep using Macs even when the featureset regresses. Like how Apple used to support OpenCL as a CUDA alternative, but gave up on it altogether to focus on simpler, mobile-friendly GPU designs. The Pytorch MPS patches are a fun appeasement for developers, but they didn't unthrone Nvidia's demand. They didn't beat Nvidia on performance per watt, they didn't match their price, their scale or CUDA's featureset, and they don't even provide basic server drivers. It's got nothing to do with what brand you prefer and everything to do with what makes actual sense in a datacenter. Apple can't take on Nvidia clusters without copying Nvidia's current architecture - Apple Silicon's current architecture is too inefficient to be a serious replacement to Nvidia clusters. If Apple wanted to have a shot at entering the cluster game, that window of opportunity closed when Apple Silicon converged on simplified GPU designs. The 2w NPUs and compute shaders aren't going to make Nvidia scared, let alone compete with AMD's market share.
- stonemetal12 2y agoAccording to Wikipedia the previous #1 was from 2022 with a peak petaflops of 2,055. This system is rated at 2,746. So about 33% faster than the old #1. Also, of the top 10, AMD has 5 systems. https://en.wikipedia.org/wiki/TOP500 https://en.wikipedia.org/wiki/TOP500
- smokel 2y ago> P.s. how much is an exaflop or petaflop 1 petaflop = 10^15 flops = 1,000,000,000,000,000 flops. 1 exaflop = 10^18 flops = 1,000,000,000,000,000,000 flops. Note that these are simply powers of 10, not powers of 2, which are used for storage for example.
- fweimer 2y agoIsn't porting software to the next generation supercomputer pretty standard for HPC?
- ok123456 2y agoPeople have been chipping away at this for a while. HIP allows source-level translation, and libraries like Jax provide a HIP version.
- vitus 2y ago> P.s. how much is an exaflop or petaflop, and how significant is it? The numbers thrown around in this article don't mean anything to me. Is this new cluster way more powerful than the last top? Nominally, a measurement in "flops" is how many (typically 32-bit) FLoating-point Operations Per Second the hardware is capable of performing, so it's an approximate measure of total available computing power. A high-end consumer-grade CPU can achieve on the order of a few hundred gigaflops (let's say 250, just for a nice round number). https://boinc.bakerlab.org/rosetta/cpu_list.php https://boinc.bakerlab.org/rosetta/cpu_list.php A petaflop is therefore about four thousand of those; multiply by another thousand to get an exaflop. For another point of comparison, a high-end GPU might be on the order of 40-80 teraflops. https://www.tomshardware.com/reviews/gpu-hierarchy,4388-2.html#legacy-gpu-hierarchy https://www.tomshardware.com/reviews/gpu-hierarchy,4388-2.ht...
- metadat 2y agoHow many teraflops in an exaflop? The tera is screwing me up.. Google not helping today, so many cards.
- aaronblohowiak 2y agohttps://en.m.wikipedia.org/wiki/Metric_prefix https://en.m.wikipedia.org/wiki/Metric_prefix
- quickthrowman 2y ago> But how will AMD or anyone else push in? CUDA is actually a whole virtualization layer on top of the hardware and isn't easily replicable, Nvidia has been at it for 17 years. NVidia currently has 80-90% gross margins on their LLM GPUs, that’s all the incentive another company needs to invest money into a CUDA alternative.
- NineStarPoint 2y agoA high grade consumer gpu a (a 4090) is about 80 teraflops. So rounding up to 100, an exaflop is about 10,000 consumer grade cards worth of compute, and a petaflop is about 10. Which doesn’t help with understanding how much more impressive these are than the last clusters, but does to me at least put the amount of compute these clusters have into focus.
- vitus 2y agoYou're off by three orders of magnitude. My point of reference is that back in undergrad (~10-15 years ago), I recall a class assignment where we had to optimize matrix multiplication on a CPU; typical good parallel implementations achieved about 100-130 gigaflops (on a... Nehalem or Westmere Xeon, I think?).
- NineStarPoint 2y agoYou are 100% correct, I lost a full prefix of performance there. Edited my message. Which does make the clusters a fair bit less impressive, but also a lot more sensibly sized.
- deleted 2y ago[deleted]
- winwang 2y ago4090 tensor performance (FP8): 660 teraflops, 1320 "with sparsity" (i.e. max theoretical with zeroes in the right places). https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvidia-ada-gpu-architecture.pdf https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvid... But at these levels of compute, the memory/interconnect bandwidth becomes the bottleneck.
- shmerl 2y agoThere is ZLUDA to break the lock-in for those who are stuck with it. The rest will use something else.
- jillesvangurp 2y agoSoftware will bridge the gap. There are simply too many competing platforms out there that are not Nvidia based. Most decent AI libraries and frameworks already need to support more than just Nvidia. There's a reason macs are popular with AI researchers: many of these platforms support Apple's chips already and they perform pretty well. Anything that doesn't support those chips, is a problem waiting to be fixed with plenty of people working on fixing that. If it can be fixed for Apple's chips, it can also be fixed for other people's chips. And of course there is some serious amount of money sloshing around in this space. Things being hard doesn't mean it's impossible. And there's no shortage of extremely well funded companies working on this stuff. All your favorite trillion $ companies basically. And most of them have their own AI chips too. And probably some reservations about perpetually handing a lot of their cash to Nvidia. If you want an example of a company that used to have a gigantic moat that is now dealing with a lot of competition, look at Intel. X86 used to be that moat. And that's looking pretty weak lately. One reason that AMD is in the news a lot lately is that they are growing at Intel's expense. Nvidia might be their next target.
- latchkey 2y agoWe donated one of our MI300x systems to the SCALE team. The moat-less future is coming more quickly than you think. https://scale-lang.com/ https://scale-lang.com/
- YetAnotherNick 2y agoCUDA moat is highly overrated for AI in the first place and sold as the reason for the failure of AMD. Almost no one in AI uses CUDA. They only use pytorch or Triton. TPUs didn't face lot of hurdle due to CUDA because they were initially better in terms of price to performance and supported pytorch, tensorflow and jax. The reason why AMD is behind is that it is behind in hardware. MI300x is more pricey per hour in all the cloud I can find compared to H100, and the MFU is order of magnitude lower compared to NVIDIA for transformers, even though transformers are fully supported. And I get same 40-50% MFU in TPU for the same code. If anyone is investing >10 million dollar for hardware, they sure can invest a million dollar to rewrite everything in whatever language AMD asks them to if it is cheaper.
- saagarjha 2y agoPeople most certainly do use CUDA
- llm_trw 2y agoAt this scale cuda is quite useless. You need to develop your own in house solution to distributing workloads. The difference to regular clusters is that all the memory is globally visible, so machine 0023 can access and modify address 0x0123456789abcdef0123456789abcdef which happens to be on machine 0999.
- yeahwhatever10 2y agoCUDA is one part, but another part of Nvidia's lead is their focus on bandwidth both memory and GPU-GPU communication. AMD dramatically falls behind Nvidia in training because of its terrible collective times (AllReduce, AllGather, etc.)
- nwgo 2y agoIt does not matter. AMD is shit when it comes to low-level processing, their algos are stuck that go nowhere. Nvidia is killing it. There is a reason why Zookerberg ordered billions in GPUs from Nvidia and not from AMD.
- ipsum2 2y agoAMD GPUs handle all inference for Llama3 at Meta btw.
- thechao 2y agoWhy is AMD shit at low-level processing? What does it mean "their algos are stuck"? Having watched "the industry" for a few decades now, the appeal for NV smells heavily like the old appeal for Xeons, and Big Blue before them. The moat appears (to me, an unknowledgeable outsider) to be just cultural, not necessarily technical.
- sqeaky 2y agoThis is just silly fanboyism, there are pros and cons to each.
- Koshkin 2y agoThis comment is somewhat more insightful: https://news.ycombinator.com/item?id=40791010 https://news.ycombinator.com/item?id=40791010
- trynumber9 2y ago>AMD said it now expects to make more than $5 billion from sales of its Instinct data center GPUs this year due to high demand from hyperscalers like Meta and Microsoft It's no Nvidia but Meta has ordered AMD GPUs.
- ipsum2 2y agoAs someone who worked in the ML infra space: Google, Meta, XAI, Oracle, Microsoft, Amazon have clusters that perform better than the highest performing cluster on Top500. They don't submit because there's no reason to, and some want to keep the size of their clusters a secret. They're all running Nvidia. (Except Google, who uses TPUs and Nvidia.) > El Capitan – we don’t yet know how big of a portion yet as we write this – with 43,808 of AMD’s “Antares-A” Instinct MI300A devices By comparison XAI announced that they have 100k H100s. MI300A and H100s have roughly similar performance. Meta says they're training on more than 100k H100s for Llama-4, and have the equivalent of 600k H100s worth of compute. (Note that compute and networking can be orthogonal). Also, Nvidia B200s are rolling out now. They offer 2-3x the performance of H100s.
- danpalmer 2y agoGoogle is running its own TPU hardware for internal workloads. I believe Nvidia is just resold for cloud customers.
- amelius 2y agoWhy the focus on AMD and Nvidia? It really isn't that hard to design a large number of ALU blocks into some silicon IP block and make them work together efficiently. The real accomplishment is fabricating them.
- georgeecollins 2y agoBut not the profits.
- talldayo 2y ago> It really isn't that hard to design a large number of ALU blocks into some silicon IP block and make them work together efficiently. It really is that hard, and the fabrication side of the issue the easy part from Nvidia's perspective - you just pay TSMC a shitload of money. Nvidia's real victory (besides leading on performance-per-watt) is that their software stack doesn't suck. They invested in complex shader units and tensor accelerators that scale with the size of the card rather than being restrained in puny and limited NPUs. CUDA unified this featureset and was industry-entrenched for almost a decade, which gave it pretty much any feature you could want be it crypto acceleration or AI/ML primitives. The ultimate tragedy is that there was a potential future where a Free and Open Source CUDA alternative existed. Apple wrote the OpenCL spec for exactly that purpose and gave it to Khronos, but later abandoned it to focus on... checks clipboard MLX and Metal Performance Shaders. Oh, what could have been if the industry weren't so stingy and shortsighted.
- amelius 2y ago> you just pay TSMC a shitload of money I guess with money you can win any argument ...
- talldayo 2y agoSure, Apple did the same thing with TSMC's 5nm node. They still lost in performance-per-watt in direct comparison with Nvidia GPUs using Samsung's 8nm node. Money isn't everything, even when you have so much of it that you can deny your competitors access to the tech you use. Nvidia's lead is not only cemented by dense silicon. Their designs are extremely competitive, perhaps even a generational leap over what their competitors offer.
- vitus 2y agoAfter skimming the article, I'm confused -- where exactly is the headline being pulled from? If you look at the table toward the bottom, no matter how you slice it, Nvidia has 50% of the total cores, 50% of the total flops, and 90% of the total systems among the Top 500, while AMD has 26% of the total cores, 27.5% of the total flops, and 7% of the total systems. Is it a matter of newly-added compute? > This time around, on the November 2024 Top500 rankings, AMD is the big winner in terms of adding capacity to the HPC base.
- Koshkin 2y ago> AMD GPUs drove 72.1 percent of the new performance added for the November 2024 rankings
- vitus 2y agoYes, I saw that, but that doesn't justify the title as written. Had it said "AMD Now Has More New Compute" I wouldn't have said anything.
- zamadatix 2y agoSaid table is titled for "Accelerated Supers" (i.e. only ones with GPUs) so the numbers can't be applied to the Top500 as a whole like that. Combining numbers from the summaries at the bottom of the table titled for "All Supers", Nvidia is more like 38% of Top500 FLOPS as they don't have any non-accelerated systems in the list. Knowing all of that it still leaves unexplained whether AMD has the needed ~70% of non-accelerated compute (assuming FLOPS) to clear the bar for the headline. It seems unlikely to me... but the article doesn't actually have enough data to be sure one way or the other.
- vitus 2y agoThat's a good point. That said, I assumed the context of the article was specifically on the topic of AMD GPUs, and not, say, Epyc processors, so if so, it's ultimately irrelevant. (Also, there are just over 3 exaflops across non-accelerated supers; AMD would need > 2/3 of the remaining share in order to surpass Nvidia on that front as well.)
- latchkey 2y agoI'm sure there is also a lot not on the Top500. I've got enough AMD MI300x compute for about 140th position, but haven't submitted numbers.
- deleted 2y ago[deleted]
- aiforecastthway 2y agoTop500 is weird for lots of reasons. It over-indexes on a few peculiar types of workloads and a few peculiar types of users (mostly gov). Historically, those workloads and users were leading indicators of certain types of things. I don't think that's true anymore. In fact, I wonder if this is mostly a story of the government agencies not being able to compete with the private sector for NVIDIA gpus.
- latchkey 2y agoI think you nailed it on the head. Companies like CoreWeave have deployed so many giant clusters (and growing), it is insane. Their IDLE compute is larger than most of the supercompuers out there. Of course, they aren't on the list either.
- ted_dunning 2y agoThere is another widespread common factor among the top machines. A large majority are based on HPE Slingshot networking (7 out of top 10 by my count). Without blindingly fast, otherwise blinding numerical performance dims quite a lot. This is why the Cerebras numbers on heavy numerical problems are competitive up to a pretty severe ceiling. Below that point, their on wafer interconnects suffice, above it they cannot scale the data communications bandwidth necessary.