5 ms·
As someone who worked in the ML infra space: Google, Meta, XAI, Oracle, Microsoft, Amazon have clusters that perform better than the highest performing cluster
by ipsum2 2y ago
As someone who worked in the ML infra space: Google, Meta, XAI, Oracle, Microsoft, Amazon have clusters that perform better than the highest performing cluster on Top500. They don't submit because there's no reason to, and some want to keep the size of their clusters a secret. They're all running Nvidia. (Except Google, who uses TPUs and Nvidia.)
> El Capitan – we don’t yet know how big of a portion yet as we write this – with 43,808 of AMD’s “Antares-A” Instinct MI300A devices
By comparison XAI announced that they have 100k H100s. MI300A and H100s have roughly similar performance. Meta says they're training on more than 100k H100s for Llama-4, and have the equivalent of 600k H100s worth of compute. (Note that compute and networking can be orthogonal).
Also, Nvidia B200s are rolling out now. They offer 2-3x the performance of H100s.
- danpalmer 2y agoGoogle is running its own TPU hardware for internal workloads. I believe Nvidia is just resold for cloud customers.
- ipsum2 2y agoNvidia GPUs are also used for inference on Google products. It just depends on availability.
- okdood64 2y agoHuh? https://cloud.google.com/tpu/docs/intro-to-tpu https://cloud.google.com/tpu/docs/intro-to-tpu
- deeth_starr_v 2y agoNot true. Apple trained some models on their TPU
- danpalmer 2y agoApologies, to be clear what I meant was that to my knowledge Google doesn't use GPUs for it's own stuff, but does sell both TPUs and GPUs to others on Cloud. Also, to be clear, I have no internal info about this, I'm going based on external stuff I've seen.
- pclmulqdq 2y agoB200s have an incremental increase in FP64 and FP32 performance over H100s. That is the number format that HPC people care about. The MI300A can get to 150% the FP64 peak performance that B200 devices can get, although AMD GPUs have historically underperformed their spec more than Nvidia GPUs. It's possible that B200 devices are actually behind for HPC.
- cayleyh 2y agoTop line comparison numbers for reference: https://www.theregister.com/2024/03/18/nvidia_turns_up_the_ai/ https://www.theregister.com/2024/03/18/nvidia_turns_up_the_a... It does seem like Nvidia is prioritizing int8 / fp8 performance over FP64, which given the current state of the ML marketplace is a great idea.
- nextos 2y agoMI300 also have decent performance in FP16 (~108 TFLOPS). Not as good as NVIDIA, but it's getting there. Anyone has experience using these on JAX? Support is said to be decent, but no idea if it's good enough for research-oriented tasks, i.e. stable enough for training and inference.
- zekrioca 2y agoThe Top500 list is useful as a public, standardized baseline that is straightforward, with a predicted periodicity for more than 30 years. It is trickier to compare cloud infras due to their heterogeneity, fast pace, and more importantly, due the lack of standardized tests, although the MLCommons [1] have been very keen on helping with that. [1] https://mlcommons.org/datasets/ https://mlcommons.org/datasets/
- makeitdouble 2y agoIf I understand your comment correctly, we're taking a stable but not that relevant metric, because the real players of the market are too secretive, fast and far ahead to allow for simple comparisons. From a distance, it kinda sounds like listening to kids brag about their allowance while the adults don't want to talk about their salary, and try to draw wider conclusions from there.
- wbl 2y agoEven the DoE posts top 500 results when they commission a supercomputer.
- makeitdouble 2y agoDoE has absolutely no incentive (nor need, I'd argue) to compare their supercomputers to commercially owned data center operations though. Comparing their crazy expensive custom built HPC to massive arrays of customer grade hardware doesn't bring them additional funds, nor help them more PR wise than being the owner of the fastest individual clusters. Being at the top of some heap is visibly one of their goal: https://www.energy.gov/science/high-performance-computing https://www.energy.gov/science/high-performance-computing
- khm 2y agoDOE clusters are also massive arrays of customer grade hardware. Private cloud can only keep up in low precision work, and that is why they're still playing with remote memory access over TCP, because it's good enough for web and ML. High precision HPC exists in the private cloud, but you only hear "we don't want to embarrass others" excuses because otherwise you would be able to calculate the cost. On prem HPC is still very, very much cheaper than hiring out.
- almostgotcaught 2y agoYa exactly - no one cares about top500 outside of academia (literally have never heard it come up at work). So this is like the gold star (participation award) of DCGPU competition.
- maratc 2y ago> Nvidia B200s ... offer 2-3x the performance of H100s For ML, not for HPC. ML and HPC are two completely different, only loosely related fields. ML tasks are doing great with low precision, 16 and 8 bit precision is fine, arguably good results can be achieved even with 4 bit precision [0][1]. That won't do for HPC tasks, like predicting global weather, computational biology, etc. -- one would need 64 to 128 bit precision for that. Nvidia needs to decide how to divide the billions of transistors on their new silicon. Greatly oversimplifying, they can choose to make one of the following: * Card A with *n* FP64 cores, or * Card B with *2n* FP32 cores, or * Card C with *4n* FP16 cores, or * Card D with *8n* FP8 cores, or (theoretically) * Card E with *16n* FP4 cores (not sure if FP4 is a thing). Card A would give HPC guys n usable cores, and it would give ML guys n usable cores. On the other end, Card E would give ML guys 16n usable cores (and zero usable cores for HPC guys). It's no wonder that HPC crowd wants Nvidia to produce Card A, while ML crowd wants Nvidia to produce Card E. Given that all the hype and the money are currently with the ML guys (and $NVDA reflects that), Nvidia will make a combination of different cores that is much much closer to Card E than it is to Card A. Their new offerings are arguably worse than their older offerings for HPC tasks, and the feeling with the HPC crowd is that "Nvidia and AMD are in the process of abandoning this market". [0] https://papers.nips.cc/paper/2020/file/13b919438259814cd5be8cb45877d577-Paper.pdf https://papers.nips.cc/paper/2020/file/13b919438259814cd5be8... [1] https://arxiv.org/abs/2212.09720 https://arxiv.org/abs/2212.09720
- touisteur 2y agoWith the B100 somehow announced to have lower scalar FP64 throughput than the H100 (did they remove the DP tensor cores ?), one will have to rely on Ozaki schemes (dgemm with int8 tensor cores) and lots of the recent body of work on mixed-precision linear algebra show there's a lot of computing power to be harnessed from Tensor Cores. One of the problems of HPC now is a level of ossification of some codebases (or the lack of availability of porting/coding/optimizing people). You shouldn't have to rewrite everything every 5 years but the hardware constructors go where they go and we still haven't found the right level of abstraction to avoid big porting efforts.
- ipsum2 2y agoYes, that's a great point that I missed. From anecdotal evidence, it seems more people are using supercomputers for ML use cases, that would have been traditionally done by HPC. (eg training models for weather forecasts)
- formerly_proven 2y agoChina has been absent from TOP500 for years as well.
- zitterbewegung 2y agoGenerally HPC compute has lower margins similar to consoles. It makes sense that AMD would fight for that contract more than NVIDIA similar to IBM stopped doing this. Its sort of comparing Apples to Raspberry Pis.
- geerlingguy 2y agoHey now I compare Apples to Raspberry Pi's regularly :)
- llm_trw 2y agoA cluster is not a super computer. The whole point of a super computer is that it act as much as a single machine as it is possible while a cluster is a soup of nearly independent machines.
- kristjansson 2y ago> soup of nearly independent machines that does a serious disservice to hyperscaler clusters.
- llm_trw 2y agoSure but it's closer to the truth than saying they have similar or more raw compute than a super computer.
- almostgotcaught 2y agoi wish people wouldn't make stuff up just to sound cool. like do you have actual experience with gov/edu HPC? i doubt it because you couldn't be more wrong - lab HPC clusters are just very very poorly (relative to FAANG) strewn together nodes. there is absolutely no sense in which they are "one single machine" (nothing is "abstracted over" except NFS). what you're saying is trivially false because no one ever requests all the machines at once (except when they're running linpack to produce top500 numbers). the rest of the time the workflow is exactly like in any industrial cluster: request some machines (through slurm), get those machines, run your job (hopefully you distributed the job across the nodes correctly), release those machines. if i still had my account i could tell you literally how many different jobs are running right now on polaris.
- bocklund 2y agoActually, LLNL (the site of El Capitan) has a process for requesting Dedicated Application Time (a DAT) where you use up to a whole machine, usually over a weekend. They occur fairly regularly. Mostly it's lots of individual users and jobs, like you said though.
- lobochrome 2y agoB200 is very much not rolling out because NVIDIA, after the respin, doesn't have the thermals under control (yet). Your other points may be valid.
- deeth_starr_v 2y agoSource?
- lobochrome 2y agoReuters!
- _zoltan_ 2y agodon't spread FUD please.
- deleted 2y ago[deleted]