8 ms·

MLPerf training tests put Nvidia ahead, Intel close, and Google well behind

shmerl 3y ago
Why is AMD not compared?
- ioedward 3y ago
  AMD didn't submit results, despite geohot/Lisa Su's plan to get AMD on MLPerf: https://twitter.com/realGeorgeHotz/status/1669803464082489347 https://twitter.com/realGeorgeHotz/status/166980346408248934...
  - dhruvdh 3y ago
    If Dr. Lisa Su, the CEO, wants AMD to submit to MLPerf they would have already submitted to MLPref. This tweet is about a discussion George had with her on AMD's open-source practices and code quality for the contributions they make.
- zwaps 3y ago
  AMD does not do ML. They don't have the capability and engineers, nor desire in their leadership, to develop the software stack.
  - shmerl 3y ago
    Makes no sense and not true most likely.
    - brucethemoose2 3y ago
      AMD does have a history of spinning their wheels on the software side. Not graphics specifically (anymore), but definitely with their compute initiatives like HSA or even OpenCL. I dunno what to think about ROCm. It does seem like they should have thrown the MI300 into the mix, but maybe its not ready.
    - zwaps 3y ago
      The proof is in the pudding. AMD has excellent hardware and has had years now to get anything going. They didn't, and they don't so our best guess is, they won't.
sillysaurusx 3y ago
One bad sign is that Google recently put out a press release that they’ve achieved the largest training job ever run, but they didn’t actually train any models. It was a bandwidth test: https://news.ycombinator.com/item?id=38227836 https://news.ycombinator.com/item?id=38227836 (Shoutout to DavidSJ who sleuthed this out based on the careful wording of the post.) This was a big shock to me since MLPerf is the gold standard for performance comparisons. The reason you want to train an actual model on real data is because it gives a precise idea of what you can expect in the field. Google’s 2019 MLPerf results are why I dove in to TPU land and fell in love with TPUs. Nowadays it’s all about language models and giant training runs and there’s not a lot of compute left over for average joes (which I very much was), but rants aren’t productive. What would be productive is for Google to focus on the basics: focus on MLPerf. It’s the standard benchmark for a reason. I can’t fault them for going where the money is, but it’s still sad to see the sorry state of Google being dead last. I’m still rooting for you, Jax team. I believe in you. TRC is looking better these days. If you need spare compute or just want to try out TPUs, give them a spin for a month. When I was complaining on Twitter in 2023 about TPUs being deleted after 30 minutes circa 2022, two people stepped in and said that they were happy with their quotas and haven’t observed similar pain points. So maybe they’ve solved the provisioning issues, and they’re starting to treat TRC members as more than third-class citizens again. But, especially in modern times, it’s hard to complain when compute is so scarce; you can do some pretty amazing things with TPUs if you focus on them for a few months. Thank you to the organizers of TRC (Jonathan in particular, their support lead, who is single handedly the most impressive support person I’ve ever seen by a huge margin; and Zak for keeping so many balls in the air while leading the TPU infra).
- kccqzy 3y ago
  > What would be productive is for Google to focus on the basics: focus on MLPerf. It’s the standard benchmark for a reason. I can’t fault them for going where the money is, but it’s still sad to see the sorry state of Google being dead last. You seem to be suggesting that Google isn't focusing on performance on MLPerf but on something else that's more lucrative. Do you want to expound on what that is?
  - 3y ago
jfim 3y ago
Oddly the article talks about training performance, but the link to the tests links to the inference benchmark. The correct link to the training benchmark results is at https://mlcommons.org/benchmarks/training/ https://mlcommons.org/benchmarks/training/
frays 3y ago
This needs to be called out anytime an AI benchmarking article with Google as one of its key comparisons is published: many of the outstanding AI researchers and innovators who gave Google the reputation it has today are no longer at the company. Startups such as Essential AI, Ideogram, Character.ai were all founded by ex-Googlers. Plenty of engineers and researchers from Google Brain/Research have pivoted into these types of start-ups, not necessarily as founders but as individual contributors continuing their engineering and research work at these companies instead. Heck, Google is even considering investing into Character.ai (ex-Google founder) based on the news released a few days ago: https://www.reuters.com/technology/google-talks-invest-ai-startup-characterai-sources-2023-11-10/ https://www.reuters.com/technology/google-talks-invest-ai-st... It's not surprising that Google isn't the leader of AI anymore. The people who put Google in that lead position are gone.
- kmeisthax 3y ago
  Google doesn't build, it buys.
  - VirusNewbie 3y ago
    Did they buy K8s, TPU, Jax or TF?
    - deaddodo 3y ago
      Or even, you know, the entire Google suite? There's plenty to criticize Alphabet/GOOG on. But pretending they don't develop things in house is just naive.
      - nine_k 3y ago
        Maybe not entire. Google Ads: built from great many pieces, one of the largest was DoubleClick, acquired in 2008. Google Docs: acquired in 2006 from Writely. Google Sheets: acquired in 2006 from XL2Web. Google Maps: acquired in 2004. I'm not trying to say that Google acquired everything and hasn't built anything; not at all. But a few crown jewels were indeed acquisitions.
joshvm 3y ago
The headline feels like clickbait. Does MLPerf also benchmark power consumption/cost per submission? It's not particularly illuminating that if you use a 10k GPU cluster, you can train things quickly and a "per chip" comparison is vague. For most people the bottom line is cost and availability anyway - does it matter if a TPU is twice as slow if it costs half as much? I think I read somewhere that Apple Silicon is actually one of the most efficient platforms to use for development (e.g. a tricked out Mini makes quite a good inference server); I've been able to train recent object detection models on my M2 without much trouble (and it's much cheaper than paying for cloud time).
- sillysaurusx 3y ago
  Nah, MLPerf is legit. I was skeptical of it too, but wanted to leave a quick note (I have to run) that they’re solid. I’d be the first to call it out if it was pointless or misleading, but it turns out to be the only way to get a true idea of comparisons across different hardware. It forces everyone to achieve the same goal, which is key; otherwise you’re left with a bunch of slight (and large) variations that tell you nothing about achieving the actual goal, which is all you care about. It’s a bit like a shared space race. Getting that top-N accuracy to 74 point yada yada percent in 37 seconds on a certain resnet architecture tells you that you can do the same thing on that hardware, which means you can do your own things just as quickly. So it’s a worthwhile time investment. If you force everyone to get that same accuracy with the same model arch, you can make informed decisions, which is especially important when throwing millions of VC dollars around. EDIT: Sorry, I completely misread you. It’s difficult to quantify what you’d like to measure: the bottom line price of who is cheaper for your expected workload. There are immense trade offs, not least of which is time spent learning a particular (esoteric) stack. CUDA knowledge doesn’t port to JAX and vice-versa. Prices are always changing, and you can usually work out a deal with the sales team to get lower than advertised. Especially if they want you to choose them instead of some competitor. So in general it’s hard to figure out what you can expect for production workloads in terms of total dev cost vs price vs speed. I will say that as a researcher, there’s no substitute for fast iteration cycles. I’m one of the few who believe in scaling down your models as much as possible when testing experimental ideas, precisely because you can try 30 runs instead of 3. So all else being equal, I’d take speed. But all else isn’t equal. The only thing I want nowadays is free plus stable. It’s looking like a 4090 might be the way to get that, which is enough to try out some interesting ideas.
huac 3y ago
10k H100's cost around $400M at sticker price. Sometimes I wonder if a massive setup like this is less expensive (volume discount) or more.
- fnbr 3y ago
  Less expensive. Big discounts exist.
- YetAnotherNick 3y ago
  It takes $3k for Nvidia to produce H100[1], so the total for Nvidia will be $30m, not small but definitely not the most expensive supercomputer. [1]: https://the-decoder.com/nvidias-h100-gpu-sells-like-hot-cakes-with-high-profit-margins https://the-decoder.com/nvidias-h100-gpu-sells-like-hot-cake...
_xkcr 3y ago
This is dumb- they’re testing against google’s v5e chip, which isn’t the H100 equivalent (the V5P is). Of course Google came out worse here.
- kaelinl 3y ago
  MLPerf does not test hardware. Companies submit results from their own hardware. This is what each company chose to submit.
  - deleted 3y ago
    [deleted]
  - emu 3y ago
    But the interpretation of the journalist is the issue here. Comparing a cost optimized part (TPU v5e) against a performance optimized part (H100) and deciding that this makes Google "behind" is just incorrect.
- dgacmu 3y ago
  There is no publicly announced v5p. Source?
londons_explore 3y ago
> "We delivered more than what was promised-a 103 percent reduction in time-to- train Can someone explain this math to me?
TheCaptain4815 3y ago
For what Google is primarily used for, search engine and so forth, do they need an extremely advanced Ai? As we've seen with other big tech companies, using giant expensive models for "simple tasks" isn't worth it financially. How advanced of a LLM does Google really need to improve search?
- jeffbee 3y ago
  Google has numerous internal applications other than websearch that use machine learning, such as spam classification (Gmail), image segmentation and recognition (photos, image search), text-to-speech and speech-to-text (assistant), very small generative tasks (smart reply, smart compose), predicting which videos will be watched (YouTube), and even operating the pumps and fans in their data centers.
- kaycebasques 3y ago
  Here's a post from Jan 2023 on the official blog: "9 Ways We Use AI In Our Products" https://blog.google/technology/ai/9-ways-we-use-ai-in-our-products/ https://blog.google/technology/ai/9-ways-we-use-ai-in-our-pr...
0-_-0 3y ago
And AMD is nowhere to be found
htrp 3y ago
Actual Results (and not the article talking about it) https://mlcommons.org/benchmarks/inference-datacenter/ https://mlcommons.org/benchmarks/inference-datacenter/

Why is AMD not compared?

ioedward 3y ago
AMD didn't submit results, despite geohot/Lisa Su's plan to get AMD on MLPerf: https://twitter.com/realGeorgeHotz/status/1669803464082489347 https://twitter.com/realGeorgeHotz/status/166980346408248934...
- dhruvdh 3y ago
  If Dr. Lisa Su, the CEO, wants AMD to submit to MLPerf they would have already submitted to MLPref. This tweet is about a discussion George had with her on AMD's open-source practices and code quality for the contributions they make.
zwaps 3y ago
AMD does not do ML. They don't have the capability and engineers, nor desire in their leadership, to develop the software stack.
- shmerl 3y ago
  Makes no sense and not true most likely.
  - brucethemoose2 3y ago
    AMD does have a history of spinning their wheels on the software side. Not graphics specifically (anymore), but definitely with their compute initiatives like HSA or even OpenCL. I dunno what to think about ROCm. It does seem like they should have thrown the MI300 into the mix, but maybe its not ready.
  - zwaps 3y ago
    The proof is in the pudding. AMD has excellent hardware and has had years now to get anything going. They didn't, and they don't so our best guess is, they won't.

One bad sign is that Google recently put out a press release that they’ve achieved the largest training job ever run, but they didn’t actually train any models. It was a bandwidth test: https://news.ycombinator.com/item?id=38227836 https://news.ycombinator.com/item?id=38227836 (Shoutout to DavidSJ who sleuthed this out based on the careful wording of the post.) This was a big shock to me since MLPerf is the gold standard for performance comparisons. The reason you want to train an actual model on real data is because it gives a precise idea of what you can expect in the field. Google’s 2019 MLPerf results are why I dove in to TPU land and fell in love with TPUs. Nowadays it’s all about language models and giant training runs and there’s not a lot of compute left over for average joes (which I very much was), but rants aren’t productive. What would be productive is for Google to focus on the basics: focus on MLPerf. It’s the standard benchmark for a reason. I can’t fault them for going where the money is, but it’s still sad to see the sorry state of Google being dead last. I’m still rooting for you, Jax team. I believe in you. TRC is looking better these days. If you need spare compute or just want to try out TPUs, give them a spin for a month. When I was complaining on Twitter in 2023 about TPUs being deleted after 30 minutes circa 2022, two people stepped in and said that they were happy with their quotas and haven’t observed similar pain points. So maybe they’ve solved the provisioning issues, and they’re starting to treat TRC members as more than third-class citizens again. But, especially in modern times, it’s hard to complain when compute is so scarce; you can do some pretty amazing things with TPUs if you focus on them for a few months. Thank you to the organizers of TRC (Jonathan in particular, their support lead, who is single handedly the most impressive support person I’ve ever seen by a huge margin; and Zak for keeping so many balls in the air while leading the TPU infra).

kccqzy 3y ago
> What would be productive is for Google to focus on the basics: focus on MLPerf. It’s the standard benchmark for a reason. I can’t fault them for going where the money is, but it’s still sad to see the sorry state of Google being dead last. You seem to be suggesting that Google isn't focusing on performance on MLPerf but on something else that's more lucrative. Do you want to expound on what that is?
- 3y ago

jfim 3y ago

Oddly the article talks about training performance, but the link to the tests links to the inference benchmark. The correct link to the training benchmark results is at https://mlcommons.org/benchmarks/training/ https://mlcommons.org/benchmarks/training/

frays 3y ago

This needs to be called out anytime an AI benchmarking article with Google as one of its key comparisons is published: many of the outstanding AI researchers and innovators who gave Google the reputation it has today are no longer at the company. Startups such as Essential AI, Ideogram, Character.ai were all founded by ex-Googlers. Plenty of engineers and researchers from Google Brain/Research have pivoted into these types of start-ups, not necessarily as founders but as individual contributors continuing their engineering and research work at these companies instead. Heck, Google is even considering investing into Character.ai (ex-Google founder) based on the news released a few days ago: https://www.reuters.com/technology/google-talks-invest-ai-startup-characterai-sources-2023-11-10/ https://www.reuters.com/technology/google-talks-invest-ai-st... It's not surprising that Google isn't the leader of AI anymore. The people who put Google in that lead position are gone.

kmeisthax 3y ago
Google doesn't build, it buys.
- VirusNewbie 3y ago
  Did they buy K8s, TPU, Jax or TF?
  - deaddodo 3y ago
    Or even, you know, the entire Google suite? There's plenty to criticize Alphabet/GOOG on. But pretending they don't develop things in house is just naive.
    - nine_k 3y ago
      Maybe not entire. Google Ads: built from great many pieces, one of the largest was DoubleClick, acquired in 2008. Google Docs: acquired in 2006 from Writely. Google Sheets: acquired in 2006 from XL2Web. Google Maps: acquired in 2004. I'm not trying to say that Google acquired everything and hasn't built anything; not at all. But a few crown jewels were indeed acquisitions.

joshvm 3y ago

The headline feels like clickbait. Does MLPerf also benchmark power consumption/cost per submission? It's not particularly illuminating that if you use a 10k GPU cluster, you can train things quickly and a "per chip" comparison is vague. For most people the bottom line is cost and availability anyway - does it matter if a TPU is twice as slow if it costs half as much? I think I read somewhere that Apple Silicon is actually one of the most efficient platforms to use for development (e.g. a tricked out Mini makes quite a good inference server); I've been able to train recent object detection models on my M2 without much trouble (and it's much cheaper than paying for cloud time).

sillysaurusx 3y ago
Nah, MLPerf is legit. I was skeptical of it too, but wanted to leave a quick note (I have to run) that they’re solid. I’d be the first to call it out if it was pointless or misleading, but it turns out to be the only way to get a true idea of comparisons across different hardware. It forces everyone to achieve the same goal, which is key; otherwise you’re left with a bunch of slight (and large) variations that tell you nothing about achieving the actual goal, which is all you care about. It’s a bit like a shared space race. Getting that top-N accuracy to 74 point yada yada percent in 37 seconds on a certain resnet architecture tells you that you can do the same thing on that hardware, which means you can do your own things just as quickly. So it’s a worthwhile time investment. If you force everyone to get that same accuracy with the same model arch, you can make informed decisions, which is especially important when throwing millions of VC dollars around. EDIT: Sorry, I completely misread you. It’s difficult to quantify what you’d like to measure: the bottom line price of who is cheaper for your expected workload. There are immense trade offs, not least of which is time spent learning a particular (esoteric) stack. CUDA knowledge doesn’t port to JAX and vice-versa. Prices are always changing, and you can usually work out a deal with the sales team to get lower than advertised. Especially if they want you to choose them instead of some competitor. So in general it’s hard to figure out what you can expect for production workloads in terms of total dev cost vs price vs speed. I will say that as a researcher, there’s no substitute for fast iteration cycles. I’m one of the few who believe in scaling down your models as much as possible when testing experimental ideas, precisely because you can try 30 runs instead of 3. So all else being equal, I’d take speed. But all else isn’t equal. The only thing I want nowadays is free plus stable. It’s looking like a 4090 might be the way to get that, which is enough to try out some interesting ideas.

huac 3y ago

10k H100's cost around $400M at sticker price. Sometimes I wonder if a massive setup like this is less expensive (volume discount) or more.

fnbr 3y ago
Less expensive. Big discounts exist.
YetAnotherNick 3y ago
It takes $3k for Nvidia to produce H100[1], so the total for Nvidia will be $30m, not small but definitely not the most expensive supercomputer. [1]: https://the-decoder.com/nvidias-h100-gpu-sells-like-hot-cakes-with-high-profit-margins https://the-decoder.com/nvidias-h100-gpu-sells-like-hot-cake...

_xkcr 3y ago

This is dumb- they’re testing against google’s v5e chip, which isn’t the H100 equivalent (the V5P is). Of course Google came out worse here.

kaelinl 3y ago
MLPerf does not test hardware. Companies submit results from their own hardware. This is what each company chose to submit.
- deleted 3y ago
  [deleted]
- emu 3y ago
  But the interpretation of the journalist is the issue here. Comparing a cost optimized part (TPU v5e) against a performance optimized part (H100) and deciding that this makes Google "behind" is just incorrect.
dgacmu 3y ago
There is no publicly announced v5p. Source?

londons_explore 3y ago

> "We delivered more than what was promised-a 103 percent reduction in time-to- train Can someone explain this math to me?

TheCaptain4815 3y ago

For what Google is primarily used for, search engine and so forth, do they need an extremely advanced Ai? As we've seen with other big tech companies, using giant expensive models for "simple tasks" isn't worth it financially. How advanced of a LLM does Google really need to improve search?

jeffbee 3y ago
Google has numerous internal applications other than websearch that use machine learning, such as spam classification (Gmail), image segmentation and recognition (photos, image search), text-to-speech and speech-to-text (assistant), very small generative tasks (smart reply, smart compose), predicting which videos will be watched (YouTube), and even operating the pumps and fans in their data centers.
kaycebasques 3y ago
Here's a post from Jan 2023 on the official blog: "9 Ways We Use AI In Our Products" https://blog.google/technology/ai/9-ways-we-use-ai-in-our-products/ https://blog.google/technology/ai/9-ways-we-use-ai-in-our-pr...

0-_-0 3y ago

And AMD is nowhere to be found

htrp 3y ago

Actual Results (and not the article talking about it) https://mlcommons.org/benchmarks/inference-datacenter/ https://mlcommons.org/benchmarks/inference-datacenter/