10 ms·

I really have to wonder if anyone can compete with this kind of systems integration capability. A core having 900GBps connectivity to the cluster memory at such

I really have to wonder if anyone can compete with this kind of systems integration capability. A core having 900GBps connectivity to the cluster memory at such relative low power is epic beyond words. 800Gbps ethernet across PCIe is uncompetitive in extreme. How the rest of the industry can respond is such a mystery. And will it be lone competitors, or will a new PC era be able to start, with an ecosystem of capabilities?

briffle 3y ago
I'm curious how you can keep that fed with data fast enough. What kind of interfaces to your network do you need to keep it busy and not just waiting on data.
makestuff 3y ago
I have to wonder why these engineers are not paid millions of dollars per year. As a lowly backend dev this seems so much more impressive than my new API that retrieves something from a database...
- akomtu 3y ago
  Engineers are really poor negotiators, probably because they neglect "people skills".
  - ksec 3y ago
    Hardware Engineers are really poor negotiators.
  - flangola7 3y ago
    No need for the quotes, it's exactly the issue
  - rektide 3y ago
    Because their scales of what are important things are so often entirely unintelligible to the rest of the planet.
- jjtheblunt 3y ago
  perhaps they are, even if in stock share price appreciation?
- GuB-42 3y ago
  Because 10 engineers paid hundreds of thousands a year do a better job than 1 engineer paid millions. And that's because it is very high skill work, otherwise, it would have been 100 engineers paid tens of thousands.
virtuallynathan 3y ago
FWIW, I believe it’s 900GB/s full duplex, so 450GB/s each way. Still a ton!
rektide 3y ago
For compare a lane of PCIe 6.0 is 7.56GBps. If all 128 lanes of a AMD cpu were able to be interconnect, that would be 967GBps.
modeless 3y ago
This seems to be competing directly with Google's TPU pods. Looks like TPU v4 has a 300 GB/s interconnect, and 32 GB HBM per chip * 4096 chips = 131 TB (which is all HBM, so higher bandwidth than the LPDDR in Nvidia's system). So yeah, Nvidia's interconnect seems better. However, TPU v4 was deployed in 2020 (!) and Nvidia's thing won't be ready until next year. I've gotta imagine that TPU v5 has already been deployed internally for a while now, but hasn't been disclosed yet. Who knows, TPU v6 might even be deployed before this Nvidia thing.
- tbruckner 3y ago
  Some of the specs seem inaccurate here, HBM has been present in NVIDIA datacenter GPUs for awhile now. LPDDR is for their gaming hardware.
  - wmf 3y ago
    The GH200 uses a combination of HBM3 for the GPU and LPDDR5 for the CPU but it's a unified memory system so the GPU can access all the RAM. Gaming GPUs use GDDR which is a third flavor.
- the_svd_doctor 3y ago
  The network topology of TPUv4 is far far inferior though. It's a torus. No switches.
- renonce 3y ago
  If TPUv4 pods were so powerful, why are most new models trained with NVIDIA cards rather than TPU?
  - llm_nerd 3y ago
    There is a mythology to Google's TPU that is not validated by real world numbers. Where we can actually test (I mean -- TPUv4 pods are available right now on their cloud) it is very good, but remains uncompetitive with the h100. I mean, Google disclaims that you shouldn't compare it, doing the classic "the h100 is on a better process node so it's unfair". People will always point at a mythical next generation that is surely way better, despite the fact that Google is currently building big supercomputers with their TPUv4. And in Google's shootout, again comparing with the last generation of nvidia hardware (the A100), Google's biggest advantage was in the connection fabric, which with this DGX GH200 nvidia not only overcame, but bested by a magnitude. More competition would be fantastic. Better pricing at scale would be fantastic. But there is absolutely no doubt that nvidia is far ahead of Google right now. Tesla made some believably pushing claims about their own efforts with their own hardware, so who knows maybe they're the real challenger.
brucethemoose2 3y ago
Cerebras supposedly can: https://www.servethehome.com/cerebras-wafer-scale-engine-2-wse-2-at-hot-chips-33/hc33-cerebras-wse-2-swarmx-interconnect/ https://www.servethehome.com/cerebras-wafer-scale-engine-2-w... In hindsight the 40GB of SRAM feels kind of quaint, but nevertheless their very fat nodes let them get away with more than Nvidia could with A100s, as you can see in the slides. CS2 is a little old now. I bet an update is just around the corner.
amelius 3y ago
Ultimately how fast their transistors can switch and at what power is determined by TSMC, which everyone else can use too. Same for density of interconnect.
epolanski 3y ago
As always in economics it is about volumes and margins. If the competitors (mainly AMD, Intel and to some extent ARM) will keep seeing growing volumes and insane margins they will be attracted to bring to invest and take part of that market. Till now gaming GPU market did not bring to AMD the necessary margins to really push them to bring a better competition to Nvidia. Even 10/15 years ago when ATI was way ahead of Nvidia technologically for 2/3 years (the HD 4000 and HD 5000 generations vs the Nvidia flops of the 9000, 200 and 400 series) Nvidia was posting billions of profits and ATI posted a whole...19 millions of profits across 3 years. But today's GPU market thanks to it's non-gaming sales is much bigger to ignore (which is why Intel entered it as well) and those players will likely react. You don't need to have the best premier product, you need to have your products good and priced well enough that they will be chosen over the competitor's.
ironbound 3y ago
Sounds like you havent seen Wafer-scale integration computing, Tesla has one and comercial companies like cerebras will sell you a cabnet without the miles of fiber networking. https://www.cerebras.net/andromeda/ https://www.cerebras.net/andromeda/
usernew 3y ago
Is there really a market for a response though? Now, I'll be honest that I know very little about this market. What I do know from doing a decade of presales before covid hit, is that people who buy GPUs go for aggregate max on a big node farm. Now, most of my clients who bought GPU-heavy scale-out nodes were in the financial industry, so maybe deep learning stuff is different. Their workloads were massively parallel, and could scale out instead of needing something singularly fast. So I guess my question is - what use case is there for a huge truck that goes 200mph and take 4 trips, when you could just buy 16 regular trucks, and move your apartment in the same amount of time at half the cost.
- rlupi 3y ago
  There are workloads that are data parallel, and scale like the GPU-heavy scale-out nodes that you describe. The other approach, which you do when models themselves are massive, is model parallelism. You split it into multiple parts that run on different nodes. In both cases, you need to distribute weight updates through the network although the traffic patterns can be different. To maximize the performance in both scenarios, systems designers optimize for all-reduce and bisection bandwidth. There are also other tricks, for example the TPUv4 ICI network is optically switched, and it is configured when a workload starts to maximize bandwidth for the requested topology ("twisting the torus" in the published paper).
- rektide 3y ago
  The exciting thing about CXL is we can start to find out if peripheral or hopefully close-networked computing fabrics can be useful & interesting, beyond the small circumstances Nvidia will offer. Having an ecosystem that everyone can participate in will let us explore. Money can't buy that. Talent can't buy that. You need to socialize to really find out the possible values. 'The street finds it's own uses for things' is the well known Gibson adage, I and typically it's a comment aimed low. But our entire era of amazing computing began with the Gang of Nine enabling lowness in a degree such that it quickly became the highest tech, the best. Sure you can still buy a mainframe & they have amazing feats but it's not where the value is, but and the value is where it is because possibility was unchained, I unleashed from corporate dominion, and spread wide. I think we can find amazing new futures with CXL & mad bandwidth connectivity.
fock 3y ago
Isn't this a lot of things which AMD has already sold as the ORNL-Frontier 2 years ago? The main difference seems to be that external bandwidth here indeed is crazy via NVLink (though it is only 450GB/s per way so the same as 64 PCIe-Gen5...) and they have two networks for communication (although I suppose the HPE Slingshot is as good as the Infiniband in here)...

briffle 3y ago

I'm curious how you can keep that fed with data fast enough. What kind of interfaces to your network do you need to keep it busy and not just waiting on data.

makestuff 3y ago

I have to wonder why these engineers are not paid millions of dollars per year. As a lowly backend dev this seems so much more impressive than my new API that retrieves something from a database...

akomtu 3y ago
Engineers are really poor negotiators, probably because they neglect "people skills".
- ksec 3y ago
  Hardware Engineers are really poor negotiators.
- flangola7 3y ago
  No need for the quotes, it's exactly the issue
- rektide 3y ago
  Because their scales of what are important things are so often entirely unintelligible to the rest of the planet.
jjtheblunt 3y ago
perhaps they are, even if in stock share price appreciation?
GuB-42 3y ago
Because 10 engineers paid hundreds of thousands a year do a better job than 1 engineer paid millions. And that's because it is very high skill work, otherwise, it would have been 100 engineers paid tens of thousands.

virtuallynathan 3y ago

FWIW, I believe it’s 900GB/s full duplex, so 450GB/s each way. Still a ton!

rektide 3y ago

For compare a lane of PCIe 6.0 is 7.56GBps. If all 128 lanes of a AMD cpu were able to be interconnect, that would be 967GBps.

modeless 3y ago

This seems to be competing directly with Google's TPU pods. Looks like TPU v4 has a 300 GB/s interconnect, and 32 GB HBM per chip * 4096 chips = 131 TB (which is all HBM, so higher bandwidth than the LPDDR in Nvidia's system). So yeah, Nvidia's interconnect seems better. However, TPU v4 was deployed in 2020 (!) and Nvidia's thing won't be ready until next year. I've gotta imagine that TPU v5 has already been deployed internally for a while now, but hasn't been disclosed yet. Who knows, TPU v6 might even be deployed before this Nvidia thing.

tbruckner 3y ago
Some of the specs seem inaccurate here, HBM has been present in NVIDIA datacenter GPUs for awhile now. LPDDR is for their gaming hardware.
- wmf 3y ago
  The GH200 uses a combination of HBM3 for the GPU and LPDDR5 for the CPU but it's a unified memory system so the GPU can access all the RAM. Gaming GPUs use GDDR which is a third flavor.
the_svd_doctor 3y ago
The network topology of TPUv4 is far far inferior though. It's a torus. No switches.
renonce 3y ago
If TPUv4 pods were so powerful, why are most new models trained with NVIDIA cards rather than TPU?
- llm_nerd 3y ago
  There is a mythology to Google's TPU that is not validated by real world numbers. Where we can actually test (I mean -- TPUv4 pods are available right now on their cloud) it is very good, but remains uncompetitive with the h100. I mean, Google disclaims that you shouldn't compare it, doing the classic "the h100 is on a better process node so it's unfair". People will always point at a mythical next generation that is surely way better, despite the fact that Google is currently building big supercomputers with their TPUv4. And in Google's shootout, again comparing with the last generation of nvidia hardware (the A100), Google's biggest advantage was in the connection fabric, which with this DGX GH200 nvidia not only overcame, but bested by a magnitude. More competition would be fantastic. Better pricing at scale would be fantastic. But there is absolutely no doubt that nvidia is far ahead of Google right now. Tesla made some believably pushing claims about their own efforts with their own hardware, so who knows maybe they're the real challenger.

brucethemoose2 3y ago

Cerebras supposedly can: https://www.servethehome.com/cerebras-wafer-scale-engine-2-wse-2-at-hot-chips-33/hc33-cerebras-wse-2-swarmx-interconnect/ https://www.servethehome.com/cerebras-wafer-scale-engine-2-w... In hindsight the 40GB of SRAM feels kind of quaint, but nevertheless their very fat nodes let them get away with more than Nvidia could with A100s, as you can see in the slides. CS2 is a little old now. I bet an update is just around the corner.

amelius 3y ago

Ultimately how fast their transistors can switch and at what power is determined by TSMC, which everyone else can use too. Same for density of interconnect.

epolanski 3y ago

As always in economics it is about volumes and margins. If the competitors (mainly AMD, Intel and to some extent ARM) will keep seeing growing volumes and insane margins they will be attracted to bring to invest and take part of that market. Till now gaming GPU market did not bring to AMD the necessary margins to really push them to bring a better competition to Nvidia. Even 10/15 years ago when ATI was way ahead of Nvidia technologically for 2/3 years (the HD 4000 and HD 5000 generations vs the Nvidia flops of the 9000, 200 and 400 series) Nvidia was posting billions of profits and ATI posted a whole...19 millions of profits across 3 years. But today's GPU market thanks to it's non-gaming sales is much bigger to ignore (which is why Intel entered it as well) and those players will likely react. You don't need to have the best premier product, you need to have your products good and priced well enough that they will be chosen over the competitor's.

ironbound 3y ago

Sounds like you havent seen Wafer-scale integration computing, Tesla has one and comercial companies like cerebras will sell you a cabnet without the miles of fiber networking. https://www.cerebras.net/andromeda/ https://www.cerebras.net/andromeda/

usernew 3y ago

Is there really a market for a response though? Now, I'll be honest that I know very little about this market. What I do know from doing a decade of presales before covid hit, is that people who buy GPUs go for aggregate max on a big node farm. Now, most of my clients who bought GPU-heavy scale-out nodes were in the financial industry, so maybe deep learning stuff is different. Their workloads were massively parallel, and could scale out instead of needing something singularly fast. So I guess my question is - what use case is there for a huge truck that goes 200mph and take 4 trips, when you could just buy 16 regular trucks, and move your apartment in the same amount of time at half the cost.

rlupi 3y ago
There are workloads that are data parallel, and scale like the GPU-heavy scale-out nodes that you describe. The other approach, which you do when models themselves are massive, is model parallelism. You split it into multiple parts that run on different nodes. In both cases, you need to distribute weight updates through the network although the traffic patterns can be different. To maximize the performance in both scenarios, systems designers optimize for all-reduce and bisection bandwidth. There are also other tricks, for example the TPUv4 ICI network is optically switched, and it is configured when a workload starts to maximize bandwidth for the requested topology ("twisting the torus" in the published paper).
rektide 3y ago
The exciting thing about CXL is we can start to find out if peripheral or hopefully close-networked computing fabrics can be useful & interesting, beyond the small circumstances Nvidia will offer. Having an ecosystem that everyone can participate in will let us explore. Money can't buy that. Talent can't buy that. You need to socialize to really find out the possible values. 'The street finds it's own uses for things' is the well known Gibson adage, I and typically it's a comment aimed low. But our entire era of amazing computing began with the Gang of Nine enabling lowness in a degree such that it quickly became the highest tech, the best. Sure you can still buy a mainframe & they have amazing feats but it's not where the value is, but and the value is where it is because possibility was unchained, I unleashed from corporate dominion, and spread wide. I think we can find amazing new futures with CXL & mad bandwidth connectivity.

fock 3y ago

Isn't this a lot of things which AMD has already sold as the ORNL-Frontier 2 years ago? The main difference seems to be that external bandwidth here indeed is crazy via NVLink (though it is only 450GB/s per way so the same as 64 PCIe-Gen5...) and they have two networks for communication (although I suppose the HPE Slingshot is as good as the Infiniband in here)...