7 ms·

One thing google et all are missing from a typical super computer is infiniband style interconnects. They provide integrations with parallel data libraries like

by bbatha 8y ago

One thing google et all are missing from a typical super computer is infiniband style interconnects. They provide integrations with parallel data libraries like mpi and offer “3d” networking that will take into account physical distance between nodes and can do single rack mesh networking to avoid the overhead of switching. Despite google having lots of compute power they probably can’t leverage it in the way that the LINPACK benchmarks need.

improbable22 8y ago
Thanks, this is interesting. It would be somehow satisfying if their LINPACK benchmarks would actually not be beaten by Google et al. (And their real workloads too.) But how tightly can you really connect 27000 GPUs? Would be curious if anyone has a more technical article handy about what's different.
- jcranmer 8y ago
  The list of top supercomputers isn't a list of which systems have the most ALUs that you can shove floats through (though that is definitely a strong correlate). The difficult part in HPC is actually being able to keep those ALUs fed with floats. In large HPC applications, the communication is the principle bottleneck in being able to scale up [1]. Communication patterns for HPC application also tend to very much have a bursty everybody-is-sending-at-the-same-time pattern, which makes it very easy to saturate a typical star-like Ethernet network configuration (supercomputers typically use a torus or mesh-style interconnect). For GPUs, one trick you can do is to do GPU-to-GPU communication that bypasses the CPU. I don't believe the hardware that extends this to do CPU-less transfer systems across different nodes is common on non-HPC systems. [1] One of the main criticisms of LINPACK as a benchmark is that it is a low-communication benchmark. Essentially, you're doing O(n^3) computation on O(n^2) communication. In many benchmarks, such as grid simulation, the ratio of computation is communication is constant with respect to size.
  - stephencanon 8y ago
    > One of the main criticisms of LINPACK as a benchmark is that it is a low-communication benchmark. Essentially, you're doing O(n^3) computation on O(n^2) communication. In many benchmarks, such as grid simulation, the ratio of computation is communication is constant with respect to size. This is critical and poorly communicated to most people outside the HPC world.
- bbatha 8y ago
  > But how tightly can you really connect 27000 GPUs? Not all that well currently, NVidia and others are working on GPU specific interconnects[0] but they don't have anywhere near the scale of traditional interconnects which have supported hundreds of thousands of nodes by the late 90s. On of the big challenges in modern super computer programming is in fact keeping the GPUs hot, which can often mean offloading work that needs high memory usage to CPUs. Unfortunately my knowledge here is a little dated, I interned at Los Alamos National Lab from 2008 - 2012 when they were doing a lot of rearchitecting of old codes for RoadRunner, the first peta-scale computer. It used Cell chips in accelerator cards and predicated a lot of the challenges in GPU programming, but did not fully elucidate them. For instance we didn't have CUDA! If I had to take my guess the first exa-scale computer is going to be the one that solves the GPU interconnect problem at scale. 0: https://www.nvidia.com/en-us/data-center/nvlink/ https://www.nvidia.com/en-us/data-center/nvlink/
  - shaklee3 8y ago
    Exactly. First there was nvswitch, which dramatically increased the bandwidth over pcie. But that didn't scale to a large number of GPUs. Then there was nvswitch, which solved the scaling problem inside a node. I wouldn't be surprised if the next leap is something like nvlink cables between nodes that don't need traditional routing capabilities.
    - shaklee3 8y ago
      First there was nvlink, rather.
  - greglindahl 8y ago
    Roadrunner was... special... in that regard, requiring even more effort and bizarreness than the typical HPC GPU setup. I remember that the pre-install plan for Roadrunner Linpack was a 50 page document. Also, it's worth noting that GPU HPC computing was already in full swing around the same time: CUDA was first released in June, 2007, which is the same time that Roadrunner released its first Top500 entry.
jedbrown 8y ago
HPL is not especially sensitive to network latency. There are many data centers that could run HPL and get on the list, but don't care to pull that (relatively expensive) stunt. Among scientific applications, a significant fraction really depend on the high-end networks while others would be fine without.
- bbgm 8y ago
  In the past EC2-based clusters have made the Top500. Somewhat unique then because they were the only virtualized systems, but those were just 10 gig ethernet and HPL runs really well on those. In all cases we did it on relatively small number of machines before they had been publicly launched (essentially we used HPL as a stress test). (Work at AWS)
  - greglindahl 8y ago
    Look at the % of peak for ethernet systems vs various Infiniband speeds, or Intel Omni-Path, and you'll see that there's a significant different in the % of peak for Ethernet vs Infiniband vs Omni-Path.
monocasa 8y ago
The interconnects have gotten waaaayyy better in datacenters over the past five years or so when compared to Infiniband. Stuff like FPGAs doing data plane routing, and all of the "converged Ethernet" standards like RoCE have really narrowed the gap between Ethernet and Infiniband.
- gnufx 8y ago
  However much Ethernet has changed, it's not appropriate for typical HPC use (c.f. IB, OPA, Aries, ...). I investigated Cisco's UCS at some length, for instance, which they had persuaded an important person to buy, and that's at least partly the baby of an MPI person. (I recall Infiniband was designed as a "datacentre" thing.)
  - monocasa 8y ago
    Can you expand on why?
InclinedPlane 8y ago
Infiniband used to be great but in an era of 40 or 100 gig ethernet it's not particularly special anymore.
- DannyBee 8y ago
  Right. As https://en.wikipedia.org/wiki/InfiniBand#cite_note-20 https://en.wikipedia.org/wiki/InfiniBand#cite_note-20 says "2009: of the top 500 supercomputers in the world, Gigabit Ethernet is the internal interconnect technology in 259 installations, compared with 181 using InfiniBand.[20]"
  - gnufx 8y ago
    There are (at least) two parameters that are relevant for the fabric, and latency is the critical one for HPC. Have you seen a materials science computation, of the sort that typically takes much of the time on HPC systems, running on 1GbE and SDR IB, even on three nodes? It sometimes happened on our system by mistake in the 2009 era. HPL is simply not a good measure of the utility of a system for HPC, which generally accepted amongst HPC people, who deserve to be listened to. There's a good reason you don't see petascale DFT calculations on one of these "cloud provider"s which comprise so much of the top 500. (That might work on Azure, or something else with an Infiniband-ish fabric, but I've still not seen it.) You can, and to some extent should, model network effects with systems like Simgrid or Dimemas.
  - InclinedPlane 8y ago
    And that was almost a decade ago.
DannyBee 8y ago
This is just wrong. 100 gigabit ethernet switches are commonplace in the datacenters of companies like google, facebook, etc. Facebook even open sourced their second generation 100 gigabit switch design years ago. Combine this with the fact that LINPACK is not a benchmark that involves a lot of communication in the relative scheme of things. If any of them big players cared, they could probably destroy the top LINPACK numbers instantly. There are HPC benchmarks that they would have a harder time with, just not LINPACK.

by bbatha 8y ago

improbable22 8y ago

Thanks, this is interesting. It would be somehow satisfying if their LINPACK benchmarks would actually not be beaten by Google et al. (And their real workloads too.) But how tightly can you really connect 27000 GPUs? Would be curious if anyone has a more technical article handy about what's different.

jcranmer 8y ago
The list of top supercomputers isn't a list of which systems have the most ALUs that you can shove floats through (though that is definitely a strong correlate). The difficult part in HPC is actually being able to keep those ALUs fed with floats. In large HPC applications, the communication is the principle bottleneck in being able to scale up [1]. Communication patterns for HPC application also tend to very much have a bursty everybody-is-sending-at-the-same-time pattern, which makes it very easy to saturate a typical star-like Ethernet network configuration (supercomputers typically use a torus or mesh-style interconnect). For GPUs, one trick you can do is to do GPU-to-GPU communication that bypasses the CPU. I don't believe the hardware that extends this to do CPU-less transfer systems across different nodes is common on non-HPC systems. [1] One of the main criticisms of LINPACK as a benchmark is that it is a low-communication benchmark. Essentially, you're doing O(n^3) computation on O(n^2) communication. In many benchmarks, such as grid simulation, the ratio of computation is communication is constant with respect to size.
- stephencanon 8y ago
  > One of the main criticisms of LINPACK as a benchmark is that it is a low-communication benchmark. Essentially, you're doing O(n^3) computation on O(n^2) communication. In many benchmarks, such as grid simulation, the ratio of computation is communication is constant with respect to size. This is critical and poorly communicated to most people outside the HPC world.
bbatha 8y ago
> But how tightly can you really connect 27000 GPUs? Not all that well currently, NVidia and others are working on GPU specific interconnects[0] but they don't have anywhere near the scale of traditional interconnects which have supported hundreds of thousands of nodes by the late 90s. On of the big challenges in modern super computer programming is in fact keeping the GPUs hot, which can often mean offloading work that needs high memory usage to CPUs. Unfortunately my knowledge here is a little dated, I interned at Los Alamos National Lab from 2008 - 2012 when they were doing a lot of rearchitecting of old codes for RoadRunner, the first peta-scale computer. It used Cell chips in accelerator cards and predicated a lot of the challenges in GPU programming, but did not fully elucidate them. For instance we didn't have CUDA! If I had to take my guess the first exa-scale computer is going to be the one that solves the GPU interconnect problem at scale. 0: https://www.nvidia.com/en-us/data-center/nvlink/ https://www.nvidia.com/en-us/data-center/nvlink/
- shaklee3 8y ago
  Exactly. First there was nvswitch, which dramatically increased the bandwidth over pcie. But that didn't scale to a large number of GPUs. Then there was nvswitch, which solved the scaling problem inside a node. I wouldn't be surprised if the next leap is something like nvlink cables between nodes that don't need traditional routing capabilities.
  - shaklee3 8y ago
    First there was nvlink, rather.
- greglindahl 8y ago
  Roadrunner was... special... in that regard, requiring even more effort and bizarreness than the typical HPC GPU setup. I remember that the pre-install plan for Roadrunner Linpack was a 50 page document. Also, it's worth noting that GPU HPC computing was already in full swing around the same time: CUDA was first released in June, 2007, which is the same time that Roadrunner released its first Top500 entry.

jedbrown 8y ago

HPL is not especially sensitive to network latency. There are many data centers that could run HPL and get on the list, but don't care to pull that (relatively expensive) stunt. Among scientific applications, a significant fraction really depend on the high-end networks while others would be fine without.

bbgm 8y ago
In the past EC2-based clusters have made the Top500. Somewhat unique then because they were the only virtualized systems, but those were just 10 gig ethernet and HPL runs really well on those. In all cases we did it on relatively small number of machines before they had been publicly launched (essentially we used HPL as a stress test). (Work at AWS)
- greglindahl 8y ago
  Look at the % of peak for ethernet systems vs various Infiniband speeds, or Intel Omni-Path, and you'll see that there's a significant different in the % of peak for Ethernet vs Infiniband vs Omni-Path.

monocasa 8y ago

The interconnects have gotten waaaayyy better in datacenters over the past five years or so when compared to Infiniband. Stuff like FPGAs doing data plane routing, and all of the "converged Ethernet" standards like RoCE have really narrowed the gap between Ethernet and Infiniband.

gnufx 8y ago
However much Ethernet has changed, it's not appropriate for typical HPC use (c.f. IB, OPA, Aries, ...). I investigated Cisco's UCS at some length, for instance, which they had persuaded an important person to buy, and that's at least partly the baby of an MPI person. (I recall Infiniband was designed as a "datacentre" thing.)
- monocasa 8y ago
  Can you expand on why?

InclinedPlane 8y ago

Infiniband used to be great but in an era of 40 or 100 gig ethernet it's not particularly special anymore.

DannyBee 8y ago
Right. As https://en.wikipedia.org/wiki/InfiniBand#cite_note-20 https://en.wikipedia.org/wiki/InfiniBand#cite_note-20 says "2009: of the top 500 supercomputers in the world, Gigabit Ethernet is the internal interconnect technology in 259 installations, compared with 181 using InfiniBand.[20]"
- gnufx 8y ago
  There are (at least) two parameters that are relevant for the fabric, and latency is the critical one for HPC. Have you seen a materials science computation, of the sort that typically takes much of the time on HPC systems, running on 1GbE and SDR IB, even on three nodes? It sometimes happened on our system by mistake in the 2009 era. HPL is simply not a good measure of the utility of a system for HPC, which generally accepted amongst HPC people, who deserve to be listened to. There's a good reason you don't see petascale DFT calculations on one of these "cloud provider"s which comprise so much of the top 500. (That might work on Azure, or something else with an Infiniband-ish fabric, but I've still not seen it.) You can, and to some extent should, model network effects with systems like Simgrid or Dimemas.
- InclinedPlane 8y ago
  And that was almost a decade ago.

DannyBee 8y ago

This is just wrong. 100 gigabit ethernet switches are commonplace in the datacenters of companies like google, facebook, etc. Facebook even open sourced their second generation 100 gigabit switch design years ago. Combine this with the fact that LINPACK is not a benchmark that involves a lot of communication in the relative scheme of things. If any of them big players cared, they could probably destroy the top LINPACK numbers instantly. There are HPC benchmarks that they would have a harder time with, just not LINPACK.