4 ms·

I did not make that error. Read the links: > 7.2 terabytes per second of bidirectional GPU-to-GPU bandwidth, 1.5X more than previous generation And... https:

I did not make that error. Read the links: > 7.2 terabytes per second of bidirectional GPU-to-GPU bandwidth, 1.5X more than previous generation And... https://www.olcf.ornl.gov/frontier/ https://www.olcf.ornl.gov/frontier/ > Multiple Slingshot NICs providing 100 GB/s network bandwidth. Slingshot network which provides adaptive routing, congestion management and quality of service. That's not "little-b bits", that's big-B Bytes. > Typically supercomputers don't have RAM or busses that are more than 2X faster than conventional machines because it's not economic. These are top-end GPUs equipped with the latest-and-greatest HBM RAM, with 2TBytes/sec RAM bandwidth or more. They put the super in "supercomputer". When you have 1TByte to 2TBytes of RAM bandwidth, you need 400GByte+ network connections to keep up with them.

dekhn 4y ago
Sure, I worked on TPUs at google, I know you need fast network connections. But "multiple slingshot NICs providing 100GB/sec"- that's ganged NICs, each one only does 200Gbps. It looks like you're conflating what's available to a single GPU, versus cross sectional bandwidth? I had to spend some time looking at the Slingshot docs because they throw around a bunch of numbers aggregated over different dimensions, leaving out the number of items they're aggregating over (common technique to make things look "bigger"). Another big issue that threw me off is that you're comparing the GPU network on the hosts to desktop computers. But in thinking, I think that's fair since the GPUs are the (primary) processing elements.