9 ms·

Is "supercomputer" still even a thing? It's not as if they're building something with a single, super-powerful core. They're combining lots and lots of CPUs and

Is "supercomputer" still even a thing? It's not as if they're building something with a single, super-powerful core. They're combining lots and lots of CPUs and GPUs. So by that measure, shouldn't AWS and Azure be the world's biggest supercomputers by far?

dragontamer 4y ago
A "supercomputer" will have far stronger communications between the CPUs / GPUs than the typical AWS network. For example, a typical DGX supercomputer system from NVidia is pushing 7.2TBps (https://www.nvidia.com/en-us/data-center/dgx-h100/ https://www.nvidia.com/en-us/data-center/dgx-h100/) GPU-to-GPU communications. In contrast, a typical DDR4 RAM on your typical desktop is 0.05 TBps, so yeah, 7.2TBps external bandwidth between GPUs is quite a lot. ----------- For Frontier, the Slingshot NICs are 100GBps each. So each node can communicate with more bandwidth to each other than your typical Desktop computer has RAM-bandwidth. The diagrams imply that there's 4x Slingshot NICs per node on Frontier, suggesting 400GBps bandwidth to the interconnect. (https://www.olcf.ornl.gov/wp-content/uploads/2020/02/frontier_node_diagram_lr.png https://www.olcf.ornl.gov/wp-content/uploads/2020/02/frontie...)
- ccday 4y ago
  You can rent a supercomputer from AWS: https://aws.amazon.com/ec2/instance-types/p4/ https://aws.amazon.com/ec2/instance-types/p4/
- deleted 4y ago
  [deleted]
- dekhn 4y ago
  I think there's a number of math errors in your comparisons. For example, you're comparing RAM bandwidth to network performance. And those network numbers are summed over a large number of NICs. You mixed up GBps (gigaBYTES per second) with Gbps (gigaBITs per second)- divide your numbers by ten to get a good idea of gigabytes for comparing to RAM. RAM is about 50GBytes/sec, fast individual nics are 400Gbps (or about 40GB/sec). Unless you have special caches or very large ram, you will run out of bits to send very quickly on a network like that. Typically supercomputers don't have RAM or busses that are more than 2X faster than conventional machines because it's not economic.
  - dragontamer 4y ago
    I did not make that error. Read the links: > 7.2 terabytes per second of bidirectional GPU-to-GPU bandwidth, 1.5X more than previous generation And... https://www.olcf.ornl.gov/frontier/ https://www.olcf.ornl.gov/frontier/ > Multiple Slingshot NICs providing 100 GB/s network bandwidth. Slingshot network which provides adaptive routing, congestion management and quality of service. That's not "little-b bits", that's big-B Bytes. > Typically supercomputers don't have RAM or busses that are more than 2X faster than conventional machines because it's not economic. These are top-end GPUs equipped with the latest-and-greatest HBM RAM, with 2TBytes/sec RAM bandwidth or more. They put the super in "supercomputer". When you have 1TByte to 2TBytes of RAM bandwidth, you need 400GByte+ network connections to keep up with them.
    - dekhn 4y ago
      Sure, I worked on TPUs at google, I know you need fast network connections. But "multiple slingshot NICs providing 100GB/sec"- that's ganged NICs, each one only does 200Gbps. It looks like you're conflating what's available to a single GPU, versus cross sectional bandwidth? I had to spend some time looking at the Slingshot docs because they throw around a bunch of numbers aggregated over different dimensions, leaving out the number of items they're aggregating over (common technique to make things look "bigger"). Another big issue that threw me off is that you're comparing the GPU network on the hosts to desktop computers. But in thinking, I think that's fair since the GPUs are the (primary) processing elements.
dekhn 4y ago
No, most of the cloud and internal infra at those orgs isn't supercomputers. Each of the major cloud vendors has supercomputer-like products. Supercomputers today are defined by having networks that allow the processing elements to reach a significant level of utilization even though the problem is partitioned over millions of processing elements. The cluster that MSFT build for OpenAI is definitely a supercomputer.

dragontamer 4y ago

A "supercomputer" will have far stronger communications between the CPUs / GPUs than the typical AWS network. For example, a typical DGX supercomputer system from NVidia is pushing 7.2TBps (https://www.nvidia.com/en-us/data-center/dgx-h100/ https://www.nvidia.com/en-us/data-center/dgx-h100/) GPU-to-GPU communications. In contrast, a typical DDR4 RAM on your typical desktop is 0.05 TBps, so yeah, 7.2TBps external bandwidth between GPUs is quite a lot. ----------- For Frontier, the Slingshot NICs are 100GBps each. So each node can communicate with more bandwidth to each other than your typical Desktop computer has RAM-bandwidth. The diagrams imply that there's 4x Slingshot NICs per node on Frontier, suggesting 400GBps bandwidth to the interconnect. (https://www.olcf.ornl.gov/wp-content/uploads/2020/02/frontier_node_diagram_lr.png https://www.olcf.ornl.gov/wp-content/uploads/2020/02/frontie...)

ccday 4y ago
You can rent a supercomputer from AWS: https://aws.amazon.com/ec2/instance-types/p4/ https://aws.amazon.com/ec2/instance-types/p4/
deleted 4y ago
[deleted]
dekhn 4y ago
I think there's a number of math errors in your comparisons. For example, you're comparing RAM bandwidth to network performance. And those network numbers are summed over a large number of NICs. You mixed up GBps (gigaBYTES per second) with Gbps (gigaBITs per second)- divide your numbers by ten to get a good idea of gigabytes for comparing to RAM. RAM is about 50GBytes/sec, fast individual nics are 400Gbps (or about 40GB/sec). Unless you have special caches or very large ram, you will run out of bits to send very quickly on a network like that. Typically supercomputers don't have RAM or busses that are more than 2X faster than conventional machines because it's not economic.
- dragontamer 4y ago
  I did not make that error. Read the links: > 7.2 terabytes per second of bidirectional GPU-to-GPU bandwidth, 1.5X more than previous generation And... https://www.olcf.ornl.gov/frontier/ https://www.olcf.ornl.gov/frontier/ > Multiple Slingshot NICs providing 100 GB/s network bandwidth. Slingshot network which provides adaptive routing, congestion management and quality of service. That's not "little-b bits", that's big-B Bytes. > Typically supercomputers don't have RAM or busses that are more than 2X faster than conventional machines because it's not economic. These are top-end GPUs equipped with the latest-and-greatest HBM RAM, with 2TBytes/sec RAM bandwidth or more. They put the super in "supercomputer". When you have 1TByte to 2TBytes of RAM bandwidth, you need 400GByte+ network connections to keep up with them.
  - dekhn 4y ago
    Sure, I worked on TPUs at google, I know you need fast network connections. But "multiple slingshot NICs providing 100GB/sec"- that's ganged NICs, each one only does 200Gbps. It looks like you're conflating what's available to a single GPU, versus cross sectional bandwidth? I had to spend some time looking at the Slingshot docs because they throw around a bunch of numbers aggregated over different dimensions, leaving out the number of items they're aggregating over (common technique to make things look "bigger"). Another big issue that threw me off is that you're comparing the GPU network on the hosts to desktop computers. But in thinking, I think that's fair since the GPUs are the (primary) processing elements.

dekhn 4y ago

No, most of the cloud and internal infra at those orgs isn't supercomputers. Each of the major cloud vendors has supercomputer-like products. Supercomputers today are defined by having networks that allow the processing elements to reach a significant level of utilization even though the problem is partitioned over millions of processing elements. The cluster that MSFT build for OpenAI is definitely a supercomputer.