4 ms·

It's not a dumb question, that's just a problem that most people don't have. If you have hundreds of 80.000$ nodes with 8 V100 each consuming thousands of Watt

It's not a dumb question, that's just a problem that most people don't have. If you have hundreds of 80.000$ nodes with 8 V100 each consuming thousands of Watts per node, you start thinking about how much money is the 90% idle time of those nodes, waiting for data from the network, is costing you (probably millions of dollars per year). So yeah, a network that's 10x faster is worth every penny if it can turn the system from a 90% idle to <10% idle. That's why people always pay extra for NVLink, HDR Infiniband or HPC Ethernet interconnects, and also why Nvidia bought Mellanox last month... Essentially, when your machine has 1 ExaFLOP/s of compute performance, every second that it remains idle waiting for the network you... well lose 1 ExaFLOP. That's just a lot. Also, imagine millions of cores writing to the same harddrive simultaneously - hell try doing `ls` on a normal linux machine on a folder with 10 million empty files. The "storage" infrastructure to support instantaneous I/O on these systems is quite complex as well. Every second that the system is writing to disk you are wasting another ExaFLOP of compute.

bno1 6y ago
But can't you make the same argument about bandwidth? Every second that the processing units are computing, terabits of bandwidth are unused.
- magicalhippo 6y ago
  Except the goal is usually to compute, not to transfer data, so the interconnects are a necessary evil.
- dooglius 6y ago
  No, computing is happening in parallel with data being sent/received
  - gnufx 6y ago
    What fraction of communication on a typical HPC system is overlapped with computation? My guess is rather little, despite the Mellanox sales pitch. I do wish people commonly made cluster-global profiling-type measurements to be able to answer that sort of question and do optimization.
    - fluffything 6y ago
      Heavily depends on the application. Many applications running on a Top 10 system of the Top 500 are required to show good single threaded performance and good weak and strong scalings up to a big part of the system. Checking all these 3 check boxes is hard to do if your communication is not fully overlapped by computation. The smaller your problem, the better your single threaded performance (e.g. at some point your problem fits in L3), and the larger fraction of execution time that communication dominates. Even the poorest applications on these systems use non-blocking MPI to overlap communication with computation nowadays.
- fluffything 6y ago
  Good question. Of course you can make the same argument about bandwidth, the question is whether you want to make such an argument. If you are building a system whose purpose is to transfer data (e.g. a router), every second that system is "computing" and not "transfering" you are paying for Watts that aren't doing anything useful. A supercomputer purpose is to compute stuff, not to move data around, so making that argument about a supercomputer doesn't make much sense because nobody cares whether the supercomputer is using all its bandwidth or not as long as its using all of its compute capabilities. So that's like making the argument that everything that isn't a plane, well, isn't a plane. Sure, my guitar isn't a plane, but it is not the purpose of my guitar to be one :D
- dekhn 6y ago
  Ultimately what everybody wants is a balanced system (that acts balanced on the workloads of interest). Put another way: you don't want any one part of the system to be massively overprovisioned in a way that causes expensive bottlenecks. The Cray T3E had fairly slow processors, but you coudl keep the entire supercomputer at very high utilization and get better throughput that machines with faster processors and a slower interconnect.
dekhn 6y ago
I was going to say it's hard to make a single server that costs $80K, but I checked with my preferred white box vendor. Two Xeon 8280 CPUs (28 cores each), 10 nVidia Tesla V100 32GB cards 1536GB DDR4-2933 memory, registered ECC, 24 DIMMs 4x 3.8TB SSD, SATA, Intel D3-S4610 Series and a Mellanox ConnectX-5 IB InfiniBand, dual port EDR 100Gb/s is ~$160K. That is indeed some very expensive idle time (although, I think I'd work as hard as possible to get my problem to fit on individual machines with data on external fast storage).
- fluffything 6y ago
  Yeah. FWIW I think it would be rare for somebody to build that kind of node. Applications that fully utilize two CPU sockets while simultaneously maxing 10x V100s are super rare. Usually for GPGPU applications the CPU is mostly idle, and only "orchestrates" the GPU usage. If one needs to support both GPGPU-only and CPU-only applications in a cluster, one would usually just build two separate partitions. One with e.g. 4-8 V100s per node, but way weaker and cheaper CPUs, and the other one with 2 beefy CPU sockets but no GPUs. Also, if you are going for that kind of interconnects, you are probably building a system with >100 nodes. For your price, that's 160 million dollars, which is on the ballpark of what these systems costs. You usually get a cheaper price per node at that scale (although not much cheaper), but there are many costs (installation, maintenance, training, etc.) that are not covered there. In particular, those nodes will be running quite hot (just add the watts that those V100s and the CPUs use!), so the cooling infrastructure required to held those in a relatively small space can also end up costing quite a bit. You don't want a water pipe breaking and destroying a rack containing 10 of these nodes costing 1.6 million dollars, right?
  - dekhn 6y ago
    On many-GPU machines, the CPU isn't idle- it's handling the data infeed and data outfeed pipelines. This can often be a bottleneck if training is really fast and you need to stream lots of examples/sec. As for your comments on water pipes breaking: I work on Google's TPUv3, which uses water cooling to the TPU chips, see https://www.nextplatform.com/2018/05/10/tearing-apart-googles-tpu-3-0-ai-coprocessor/ https://www.nextplatform.com/2018/05/10/tearing-apart-google... for some speculation on the details.

bno1 6y ago

But can't you make the same argument about bandwidth? Every second that the processing units are computing, terabits of bandwidth are unused.

magicalhippo 6y ago
Except the goal is usually to compute, not to transfer data, so the interconnects are a necessary evil.
dooglius 6y ago
No, computing is happening in parallel with data being sent/received
- gnufx 6y ago
  What fraction of communication on a typical HPC system is overlapped with computation? My guess is rather little, despite the Mellanox sales pitch. I do wish people commonly made cluster-global profiling-type measurements to be able to answer that sort of question and do optimization.
  - fluffything 6y ago
    Heavily depends on the application. Many applications running on a Top 10 system of the Top 500 are required to show good single threaded performance and good weak and strong scalings up to a big part of the system. Checking all these 3 check boxes is hard to do if your communication is not fully overlapped by computation. The smaller your problem, the better your single threaded performance (e.g. at some point your problem fits in L3), and the larger fraction of execution time that communication dominates. Even the poorest applications on these systems use non-blocking MPI to overlap communication with computation nowadays.
fluffything 6y ago
Good question. Of course you can make the same argument about bandwidth, the question is whether you want to make such an argument. If you are building a system whose purpose is to transfer data (e.g. a router), every second that system is "computing" and not "transfering" you are paying for Watts that aren't doing anything useful. A supercomputer purpose is to compute stuff, not to move data around, so making that argument about a supercomputer doesn't make much sense because nobody cares whether the supercomputer is using all its bandwidth or not as long as its using all of its compute capabilities. So that's like making the argument that everything that isn't a plane, well, isn't a plane. Sure, my guitar isn't a plane, but it is not the purpose of my guitar to be one :D
dekhn 6y ago
Ultimately what everybody wants is a balanced system (that acts balanced on the workloads of interest). Put another way: you don't want any one part of the system to be massively overprovisioned in a way that causes expensive bottlenecks. The Cray T3E had fairly slow processors, but you coudl keep the entire supercomputer at very high utilization and get better throughput that machines with faster processors and a slower interconnect.

dekhn 6y ago

I was going to say it's hard to make a single server that costs $80K, but I checked with my preferred white box vendor. Two Xeon 8280 CPUs (28 cores each), 10 nVidia Tesla V100 32GB cards 1536GB DDR4-2933 memory, registered ECC, 24 DIMMs 4x 3.8TB SSD, SATA, Intel D3-S4610 Series and a Mellanox ConnectX-5 IB InfiniBand, dual port EDR 100Gb/s is ~$160K. That is indeed some very expensive idle time (although, I think I'd work as hard as possible to get my problem to fit on individual machines with data on external fast storage).

fluffything 6y ago
Yeah. FWIW I think it would be rare for somebody to build that kind of node. Applications that fully utilize two CPU sockets while simultaneously maxing 10x V100s are super rare. Usually for GPGPU applications the CPU is mostly idle, and only "orchestrates" the GPU usage. If one needs to support both GPGPU-only and CPU-only applications in a cluster, one would usually just build two separate partitions. One with e.g. 4-8 V100s per node, but way weaker and cheaper CPUs, and the other one with 2 beefy CPU sockets but no GPUs. Also, if you are going for that kind of interconnects, you are probably building a system with >100 nodes. For your price, that's 160 million dollars, which is on the ballpark of what these systems costs. You usually get a cheaper price per node at that scale (although not much cheaper), but there are many costs (installation, maintenance, training, etc.) that are not covered there. In particular, those nodes will be running quite hot (just add the watts that those V100s and the CPUs use!), so the cooling infrastructure required to held those in a relatively small space can also end up costing quite a bit. You don't want a water pipe breaking and destroying a rack containing 10 of these nodes costing 1.6 million dollars, right?
- dekhn 6y ago
  On many-GPU machines, the CPU isn't idle- it's handling the data infeed and data outfeed pipelines. This can often be a bottleneck if training is really fast and you need to stream lots of examples/sec. As for your comments on water pipes breaking: I work on Google's TPUv3, which uses water cooling to the TPU chips, see https://www.nextplatform.com/2018/05/10/tearing-apart-googles-tpu-3-0-ai-coprocessor/ https://www.nextplatform.com/2018/05/10/tearing-apart-google... for some speculation on the details.