4 ms·

I would be nice if the article had indicated 1) the incremental improvement over renting ~200K cores and similar memory from AWS, and 2) the cost of this behemo

by JackFr 8y ago

I would be nice if the article had indicated 1) the incremental improvement over renting ~200K cores and similar memory from AWS, and 2) the cost of this behemoth. I assume there is a significant advantage -- it would be nice to know how much and at what cost.

evanb 8y ago
It's not remotely a fair comparison, as typical scientific computing requires high-performance communication to run with any modicum of efficiency. It's not just a matter of scale---the architecture of a machine like this differs dramatically from AWS of a similar size. [Never mind the national security concerns.]
- halbritt 8y ago
  How, specifically does the architecture differ dramatically?
  - scrooched_moose 8y ago
    Generally Infiniband [1] or Omni-Path [2] interconnects between the compute nodes - extremely high throughput and low latency communication on the order of 100 Gb/s and 1 microsecond. These problems aren't trillions of completely independent operations which may work for cloud/distributed architecture. For these types of calculations compute node 6523 needs to know the data coming from compute node 2447 in near-realtime to continue working. https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/Omni-Path https://en.wikipedia.org/wiki/Omni-Path
    - halbritt 8y ago
      Thanks for the detail. I can clearly see the benefit of a couple orders of magnitude less latency.
SEJeff 8y ago
The cost of Amazon for this type of thing is only useful to burst to 200K cores, this is a supercomputer, which will be heavily used all the time. From a pure economics standpoint, the cloud makes zero sense for this sort of thing. Also, the performance of 200K cores on Amazon in VMs compared to 200K physical cores is a lot different. These HPC systems are designed to eek every last 3-5% of performance out of the entire thing, something that you simply can not do even if you try using virtual machines or the cloud.
- scrooched_moose 8y ago
  I maintain a much (much much much much.....much) smaller HPC system at my company and this seems to be a quarterly battle I'm beyond sick of. Every time a new IT manager sees our costs they immediately declare 'I can get you that on the cloud for a fraction of the cost' and starts trying to decommission the system. Never mind the 20-100x increase in solve time and the multi-TB a day data transfer required. Waste 10 hours in meetings, stave it off once again, and gear up for the next round in January...
dekhn 8y ago
amazon does not have a product that competes with this. they don't have a low-latency network. Without that, you're limited to coarse-grained parallelism. Supercomputers like this are built for getting near-peak performance across the entire machine by using a low-latency network to accelerate tightly coupled codes. Modern supercomputers aren't particularly expensive- in the several tens of millions for the capital cost of the machine, several tens of millions for the storage system, several tens of millions for the space, several tens of millions for the power, and a few million for the support staff. In this case, Sierra probably cost $100M for the base machine and storage services.
- halbritt 8y ago
  What is the distinction between low latency networking commonly used in supercomputers today and cloud computing? What is the cutoff? I've commonly seen 500 microsecond latency between certain AWS zones. That's pretty impressive. Inside a given zone, 200-300 microseconds isn't uncommon.
  - wmf 8y ago
    HPC networking is more like 2-3 microseconds. Also it's 200 Gbps instead of 25 Gbps.
  - scrooched_moose 8y ago
    Infiniband is on the order of 1 microsecond between compute nodes and 100 Gb/s throughput. It's often still the bottleneck in HPC systems. https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/InfiniBand
fulafel 8y ago
Many hpc codes work on big data in lockstep an all nodes, so virtualization risks slowdown every tine one nodebis preempted in shared virtualization hosting.

by JackFr 8y ago

evanb 8y ago

It's not remotely a fair comparison, as typical scientific computing requires high-performance communication to run with any modicum of efficiency. It's not just a matter of scale---the architecture of a machine like this differs dramatically from AWS of a similar size. [Never mind the national security concerns.]

halbritt 8y ago
How, specifically does the architecture differ dramatically?
- scrooched_moose 8y ago
  Generally Infiniband [1] or Omni-Path [2] interconnects between the compute nodes - extremely high throughput and low latency communication on the order of 100 Gb/s and 1 microsecond. These problems aren't trillions of completely independent operations which may work for cloud/distributed architecture. For these types of calculations compute node 6523 needs to know the data coming from compute node 2447 in near-realtime to continue working. https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/Omni-Path https://en.wikipedia.org/wiki/Omni-Path
  - halbritt 8y ago
    Thanks for the detail. I can clearly see the benefit of a couple orders of magnitude less latency.

SEJeff 8y ago

The cost of Amazon for this type of thing is only useful to burst to 200K cores, this is a supercomputer, which will be heavily used all the time. From a pure economics standpoint, the cloud makes zero sense for this sort of thing. Also, the performance of 200K cores on Amazon in VMs compared to 200K physical cores is a lot different. These HPC systems are designed to eek every last 3-5% of performance out of the entire thing, something that you simply can not do even if you try using virtual machines or the cloud.

scrooched_moose 8y ago
I maintain a much (much much much much.....much) smaller HPC system at my company and this seems to be a quarterly battle I'm beyond sick of. Every time a new IT manager sees our costs they immediately declare 'I can get you that on the cloud for a fraction of the cost' and starts trying to decommission the system. Never mind the 20-100x increase in solve time and the multi-TB a day data transfer required. Waste 10 hours in meetings, stave it off once again, and gear up for the next round in January...

dekhn 8y ago

amazon does not have a product that competes with this. they don't have a low-latency network. Without that, you're limited to coarse-grained parallelism. Supercomputers like this are built for getting near-peak performance across the entire machine by using a low-latency network to accelerate tightly coupled codes. Modern supercomputers aren't particularly expensive- in the several tens of millions for the capital cost of the machine, several tens of millions for the storage system, several tens of millions for the space, several tens of millions for the power, and a few million for the support staff. In this case, Sierra probably cost $100M for the base machine and storage services.

halbritt 8y ago
What is the distinction between low latency networking commonly used in supercomputers today and cloud computing? What is the cutoff? I've commonly seen 500 microsecond latency between certain AWS zones. That's pretty impressive. Inside a given zone, 200-300 microseconds isn't uncommon.
- wmf 8y ago
  HPC networking is more like 2-3 microseconds. Also it's 200 Gbps instead of 25 Gbps.
- scrooched_moose 8y ago
  Infiniband is on the order of 1 microsecond between compute nodes and 100 Gb/s throughput. It's often still the bottleneck in HPC systems. https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/InfiniBand

fulafel 8y ago

Many hpc codes work on big data in lockstep an all nodes, so virtualization risks slowdown every tine one nodebis preempted in shared virtualization hosting.