6 ms·

The article is a bit light on technical details. I found some specs here: https://hpc.llnl.gov/hardware/platforms/sierra https://hpc.llnl.gov/hardware/platform

by rdtsc 8y ago

The article is a bit light on technical details. I found some specs here: https://hpc.llnl.gov/hardware/platforms/sierra https://hpc.llnl.gov/hardware/platforms/sierra Summary: * 190,080 IBM Power9 CPU cores * 17,280 NVIDIA V100 (Volta) GPUs * 125,626 peak TFLOPS (CPUs+GPUs)

swarnie_ 8y ago
Can you see a price quoted anywhere? The original site is a dumpster fire to read.
- pp19dd 8y ago
  $162 million. Though it's a bit fuzzy. This article is about "Sierra." Department of Energy awarded $325 million to build two supercomputers, "Summit" and "Sierra" both at 150 petaflops each. IBM took $325M, so, it's $162.5M until someone corrects me. However, the actual proposal it's under (CORAL) included more than that, approaching almost $2 billion in the RFP. They also added $100M for R&D. All these machine systems are considered NRE projects (non-recurring engineering) and contain more than just the hardware and each is estimated to be budgeted at about $400M per system (operational expenses, etc).
JackFr 8y ago
I would be nice if the article had indicated 1) the incremental improvement over renting ~200K cores and similar memory from AWS, and 2) the cost of this behemoth. I assume there is a significant advantage -- it would be nice to know how much and at what cost.
- evanb 8y ago
  It's not remotely a fair comparison, as typical scientific computing requires high-performance communication to run with any modicum of efficiency. It's not just a matter of scale---the architecture of a machine like this differs dramatically from AWS of a similar size. [Never mind the national security concerns.]
  - halbritt 8y ago
    How, specifically does the architecture differ dramatically?
    - scrooched_moose 8y ago
      Generally Infiniband [1] or Omni-Path [2] interconnects between the compute nodes - extremely high throughput and low latency communication on the order of 100 Gb/s and 1 microsecond. These problems aren't trillions of completely independent operations which may work for cloud/distributed architecture. For these types of calculations compute node 6523 needs to know the data coming from compute node 2447 in near-realtime to continue working. https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/Omni-Path https://en.wikipedia.org/wiki/Omni-Path
      - halbritt 8y ago
        Thanks for the detail. I can clearly see the benefit of a couple orders of magnitude less latency.
- SEJeff 8y ago
  The cost of Amazon for this type of thing is only useful to burst to 200K cores, this is a supercomputer, which will be heavily used all the time. From a pure economics standpoint, the cloud makes zero sense for this sort of thing. Also, the performance of 200K cores on Amazon in VMs compared to 200K physical cores is a lot different. These HPC systems are designed to eek every last 3-5% of performance out of the entire thing, something that you simply can not do even if you try using virtual machines or the cloud.
  - scrooched_moose 8y ago
    I maintain a much (much much much much.....much) smaller HPC system at my company and this seems to be a quarterly battle I'm beyond sick of. Every time a new IT manager sees our costs they immediately declare 'I can get you that on the cloud for a fraction of the cost' and starts trying to decommission the system. Never mind the 20-100x increase in solve time and the multi-TB a day data transfer required. Waste 10 hours in meetings, stave it off once again, and gear up for the next round in January...
- dekhn 8y ago
  amazon does not have a product that competes with this. they don't have a low-latency network. Without that, you're limited to coarse-grained parallelism. Supercomputers like this are built for getting near-peak performance across the entire machine by using a low-latency network to accelerate tightly coupled codes. Modern supercomputers aren't particularly expensive- in the several tens of millions for the capital cost of the machine, several tens of millions for the storage system, several tens of millions for the space, several tens of millions for the power, and a few million for the support staff. In this case, Sierra probably cost $100M for the base machine and storage services.
  - halbritt 8y ago
    What is the distinction between low latency networking commonly used in supercomputers today and cloud computing? What is the cutoff? I've commonly seen 500 microsecond latency between certain AWS zones. That's pretty impressive. Inside a given zone, 200-300 microseconds isn't uncommon.
    - wmf 8y ago
      HPC networking is more like 2-3 microseconds. Also it's 200 Gbps instead of 25 Gbps.
    - scrooched_moose 8y ago
      Infiniband is on the order of 1 microsecond between compute nodes and 100 Gb/s throughput. It's often still the bottleneck in HPC systems. https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/InfiniBand
- fulafel 8y ago
  Many hpc codes work on big data in lockstep an all nodes, so virtualization risks slowdown every tine one nodebis preempted in shared virtualization hosting.
theandrewbailey 8y ago
Firefox Dev Edition doesn't like the site: Certificates issued by GeoTrust, RapidSSL, Symantec, Thawte, and VeriSign are no longer considered safe because these certificate authorities failed to follow security practices in the past. Error code: MOZILLA_PKIX_ERROR_ADDITIONAL_POLICY_CONSTRAINT_FAILED

by rdtsc 8y ago

swarnie_ 8y ago

Can you see a price quoted anywhere? The original site is a dumpster fire to read.

pp19dd 8y ago
$162 million. Though it's a bit fuzzy. This article is about "Sierra." Department of Energy awarded $325 million to build two supercomputers, "Summit" and "Sierra" both at 150 petaflops each. IBM took $325M, so, it's $162.5M until someone corrects me. However, the actual proposal it's under (CORAL) included more than that, approaching almost $2 billion in the RFP. They also added $100M for R&D. All these machine systems are considered NRE projects (non-recurring engineering) and contain more than just the hardware and each is estimated to be budgeted at about $400M per system (operational expenses, etc).

JackFr 8y ago

I would be nice if the article had indicated 1) the incremental improvement over renting ~200K cores and similar memory from AWS, and 2) the cost of this behemoth. I assume there is a significant advantage -- it would be nice to know how much and at what cost.

evanb 8y ago
It's not remotely a fair comparison, as typical scientific computing requires high-performance communication to run with any modicum of efficiency. It's not just a matter of scale---the architecture of a machine like this differs dramatically from AWS of a similar size. [Never mind the national security concerns.]
- halbritt 8y ago
  How, specifically does the architecture differ dramatically?
  - scrooched_moose 8y ago
    Generally Infiniband [1] or Omni-Path [2] interconnects between the compute nodes - extremely high throughput and low latency communication on the order of 100 Gb/s and 1 microsecond. These problems aren't trillions of completely independent operations which may work for cloud/distributed architecture. For these types of calculations compute node 6523 needs to know the data coming from compute node 2447 in near-realtime to continue working. https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/Omni-Path https://en.wikipedia.org/wiki/Omni-Path
    - halbritt 8y ago
      Thanks for the detail. I can clearly see the benefit of a couple orders of magnitude less latency.
SEJeff 8y ago
The cost of Amazon for this type of thing is only useful to burst to 200K cores, this is a supercomputer, which will be heavily used all the time. From a pure economics standpoint, the cloud makes zero sense for this sort of thing. Also, the performance of 200K cores on Amazon in VMs compared to 200K physical cores is a lot different. These HPC systems are designed to eek every last 3-5% of performance out of the entire thing, something that you simply can not do even if you try using virtual machines or the cloud.
- scrooched_moose 8y ago
  I maintain a much (much much much much.....much) smaller HPC system at my company and this seems to be a quarterly battle I'm beyond sick of. Every time a new IT manager sees our costs they immediately declare 'I can get you that on the cloud for a fraction of the cost' and starts trying to decommission the system. Never mind the 20-100x increase in solve time and the multi-TB a day data transfer required. Waste 10 hours in meetings, stave it off once again, and gear up for the next round in January...
dekhn 8y ago
amazon does not have a product that competes with this. they don't have a low-latency network. Without that, you're limited to coarse-grained parallelism. Supercomputers like this are built for getting near-peak performance across the entire machine by using a low-latency network to accelerate tightly coupled codes. Modern supercomputers aren't particularly expensive- in the several tens of millions for the capital cost of the machine, several tens of millions for the storage system, several tens of millions for the space, several tens of millions for the power, and a few million for the support staff. In this case, Sierra probably cost $100M for the base machine and storage services.
- halbritt 8y ago
  What is the distinction between low latency networking commonly used in supercomputers today and cloud computing? What is the cutoff? I've commonly seen 500 microsecond latency between certain AWS zones. That's pretty impressive. Inside a given zone, 200-300 microseconds isn't uncommon.
  - wmf 8y ago
    HPC networking is more like 2-3 microseconds. Also it's 200 Gbps instead of 25 Gbps.
  - scrooched_moose 8y ago
    Infiniband is on the order of 1 microsecond between compute nodes and 100 Gb/s throughput. It's often still the bottleneck in HPC systems. https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/InfiniBand
fulafel 8y ago
Many hpc codes work on big data in lockstep an all nodes, so virtualization risks slowdown every tine one nodebis preempted in shared virtualization hosting.

theandrewbailey 8y ago

Firefox Dev Edition doesn't like the site: Certificates issued by GeoTrust, RapidSSL, Symantec, Thawte, and VeriSign are no longer considered safe because these certificate authorities failed to follow security practices in the past. Error code: MOZILLA_PKIX_ERROR_ADDITIONAL_POLICY_CONSTRAINT_FAILED