5 ms·

Is there anyone here who works on weather models who would be able to explain why custom supercomputers are still needed vs. large commodity clusters? Thanks!

by matt2000 11y ago

GFK_of_xmaspast 11y ago
I don't do weather, but probably it has a lot to do with inter-node bandwidth.
mispeled 11y ago
Just a guess... high fidelity weather modelling algorithms are highly constrained by node-to-node communication latency. So their supercomputer probably has some networked-direct-memory-access or another non-standard setup tuned toward passing data between adjacent nodes quickly.
stephengillie 11y ago
The branding is still very valuable. This article makes them sound like common blades, ready for virtualization or containerization: http://m.theregister.co.uk/2014/09/30/cray_adds_haswell_and_burst_buffer_tech_to_supercomputer_line http://m.theregister.co.uk/2014/09/30/cray_adds_haswell_and_...
paulmd 11y ago
Modern supercomputers essentially are large commodity clusters. Nobody builds a single fast machine ala Cray-1 anymore. It's just not scaleable past a certain point and we're long since past that. However, they still like to give them fancy names as marketing, because they cost tens of millions of dollars. The main differentiation vs a standard datacenter is low-latency high-throughput interconnects, often with a specific or reconfigurable topology. Something like InfiniBand, where you can ensure very consistent, very favorable bounds on your latency and throughput. So think of it as being a "custom installation" or "custom cluster" instead. Nowadays there really are only a few real choices left in high-performance computing hardware: CPU vs GPU, processor architecture (eg POWER vs x86), and interconnection. Everything else is more or less irrelevant. I have to admit though, I've always wanted to see what you'd get if you built a Cray-2 with GPU chips (eg Pascal) or Xeon Phi as processing modules. The sheer density of chips Cray managed to cram in that case is impressive.
- thrownaway2424 11y ago
  What kind of latency do they expect from these things? Less than 50 microseconds for a small RPC or small RDMA access? I'm wondering how far these things really are from commercial cloud clusters.
  - semi-extrinsic 11y ago
    For a relatively modern cluster the standard is FDR Infiniband, which gives you 56 Gbit/s throughput. The latency for a small (< 32 kbytes) packet is less than 10 microseconds, down towards 1 microsecond for a 8 byte packet. Glenn Lockwood has a nice blogpost with much more detail: http://glennklockwood.blogspot.com/2013/05/fdr-infiniband-vs-dual-rail-qdr.html http://glennklockwood.blogspot.com/2013/05/fdr-infiniband-vs...
  - bashinator 11y ago
    Less than one microsecond. https://en.wikipedia.org/wiki/InfiniBand#Performance https://en.wikipedia.org/wiki/InfiniBand#Performance
  - mturmon 11y ago
    For more on Cray's Aries interconnect used on the Cray XC-30 referred to in the OP, see this paper from NERSC, which has one: https://www.nersc.gov/assets/pubs_presos/NUG2014Aries.pdf https://www.nersc.gov/assets/pubs_presos/NUG2014Aries.pdf Depending on topology, you can get ~1 µs latency with Aries. It's based on PCIe-3, so it's different from InfiniBand.
- superpc 11y ago
  Does this count, just curious. It is a large single machine not cluster. https://www.sgi.com/pdfs/4555.pdf https://www.sgi.com/pdfs/4555.pdf
  - sithadmin 11y ago
    A cursory glance at that PDF suggests the exact opposite of your claim.
mturmon 11y ago
I think the comments nearby about high communications bandwidth required by dense systems of equations are true -- basically they distinguish a tuned, integrated system sold by a vendor like Cray from a row of racks with commodity systems. The point I wanted to add is that ECMWF, generally acknowledged to be the leader in NWP, also uses a pair of large Cray supercomputers (http://www.ecmwf.int/en/computing/our-facilities/supercomputer http://www.ecmwf.int/en/computing/our-facilities/supercomput...). Note that the Cray XC30 uses Xeon processors and a custom interconnect. The old ECMWF system used IBM Power chips.
- mrow84 11y ago
  Yes, it is the interconnects that are important. The UK Met Office has also recently upgraded to a Cray XC40 [0], and there are other XC30 systems in the UK NSS [1] and the US NERSC [2], both of which are used for weather/climate models, amongst other things. [0] http://www.metoffice.gov.uk/news/releases/archive/2014/new-hpc http://www.metoffice.gov.uk/news/releases/archive/2014/new-h... [1] http://www.archer.ac.uk/about-archer/ http://www.archer.ac.uk/about-archer/ [2] http://www.nersc.gov/systems/edison-cray-xc30/ http://www.nersc.gov/systems/edison-cray-xc30/
al_biglan 11y ago
I work for the company that provided the storage for this. Our product is not a collection of commodity components but instead something very much more like an "appliance" that provides the performance at scale that NOAA needs. In general, one of the trends we see/fight is the "Hey, we can get components ourselves and use Lustre to build what we need" And.... they are spot on right. What you need to consider doing this is the amount of maintaining and administering that comes with this solution. The trade-off we see is that some groups, "Academia" for example, will prefer the cheaper solution because they have Grad Students, Post Docs and Undergrad research folks to help make the people cost free. Others, who just want to buy something that works in their infrastructure without adding the people costs would prefer to buy something ready made that "just works" Bottom line: If you are using your system to get work done, the trend is to spend the money for reliability and availability. If you want to save money, it comes at a price (but is certainly do-able) Related: Look at Apple, Google, Facebook, Amazon, Microsoft... they are large enough to build their custom systems themselves. Not everyone operates at this scale. Individuals can buy a cheap PC and build a NAS... or they can buy one from Netgear, Buffalo, QNAP that "jsut works" at a price premium.
khm 11y ago
There is no such thing as a 'large commodity cluster.' You can order a pile of commodity parts and spend the next five years fiddling with it as you watch your metrics fall into the toilet, or you can contract with a systems integrator with domain experience to make sure that you can actually accomplish the work you set out to do. When you're moving an established set of software from a known environment into a new cluster, it helps to have a wide array of technical experts and SLAs in place to ensure that the environment -- here meaning the physical compute infrastructure, storage service, and software installation -- is reliable and maintainable, so that your team can focus on their core mission -- in this case, the modeling software. "Chaos monkey" works fine when you're dealing with random cloudstuff and you are in a position to duck-punch your production code. When you have a large dataset that you need to process reliably and predictably, the mantra shifts to "fail never", and that requires engineering up front in combination with carefully-planned maintenance. Full disclosure: I've worked on this specific supercomputer. While I'm not on the admin team in question, I do work with them occasionally and I have a lot of respect for them -- their job is not an easy one.

Is there anyone here who works on weather models who would be able to explain why custom supercomputers are still needed vs. large commodity clusters? Thanks!

GFK_of_xmaspast 11y ago

I don't do weather, but probably it has a lot to do with inter-node bandwidth.

mispeled 11y ago

Just a guess... high fidelity weather modelling algorithms are highly constrained by node-to-node communication latency. So their supercomputer probably has some networked-direct-memory-access or another non-standard setup tuned toward passing data between adjacent nodes quickly.

stephengillie 11y ago

The branding is still very valuable. This article makes them sound like common blades, ready for virtualization or containerization: http://m.theregister.co.uk/2014/09/30/cray_adds_haswell_and_burst_buffer_tech_to_supercomputer_line http://m.theregister.co.uk/2014/09/30/cray_adds_haswell_and_...

paulmd 11y ago

Modern supercomputers essentially are large commodity clusters. Nobody builds a single fast machine ala Cray-1 anymore. It's just not scaleable past a certain point and we're long since past that. However, they still like to give them fancy names as marketing, because they cost tens of millions of dollars. The main differentiation vs a standard datacenter is low-latency high-throughput interconnects, often with a specific or reconfigurable topology. Something like InfiniBand, where you can ensure very consistent, very favorable bounds on your latency and throughput. So think of it as being a "custom installation" or "custom cluster" instead. Nowadays there really are only a few real choices left in high-performance computing hardware: CPU vs GPU, processor architecture (eg POWER vs x86), and interconnection. Everything else is more or less irrelevant. I have to admit though, I've always wanted to see what you'd get if you built a Cray-2 with GPU chips (eg Pascal) or Xeon Phi as processing modules. The sheer density of chips Cray managed to cram in that case is impressive.

thrownaway2424 11y ago
What kind of latency do they expect from these things? Less than 50 microseconds for a small RPC or small RDMA access? I'm wondering how far these things really are from commercial cloud clusters.
- semi-extrinsic 11y ago
  For a relatively modern cluster the standard is FDR Infiniband, which gives you 56 Gbit/s throughput. The latency for a small (< 32 kbytes) packet is less than 10 microseconds, down towards 1 microsecond for a 8 byte packet. Glenn Lockwood has a nice blogpost with much more detail: http://glennklockwood.blogspot.com/2013/05/fdr-infiniband-vs-dual-rail-qdr.html http://glennklockwood.blogspot.com/2013/05/fdr-infiniband-vs...
- bashinator 11y ago
  Less than one microsecond. https://en.wikipedia.org/wiki/InfiniBand#Performance https://en.wikipedia.org/wiki/InfiniBand#Performance
- mturmon 11y ago
  For more on Cray's Aries interconnect used on the Cray XC-30 referred to in the OP, see this paper from NERSC, which has one: https://www.nersc.gov/assets/pubs_presos/NUG2014Aries.pdf https://www.nersc.gov/assets/pubs_presos/NUG2014Aries.pdf Depending on topology, you can get ~1 µs latency with Aries. It's based on PCIe-3, so it's different from InfiniBand.
superpc 11y ago
Does this count, just curious. It is a large single machine not cluster. https://www.sgi.com/pdfs/4555.pdf https://www.sgi.com/pdfs/4555.pdf
- sithadmin 11y ago
  A cursory glance at that PDF suggests the exact opposite of your claim.

mturmon 11y ago

I think the comments nearby about high communications bandwidth required by dense systems of equations are true -- basically they distinguish a tuned, integrated system sold by a vendor like Cray from a row of racks with commodity systems. The point I wanted to add is that ECMWF, generally acknowledged to be the leader in NWP, also uses a pair of large Cray supercomputers (http://www.ecmwf.int/en/computing/our-facilities/supercomputer http://www.ecmwf.int/en/computing/our-facilities/supercomput...). Note that the Cray XC30 uses Xeon processors and a custom interconnect. The old ECMWF system used IBM Power chips.

mrow84 11y ago
Yes, it is the interconnects that are important. The UK Met Office has also recently upgraded to a Cray XC40 [0], and there are other XC30 systems in the UK NSS [1] and the US NERSC [2], both of which are used for weather/climate models, amongst other things. [0] http://www.metoffice.gov.uk/news/releases/archive/2014/new-hpc http://www.metoffice.gov.uk/news/releases/archive/2014/new-h... [1] http://www.archer.ac.uk/about-archer/ http://www.archer.ac.uk/about-archer/ [2] http://www.nersc.gov/systems/edison-cray-xc30/ http://www.nersc.gov/systems/edison-cray-xc30/

al_biglan 11y ago

I work for the company that provided the storage for this. Our product is not a collection of commodity components but instead something very much more like an "appliance" that provides the performance at scale that NOAA needs. In general, one of the trends we see/fight is the "Hey, we can get components ourselves and use Lustre to build what we need" And.... they are spot on right. What you need to consider doing this is the amount of maintaining and administering that comes with this solution. The trade-off we see is that some groups, "Academia" for example, will prefer the cheaper solution because they have Grad Students, Post Docs and Undergrad research folks to help make the people cost free. Others, who just want to buy something that works in their infrastructure without adding the people costs would prefer to buy something ready made that "just works" Bottom line: If you are using your system to get work done, the trend is to spend the money for reliability and availability. If you want to save money, it comes at a price (but is certainly do-able) Related: Look at Apple, Google, Facebook, Amazon, Microsoft... they are large enough to build their custom systems themselves. Not everyone operates at this scale. Individuals can buy a cheap PC and build a NAS... or they can buy one from Netgear, Buffalo, QNAP that "jsut works" at a price premium.

khm 11y ago

There is no such thing as a 'large commodity cluster.' You can order a pile of commodity parts and spend the next five years fiddling with it as you watch your metrics fall into the toilet, or you can contract with a systems integrator with domain experience to make sure that you can actually accomplish the work you set out to do. When you're moving an established set of software from a known environment into a new cluster, it helps to have a wide array of technical experts and SLAs in place to ensure that the environment -- here meaning the physical compute infrastructure, storage service, and software installation -- is reliable and maintainable, so that your team can focus on their core mission -- in this case, the modeling software. "Chaos monkey" works fine when you're dealing with random cloudstuff and you are in a position to duck-punch your production code. When you have a large dataset that you need to process reliably and predictably, the mantra shifts to "fail never", and that requires engineering up front in combination with carefully-planned maintenance. Full disclosure: I've worked on this specific supercomputer. While I'm not on the admin team in question, I do work with them occasionally and I have a lot of respect for them -- their job is not an easy one.