6 ms·

The Frontier computer, which broke the exascale barrier in 2022

green-salt 4y ago
I worked with a Cray back in 2012 and the amount of engineering that goes into each rack and down to each blade is incredible. The biggest thing that got me was the geometry of the heatsinks that were designed to slightly compress air as it left each blade into the one above it.
LadyCailin 4y ago
What kinds of problems is this used to solve? Is it a different class of problems that couldn’t be solved with an equivalent number of commodity servers?
- foota 4y ago
  Generally, supercomputers are used in problems where you need fast access to all or lots of a large dataset. I think a common example is weather simulation, where each grid cell needs to update based on its neighbors at each timestep.
- yjftsjthsd-h 4y ago
  I expect you could solve the same problems on commodity hardware, but it'd be slower and/or more expensive.
  - NegativeLatency 4y ago
    From what I've heard it's the high speed/throughput low latency interconnect that's the differentiator compared to thousands of commodity machines. (Weather and bomb explosions being two big ones that are hard to do concurrently, as the results depend on each other)
    - jgtrosh 4y ago
      Depends: if your application has strong scaling then just throw more power whichever way at it. Tightly coupled hpc systems aim to improve execution speed (or application resolution) to weak scaling applications.
- green-salt 4y ago
  The primary ones I worked with were for working on atmospheric science: https://www2.cisl.ucar.edu/about https://www2.cisl.ucar.edu/about https://www2.cisl.ucar.edu/ncar-supercomputing-history/bluefire https://www2.cisl.ucar.edu/ncar-supercomputing-history/bluef...
- reportingsjr 4y ago
  Most of the top supercomputers are used to model nuclear weapons, weather and climate, and various sciencey things. I don't know if the top ones are still used for oil and gas exploration (crunching data to provide higher resolution and higher accuracy oil field maps), but they have been in the past.
- areoform 4y ago
  Politely, this computer exists because the DoE performs stockpile stewardship. Less politely, it exists because of The Bombs. Everything else it will do is a hobby. It is specifically designed for whatever compute needs their classified program has. https://en.wikipedia.org/wiki/Stockpile_stewardship https://en.wikipedia.org/wiki/Stockpile_stewardship https://www.energy.gov/nnsa/articles/stockpile-stewardship-and-management-plan-ssmp https://www.energy.gov/nnsa/articles/stockpile-stewardship-a... All of this capital is being spent because the test ban treaty prohibits the detonation of these weapons, but the US stockpile and its efficacy must be maintained and new weapons must be designed. So how do you know if an old weapon or a new design will work or not? Some part of the answer is real data + simulations. Lots and lots of simulations. It's why Frontier exists and why the DoE has pursued exascale compute for quite some time. (as for the real data part, it's why places like the National Ignition Factory exist, and why there's an active experimental program that exists to study dummy warheads in unexpected ways, https://www.youtube.com/watch?v=FYdAT0v4DHs https://www.youtube.com/watch?v=FYdAT0v4DHs ) ——— Addendum, re: comments arguing that the NNSA computers are separate etc., the points raised are both true in the specific sense, but not true holistically. Quotes from the most recent report to congress, published in March 2022 about the Stockpile Stewardship and Management Plan, Near-Term and Out-Year Mission Goals: ◼ Advance the innovative experimental platforms, diagnostic equipment, and computational capabilities necessary to ensure stockpile safety, security, reliability, and effectiveness: – Achieve exascale computing by delivering an exascale-capable machine and modernizing the nuclear weapons code base – Develop an operational enhanced capability (advanced radiography and reactivity measurements) for subcritical experiments – Quantify the effects of plutonium aging on weapon performance over time – Assure an enduring, trusted supply of strategic radiation-hardened microsystems and, The weapons comprising the U.S. nuclear stockpile are assessed to be safe, reliable, effective, and secure. DOE/NNSA’s scientific infrastructure is currently adequate to support stockpile actions. The DOE/NNSA plans to address near-term gaps in required scientific capabilities by deploying the DOE/NNSA’s first exascale computing platform in fiscal year (FY)2023 and by improving capabilities to conduct subcritical experiments through the Enhanced Capabilities for Subcritical Experiments project by FY 2026. It seems they're building three of these computers, Frontier is one of them, from the budget requests and a press release, The budget request for Advanced Simulation and Computing increased to support pursuing new validated integrated design codes and advanced high-performance computing capabilities, including the El Capitan exascale system procurement. The press release, Featuring advanced capabilities for modeling, simulation and artificial intelligence (AI), based on Cray’s new Shasta architecture, El Capitan is projected to run national nuclear security applications at more than 50 times the speed of LLNL’s Sequoia system. Depending on the application, El Capitan will run roughly 10 times faster on average than LLNL’s Sierra system, currently the world’s second most powerful supercomputer at 125 petaflops of peak performance. Projected to be at least four times more energy efficient than Sierra, El Capitan is expected to go into production by late 2023, servicing the needs of NNSA’s Tri-Laboratory community: Lawrence Livermore National Laboratory, Los Alamos National Laboratory and Sandia National Laboratories El Capitan will be DOE’s third exascale-class supercomputer, following Argonne National Laboratory’s "Aurora" and Oak Ridge National Laboratory’s "Frontier" system. All three DOE exascale supercomputers will be built by Cray utilizing their Shasta architecture, Slingshot interconnect and new software platform. Link to the congressional report, https://www.energy.gov/sites/default/files/2022-03/FY%202022%20SSMP%20March%202022.pdf https://www.energy.gov/sites/default/files/2022-03/FY%202022... Link to the press release, https://www.llnl.gov/news/doennsa-lab-announce-partnership-cray-develop-nnsas-first-exascale-supercomputer https://www.llnl.gov/news/doennsa-lab-announce-partnership-c... - It does seem to me that it's an accurate assessment to say that this project exists because of the nuclear weapons research mandate. They're the same computer design, replicated. Are they using these to test/mature their software architecture? Hoping that many eyes will make bugs shallow? Or, is it something else? Perhaps it's seeing patterns where there are none, but for me the link between these projects and stockpile stewardship seems to be undeniable.
- mhh__ 4y ago
  It's not the servers as much as joining them together in a very fast network and with fast storage. A HPC cluster might also have men with guns outside which means they don't need quite the same threat model as a cloud provider (say)
- jandrewrogers 4y ago
  These essentially are commodity servers that have been hyper-optimized for maximum bandwidth and minimum latency. Many types of large-scale computation are effectively bandwidth limited. This implies needing to pack all of that compute as closely together physically as possible to minimize latency due to speed-of-light limits, which is substantial in massive compute clusters. It takes a long time for light to get from one side of the cluster to the other from the perspective of a computer. This, in turn, implies needing sophisticated custom cooling since the power density is very high due to packing so much silicon into such a small footprint. Another issue is that systems this large are at high risk of producing erroneous results due to the quantity of silicon involved creating a large attack surface for bit flips. It is very expensive to run a large supercomputing job only to find out that the output is incorrect and you'll need to start over, so aggressive mitigation, detection, and correction of sporadic data and compute errors is important. You could run all of these codes on a stack of beige boxes. It would take so long for the computation to complete that it would be extraordinarily expensive in both time and money. These systems focus on codes that would be effectively computationally intractable if you tried to run them in, say, AWS.
  - moab 4y ago
    > codes that would be effectively computationally intractable if you tried to run them in, say, AWS. such as?
    - nine_k 4y ago
      Maybe weather prediction and other problems of fluid dynamics?
    - jandrewrogers 4y ago
      Large-scale problems that look like computational fluid dynamics, graph analysis, spatiotemporal analysis, various kinds of computational science and physics models, etc. Just about anything that involves modeling and analyzing behaviors in the physical world. All of these involve high-bandwidth parallel orchestration at the data structures and algorithms level. The quality of the network is everything if your code is competently parallel. They don’t run these things on supercomputers for fun, it would be much easier if you could run these codes on AWS. The cloud has many advantages, but high-quality inter-node bandwidth and topology isn’t one of them. In HPC, the network is the most important part of the system.
- ipsum2 4y ago
  With AWS's elastic network adapter (ENA) providing fast interconnect, there's practically not much difference, assuming you can purchase the total amount of compute.
  - pas 4y ago
    that's too slow, there's EFA https://aws.amazon.com/blogs/aws/now-available-elastic-fabric-adapter-efa-for-tightly-coupled-hpc-workloads/ https://aws.amazon.com/blogs/aws/now-available-elastic-fabri...
    - ipsum2 4y ago
      Thanks for the correction, I meant EFA but misremembered the name.
- ballen 4y ago
  You can find the list of 2023 project allocations from the DOE INCITE program here [1]. INCITE allocates the majority of compute hours at the Leadership Computing Facilities, which includes the systems at Argonne and Oak Ridge. [1] https://www.doeleadershipcomputing.org/wp-content/uploads/2023INCITEFactSheets.pdf https://www.doeleadershipcomputing.org/wp-content/uploads/20...
- mixmastamyk 4y ago
  Running two or three Electron apps at a time.
  - agumonkey 4y ago
    I was about to say "resolve npm dependencies"
  - speed_spread 4y ago
    Rebuild it's own Gentoo with -O3
- dragontamer 4y ago
  They are commodity chips. But the interconnect is extremely beefed up. The bandwidth and communications are always huge on these supercomputers. That's really the difference between a typical cluster and a supercomputer... The interconnect.
nsxwolf 4y ago
~9 million Zen 3 cores is a lot of cores. The TOP500 site says it uses 21,100 kilowatts of power.
- luhn 4y ago
  It's only 600k Zen 3 cores. Still a massive number. > Frontier uses 9,472 AMD Epyc 7A53s "Trento" 64 core 2 GHz CPUs (606,208 cores) and 37,888 Radeon Instinct MI250X GPUs (8,335,360 cores).
neonate 4y ago
https://archive.ph/iz8zJ https://archive.ph/iz8zJ
russianGuy83829 4y ago
China had two exascale supercomputers back then already. I wonder what the three-letter agencies are deliberately not publishing. https://www.tomshardware.com/news/two-chinese-exascale-supercomputers https://www.tomshardware.com/news/two-chinese-exascale-super...
gttalbot 4y ago
Isn't the average Google data center larger than this?
- pclmulqdq 4y ago
  The average Google/AWS datacenter doesn't have the networking to support the kinds of high-adjacency work that supercomputers do. The biggest ones may be able to score higher on Linpack if they tried, but at much higher power.
  - dekhn 4y ago
    With the exception of double precision performance, TPUv4 is in every way a supercomputer in terms of networking.
mrtweetyhack 4y ago
[dead]

I worked with a Cray back in 2012 and the amount of engineering that goes into each rack and down to each blade is incredible. The biggest thing that got me was the geometry of the heatsinks that were designed to slightly compress air as it left each blade into the one above it.

LadyCailin 4y ago

What kinds of problems is this used to solve? Is it a different class of problems that couldn’t be solved with an equivalent number of commodity servers?

foota 4y ago
Generally, supercomputers are used in problems where you need fast access to all or lots of a large dataset. I think a common example is weather simulation, where each grid cell needs to update based on its neighbors at each timestep.
yjftsjthsd-h 4y ago
I expect you could solve the same problems on commodity hardware, but it'd be slower and/or more expensive.
- NegativeLatency 4y ago
  From what I've heard it's the high speed/throughput low latency interconnect that's the differentiator compared to thousands of commodity machines. (Weather and bomb explosions being two big ones that are hard to do concurrently, as the results depend on each other)
  - jgtrosh 4y ago
    Depends: if your application has strong scaling then just throw more power whichever way at it. Tightly coupled hpc systems aim to improve execution speed (or application resolution) to weak scaling applications.
green-salt 4y ago
The primary ones I worked with were for working on atmospheric science: https://www2.cisl.ucar.edu/about https://www2.cisl.ucar.edu/about https://www2.cisl.ucar.edu/ncar-supercomputing-history/bluefire https://www2.cisl.ucar.edu/ncar-supercomputing-history/bluef...
reportingsjr 4y ago
Most of the top supercomputers are used to model nuclear weapons, weather and climate, and various sciencey things. I don't know if the top ones are still used for oil and gas exploration (crunching data to provide higher resolution and higher accuracy oil field maps), but they have been in the past.
areoform 4y ago
Politely, this computer exists because the DoE performs stockpile stewardship. Less politely, it exists because of The Bombs. Everything else it will do is a hobby. It is specifically designed for whatever compute needs their classified program has. https://en.wikipedia.org/wiki/Stockpile_stewardship https://en.wikipedia.org/wiki/Stockpile_stewardship https://www.energy.gov/nnsa/articles/stockpile-stewardship-and-management-plan-ssmp https://www.energy.gov/nnsa/articles/stockpile-stewardship-a... All of this capital is being spent because the test ban treaty prohibits the detonation of these weapons, but the US stockpile and its efficacy must be maintained and new weapons must be designed. So how do you know if an old weapon or a new design will work or not? Some part of the answer is real data + simulations. Lots and lots of simulations. It's why Frontier exists and why the DoE has pursued exascale compute for quite some time. (as for the real data part, it's why places like the National Ignition Factory exist, and why there's an active experimental program that exists to study dummy warheads in unexpected ways, https://www.youtube.com/watch?v=FYdAT0v4DHs https://www.youtube.com/watch?v=FYdAT0v4DHs ) ——— Addendum, re: comments arguing that the NNSA computers are separate etc., the points raised are both true in the specific sense, but not true holistically. Quotes from the most recent report to congress, published in March 2022 about the Stockpile Stewardship and Management Plan, Near-Term and Out-Year Mission Goals: ◼ Advance the innovative experimental platforms, diagnostic equipment, and computational capabilities necessary to ensure stockpile safety, security, reliability, and effectiveness: – Achieve exascale computing by delivering an exascale-capable machine and modernizing the nuclear weapons code base – Develop an operational enhanced capability (advanced radiography and reactivity measurements) for subcritical experiments – Quantify the effects of plutonium aging on weapon performance over time – Assure an enduring, trusted supply of strategic radiation-hardened microsystems and, The weapons comprising the U.S. nuclear stockpile are assessed to be safe, reliable, effective, and secure. DOE/NNSA’s scientific infrastructure is currently adequate to support stockpile actions. The DOE/NNSA plans to address near-term gaps in required scientific capabilities by deploying the DOE/NNSA’s first exascale computing platform in fiscal year (FY)2023 and by improving capabilities to conduct subcritical experiments through the Enhanced Capabilities for Subcritical Experiments project by FY 2026. It seems they're building three of these computers, Frontier is one of them, from the budget requests and a press release, The budget request for Advanced Simulation and Computing increased to support pursuing new validated integrated design codes and advanced high-performance computing capabilities, including the El Capitan exascale system procurement. The press release, Featuring advanced capabilities for modeling, simulation and artificial intelligence (AI), based on Cray’s new Shasta architecture, El Capitan is projected to run national nuclear security applications at more than 50 times the speed of LLNL’s Sequoia system. Depending on the application, El Capitan will run roughly 10 times faster on average than LLNL’s Sierra system, currently the world’s second most powerful supercomputer at 125 petaflops of peak performance. Projected to be at least four times more energy efficient than Sierra, El Capitan is expected to go into production by late 2023, servicing the needs of NNSA’s Tri-Laboratory community: Lawrence Livermore National Laboratory, Los Alamos National Laboratory and Sandia National Laboratories El Capitan will be DOE’s third exascale-class supercomputer, following Argonne National Laboratory’s "Aurora" and Oak Ridge National Laboratory’s "Frontier" system. All three DOE exascale supercomputers will be built by Cray utilizing their Shasta architecture, Slingshot interconnect and new software platform. Link to the congressional report, https://www.energy.gov/sites/default/files/2022-03/FY%202022%20SSMP%20March%202022.pdf https://www.energy.gov/sites/default/files/2022-03/FY%202022... Link to the press release, https://www.llnl.gov/news/doennsa-lab-announce-partnership-cray-develop-nnsas-first-exascale-supercomputer https://www.llnl.gov/news/doennsa-lab-announce-partnership-c... - It does seem to me that it's an accurate assessment to say that this project exists because of the nuclear weapons research mandate. They're the same computer design, replicated. Are they using these to test/mature their software architecture? Hoping that many eyes will make bugs shallow? Or, is it something else? Perhaps it's seeing patterns where there are none, but for me the link between these projects and stockpile stewardship seems to be undeniable.
mhh__ 4y ago
It's not the servers as much as joining them together in a very fast network and with fast storage. A HPC cluster might also have men with guns outside which means they don't need quite the same threat model as a cloud provider (say)
jandrewrogers 4y ago
These essentially are commodity servers that have been hyper-optimized for maximum bandwidth and minimum latency. Many types of large-scale computation are effectively bandwidth limited. This implies needing to pack all of that compute as closely together physically as possible to minimize latency due to speed-of-light limits, which is substantial in massive compute clusters. It takes a long time for light to get from one side of the cluster to the other from the perspective of a computer. This, in turn, implies needing sophisticated custom cooling since the power density is very high due to packing so much silicon into such a small footprint. Another issue is that systems this large are at high risk of producing erroneous results due to the quantity of silicon involved creating a large attack surface for bit flips. It is very expensive to run a large supercomputing job only to find out that the output is incorrect and you'll need to start over, so aggressive mitigation, detection, and correction of sporadic data and compute errors is important. You could run all of these codes on a stack of beige boxes. It would take so long for the computation to complete that it would be extraordinarily expensive in both time and money. These systems focus on codes that would be effectively computationally intractable if you tried to run them in, say, AWS.
- moab 4y ago
  > codes that would be effectively computationally intractable if you tried to run them in, say, AWS. such as?
  - nine_k 4y ago
    Maybe weather prediction and other problems of fluid dynamics?
  - jandrewrogers 4y ago
    Large-scale problems that look like computational fluid dynamics, graph analysis, spatiotemporal analysis, various kinds of computational science and physics models, etc. Just about anything that involves modeling and analyzing behaviors in the physical world. All of these involve high-bandwidth parallel orchestration at the data structures and algorithms level. The quality of the network is everything if your code is competently parallel. They don’t run these things on supercomputers for fun, it would be much easier if you could run these codes on AWS. The cloud has many advantages, but high-quality inter-node bandwidth and topology isn’t one of them. In HPC, the network is the most important part of the system.
ipsum2 4y ago
With AWS's elastic network adapter (ENA) providing fast interconnect, there's practically not much difference, assuming you can purchase the total amount of compute.
- pas 4y ago
  that's too slow, there's EFA https://aws.amazon.com/blogs/aws/now-available-elastic-fabric-adapter-efa-for-tightly-coupled-hpc-workloads/ https://aws.amazon.com/blogs/aws/now-available-elastic-fabri...
  - ipsum2 4y ago
    Thanks for the correction, I meant EFA but misremembered the name.
ballen 4y ago
You can find the list of 2023 project allocations from the DOE INCITE program here [1]. INCITE allocates the majority of compute hours at the Leadership Computing Facilities, which includes the systems at Argonne and Oak Ridge. [1] https://www.doeleadershipcomputing.org/wp-content/uploads/2023INCITEFactSheets.pdf https://www.doeleadershipcomputing.org/wp-content/uploads/20...
mixmastamyk 4y ago
Running two or three Electron apps at a time.
- agumonkey 4y ago
  I was about to say "resolve npm dependencies"
- speed_spread 4y ago
  Rebuild it's own Gentoo with -O3
dragontamer 4y ago
They are commodity chips. But the interconnect is extremely beefed up. The bandwidth and communications are always huge on these supercomputers. That's really the difference between a typical cluster and a supercomputer... The interconnect.

nsxwolf 4y ago

~9 million Zen 3 cores is a lot of cores. The TOP500 site says it uses 21,100 kilowatts of power.

luhn 4y ago
It's only 600k Zen 3 cores. Still a massive number. > Frontier uses 9,472 AMD Epyc 7A53s "Trento" 64 core 2 GHz CPUs (606,208 cores) and 37,888 Radeon Instinct MI250X GPUs (8,335,360 cores).

neonate 4y ago

https://archive.ph/iz8zJ https://archive.ph/iz8zJ

russianGuy83829 4y ago

China had two exascale supercomputers back then already. I wonder what the three-letter agencies are deliberately not publishing. https://www.tomshardware.com/news/two-chinese-exascale-supercomputers https://www.tomshardware.com/news/two-chinese-exascale-super...

gttalbot 4y ago

Isn't the average Google data center larger than this?

pclmulqdq 4y ago
The average Google/AWS datacenter doesn't have the networking to support the kinds of high-adjacency work that supercomputers do. The biggest ones may be able to score higher on Linpack if they tried, but at much higher power.
- dekhn 4y ago
  With the exception of double precision performance, TPUv4 is in every way a supercomputer in terms of networking.

mrtweetyhack 4y ago

[dead]