10 ms·

LUMI, Europe’s most powerful supercomputer

118 points by Sami_Lehtinen 4y ago

hulitu 4y ago
This looks like a CIA briefing: A very powerful supercomputer. - But what can it do ? - I tell you, it's very powerful.
drallison 4y ago
The full article title is "LUMI, Europe’s most powerful supercomputer, is solving global challenges and promoting a green transformation". It's an irrational response, but I find it difficult to trust an article about a computer "solving global challenges" and "promoting a green transformation". The English usage is wrong and that makes me wonder whether the content of the article is correct.
robinhoodexe 4y ago
For a more technical overview: https://www.lumi-supercomputer.eu/may-we-introduce-lumi/ https://www.lumi-supercomputer.eu/may-we-introduce-lumi/
- oittaa 4y ago
  And full specs at https://www.lumi-supercomputer.eu/lumis-full-system-architecture-revealed/ https://www.lumi-supercomputer.eu/lumis-full-system-architec...
- martyvis 4y ago
  And it looks like the programming environment is running on that hobbyist OS kernel, Linux, that will come to nothing ;-) https://rocmdocs.amd.com/en/latest/index.html https://rocmdocs.amd.com/en/latest/index.html
  - pjmlp 4y ago
    > 1998: Many major companies such as IBM, Compaq and Oracle announce their support for Linux. > 2000: Dell announces that it is now the No. 2 provider of Linux-based systems worldwide and the first major manufacturer to offer Linux across its full product line https://en.wikipedia.org/wiki/History_of_Linux https://en.wikipedia.org/wiki/History_of_Linux 2019 => https://newsroom.ibm.com/2019-07-09-IBM-Closes-Landmark-Acquisition-of-Red-Hat-for-34-Billion-Defines-Open-Hybrid-Cloud-Future https://newsroom.ibm.com/2019-07-09-IBM-Closes-Landmark-Acqu... Linux never had any issues becoming yet another UNIX clone, hence why its strong points are CLI and headless applications. I bet LUMI researchers won't be doing data analysis on their laptops from Linux distributions.
    - asdf4564446 4y ago
      >I bet LUMI researchers won't be doing data analysis on their laptops from Linux distributions. I did just that when I used one of LUMI's predecessors. Linux desktops weren't uncommon in my corner of academia.
      - pjmlp 4y ago
        While at CERN and Fermilab everyone was using mostly Windows 2000 and OS X on their desktops, and many were doing their papers in Framemaker instead of LaTeX. The creation of the Scientific Linux distribution hardly changed this during the times I was there, and during my last visit for the Alumni Network creation event, it seems to have hardly changed. This is where I am coming from.
  - hrgiger 4y ago
    Afaik ROCM only provides fine tuned libs/drivers running on linux. At least it was provided that way years ago when I used and shipped with kinda soft vendor lock only main distros supported but if they provide now with the some tuned hpc OS that would be pretty good idea.
geoalchimista 4y ago
Its peak flops performance seems on par with DOE's Summit and 15% of Frontier, according to the top 500 supercomputer list: https://www.top500.org/lists/top500/2022/06/ https://www.top500.org/lists/top500/2022/06/.
Barrin92 4y ago
>"In addition, the waste heat produced by LUMI will be utilised in the district heating network of Kajaani, which means that its overall carbon footprint is negative. The waste heat produced by LUMI will provide 20 percent of Kajaani’s annual demand for district heat" Pretty cool honestly. Reminds me of the datacenter that Microsoft built in a harbor to cool with the surrounding seawater.
- nabla9 4y ago
  There are now several datacenters in Finland that link into local district heating. Microsoft recently announced that they build similar data center in Finland too https://www.fortum.com/media/2022/03/fortum-and-microsoft-announce-worlds-largest-collaboration-heat-homes-services-and-businesses-sustainable-waste-heat-new-data-centre-region https://www.fortum.com/media/2022/03/fortum-and-microsoft-an...
- alkonaut 4y ago
  I hope no datacenters these days are built on the idea of just running cooling with straight electricity (e.g. no cooling water) and shifting the heat straight out to the air (no waste heat recovery). Even in the late 90's that sounds like a poor design.
  - why_only_15 4y ago
    that's how all of Google's datacenters are built in my understanding. Water cooling is very expensive compared to air cooling, and only used for their supercomputer-esque applications like TPU pods. I don't know about waste heat recovery but I don't think they use that either.
    - RobertoG 4y ago
      There is also immersion cooling. The liquid is not water and it seems is pretty efficient: https://submer.com/immersion-cooling/ https://submer.com/immersion-cooling/
      - 4y ago
saddlerustle 4y ago
A supercomputer comparable to mid-size hyperscaler DC. (and no, it doesn't have uniquely good interconnect, it's broadly on par with the HPC GPU instances available from AWS and Azure)
- ClumsyPilot 4y ago
  Whats is the difference between a DC and why don't DCs appear in supercomputer rankings?
  - freemint 4y ago
    Optimization for different workloads, scheduling is per workload not renting per machine and that they are submitted the TOP500 list and run benchmarks. Why do not DCs appear? Because they have not submitted benchmarks and power measurements.
    - SiempreViernes 4y ago
      > scheduling is per workload This is really the key: a supercomputer has the (software) facilities that makes it possible to launch one coordinate job that runs across all nodes. A data centre is just a bunch of computers placed next to each other, with no affordances to coordinate things across them. At on point in time the hardware differences were much greater between the two, but the fundamental distinction where a supercomputer really is concerned with having the ability to be "one" computer remains.
      - zekrioca 4y ago
        Indeed, and that’s why it is called a “supercomputer”.
  - wongarsu 4y ago
    I think the main difference is that on a supercomputer you generally run one task at a time, while in a DC you have computers that do different, unrelated things. The rest kind of follows from that, like how a supercomputer that consists of multiple computers needs a fast, low-latency interconnect between them to coordinate and exchange results, while computers in a DC care a lot less about each other. On the other hand the distinction is fluid. Google could call the indexers that power their search engine a supercomputer, but they prefer to talk about datacenters
anttiharju 4y ago
Lumi is the Finnish word for snow in case anyone's wondering.
- kgwgk 4y ago
  Good to know the inspiration was not this: https://www.collinsdictionary.com/dictionary/spanish-english/lumi https://www.collinsdictionary.com/dictionary/spanish-english...
  - Hamuko 4y ago
    It's so hard to name anything without it being rude in some language. See also: Mitsubishi Pajero, Mazda Laputa.
- jjtheblunt 4y ago
  is there a cognate in an indo-european neighboring language, perhaps by loanword?
  - user_7832 4y ago
    https://en.m.wiktionary.org/wiki/lumi https://en.m.wiktionary.org/wiki/lumi Doesn't look like in a quick glance
    - jjtheblunt 4y ago
      holy cow, I didn't realize that was searchable or i'd have. thank you.
      - user_7832 4y ago
        Haha I just googled lumi etymology I think
        jjtheblunt 4y ago
        true : right under our noses is Google.
deleted 4y ago
[deleted]
occamrazor 4y ago
What is a goid benchmark today for supercomputers? TFLOPS don’t seem to be a good measure, because it’s relatively easy to deploy tens of thousands of servers. Is it the latency of the interconnection? Or the bandwidth? Or something entirely different?
- nabla9 4y ago
  They never have used flops to measure supercomputer performance. Its GFLOPS in HPLinpack (dense matrice multiplication)
- the_svd_doctor 4y ago
  Hard/impossible to respond. Every workload is different. It can be mostly compute bound (Linpack), communication bound, a mix, very latency sensitive, etc. Imho, it just depends on the workflow and we should probably use multiple metrics instead of just Linpack peak TFLOPS.
- alar44 4y ago
  That's like asking what the benchmark for an engine is. It all depends on what you're trying to do with it. There's no single metric to compare a diesel semi truck engine to a 2 stroke golf cart. You need multiple measures and the importance of each is dependent on your workload.
jpgvm 4y ago
Interesting to see Ceph mixed into the storage options. Lustre still king of the hill though.
- asciimike 4y ago
  My assumption is that Ceph is just there for easy/cheap block storage, while Lustre is doing the majority of the heavy lifting for the "supercomputing." Ceph file storage performance is abysmal, so it doesn't make sense to try and offer it for everything.
throw0101a 4y ago
Using AMD GPUs. How popular are they compared to Nvidia for HPC?
- cameronperot 4y ago
  NVIDIA has a significantly larger market share for HPC [1] (select accelerator for category). [1] https://top500.org/statistics/list/ https://top500.org/statistics/list/
  - brandmeyer 4y ago
    That's not my take-away from the chart, especially if you normalize by performance share. "Other" is the clear winner, and AMD has slightly more performance share than NVIDIA.
    - incrudible 4y ago
      AMD hardware is much more cost efficient on software that has not yet been developed. If you are planning on not using GPUs, definitely go with AMD.
    - cameronperot 4y ago
      Good point. I was looking at the "Family" aggregation which doesn't list AMD in the performance share chart, which was a bit misleading.
fancyfredbot 4y ago
I really love supercomputing but I worry whether, with a machine like this one, we get the right balance between spending on software optimization Vs spending on hardware. It used to be the case that fast hardware made sense because it was cheaper than optimising hundreds of applications but these days with unforgiving GPU architectures the penalty for poor optimisation is so high...
- jbjbjbjb 4y ago
  I wonder if anyone on HN could tell us how well optimised the code is on these? I imagine the simulations are complicated enough without someone going in and adding some performance optimisation.
  - nestorD 4y ago
    I am not familiar with that particular one but I have used other supercomputers and those people are not waiting for better hardware, they are trying to squeeze the best performance they can out of what they have. The end result mostly depends on the balance between scientists and engineers in the development team, it will oscillates between "this is python because the scientists working on the code know only that but we are using MPI to at least use several cores" and "we have a direct line with the hardware vendors in order to help us write the best software possible for this thing".
  - SiempreViernes 4y ago
    It varies quite a lot depending on the exact project and how much is expected to be purely waiting on one big compute job to finish. For something like climate simulations where a project is running big long jobs repeatedly I imagine they spend quite a bit of time on making it fast. For something like detector development where you run the hardware simulation production once and then spend three years trying to find the best way to reconstruct events less effort is put into making it fast. Saving two months from a six months job you run once isn't worth it if you have to spend more than a few weeks optimising it, and as these type of jobs need to write a lot to disk there's a limit to how much you'll get from optimising the hot loop.
  - 4y ago
sampo 4y ago
The computer as a whole has an entry (3.) in the top500 list. And then the cpu-only part of the computer has another entry (84.). The whole computer does about 150 PFlop/s, and the cpu-only part about 6 PFlop/s. So 96% of the computing power comes from the GPU cards. https://www.top500.org/lists/top500/list/2022/06/ https://www.top500.org/lists/top500/list/2022/06/
- credit_guy 4y ago
  This is impressive, but not exceedingly so. A single NVidia H100 GPU gets 1 PFlops/s [1] for single float (32 bit) precision. There's an asterisk which says Shown with sparsity. Specifications are one-half lower without sparsity I have to admit that I don't understand what this means, but the apparent takeaway is that the "non-sparse" speed is 0.5 PFlops/s. So this supercomputer has a speed equal to about 300 H100 GPUs. Not bad, but not something where one would start worrying about renewable electricity or not. [1] https://www.nvidia.com/en-us/data-center/h100/ https://www.nvidia.com/en-us/data-center/h100/
  - ajtulloch 4y ago
    TF32 is not IEEE-754 float32, it is a reduced precision format designed for machine learning usecases. The correct specsheet number for FP32 (and FP64 which is the relevant precision here) throughput on H100 is more like 60TFLOP/s, so your number is off by roughly an order of magnitude.
jp0d 4y ago
They've also partnered with the RIKEN Centre for Computation Science (Developer of the fastest Super computer on earth). Quite impressive and interesting at the same time as they use very different architectures. https://www.r-ccs.riken.jp/en/outreach/topics/20220518-1/ https://www.r-ccs.riken.jp/en/outreach/topics/20220518-1/ https://top500.org/lists/hpcg/2022/06/ https://top500.org/lists/hpcg/2022/06/
- Wuzado 4y ago
  Technically, according to Top500, they're not the fastest anymore. The #1 right now is held by Frontier over at the Oak Ridge National Laboratory, with Rmax 1.102 EFlop/s. That doesn't mean that it's not an extremely capable system, though. (And Fugaku's performance seems not that well reflected in the HPL benchmark.)
  - jp0d 4y ago
    Thanks for clarifying. It's also interesting to see the presence of ARM64 and Power architecture in top 10. Also the prominence of AMD EPYC chips.
formerkrogemp 4y ago
The most powerful computer is the one that can launch nuclear weapons. "Shall we play a game?"
robocat 4y ago
> LUMI gets all of its electricity from 100 percent renewable hydroelectric power. 100% pure greenwashing bullshit at a guess. Power usage has gone up in Finland, and that extra power does not come from hydroelectricity, because all[1] the hydroelectric power is already used domestically. Instead the extra power usually comes from non-renewable sources. I am guessing they signed an agreement with a hydroelectric producer to buy “hydroelectric” electricity, fungibly delivered over the transmission network. Even if they got the electricity delivered directly from hydro, that would leave a network shortfall to be filled by non-renewables. Avoiding aircon is good, and the heat reuse is good since it presumably reduces non-renewable resource use by domestic heating. [1] https://www.stat.fi/til/salatuo/2020/salatuo_2020_2021-11-02_tie_001_en.html https://www.stat.fi/til/salatuo/2020/salatuo_2020_2021-11-02...
- teruakohatu 4y ago
  If every large producer insists on hydro/green energy, the price of hydro will go up encouraging investment in more green energy. It's like arguing planting a tree just denied someone else the opportunity to plant a tree in that square foot of dirt. I am sure they are paying a premium. Here in NZ we have an aluminium smelter that is directly connected to a huge hydro dam, rather than the grid, so they can claim it is powered by green energy regardless of how you look at it.
  - robocat 4y ago
    > If every large producer insists on hydro/green energy, the price of hydro will go up encouraging investment in more green energy. I don’t think that is realistic, and your word “every” is doing some very heavy lifting to give an overly simplistic answer. Renewable electricity is over 50% in NZ[0]. From a pure economic argument, green demand would need to approach green supply before before hydro could gain much of an excess price. Paradoxically, it seems plausible to me that electricity generation companies have enough moat to be able to just squeeze profits out, and not increase green generation at all (regardless of your hypothetical financial incentives). Most renewable requires huge capital investments, and a few percent extra profit can easily not move the profitability needle enough to change a project from a no to a go. I would like to see a good economic and ecological comparison of the regulatory/tax choices the government could make, because I am somewhat suspicious about the recent electric-car subsidy. I have an acquaintance who has been part of a very well planned ($x million invested) solar project for NZ with an expected capital cost in the hundreds of millions, and anecdotally the government doesn’t seem to give a shit about it. Maybe they have the wrong political connections? Hydro is the same as a battery, so solar appears to be very sensible in NZ. > aluminium smelter [claims] it is powered by green energy I already answered that one in the comment you are replying to. More specifically: our NZ government has decided for decades to give Tiwai Smelter cheap electricity contracts (maybe for the sake of 2500 jobs), and finally decided to pull the plug on them. NZ consumers have had to buy marginally expensive and mostly non-renewable electricity over many decades due to that government choice. I haven’t looked at the country economics for the smelter, but I strongly suspect our government has been making uneconomic political decisions that have cost the country dearly. I presume they will put in a link from Manapouri to Benmore (connect to the Benmore to Wellington DC link, which isn’t 100% capacity constrained?[3]). I would also guess that the government was politically pressured to have more green generation for NZ, and shutting down Tiwai finally made political sense on the world stage to achieve “more” renewable generation. Who wants another 1992 due to mis-planning for our security of supply? 1992 was perhaps historical blindness given that: “the power crises of the early 50s when serious shortfalls in supply created shutdowns and blackouts throughout the country. These were a regular occurrence in the years between 1954 and 1957”[1]. Although I guess I would need to see a past analysis of the risks versus the costs to judge whether the 1992 economic GDP reduction was just an acceptable cost versus the costs of upgrading generation. Governments generally are motivated by quick wins, and are sometimes not motivated to use regulation to encourage long term risk reduction against fat-tail events. [0] https://www.mbie.govt.nz/assets/Data-Files/Energy/nz-energy-quarterly-and-energy-in-nz/electricity.xlsx https://www.mbie.govt.nz/assets/Data-Files/Energy/nz-energy-... [1] https://www.engineeringnz.org/programmes/heritage/heritage-records/new-zealand-electricity-system/ https://www.engineeringnz.org/programmes/heritage/heritage-r... [2] 2020: “Transpower has indicated it will cost $600 million to put the transmission structures in place to take the power from Southland to Auckland. That work was due to be completed over the next five to seven years.” The majority of Auckland electricity consumption is residential, the remainder of usage mostly commercial, and Glenbrook mill is less than 10%. https://www.stuff.co.nz/national/122113863/the-power-game-where-will-manapouris-electricity-go https://www.stuff.co.nz/national/122113863/the-power-game-wh... [3] https://www.transpower.co.nz/power-system-live-data https://www.transpower.co.nz/power-system-live-data
ValtteriL 4y ago
CSC <3

118 points by Sami_Lehtinen 4y ago

This looks like a CIA briefing: A very powerful supercomputer. - But what can it do ? - I tell you, it's very powerful.

The full article title is "LUMI, Europe’s most powerful supercomputer, is solving global challenges and promoting a green transformation". It's an irrational response, but I find it difficult to trust an article about a computer "solving global challenges" and "promoting a green transformation". The English usage is wrong and that makes me wonder whether the content of the article is correct.

robinhoodexe 4y ago

For a more technical overview: https://www.lumi-supercomputer.eu/may-we-introduce-lumi/ https://www.lumi-supercomputer.eu/may-we-introduce-lumi/

oittaa 4y ago
And full specs at https://www.lumi-supercomputer.eu/lumis-full-system-architecture-revealed/ https://www.lumi-supercomputer.eu/lumis-full-system-architec...
martyvis 4y ago
And it looks like the programming environment is running on that hobbyist OS kernel, Linux, that will come to nothing ;-) https://rocmdocs.amd.com/en/latest/index.html https://rocmdocs.amd.com/en/latest/index.html
- pjmlp 4y ago
  > 1998: Many major companies such as IBM, Compaq and Oracle announce their support for Linux. > 2000: Dell announces that it is now the No. 2 provider of Linux-based systems worldwide and the first major manufacturer to offer Linux across its full product line https://en.wikipedia.org/wiki/History_of_Linux https://en.wikipedia.org/wiki/History_of_Linux 2019 => https://newsroom.ibm.com/2019-07-09-IBM-Closes-Landmark-Acquisition-of-Red-Hat-for-34-Billion-Defines-Open-Hybrid-Cloud-Future https://newsroom.ibm.com/2019-07-09-IBM-Closes-Landmark-Acqu... Linux never had any issues becoming yet another UNIX clone, hence why its strong points are CLI and headless applications. I bet LUMI researchers won't be doing data analysis on their laptops from Linux distributions.
  - asdf4564446 4y ago
    >I bet LUMI researchers won't be doing data analysis on their laptops from Linux distributions. I did just that when I used one of LUMI's predecessors. Linux desktops weren't uncommon in my corner of academia.
    - pjmlp 4y ago
      While at CERN and Fermilab everyone was using mostly Windows 2000 and OS X on their desktops, and many were doing their papers in Framemaker instead of LaTeX. The creation of the Scientific Linux distribution hardly changed this during the times I was there, and during my last visit for the Alumni Network creation event, it seems to have hardly changed. This is where I am coming from.
- hrgiger 4y ago
  Afaik ROCM only provides fine tuned libs/drivers running on linux. At least it was provided that way years ago when I used and shipped with kinda soft vendor lock only main distros supported but if they provide now with the some tuned hpc OS that would be pretty good idea.

geoalchimista 4y ago

Its peak flops performance seems on par with DOE's Summit and 15% of Frontier, according to the top 500 supercomputer list: https://www.top500.org/lists/top500/2022/06/ https://www.top500.org/lists/top500/2022/06/.

Barrin92 4y ago

>"In addition, the waste heat produced by LUMI will be utilised in the district heating network of Kajaani, which means that its overall carbon footprint is negative. The waste heat produced by LUMI will provide 20 percent of Kajaani’s annual demand for district heat" Pretty cool honestly. Reminds me of the datacenter that Microsoft built in a harbor to cool with the surrounding seawater.

nabla9 4y ago
There are now several datacenters in Finland that link into local district heating. Microsoft recently announced that they build similar data center in Finland too https://www.fortum.com/media/2022/03/fortum-and-microsoft-announce-worlds-largest-collaboration-heat-homes-services-and-businesses-sustainable-waste-heat-new-data-centre-region https://www.fortum.com/media/2022/03/fortum-and-microsoft-an...
alkonaut 4y ago
I hope no datacenters these days are built on the idea of just running cooling with straight electricity (e.g. no cooling water) and shifting the heat straight out to the air (no waste heat recovery). Even in the late 90's that sounds like a poor design.
- why_only_15 4y ago
  that's how all of Google's datacenters are built in my understanding. Water cooling is very expensive compared to air cooling, and only used for their supercomputer-esque applications like TPU pods. I don't know about waste heat recovery but I don't think they use that either.
  - RobertoG 4y ago
    There is also immersion cooling. The liquid is not water and it seems is pretty efficient: https://submer.com/immersion-cooling/ https://submer.com/immersion-cooling/
    - 4y ago

saddlerustle 4y ago

A supercomputer comparable to mid-size hyperscaler DC. (and no, it doesn't have uniquely good interconnect, it's broadly on par with the HPC GPU instances available from AWS and Azure)

ClumsyPilot 4y ago
Whats is the difference between a DC and why don't DCs appear in supercomputer rankings?
- freemint 4y ago
  Optimization for different workloads, scheduling is per workload not renting per machine and that they are submitted the TOP500 list and run benchmarks. Why do not DCs appear? Because they have not submitted benchmarks and power measurements.
  - SiempreViernes 4y ago
    > scheduling is per workload This is really the key: a supercomputer has the (software) facilities that makes it possible to launch one coordinate job that runs across all nodes. A data centre is just a bunch of computers placed next to each other, with no affordances to coordinate things across them. At on point in time the hardware differences were much greater between the two, but the fundamental distinction where a supercomputer really is concerned with having the ability to be "one" computer remains.
    - zekrioca 4y ago
      Indeed, and that’s why it is called a “supercomputer”.
- wongarsu 4y ago
  I think the main difference is that on a supercomputer you generally run one task at a time, while in a DC you have computers that do different, unrelated things. The rest kind of follows from that, like how a supercomputer that consists of multiple computers needs a fast, low-latency interconnect between them to coordinate and exchange results, while computers in a DC care a lot less about each other. On the other hand the distinction is fluid. Google could call the indexers that power their search engine a supercomputer, but they prefer to talk about datacenters

anttiharju 4y ago

Lumi is the Finnish word for snow in case anyone's wondering.

kgwgk 4y ago
Good to know the inspiration was not this: https://www.collinsdictionary.com/dictionary/spanish-english/lumi https://www.collinsdictionary.com/dictionary/spanish-english...
- Hamuko 4y ago
  It's so hard to name anything without it being rude in some language. See also: Mitsubishi Pajero, Mazda Laputa.
jjtheblunt 4y ago
is there a cognate in an indo-european neighboring language, perhaps by loanword?
- user_7832 4y ago
  https://en.m.wiktionary.org/wiki/lumi https://en.m.wiktionary.org/wiki/lumi Doesn't look like in a quick glance
  - jjtheblunt 4y ago
    holy cow, I didn't realize that was searchable or i'd have. thank you.
    - user_7832 4y ago
      Haha I just googled lumi etymology I think
      - jjtheblunt 4y ago
        true : right under our noses is Google.

deleted 4y ago

[deleted]

occamrazor 4y ago

What is a goid benchmark today for supercomputers? TFLOPS don’t seem to be a good measure, because it’s relatively easy to deploy tens of thousands of servers. Is it the latency of the interconnection? Or the bandwidth? Or something entirely different?

nabla9 4y ago
They never have used flops to measure supercomputer performance. Its GFLOPS in HPLinpack (dense matrice multiplication)
the_svd_doctor 4y ago
Hard/impossible to respond. Every workload is different. It can be mostly compute bound (Linpack), communication bound, a mix, very latency sensitive, etc. Imho, it just depends on the workflow and we should probably use multiple metrics instead of just Linpack peak TFLOPS.
alar44 4y ago
That's like asking what the benchmark for an engine is. It all depends on what you're trying to do with it. There's no single metric to compare a diesel semi truck engine to a 2 stroke golf cart. You need multiple measures and the importance of each is dependent on your workload.

jpgvm 4y ago

Interesting to see Ceph mixed into the storage options. Lustre still king of the hill though.

asciimike 4y ago
My assumption is that Ceph is just there for easy/cheap block storage, while Lustre is doing the majority of the heavy lifting for the "supercomputing." Ceph file storage performance is abysmal, so it doesn't make sense to try and offer it for everything.

throw0101a 4y ago

Using AMD GPUs. How popular are they compared to Nvidia for HPC?

cameronperot 4y ago
NVIDIA has a significantly larger market share for HPC [1] (select accelerator for category). [1] https://top500.org/statistics/list/ https://top500.org/statistics/list/
- brandmeyer 4y ago
  That's not my take-away from the chart, especially if you normalize by performance share. "Other" is the clear winner, and AMD has slightly more performance share than NVIDIA.
  - incrudible 4y ago
    AMD hardware is much more cost efficient on software that has not yet been developed. If you are planning on not using GPUs, definitely go with AMD.
  - cameronperot 4y ago
    Good point. I was looking at the "Family" aggregation which doesn't list AMD in the performance share chart, which was a bit misleading.

fancyfredbot 4y ago

I really love supercomputing but I worry whether, with a machine like this one, we get the right balance between spending on software optimization Vs spending on hardware. It used to be the case that fast hardware made sense because it was cheaper than optimising hundreds of applications but these days with unforgiving GPU architectures the penalty for poor optimisation is so high...

jbjbjbjb 4y ago
I wonder if anyone on HN could tell us how well optimised the code is on these? I imagine the simulations are complicated enough without someone going in and adding some performance optimisation.
- nestorD 4y ago
  I am not familiar with that particular one but I have used other supercomputers and those people are not waiting for better hardware, they are trying to squeeze the best performance they can out of what they have. The end result mostly depends on the balance between scientists and engineers in the development team, it will oscillates between "this is python because the scientists working on the code know only that but we are using MPI to at least use several cores" and "we have a direct line with the hardware vendors in order to help us write the best software possible for this thing".
- SiempreViernes 4y ago
  It varies quite a lot depending on the exact project and how much is expected to be purely waiting on one big compute job to finish. For something like climate simulations where a project is running big long jobs repeatedly I imagine they spend quite a bit of time on making it fast. For something like detector development where you run the hardware simulation production once and then spend three years trying to find the best way to reconstruct events less effort is put into making it fast. Saving two months from a six months job you run once isn't worth it if you have to spend more than a few weeks optimising it, and as these type of jobs need to write a lot to disk there's a limit to how much you'll get from optimising the hot loop.
- 4y ago

sampo 4y ago

The computer as a whole has an entry (3.) in the top500 list. And then the cpu-only part of the computer has another entry (84.). The whole computer does about 150 PFlop/s, and the cpu-only part about 6 PFlop/s. So 96% of the computing power comes from the GPU cards. https://www.top500.org/lists/top500/list/2022/06/ https://www.top500.org/lists/top500/list/2022/06/

credit_guy 4y ago
This is impressive, but not exceedingly so. A single NVidia H100 GPU gets 1 PFlops/s [1] for single float (32 bit) precision. There's an asterisk which says Shown with sparsity. Specifications are one-half lower without sparsity I have to admit that I don't understand what this means, but the apparent takeaway is that the "non-sparse" speed is 0.5 PFlops/s. So this supercomputer has a speed equal to about 300 H100 GPUs. Not bad, but not something where one would start worrying about renewable electricity or not. [1] https://www.nvidia.com/en-us/data-center/h100/ https://www.nvidia.com/en-us/data-center/h100/
- ajtulloch 4y ago
  TF32 is not IEEE-754 float32, it is a reduced precision format designed for machine learning usecases. The correct specsheet number for FP32 (and FP64 which is the relevant precision here) throughput on H100 is more like 60TFLOP/s, so your number is off by roughly an order of magnitude.

jp0d 4y ago

They've also partnered with the RIKEN Centre for Computation Science (Developer of the fastest Super computer on earth). Quite impressive and interesting at the same time as they use very different architectures. https://www.r-ccs.riken.jp/en/outreach/topics/20220518-1/ https://www.r-ccs.riken.jp/en/outreach/topics/20220518-1/ https://top500.org/lists/hpcg/2022/06/ https://top500.org/lists/hpcg/2022/06/

Wuzado 4y ago
Technically, according to Top500, they're not the fastest anymore. The #1 right now is held by Frontier over at the Oak Ridge National Laboratory, with Rmax 1.102 EFlop/s. That doesn't mean that it's not an extremely capable system, though. (And Fugaku's performance seems not that well reflected in the HPL benchmark.)
- jp0d 4y ago
  Thanks for clarifying. It's also interesting to see the presence of ARM64 and Power architecture in top 10. Also the prominence of AMD EPYC chips.

formerkrogemp 4y ago

The most powerful computer is the one that can launch nuclear weapons. "Shall we play a game?"

robocat 4y ago

> LUMI gets all of its electricity from 100 percent renewable hydroelectric power. 100% pure greenwashing bullshit at a guess. Power usage has gone up in Finland, and that extra power does not come from hydroelectricity, because all[1] the hydroelectric power is already used domestically. Instead the extra power usually comes from non-renewable sources. I am guessing they signed an agreement with a hydroelectric producer to buy “hydroelectric” electricity, fungibly delivered over the transmission network. Even if they got the electricity delivered directly from hydro, that would leave a network shortfall to be filled by non-renewables. Avoiding aircon is good, and the heat reuse is good since it presumably reduces non-renewable resource use by domestic heating. [1] https://www.stat.fi/til/salatuo/2020/salatuo_2020_2021-11-02_tie_001_en.html https://www.stat.fi/til/salatuo/2020/salatuo_2020_2021-11-02...

teruakohatu 4y ago
If every large producer insists on hydro/green energy, the price of hydro will go up encouraging investment in more green energy. It's like arguing planting a tree just denied someone else the opportunity to plant a tree in that square foot of dirt. I am sure they are paying a premium. Here in NZ we have an aluminium smelter that is directly connected to a huge hydro dam, rather than the grid, so they can claim it is powered by green energy regardless of how you look at it.
- robocat 4y ago
  > If every large producer insists on hydro/green energy, the price of hydro will go up encouraging investment in more green energy. I don’t think that is realistic, and your word “every” is doing some very heavy lifting to give an overly simplistic answer. Renewable electricity is over 50% in NZ[0]. From a pure economic argument, green demand would need to approach green supply before before hydro could gain much of an excess price. Paradoxically, it seems plausible to me that electricity generation companies have enough moat to be able to just squeeze profits out, and not increase green generation at all (regardless of your hypothetical financial incentives). Most renewable requires huge capital investments, and a few percent extra profit can easily not move the profitability needle enough to change a project from a no to a go. I would like to see a good economic and ecological comparison of the regulatory/tax choices the government could make, because I am somewhat suspicious about the recent electric-car subsidy. I have an acquaintance who has been part of a very well planned ($x million invested) solar project for NZ with an expected capital cost in the hundreds of millions, and anecdotally the government doesn’t seem to give a shit about it. Maybe they have the wrong political connections? Hydro is the same as a battery, so solar appears to be very sensible in NZ. > aluminium smelter [claims] it is powered by green energy I already answered that one in the comment you are replying to. More specifically: our NZ government has decided for decades to give Tiwai Smelter cheap electricity contracts (maybe for the sake of 2500 jobs), and finally decided to pull the plug on them. NZ consumers have had to buy marginally expensive and mostly non-renewable electricity over many decades due to that government choice. I haven’t looked at the country economics for the smelter, but I strongly suspect our government has been making uneconomic political decisions that have cost the country dearly. I presume they will put in a link from Manapouri to Benmore (connect to the Benmore to Wellington DC link, which isn’t 100% capacity constrained?[3]). I would also guess that the government was politically pressured to have more green generation for NZ, and shutting down Tiwai finally made political sense on the world stage to achieve “more” renewable generation. Who wants another 1992 due to mis-planning for our security of supply? 1992 was perhaps historical blindness given that: “the power crises of the early 50s when serious shortfalls in supply created shutdowns and blackouts throughout the country. These were a regular occurrence in the years between 1954 and 1957”[1]. Although I guess I would need to see a past analysis of the risks versus the costs to judge whether the 1992 economic GDP reduction was just an acceptable cost versus the costs of upgrading generation. Governments generally are motivated by quick wins, and are sometimes not motivated to use regulation to encourage long term risk reduction against fat-tail events. [0] https://www.mbie.govt.nz/assets/Data-Files/Energy/nz-energy-quarterly-and-energy-in-nz/electricity.xlsx https://www.mbie.govt.nz/assets/Data-Files/Energy/nz-energy-... [1] https://www.engineeringnz.org/programmes/heritage/heritage-records/new-zealand-electricity-system/ https://www.engineeringnz.org/programmes/heritage/heritage-r... [2] 2020: “Transpower has indicated it will cost $600 million to put the transmission structures in place to take the power from Southland to Auckland. That work was due to be completed over the next five to seven years.” The majority of Auckland electricity consumption is residential, the remainder of usage mostly commercial, and Glenbrook mill is less than 10%. https://www.stuff.co.nz/national/122113863/the-power-game-where-will-manapouris-electricity-go https://www.stuff.co.nz/national/122113863/the-power-game-wh... [3] https://www.transpower.co.nz/power-system-live-data https://www.transpower.co.nz/power-system-live-data

ValtteriL 4y ago

CSC <3