5 ms·

Serious question: my understanding of HPC is that there are many workloads running on a given supercomputer at any time. There is no singular workload that take

Serious question: my understanding of HPC is that there are many workloads running on a given supercomputer at any time. There is no singular workload that takes up the entire or most of the resources of a supercomputer. Is my understanding correct? If yes, then why is it important to build supercomputers with more and more compute? Wouldn't it be better to build smaller systems that focus more on power/cost/space efficiency?

sseagull 2y ago
You are generally correct, however there are workloads that do use larger portions of a supercomputer that wouldn't be feasible on smaller systems. Also, I guess I'm not sure what you mean by "smaller systems that focus more on power/cost/space". A proper queueing system generally efficiently allocates the resources of a large supercomputer to smaller tasks, while also making larger tasks possible in the first place. And I imagine there's somewhat an efficiency of scale in a large installation like this. There are, of course, many many smaller supercomputers, such as at most medium to large universities. But even those often have 10-50k cores or so. (In general, efficiency is a consideration when building/running, but not of using. Scientists want the most computational power they can get, power usage be damned :) ) edit: A related topic is capacity vs. capability: https://en.wikipedia.org/wiki/Supercomputer#Capability_versus_capacity https://en.wikipedia.org/wiki/Supercomputer#Capability_versu...
tkuraku 2y ago
While in general there can be many smaller workloads running in parallel. However, periodically the whole supercomputer can be reserved for a "hero" run.
- deleted 2y ago
  [deleted]
bitfilped 2y ago
Most of the time yes, HPC systems are shared among many users. Sometimes though the whole system (or near it) will be used in a single run. These are sometimes referred to as "hero runs" and while they're more common for benchmarking and burn-ins there are some tightly-coupled workloads that perform well in that style of execution. It really depends on a number of factors like the workloads being run, the number of users, and what the primary business purpose of the HPC resource is. Sites that have to run both types of jobs will typically allow any user to schedule jobs most of the time but then pre-reserve blocks of time for hero runs to take place where other user jobs are held until the primary scheduled run is over.
- Harmohit 2y ago
  Thanks for the reply! Can you give some examples of these "hero runs"?
  - magicalhippo 2y ago
    At our university, at least when I studied there some 15 years ago, the whole cluster was occupied doing weather predictions each and every night. No point in staying up waiting for a job, it'd get rescheduled in the early morning at best. It wasn't the largest cluster around, IIRC 768 quad-core nodes, but I'm sure the meteorological department would find a way to utilize any extra capacity, so still requiring the whole thing all night.
    - moomin 2y ago
      Feels like this might be an invariant in computer science. I once worked on an IBM/360. The majority of the compute was taken up by a single person who did weather simulation. An IBM/360 has laughably less compute than your phone.
      - phinnaeus 2y ago
        It might still have produced a better weather prediction than my phone, though /s
      - nicklecompte 2y ago
        "Computational fluid dynamics is hard" seems like a true bona fide CS invariant.
      - Keyframe 2y ago
        ENIAC ran weather as well. It's a kind of numerical simulation that will always be in need of compute cycles. We'll need more of it as we go deeper into climate change.
alephnerd 2y ago
> my understanding correct Yes > why is it important to build supercomputers with more and more compute A mix of - research in distributed systems (there are plenty of open questions in Concurrency, Parallelization, Computer Architecture, etc) - a way to maintain an ecosystem of large vendors (Intel, AMD, Nvidia and plenty of smaller vendors all get a piece of the pie to subsidize R&D) - some problems are EXTREMELY computationally and financially expensive, so they require large On-Prem compute capabilities (eg. Protein folding, machine learning when I was in undergrad [DGX-100s were subsidized by Aurora], etc) - some problems are extremely sensitive for national security reasons and it's best to keep all personnel in a single region (eg. Nuclear simulations, turbine simulations, some niche ML work, etc) In reality you need to do both, and planners know this fact, and have known this fact for decades
trueismywork 2y ago
Bigger systems when utilized at 100% are more efficient than multiple smaller systems when utilized at 100%, in terms of engineering work, software, etc. But also, bigger systems have more opportunities to achieve higher utilization than smaller systems due to the dynamics of bin packing problem.
prpl 2y ago
there’s also Bell Prize submissions, which is the only time some machines get completely reserved
kkielhofner 2y ago
I have a project on Frontier. Generally these systems (including Frontier) use slurm[0] for scheduling and workload management. The OLCF Frontier user guide[1] has some information on scheduling and Frontier specific quirks (very minor). Current status of jobs on Frontier: [kkielhofner@login11.frontier ~]$ squeue -h -t running -r | wc -l 137 [kkielhofner@login11.frontier ~]$ squeue -h -t pending -r | wc -l 1016 The running jobs are relatively low because there are some massive jobs using a significant number of nodes ATM. [0] - https://slurm.schedmd.com/documentation.html https://slurm.schedmd.com/documentation.html [1] - https://docs.olcf.ornl.gov/systems/frontier_user_guide.html https://docs.olcf.ornl.gov/systems/frontier_user_guide.html EDIT: I give up on HN code formatting
- SushiHippie 2y ago
  > EDIT: I give up on HN code formatting Just FYI: https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc > Text after a blank line that is indented by two or more spaces is reproduced verbatim. (This is intended for code.) [kkielhofner@login11.frontier ~]$ squeue -h -t running -r | wc -l 137 [kkielhofner@login11.frontier ~]$ squeue -h -t pending -r | wc -l 1016
  - kkielhofner 2y ago
    Yeah I've seen that but was annoyed by not being able to just use backticks like everywhere else. Oh the irony of using Frontier but not "understanding" HF formatting ;).
dekhn 2y ago
There's many variables that go into supercomputers, of which "company/country propaganda" is just one of them. Supercomputer admins would love to have a single code that used the whole machine, both the compute elements and the network elements, at close to 100%. In fact they spend a significant fraction on network elements to unblock the compute elements, but few codes are really so light on networking that the program scales to the full core count of the machine. So, instead they usually have several codes which can scale up to a significant fraction of the machine and then backfill with smaller jobs to keep the utilization up (because the acquisition cost and the running cost are so high). Supercomputers have limited utility- beyond country bragging rights, only a few problems really justify spending this kind of resource. I intentionally switched my own research in molecular dynamics away from supercomputers (where I'd run one job on 64-128 processors for a 96X speedup) to closet cllusters, where I'd run 128 indpendent jobs for a 128X speedup, but then have to do a bunch of post-processing to make the results comparable to the long, large runs on the supercomputer (https://research.google/pubs/cloud-based-simulations-on-google-exacycle-reveal-ligand-modulation-of-gpcr-activation-pathways/ https://research.google/pubs/cloud-based-simulations-on-goog...). I actually was really relieved when my work no longer depended on expensive resources with little support, as my scientific productivity went up and my costs went way down. I feel that supercomputers are good at one thing: if you need to make your country's flagship submarine about 10% faster/quieter than the competition.
- alephnerd 2y ago
  Ever used GPT-3, DALL-E, or other LLMs? The GPUs used to train them only existed because the DoE explicitly worked with Nvidia on a decade-long roadmap for delivery in it's various supercomputers, and would often work in tandem with private sector players to coordinate purchases and R&D (for example, protein folding and just about every Big Pharma company). Hell, the only reason AMD EPYC exists is for the same reason.
  - dekhn 2y ago
    Yes. In my computational history I have: used the largest non-classified DOE supercomputers, built my own modest closet clusters, developed an embarassingly parallel computing system using Google's idle prod cycles, and helped debug training of LLMs when I worked on the TPU team at Google. I work for big pharma now (and my phd is in biophysics) and I'm also comfortable with other HPC domains. I know the DOE/Nvidia history quite well as the Chief Scientist of NVIDIA visited LBL around 2005(6? 7?) and talked about their new hardware they were just starting to build and sell, with the goal of getting them into supercomputers. We asked if they had double precision performance yet (because that was a must for many supercomputer jobs), but at the time, nvidia DP was still lagging SP (I guess it still does?) and we also quibbled about their non-compliance with some esoteric details in IEEE 754. The best part of the whole talk was when he walked us through the idea of visualizing our operations by drawing the matrices as textures, because you can easily see the NaNs- they render as nvidia Green! I left DOE (Berkeley Lab) shortly after to work in industry because it was clear that ML wasn't going to be innovated in the government labs.
ThinkBeat 2y ago
From my experience they are running whole cluster dedicated jobs quite frequently. Climate models can use whatever resources they get, nuclear weapons modelling esp for old warheads can use a lot.
- MaxikCZ 2y ago
  What is being calculated with nuclear weapons? I understand it must have been computationally expensive to get them working, but once completed, what is there left to calculate?
  - Majromax 2y ago
    I don't work in this area, but think about all of the variables that go into warhead maintenance. Your single supercomputer simulation can show that the warhead should work as-designed, but what will happen after the Plutonium pit has been sitting for a decade, slowly decaying? Will satisfactory implosion still happen if storage conditions slightly change the performance of the conventional explosives? Since modern warheads are all fusion-type warheads, there's also the fusion stage to consider with even more highly classified top-secret sauce. It appears that the conditions for fusion are triggered by radiation pressure, and that likely makes things even more complicated. Now, you need not just a successful supercritical fission event, but one of the right shape(?), timing, and interaction with other secret-sauce materials that might have their own degradation curves. So, rather than simulate one design, now you need to simulate hundreds to thousands to explore the full decay-over-time space. Getting the answer wrong means either very expensive premature warhead refurbishments or a nuclear stockpile that wouldn't work properly.
  - bargle0 2y ago
    You can’t test idle weapons to make sure they still go boom in real life, so you have to simulate it.
  - ThinkBeat 2y ago
    The original nuclear powers are accumulating really old atomic bombs / rockets. We dont know for sure what is going on inside the warheads. (or possibly a sub section of the war head) Cracking them open to have a look may not be a good idea. but leaving them alone for another few decades might not be wise either. Funny fact: A lot of the nuclear weapons that have been destroyed, have removed them from bombs or rockets. But in many case the warheads were moved into storage. Ready to slap them on a rocket if that should become needed.
Xcelerate 2y ago
> There is no singular workload that takes up the entire or most of the resources of a supercomputer. I performed molecular dynamics simulations on the Titan supercomputer at ORNL during grad school. At the time, this supercomputer was the fastest in the world. At least back then around 2012, ORNL really wanted projects that uniquely showcased the power of the machine. Many proposals for compute time were turned down for workloads that were “embarrassingly parallel” because these computations could be split up across multiple traditional compute clusters. However, research that involved MD simulations or lattice QCD required the fast Infiniband interconnects and the large amount of memory that Titan had, so these efforts were more likely to be approved. The lab did in fact want projects that utilized the whole machine at once to take maximum advantage of its capabilities. It’s just that oftentimes this wasn’t possible, and smaller jobs would be slotted into the “gaps” between the bigger ones.

sseagull 2y ago

You are generally correct, however there are workloads that do use larger portions of a supercomputer that wouldn't be feasible on smaller systems. Also, I guess I'm not sure what you mean by "smaller systems that focus more on power/cost/space". A proper queueing system generally efficiently allocates the resources of a large supercomputer to smaller tasks, while also making larger tasks possible in the first place. And I imagine there's somewhat an efficiency of scale in a large installation like this. There are, of course, many many smaller supercomputers, such as at most medium to large universities. But even those often have 10-50k cores or so. (In general, efficiency is a consideration when building/running, but not of using. Scientists want the most computational power they can get, power usage be damned :) ) edit: A related topic is capacity vs. capability: https://en.wikipedia.org/wiki/Supercomputer#Capability_versus_capacity https://en.wikipedia.org/wiki/Supercomputer#Capability_versu...

tkuraku 2y ago

While in general there can be many smaller workloads running in parallel. However, periodically the whole supercomputer can be reserved for a "hero" run.

deleted 2y ago
[deleted]

bitfilped 2y ago

Most of the time yes, HPC systems are shared among many users. Sometimes though the whole system (or near it) will be used in a single run. These are sometimes referred to as "hero runs" and while they're more common for benchmarking and burn-ins there are some tightly-coupled workloads that perform well in that style of execution. It really depends on a number of factors like the workloads being run, the number of users, and what the primary business purpose of the HPC resource is. Sites that have to run both types of jobs will typically allow any user to schedule jobs most of the time but then pre-reserve blocks of time for hero runs to take place where other user jobs are held until the primary scheduled run is over.

Harmohit 2y ago
Thanks for the reply! Can you give some examples of these "hero runs"?
- magicalhippo 2y ago
  At our university, at least when I studied there some 15 years ago, the whole cluster was occupied doing weather predictions each and every night. No point in staying up waiting for a job, it'd get rescheduled in the early morning at best. It wasn't the largest cluster around, IIRC 768 quad-core nodes, but I'm sure the meteorological department would find a way to utilize any extra capacity, so still requiring the whole thing all night.
  - moomin 2y ago
    Feels like this might be an invariant in computer science. I once worked on an IBM/360. The majority of the compute was taken up by a single person who did weather simulation. An IBM/360 has laughably less compute than your phone.
    - phinnaeus 2y ago
      It might still have produced a better weather prediction than my phone, though /s
    - nicklecompte 2y ago
      "Computational fluid dynamics is hard" seems like a true bona fide CS invariant.
    - Keyframe 2y ago
      ENIAC ran weather as well. It's a kind of numerical simulation that will always be in need of compute cycles. We'll need more of it as we go deeper into climate change.

alephnerd 2y ago

> my understanding correct Yes > why is it important to build supercomputers with more and more compute A mix of - research in distributed systems (there are plenty of open questions in Concurrency, Parallelization, Computer Architecture, etc) - a way to maintain an ecosystem of large vendors (Intel, AMD, Nvidia and plenty of smaller vendors all get a piece of the pie to subsidize R&D) - some problems are EXTREMELY computationally and financially expensive, so they require large On-Prem compute capabilities (eg. Protein folding, machine learning when I was in undergrad [DGX-100s were subsidized by Aurora], etc) - some problems are extremely sensitive for national security reasons and it's best to keep all personnel in a single region (eg. Nuclear simulations, turbine simulations, some niche ML work, etc) In reality you need to do both, and planners know this fact, and have known this fact for decades

trueismywork 2y ago

Bigger systems when utilized at 100% are more efficient than multiple smaller systems when utilized at 100%, in terms of engineering work, software, etc. But also, bigger systems have more opportunities to achieve higher utilization than smaller systems due to the dynamics of bin packing problem.

prpl 2y ago

there’s also Bell Prize submissions, which is the only time some machines get completely reserved

kkielhofner 2y ago

I have a project on Frontier. Generally these systems (including Frontier) use slurm[0] for scheduling and workload management. The OLCF Frontier user guide[1] has some information on scheduling and Frontier specific quirks (very minor). Current status of jobs on Frontier: [kkielhofner@login11.frontier ~]$ squeue -h -t running -r | wc -l 137 [kkielhofner@login11.frontier ~]$ squeue -h -t pending -r | wc -l 1016 The running jobs are relatively low because there are some massive jobs using a significant number of nodes ATM. [0] - https://slurm.schedmd.com/documentation.html https://slurm.schedmd.com/documentation.html [1] - https://docs.olcf.ornl.gov/systems/frontier_user_guide.html https://docs.olcf.ornl.gov/systems/frontier_user_guide.html EDIT: I give up on HN code formatting

SushiHippie 2y ago
> EDIT: I give up on HN code formatting Just FYI: https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc > Text after a blank line that is indented by two or more spaces is reproduced verbatim. (This is intended for code.) [kkielhofner@login11.frontier ~]$ squeue -h -t running -r | wc -l 137 [kkielhofner@login11.frontier ~]$ squeue -h -t pending -r | wc -l 1016
- kkielhofner 2y ago
  Yeah I've seen that but was annoyed by not being able to just use backticks like everywhere else. Oh the irony of using Frontier but not "understanding" HF formatting ;).

dekhn 2y ago

There's many variables that go into supercomputers, of which "company/country propaganda" is just one of them. Supercomputer admins would love to have a single code that used the whole machine, both the compute elements and the network elements, at close to 100%. In fact they spend a significant fraction on network elements to unblock the compute elements, but few codes are really so light on networking that the program scales to the full core count of the machine. So, instead they usually have several codes which can scale up to a significant fraction of the machine and then backfill with smaller jobs to keep the utilization up (because the acquisition cost and the running cost are so high). Supercomputers have limited utility- beyond country bragging rights, only a few problems really justify spending this kind of resource. I intentionally switched my own research in molecular dynamics away from supercomputers (where I'd run one job on 64-128 processors for a 96X speedup) to closet cllusters, where I'd run 128 indpendent jobs for a 128X speedup, but then have to do a bunch of post-processing to make the results comparable to the long, large runs on the supercomputer (https://research.google/pubs/cloud-based-simulations-on-google-exacycle-reveal-ligand-modulation-of-gpcr-activation-pathways/ https://research.google/pubs/cloud-based-simulations-on-goog...). I actually was really relieved when my work no longer depended on expensive resources with little support, as my scientific productivity went up and my costs went way down. I feel that supercomputers are good at one thing: if you need to make your country's flagship submarine about 10% faster/quieter than the competition.

alephnerd 2y ago
Ever used GPT-3, DALL-E, or other LLMs? The GPUs used to train them only existed because the DoE explicitly worked with Nvidia on a decade-long roadmap for delivery in it's various supercomputers, and would often work in tandem with private sector players to coordinate purchases and R&D (for example, protein folding and just about every Big Pharma company). Hell, the only reason AMD EPYC exists is for the same reason.
- dekhn 2y ago
  Yes. In my computational history I have: used the largest non-classified DOE supercomputers, built my own modest closet clusters, developed an embarassingly parallel computing system using Google's idle prod cycles, and helped debug training of LLMs when I worked on the TPU team at Google. I work for big pharma now (and my phd is in biophysics) and I'm also comfortable with other HPC domains. I know the DOE/Nvidia history quite well as the Chief Scientist of NVIDIA visited LBL around 2005(6? 7?) and talked about their new hardware they were just starting to build and sell, with the goal of getting them into supercomputers. We asked if they had double precision performance yet (because that was a must for many supercomputer jobs), but at the time, nvidia DP was still lagging SP (I guess it still does?) and we also quibbled about their non-compliance with some esoteric details in IEEE 754. The best part of the whole talk was when he walked us through the idea of visualizing our operations by drawing the matrices as textures, because you can easily see the NaNs- they render as nvidia Green! I left DOE (Berkeley Lab) shortly after to work in industry because it was clear that ML wasn't going to be innovated in the government labs.

ThinkBeat 2y ago

From my experience they are running whole cluster dedicated jobs quite frequently. Climate models can use whatever resources they get, nuclear weapons modelling esp for old warheads can use a lot.

MaxikCZ 2y ago
What is being calculated with nuclear weapons? I understand it must have been computationally expensive to get them working, but once completed, what is there left to calculate?
- Majromax 2y ago
  I don't work in this area, but think about all of the variables that go into warhead maintenance. Your single supercomputer simulation can show that the warhead should work as-designed, but what will happen after the Plutonium pit has been sitting for a decade, slowly decaying? Will satisfactory implosion still happen if storage conditions slightly change the performance of the conventional explosives? Since modern warheads are all fusion-type warheads, there's also the fusion stage to consider with even more highly classified top-secret sauce. It appears that the conditions for fusion are triggered by radiation pressure, and that likely makes things even more complicated. Now, you need not just a successful supercritical fission event, but one of the right shape(?), timing, and interaction with other secret-sauce materials that might have their own degradation curves. So, rather than simulate one design, now you need to simulate hundreds to thousands to explore the full decay-over-time space. Getting the answer wrong means either very expensive premature warhead refurbishments or a nuclear stockpile that wouldn't work properly.
- bargle0 2y ago
  You can’t test idle weapons to make sure they still go boom in real life, so you have to simulate it.
- ThinkBeat 2y ago
  The original nuclear powers are accumulating really old atomic bombs / rockets. We dont know for sure what is going on inside the warheads. (or possibly a sub section of the war head) Cracking them open to have a look may not be a good idea. but leaving them alone for another few decades might not be wise either. Funny fact: A lot of the nuclear weapons that have been destroyed, have removed them from bombs or rockets. But in many case the warheads were moved into storage. Ready to slap them on a rocket if that should become needed.

Xcelerate 2y ago

> There is no singular workload that takes up the entire or most of the resources of a supercomputer. I performed molecular dynamics simulations on the Titan supercomputer at ORNL during grad school. At the time, this supercomputer was the fastest in the world. At least back then around 2012, ORNL really wanted projects that uniquely showcased the power of the machine. Many proposals for compute time were turned down for workloads that were “embarrassingly parallel” because these computations could be split up across multiple traditional compute clusters. However, research that involved MD simulations or lattice QCD required the fast Infiniband interconnects and the large amount of memory that Titan had, so these efforts were more likely to be approved. The lab did in fact want projects that utilized the whole machine at once to take maximum advantage of its capabilities. It’s just that oftentimes this wasn’t possible, and smaller jobs would be slotted into the “gaps” between the bigger ones.