5 ms·

"If every computation were represented by a single grain of sand, you could fill up the Houston Astrodome with sand 350 times in a single second." There's a ne

"If every computation were represented by a single grain of sand, you could fill up the Houston Astrodome with sand 350 times in a single second." There's a new one. Can anyone comment on the utilization of these megacomputers? Do they have somewhere near 100% usage, with a queue extending weeks? Also, is all this computational power really... necessary? I've seen some intensely inefficient simulation code in my time.

Xcelerate 8y ago
Yes and yes. I used Titan for my research in grad school. There’s dozens of researchers waiting for as much compute power as is available. Tasks like quantum Monte Carlo simulation make extremely accurate chemical predictions but require a massive amount of compute power to do so. You may think, well just design a better algorithm, but we’re starting to run up against the limits of computational complexity. Further fundamental improvement for simulating fermion systems accurately would require quantum computing.
auvi 8y ago
Simulation of turbulence at very high Reynolds numbers is something that requires these mega computers. You can have a look at DOE's INCITE program awards to get an idea about these simulations: [0] http://www.doeleadershipcomputing.org/incite-awards/
mutagen 8y ago
Several years ago I had the opportunity to attend a non-classified project coordination conference / catchup meeting with some DoE scientists, among them attendees from LLNL and Los Alamos. I overheard discussion and some bragging about the number of cycles they had their simulation kernels at. At least some of them are dedicated to extracting every last bit of performance out of their code.
- darkmighty 8y ago
  I wonder what portion of time on those supercomputers is used in optimizing the code itself; and how much could be done. With those massive clusters you could afford testing in parallel trillions of mutations to your simulation kernels and prove the correctness of the fastest ones -- pruning should be extremely fast by finding counterexamples. Or even higher level architectural optimizations. Surely they could afford at least a few % of the total time on this pre-optimization (although the tools to achieve this automatically would need to be quite sophisticated!).
  - inteleng 8y ago
    What do you do when these mutations end up crashing constantly?
  - hedora 8y ago
    http://fftw.org http://fftw.org is optimized by doing massive parameter sweeps on each architecture (it only considers correct implementations). There are also a few “software synthesis” and “sketching” approaches that use a constraint solver to find all correct implementations of a high level spec, subject to some implementation pattern. Then they either try them all with brute force or pick the one that optimizes some objective function.
    - stochastic_monk 8y ago
      fftw isn't exactly developing an optimal kernel from scratch. It's testing a range of different methods and simply choosing the best for one's parameters. Facebook's Tensor Comprehensions framework, which generates CUDA kernels through a genetic algorithm, is closer to the sort of approach which would take greatest advantage of the hardware it's on.
- Fomite 8y ago
  Yeah, I was in a very HPC focused group run by ex-national lab people, and there were several PhDs worth of work going on at any time trying to speed things up. Compute, I/O...all of it.
xiii1408 8y ago
Yes, the utilization for these supercomputers sits upwards of 99%. There's a lot of research into supercomputer queueing that helps maintain this high throughput. Larger jobs usually take days to weeks to run. There are separate debug queues for smaller development and test jobs that prioritize latency over throughput, and this is a main source of inefficiency (but, as someone who develops on supercomputers, trust me, it's necessary to make them usable). As for the efficiency of the codes--there are a lot of barnacle-encrusted Fortran codes in astrophysics, chemistry, and elsewhere. Some of them probably contain plenty of inefficiency, but the cost of re-writing them is high. Oftentimes either the people writing the codes aren't interested in maintaining good software quality (because they're focused on the science) or the incentives aren't aligned to re-write code that's already established, but from the 80s (re-writing old code is generally not how one gets a Ph.D.). But they do very important things, like simulate nuclear fusion or the interaction of chemicals in the atmosphere that create weather patterns and storms, so it's very important to run them.
- wickberg 8y ago
  99% is a bit higher than most systems expect to run at - any mix of job sizes in the queue will tend to leave small gaps when nodes need to stay idle until jobs "fit" perfectly again. 95% is a pretty common target for these larger systems. There isn't necessarily a lot of research into packing these better - the basic algorithms have been unchanged for quite some time, and a lot more effort goes into deciding how to prioritize different groups that are sharing access into the same system.
  - quadruplebond 8y ago
    Most machines will allow for back filling if your jobs are short enough and small enough. But that will still leave some of the machine under utilized.
    - wickberg 8y ago
      I don't have any customer data on hand at the moment, but I'd roughly describe it as basic scheduling on strict priority order getting the system to 80+% usage, and then backfill boosting that up to 90%+. Careful backfill tuning, and care in defining the priority structure gets you to 95+%. 97% is the highest specific value I can recall for any of the larger sites with a heavily mixed workload, absent having a nearly infinite supply of short+small jobs at hand to use to fill those gaps. And a lot of users aren't trying to chase that - for these large scale "capability" systems the goal is to scale out as large as you can anyways, the smaller stuff is usually relegated to "capacity" systems elsewhere with a less expensive architecture. One thing that at least some schedulers can manage is the idea of a min+max runtime for a job, combined with a min+max node/cpu count. If you have users willing to 'scavenge' otherwise wasted time by running under such a regime that can put you closer to full usage.
neutronicus 8y ago
I use Edison (a DoE supercomputer), and getting through the queue there is kind of a nightmare.
- BooneJS 8y ago
  I was on the team that made the Aries interconnect. I hope all is well once your job is placed. :)
  - wickberg 8y ago
    If you're the one that came up with the idea of the "quiesce", I feel like you owe quite a few of us in the systems software community a beer or three...

Xcelerate 8y ago

Yes and yes. I used Titan for my research in grad school. There’s dozens of researchers waiting for as much compute power as is available. Tasks like quantum Monte Carlo simulation make extremely accurate chemical predictions but require a massive amount of compute power to do so. You may think, well just design a better algorithm, but we’re starting to run up against the limits of computational complexity. Further fundamental improvement for simulating fermion systems accurately would require quantum computing.

auvi 8y ago

Simulation of turbulence at very high Reynolds numbers is something that requires these mega computers. You can have a look at DOE's INCITE program awards to get an idea about these simulations: [0] http://www.doeleadershipcomputing.org/incite-awards/

mutagen 8y ago

Several years ago I had the opportunity to attend a non-classified project coordination conference / catchup meeting with some DoE scientists, among them attendees from LLNL and Los Alamos. I overheard discussion and some bragging about the number of cycles they had their simulation kernels at. At least some of them are dedicated to extracting every last bit of performance out of their code.

darkmighty 8y ago
I wonder what portion of time on those supercomputers is used in optimizing the code itself; and how much could be done. With those massive clusters you could afford testing in parallel trillions of mutations to your simulation kernels and prove the correctness of the fastest ones -- pruning should be extremely fast by finding counterexamples. Or even higher level architectural optimizations. Surely they could afford at least a few % of the total time on this pre-optimization (although the tools to achieve this automatically would need to be quite sophisticated!).
- inteleng 8y ago
  What do you do when these mutations end up crashing constantly?
- hedora 8y ago
  http://fftw.org http://fftw.org is optimized by doing massive parameter sweeps on each architecture (it only considers correct implementations). There are also a few “software synthesis” and “sketching” approaches that use a constraint solver to find all correct implementations of a high level spec, subject to some implementation pattern. Then they either try them all with brute force or pick the one that optimizes some objective function.
  - stochastic_monk 8y ago
    fftw isn't exactly developing an optimal kernel from scratch. It's testing a range of different methods and simply choosing the best for one's parameters. Facebook's Tensor Comprehensions framework, which generates CUDA kernels through a genetic algorithm, is closer to the sort of approach which would take greatest advantage of the hardware it's on.
Fomite 8y ago
Yeah, I was in a very HPC focused group run by ex-national lab people, and there were several PhDs worth of work going on at any time trying to speed things up. Compute, I/O...all of it.

xiii1408 8y ago

Yes, the utilization for these supercomputers sits upwards of 99%. There's a lot of research into supercomputer queueing that helps maintain this high throughput. Larger jobs usually take days to weeks to run. There are separate debug queues for smaller development and test jobs that prioritize latency over throughput, and this is a main source of inefficiency (but, as someone who develops on supercomputers, trust me, it's necessary to make them usable). As for the efficiency of the codes--there are a lot of barnacle-encrusted Fortran codes in astrophysics, chemistry, and elsewhere. Some of them probably contain plenty of inefficiency, but the cost of re-writing them is high. Oftentimes either the people writing the codes aren't interested in maintaining good software quality (because they're focused on the science) or the incentives aren't aligned to re-write code that's already established, but from the 80s (re-writing old code is generally not how one gets a Ph.D.). But they do very important things, like simulate nuclear fusion or the interaction of chemicals in the atmosphere that create weather patterns and storms, so it's very important to run them.

wickberg 8y ago
99% is a bit higher than most systems expect to run at - any mix of job sizes in the queue will tend to leave small gaps when nodes need to stay idle until jobs "fit" perfectly again. 95% is a pretty common target for these larger systems. There isn't necessarily a lot of research into packing these better - the basic algorithms have been unchanged for quite some time, and a lot more effort goes into deciding how to prioritize different groups that are sharing access into the same system.
- quadruplebond 8y ago
  Most machines will allow for back filling if your jobs are short enough and small enough. But that will still leave some of the machine under utilized.
  - wickberg 8y ago
    I don't have any customer data on hand at the moment, but I'd roughly describe it as basic scheduling on strict priority order getting the system to 80+% usage, and then backfill boosting that up to 90%+. Careful backfill tuning, and care in defining the priority structure gets you to 95+%. 97% is the highest specific value I can recall for any of the larger sites with a heavily mixed workload, absent having a nearly infinite supply of short+small jobs at hand to use to fill those gaps. And a lot of users aren't trying to chase that - for these large scale "capability" systems the goal is to scale out as large as you can anyways, the smaller stuff is usually relegated to "capacity" systems elsewhere with a less expensive architecture. One thing that at least some schedulers can manage is the idea of a min+max runtime for a job, combined with a min+max node/cpu count. If you have users willing to 'scavenge' otherwise wasted time by running under such a regime that can put you closer to full usage.

neutronicus 8y ago

I use Edison (a DoE supercomputer), and getting through the queue there is kind of a nightmare.

BooneJS 8y ago
I was on the team that made the Aries interconnect. I hope all is well once your job is placed. :)
- wickberg 8y ago
  If you're the one that came up with the idea of the "quiesce", I feel like you owe quite a few of us in the systems software community a beer or three...