6 ms·

> especially when the community has little or no skill at building NG Internet HPC systems I would argue that the community of people who actually have the ski

> especially when the community has little or no skill at building NG Internet HPC systems I would argue that the community of people who actually have the skills to take advantage of the interconnects in a classic HPC system is vanishingly small, and in consequence we've overbuilt them on an epic scale. Allow me to vent. I had the good fortune to have a login on a "petascale" HPC system, and access to an allocation of hours. The /scratch filesystem would fail weekly, which killed everybody's jobs. If you had a big run going when /scratch failed, you lost everything. Scratch failed so much because the models that were being used often did wildly inappropriate amounts of file IO --- debugging print statements, detailed intermediate calculations, excessively verbose output --- that worked all right in development but when run in parallel brought the filesystem to its knees. Furthermore, the login nodes were almost unusably slow because of all the Python and Perl post-processing scripts running on them. This isn't even a matter of users being cheap with their hours --- post-processing would have been a tiny fraction of their allocations. Instead, it's that many of them gave no thought at all to how the post-processing might be structured and run through the batch scheduler, and saw no downside to abusing the login nodes for that purpose. In conclusion, I can attest to at least one HPC system that was badly mismatched to its users' needs and level of sophistication, despite allocations of hours being awarded only to a small number of researchers from across the country through a highly competitive process. Building these things serves national and institutional pride far more than any utilitarian interest.

khm 11y ago
Ah, a Cray. Buck up; there are better supercomputers.
- sevensor 11y ago
  It was, in fact. How did you infer that?
  - ams6110 11y ago
    /scratch on a Cray is (last I used one) is a Lustre filesystem. It is the only file storage available to the compute nodes, which have no local storage of their own. No spinning disk, no SSD. So if the code is written such that it makes frequent small writes (e.g. it's peppered with print statements), the lustre nodes get hammered by all the compute nodes and become the bottleneck and they will eventually fall over.
    - sevensor 11y ago
      Interesting! This was exactly the case on the system I used. I didn't realize Cray was the only vendor who went the no-local-storage route.
      - khm 11y ago
        They're not. They're just the only ones who insist on doing it with a filesystem that can't handle the load. The other problem is that their interconnect is relatively fragile. It's comparatively easy to crash the entire network, at which time your filesystem goes away and processing stops. But thanks to Lustre, even when it's good, it's bad.
dekhn 11y ago
You're describing an exceptionally poorly built and used system. That said, it's not inconsistent with what I've seen as well. My claim is that the design of classic interconnects is a big waste of money, because only a few codes need it, yet the cost dominates (>50%) of the cluster. I've learned, from years of studying Google's papers, that there are better ways to build code that communicates, and those mechanisms are much easier to teach to scientists and computer scientists than MPI.
- rwallace 11y ago
  That's true for many purposes, but is it true for physical simulations? Don't they need communication on every time step?
  - dekhn 11y ago
    I would say no. Here is my argument: when I worked for DOE, everybody told me I had to run my MD simulations on a super computer using all the processors, and I would judged on my parallel efficiency. This meant using a code that used MPI to communicate at every (or every N) timesteps. I asked, instead, "Why not just run N independent simulations, and pool the results?" In this case, you run an M-thread simulation on each machine (where M = number of cores on the machine) with no internode communication at all except to read input files and write output files. The short answer is, that approach works just fine, but the DOE supercomputer people won't let you run embarassingly parallel codes because they already spent money on the interconnect to run tightly coupled codes. In reponse to this, I went to Google, built Exacycle (loosely coupled HPC) and published this well-cited paper: http://www.ncbi.nlm.nih.gov/pubmed/24345941 http://www.ncbi.nlm.nih.gov/pubmed/24345941 which in my opinion put the last nail in the coffin of DOE-style physics simulations for molecular dynamics. That said, there are systems which are so large you can't practically simulate a single instance of the system on a single machine, so you have to partition. Simulating the ribosome is a nice example. However, simulating the ribosome currently provides no valuable scientific data except to tell us that we have major problems with our simulation systems (force field errors, missing QM, electrostatic approximations,e tc).
lorenzhs 11y ago
I would argue that many of the failures you describe are due to a lack of libraries that can exploit these systems. You can't expect every biologist to be an expert in distributed computing. The HPC and algorithms communities failed to consider the users of these systems and happily tinkered in their academic niches. All the while, what was actually needed were easy to use libraries that allowed non-experts to benefit from the advances made. (Also, the algorithms community has lost interest in distributed memory computing). The abuse of /scratch and the login nodes sounds like a classical mixture of not knowing, not caring, and limited time. That isn't something that has a technical solution.
- sevensor 11y ago
  I completely agree. My point is, the workload was lots of jobs using 100 nodes or less and doing massive amounts of file IO, while the cluster was essentially designed to run very large physics simulations, one at a time. This was a cluster that didn't need to exist.
pinewurst 11y ago
The problem you were seeing wasn't the supercomputer per se, it was the sheer awfulness of Lustre.

khm 11y ago

Ah, a Cray. Buck up; there are better supercomputers.

sevensor 11y ago
It was, in fact. How did you infer that?
- ams6110 11y ago
  /scratch on a Cray is (last I used one) is a Lustre filesystem. It is the only file storage available to the compute nodes, which have no local storage of their own. No spinning disk, no SSD. So if the code is written such that it makes frequent small writes (e.g. it's peppered with print statements), the lustre nodes get hammered by all the compute nodes and become the bottleneck and they will eventually fall over.
  - sevensor 11y ago
    Interesting! This was exactly the case on the system I used. I didn't realize Cray was the only vendor who went the no-local-storage route.
    - khm 11y ago
      They're not. They're just the only ones who insist on doing it with a filesystem that can't handle the load. The other problem is that their interconnect is relatively fragile. It's comparatively easy to crash the entire network, at which time your filesystem goes away and processing stops. But thanks to Lustre, even when it's good, it's bad.

dekhn 11y ago

You're describing an exceptionally poorly built and used system. That said, it's not inconsistent with what I've seen as well. My claim is that the design of classic interconnects is a big waste of money, because only a few codes need it, yet the cost dominates (>50%) of the cluster. I've learned, from years of studying Google's papers, that there are better ways to build code that communicates, and those mechanisms are much easier to teach to scientists and computer scientists than MPI.

rwallace 11y ago
That's true for many purposes, but is it true for physical simulations? Don't they need communication on every time step?
- dekhn 11y ago
  I would say no. Here is my argument: when I worked for DOE, everybody told me I had to run my MD simulations on a super computer using all the processors, and I would judged on my parallel efficiency. This meant using a code that used MPI to communicate at every (or every N) timesteps. I asked, instead, "Why not just run N independent simulations, and pool the results?" In this case, you run an M-thread simulation on each machine (where M = number of cores on the machine) with no internode communication at all except to read input files and write output files. The short answer is, that approach works just fine, but the DOE supercomputer people won't let you run embarassingly parallel codes because they already spent money on the interconnect to run tightly coupled codes. In reponse to this, I went to Google, built Exacycle (loosely coupled HPC) and published this well-cited paper: http://www.ncbi.nlm.nih.gov/pubmed/24345941 http://www.ncbi.nlm.nih.gov/pubmed/24345941 which in my opinion put the last nail in the coffin of DOE-style physics simulations for molecular dynamics. That said, there are systems which are so large you can't practically simulate a single instance of the system on a single machine, so you have to partition. Simulating the ribosome is a nice example. However, simulating the ribosome currently provides no valuable scientific data except to tell us that we have major problems with our simulation systems (force field errors, missing QM, electrostatic approximations,e tc).

lorenzhs 11y ago

I would argue that many of the failures you describe are due to a lack of libraries that can exploit these systems. You can't expect every biologist to be an expert in distributed computing. The HPC and algorithms communities failed to consider the users of these systems and happily tinkered in their academic niches. All the while, what was actually needed were easy to use libraries that allowed non-experts to benefit from the advances made. (Also, the algorithms community has lost interest in distributed memory computing). The abuse of /scratch and the login nodes sounds like a classical mixture of not knowing, not caring, and limited time. That isn't something that has a technical solution.

sevensor 11y ago
I completely agree. My point is, the workload was lots of jobs using 100 nodes or less and doing massive amounts of file IO, while the cluster was essentially designed to run very large physics simulations, one at a time. This was a cluster that didn't need to exist.

pinewurst 11y ago

The problem you were seeing wasn't the supercomputer per se, it was the sheer awfulness of Lustre.