4 ms·

You'd be surprised how actually backwards and primitive are the tools used in HPC. Take for instance the so-called workload managers, of which the most popular

You'd be surprised how actually backwards and primitive are the tools used in HPC. Take for instance the so-called workload managers, of which the most popular ones are Slurm, PBS, UGE, LSF. Only Slurm is really open-source, PBS has a community edition, the rest is proprietary stuff executed in the best traditions of enterprise software which locks you into using pathetically bad tools, ancient and backwards tech with crappy / nonexistent documentation and inept tech support. The interface between WLMs and the user who wants to use some resources is through submitting "jobs". These jobs can be interactive, but most often they are the so-called "batch jobs". A batch job is usually defined as... a Unix Shell script, where the comments are parsed to interpret those as instructions to the WLM. In the world with dozens of configuration formats... they chose to do this: embed configuration into Shell comments. Debugging job failures is a nightmare, mostly because WLM software has really poor quality of execution. Pathetic error reporting. Idiotic defaults. Everything is so fragile it falls apart if you just as much as look at it in the wrong way. Working with it reminds me the very early days of Linux, when sometimes things just won't build, or would segfault right after you've tried running them, and there wasn't much you could do beside spending days or weeks trying to debug it just to get some basic functionality going. When I have to deal with it, I feel kind of like in a steam-punk movie. Some stuff is really advanced, and then you find out that this advanced stuff is propped by some DIY retro nonsense you thought have died off decades ago. The advanced stuff is usually more on the side of hardware, while software is not keeping up with it for the most part.

StableAlkyne 3y ago
> Working with it reminds me the very early days of Linux The other cool thing about HPC is it is one of the last areas where multi-user Unix is used! At least, if you're using a university or NSF cluster that is! Only other place I really see multiple humans using the same machine is SDF or the Tildes
- victotronics 3y ago
  It's saturday afternoon. [login1 ~:3] who | cut -d ' ' -f 1 | sort -u | wc -l 41
bee_rider 3y ago
Having switched from LSF to slurm, I have to appreciate that the ecosystem is so bash-centric. Lots of re-use in the conversion. If I’d had to learn some kind of slurm-markup-language or slurmScript or find buttons in some SlurmWizard, it would have been a nightmare.
- crabbone 3y ago
  Oh LSF... I don't know if you know this. LSF is perhaps the only system alive today that I know of that uses literal patches as a means of software distribution. Fist time I saw it, I had a flashback to the times when I worked for HP, and they were making some huge SAP knock-off, and that system was so labor-intensive to deploy that their QA process involved actual patches. As in pre-release QA cycle involved installing the system, validating it (which could take a few weeks) and if it's not considered DoD, then the developers are given the final list of things they need to fix and those fixes would have to be submitted as patches (sometimes, literal diffs that need to be applied to the deployed system with the patch tool). This is, I guess, how the "patch version component" came to be in SemVer spec. It's kind of funny how lots of tools are using this component today for completely unrelated purposes... but yeah, LSF feels like the time is ticking there at a different pace :)
OPA100 3y ago
I've dug deeply into LSF in the last few years and it's like a car crash - you can't look away. It feels like something that started in the early unix days but was developed into perhaps the late 90s, but in reality LSF was only started in the 90s (in academia). As far as I can tell development all but stopped when IBM acquired it some ten years ago.
convolvatron 3y ago
HPC software is one area where we have arguably regressed in the last 30 years. Chapel is the only light I see in the darkness
- trentnelson 3y ago
  Want to elaborate more on Chapel? I’ve recently being tasked with integrating Chapel into our system and it’s quite interesting.
victotronics 3y ago
You do a lot of scare quotes. Do you have any suggestions on how things could be different? You need batch jobs because the scheduler has to wait for resources to be available. It's kinda like Tetris in processor/time space. (In fact, that's my personal "proof" that workload scheduling is NP-complete: it's isomorphic to Tetris.) And what's wrong with shell scripts? It's a lingua franca, generally accepted across scientific disciplines, cluster vendors, workload managers, .... Considering the complexity of some setups (copy data to node-local file systems; run multiple programs, post-process results, ... ) I don't see how you could set up things other than in some scripting language. And then unix shell scripts are not the worst idea. Debugging failures: yeah. Too many levels where something can go wrong, and it can be a pain to debug. Still, your average cluster processes a few million jobs in its lifetime. If more than a microscopic portion of that would fail, computing centers would need way more personnel than they have.
- crabbone 3y ago
  > And what's wrong with shell scripts? When used as configuration? Here are some things that are wrong: * Configuration forced into a single line makes writing long lines inconvenient (for example, if you want Slurm with Pyxis, and you need to specify the image name -- it will most likely not fit on the screen. * Oh, and since we mentioning Pyxis -- their image names have pound sign in them, and now you also need to figure out how to escape it, because for some reason if used literally it breaks the comments parser. * No syntax highlighting (because it's all comments). * No way to create more complex configuration, i.e. no way to have any types other than strings, no way to have variables, no way to have collections of things. * No way to reuse configuration (you have to copy it from one job file to another). I honestly don't even know what happens if you try to source a job configuration file from another job configuration. All in all, it's really hard to imagine a worse configuration format. This sounds like a solution from some sort of a code-golfing competition where the goal was to make it as bad as possible, while still retaining some shreds of functionality.
romanows 3y ago
I really like using Slurm, the documentation is great (https://slurm.schedmd.com https://slurm.schedmd.com) and the model is pretty straightforward, at least for the mostly-single-node jobs I used it for. You can launch a job(s) via command-line, config in Bash comments, REST APIs, linking to their library, and I think a few more ways. I found it pretty easy to setup and admin. Scaling in the cloud was way less developed when I used it, so I just hacked in a simple script that allowed scaling up and down based on the job queue size. What do you like better and for what use-case? Mine was for a group of researchers training models, and the feature I desired most was an approximately fair distribution of resources (cores, GPU hours, etc.).
_naj 3y ago
Well, working in HPC, in my view the reason is the users (researchers/scientists) just want to use the machine as they have used it in the past, and this has been going on for many years now. That makes sense, as it allows them to focus on the scientific questions, not on IT stuff. They keep on using the tooling with which the software package they use came, and in some cases that is already 30 years old.

StableAlkyne 3y ago

> Working with it reminds me the very early days of Linux The other cool thing about HPC is it is one of the last areas where multi-user Unix is used! At least, if you're using a university or NSF cluster that is! Only other place I really see multiple humans using the same machine is SDF or the Tildes

victotronics 3y ago
It's saturday afternoon. [login1 ~:3] who | cut -d ' ' -f 1 | sort -u | wc -l 41

bee_rider 3y ago

Having switched from LSF to slurm, I have to appreciate that the ecosystem is so bash-centric. Lots of re-use in the conversion. If I’d had to learn some kind of slurm-markup-language or slurmScript or find buttons in some SlurmWizard, it would have been a nightmare.

crabbone 3y ago
Oh LSF... I don't know if you know this. LSF is perhaps the only system alive today that I know of that uses literal patches as a means of software distribution. Fist time I saw it, I had a flashback to the times when I worked for HP, and they were making some huge SAP knock-off, and that system was so labor-intensive to deploy that their QA process involved actual patches. As in pre-release QA cycle involved installing the system, validating it (which could take a few weeks) and if it's not considered DoD, then the developers are given the final list of things they need to fix and those fixes would have to be submitted as patches (sometimes, literal diffs that need to be applied to the deployed system with the patch tool). This is, I guess, how the "patch version component" came to be in SemVer spec. It's kind of funny how lots of tools are using this component today for completely unrelated purposes... but yeah, LSF feels like the time is ticking there at a different pace :)

OPA100 3y ago

I've dug deeply into LSF in the last few years and it's like a car crash - you can't look away. It feels like something that started in the early unix days but was developed into perhaps the late 90s, but in reality LSF was only started in the 90s (in academia). As far as I can tell development all but stopped when IBM acquired it some ten years ago.

convolvatron 3y ago

HPC software is one area where we have arguably regressed in the last 30 years. Chapel is the only light I see in the darkness

trentnelson 3y ago
Want to elaborate more on Chapel? I’ve recently being tasked with integrating Chapel into our system and it’s quite interesting.

victotronics 3y ago

You do a lot of scare quotes. Do you have any suggestions on how things could be different? You need batch jobs because the scheduler has to wait for resources to be available. It's kinda like Tetris in processor/time space. (In fact, that's my personal "proof" that workload scheduling is NP-complete: it's isomorphic to Tetris.) And what's wrong with shell scripts? It's a lingua franca, generally accepted across scientific disciplines, cluster vendors, workload managers, .... Considering the complexity of some setups (copy data to node-local file systems; run multiple programs, post-process results, ... ) I don't see how you could set up things other than in some scripting language. And then unix shell scripts are not the worst idea. Debugging failures: yeah. Too many levels where something can go wrong, and it can be a pain to debug. Still, your average cluster processes a few million jobs in its lifetime. If more than a microscopic portion of that would fail, computing centers would need way more personnel than they have.

crabbone 3y ago
> And what's wrong with shell scripts? When used as configuration? Here are some things that are wrong: * Configuration forced into a single line makes writing long lines inconvenient (for example, if you want Slurm with Pyxis, and you need to specify the image name -- it will most likely not fit on the screen. * Oh, and since we mentioning Pyxis -- their image names have pound sign in them, and now you also need to figure out how to escape it, because for some reason if used literally it breaks the comments parser. * No syntax highlighting (because it's all comments). * No way to create more complex configuration, i.e. no way to have any types other than strings, no way to have variables, no way to have collections of things. * No way to reuse configuration (you have to copy it from one job file to another). I honestly don't even know what happens if you try to source a job configuration file from another job configuration. All in all, it's really hard to imagine a worse configuration format. This sounds like a solution from some sort of a code-golfing competition where the goal was to make it as bad as possible, while still retaining some shreds of functionality.

romanows 3y ago

I really like using Slurm, the documentation is great (https://slurm.schedmd.com https://slurm.schedmd.com) and the model is pretty straightforward, at least for the mostly-single-node jobs I used it for. You can launch a job(s) via command-line, config in Bash comments, REST APIs, linking to their library, and I think a few more ways. I found it pretty easy to setup and admin. Scaling in the cloud was way less developed when I used it, so I just hacked in a simple script that allowed scaling up and down based on the job queue size. What do you like better and for what use-case? Mine was for a group of researchers training models, and the feature I desired most was an approximately fair distribution of resources (cores, GPU hours, etc.).

_naj 3y ago

Well, working in HPC, in my view the reason is the users (researchers/scientists) just want to use the machine as they have used it in the past, and this has been going on for many years now. That makes sense, as it allows them to focus on the scientific questions, not on IT stuff. They keep on using the tooling with which the software package they use came, and in some cases that is already 30 years old.