4 ms·
> I also kind of object to the term HPC... with the exception of a small number of shared memory clusters used for physics simulations they're usually just a bu
by j33zusjuice 4y ago
> I also kind of object to the term HPC... with the exception of a small number of shared memory clusters used for physics simulations they're usually just a bunch of standard servers often with incredibly slow network storage. Nothing high performance about them.
I’m sure there are plenty of places who have production clusters equivalent to our “test cluster” that we ran in VMWare (which we only used to check version compatibility during upgrades), but in my experience working on an HPC team at a research university is that most universities are using real HPC clusters. They’re not all equally built and managed, but they have 50+ compute nodes using Infiniband (or equivalent) for interconnectivity between nodes, and to connect to the back end SAN, which runs a distributed, parallel file system (usually GPFS, sometimes Lustre, or BeeGFS).
Apptainer (formerly Singularity) is aimed at containerizing HPC workloads. You can build it with Docker commands, and then convert to Apptainer’s format, so it’s pretty easy to use. You don’t run Docker directly, in any case.
Correct that compatibility is a huge issue. It got ugly at the end of our EL6 cluster’s life. It didn’t run containers well, and our cluster was so entrenched in the old way of managing software (modules and Conda envs) that converting would have been a massive effort, and it may not have worked at all! There was a lot that had to get rescheduled or find a different place to run while we dealt with supply chain slowness.