3 ms·
When joining a small company supporting the engineers of the HPC of a large car manufacturer, I was surprised to see so many in-house developed scripts around t
by guenthert 3y ago
When joining a small company supporting the engineers of the HPC of a large car manufacturer, I was surprised to see so many in-house developed scripts around the scheduler (LSF). Only much later, when playing myself on a private miniature cluster with SLURM, I noticed that different versions of the scheduler software were generally incompatible to each other, i.e. one couldn't use one inside the cluster and another on external client machine. Hence the need for glue software to inject jobs into the scheduler from outside and retrieve the results later on (IMHO devaluating the scheduler).
I would have thought, that after some 30 years of high performance distributed computing, the requirements were well known and at least the protocol for command and data exchange could be fixed. Apparently not so.
- wickberg 3y agoCluster authentication actually remains the sore point for all interactions - for most systems the least-common denominator remains whether one can SSH into a login node. At which point the main job commands - sbatch/squeue - are pretty stable. There have been some attempts to standardize basic job management APIs in the past - DRMAA being one noteworthy example. Although DRMAA v2 was only ever implemented by Grid Engine, and is effectively an lightly-abstracted version of their internal APIs, that has never really seen first-class adoption by Slurm/PBS/LSF. For Slurm, the REST API is meant to be the way forward. It punts the authentication problem to, potentially, anything the admins may care to wire up through an Apache / NGINX proxy. And the basic job submission and status APIs have stablizied to the point that a client application should be able to consume nearly any version going forward.