3 ms·
The idea for cluster support has so far been to implement connectors for resource managers like SLURM, and basically keeping scipipe as an orchestrator. That i
by samuell 11y ago
The idea for cluster support has so far been to implement connectors for resource managers like SLURM, and basically keeping scipipe as an orchestrator.
That is, for multi-node jobs. As long as you can stay within the 16-32 or so cores on a typical HPC node (in our cluster at least), scipipe should be great for that. I think some means of simple resource management (to not overstress these 16-32 cores) is needed, but that can be done in a simple way by e.g. using a central go-routine that lends out "the right to use a cpu core" on demand.
Thanks for interesting feedback. I will think about this!