6 ms·

Slurm is not great. I found recently that when submitting an array of jobs, it will reject you if the upper bound is too high, regardless of the size of the arr

Slurm is not great. I found recently that when submitting an array of jobs, it will reject you if the upper bound is too high, regardless of the size of the array. I filed a bug but received no response.

bonoboTP 7y ago
That's according to documentation (https://slurm.schedmd.com/job_array.html https://slurm.schedmd.com/job_array.html). > Note that the minimum index value is zero and the maximum value is a Slurm configuration parameter (MaxArraySize minus one). Note that they say "the maximum value is a Slurm configuration parameter", not "the maximum value minus minimum value is a Slurm configuration parameter". The name of the parameter is confusing though, I admit. Also, they prioritize paying customers when responding to bug tickets. Slurm is free software but if you want to interact with them, you need to pay. Nevertheless, for simple use cases (just scheduling a bunch of jobs) it's very comfortable to use. If you get into more complex territory (like array jobs), you'll need to read more docs. And it scales very nicely from just one workstation up to the largest HPC clusters on the planet with complex accounting, multi-node jobs and so on.
- rgovostes 7y ago
  It is documented, but it strikes me as simply being a lazy implementation. The consequence of this is that you cannot submit jobs 0-999, then 1000-1999, etc. You must submit 0-999, 0-999, ..., with an additional argument. It doesn't really prevent you from doing anything, it just gets in your way. (The name MaxArraySize definitely makes it seem more like this is a bug than intended, albeit documented, behavior.) I looked into submitting a patch but unfortunately MaxArraySize is used in more places than I would have guessed, and lost interest in trying to test all of them.
  - bonoboTP 7y ago
    We're talking about installing Slurm for one's own use, so you can just change MaxArraySize in the configuration as an easy solution.

bonoboTP 7y ago

That's according to documentation (https://slurm.schedmd.com/job_array.html https://slurm.schedmd.com/job_array.html). > Note that the minimum index value is zero and the maximum value is a Slurm configuration parameter (MaxArraySize minus one). Note that they say "the maximum value is a Slurm configuration parameter", not "the maximum value minus minimum value is a Slurm configuration parameter". The name of the parameter is confusing though, I admit. Also, they prioritize paying customers when responding to bug tickets. Slurm is free software but if you want to interact with them, you need to pay. Nevertheless, for simple use cases (just scheduling a bunch of jobs) it's very comfortable to use. If you get into more complex territory (like array jobs), you'll need to read more docs. And it scales very nicely from just one workstation up to the largest HPC clusters on the planet with complex accounting, multi-node jobs and so on.

rgovostes 7y ago
It is documented, but it strikes me as simply being a lazy implementation. The consequence of this is that you cannot submit jobs 0-999, then 1000-1999, etc. You must submit 0-999, 0-999, ..., with an additional argument. It doesn't really prevent you from doing anything, it just gets in your way. (The name MaxArraySize definitely makes it seem more like this is a bug than intended, albeit documented, behavior.) I looked into submitting a patch but unfortunately MaxArraySize is used in more places than I would have guessed, and lost interest in trying to test all of them.
- bonoboTP 7y ago
  We're talking about installing Slurm for one's own use, so you can just change MaxArraySize in the configuration as an easy solution.