4 ms·

A month in the lab saves a day in the library.

semi-extrinsic 7y ago
I also have the feeling that this should be "easy" with a traditional scheduler. But could you (or someone else) point me in the direction of something that's easy to set up for a quad-GPU machine? I spent two days trying to get SLURM to work, at one point. Currently I'm running with GNU Parallel, but it's much less flexible of course.
- moonbug 7y ago
  try PBS Pro.
- petschge 7y ago
  My favourite scheduler is still "Son of Grid Engine", the open source continuation of the "Sun Grid Engine". I never had much trouble setting it up, but it is not really maintained at this point. PBS as suggested by others never worked well for me. But Slurm didn't give me much trouble, certainly not 2 days worth.
  - gravypod 7y ago
    Son of grid engine definitely shows its age but I never had troubles submitting jobs to my school's cluster.
- sgt101 7y ago
  This is fundamentally why people buy DGX machines from Nvidia. It's preloaded. You have to figure out how to use them though.
- bonoboTP 7y ago
  Slurm works well even for a single machine. It does require some configuration but you only need to do it once and then it's very robust. The packages are available in standard repositories (like Ubuntu's) and there's a web-based configuration generator available. And if you ever want to add a second machine or more, it will be very easy to expand. What stopped you from using Slurm?
  - ExpectationMax 7y ago
    Installing Slurm requires root access to the server/workstation, which is not always available. In academia the management of computational resources is often the responsibility of a separate IT unit which is not part of the research group itself. Further the IT unit might not be willing to implement desired changes to the server infrastructure or might take _extremely_ long to do so. Often the setup of a scheduler is also significantly more complicated in this context, for example due to authentication via a University LDAP or oddities of the server setup.
    - bonoboTP 7y ago
      Luckily in our academic research group we administer our own workstations and servers ourselves. Has downsides as well of course, but at least we can fix or change anything we want hardware or software-wise. I can imagine it would be excruciating to explain our requirements to an external IT department and then go back and forth clarifying what we want, make them learn stuff they'd otherwise not need, convince them that we really need it, wait until they have time... The downside is that things are not "professionally set up" (we're computer scientists but not pro admins), but at least if there's a deadline and something isn't working, we'll surely fix it because we have skin in the game.
  - rgovostes 7y ago
    Slurm is not great. I found recently that when submitting an array of jobs, it will reject you if the upper bound is too high, regardless of the size of the array. I filed a bug but received no response.
    - bonoboTP 7y ago
      That's according to documentation (https://slurm.schedmd.com/job_array.html https://slurm.schedmd.com/job_array.html). > Note that the minimum index value is zero and the maximum value is a Slurm configuration parameter (MaxArraySize minus one). Note that they say "the maximum value is a Slurm configuration parameter", not "the maximum value minus minimum value is a Slurm configuration parameter". The name of the parameter is confusing though, I admit. Also, they prioritize paying customers when responding to bug tickets. Slurm is free software but if you want to interact with them, you need to pay. Nevertheless, for simple use cases (just scheduling a bunch of jobs) it's very comfortable to use. If you get into more complex territory (like array jobs), you'll need to read more docs. And it scales very nicely from just one workstation up to the largest HPC clusters on the planet with complex accounting, multi-node jobs and so on.
Topolomancer 7y ago
Did you actually _look_ at the repo? The author does not claim that this is the only way of doing it; and in fact, they seem to be quite aware of the existence of other schedulers. This is just a very simple one that does not require a lot of overhead and installation time. In fact, I would go as far as to say that this scheduler is written in the spirit of Unix: make a task easy and provide a simple, chainable interface.
deleted 7y ago
[deleted]
dang 7y ago
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

I also have the feeling that this should be "easy" with a traditional scheduler. But could you (or someone else) point me in the direction of something that's easy to set up for a quad-GPU machine? I spent two days trying to get SLURM to work, at one point. Currently I'm running with GNU Parallel, but it's much less flexible of course.

moonbug 7y ago
try PBS Pro.
petschge 7y ago
My favourite scheduler is still "Son of Grid Engine", the open source continuation of the "Sun Grid Engine". I never had much trouble setting it up, but it is not really maintained at this point. PBS as suggested by others never worked well for me. But Slurm didn't give me much trouble, certainly not 2 days worth.
- gravypod 7y ago
  Son of grid engine definitely shows its age but I never had troubles submitting jobs to my school's cluster.
sgt101 7y ago
This is fundamentally why people buy DGX machines from Nvidia. It's preloaded. You have to figure out how to use them though.
bonoboTP 7y ago
Slurm works well even for a single machine. It does require some configuration but you only need to do it once and then it's very robust. The packages are available in standard repositories (like Ubuntu's) and there's a web-based configuration generator available. And if you ever want to add a second machine or more, it will be very easy to expand. What stopped you from using Slurm?
- ExpectationMax 7y ago
  Installing Slurm requires root access to the server/workstation, which is not always available. In academia the management of computational resources is often the responsibility of a separate IT unit which is not part of the research group itself. Further the IT unit might not be willing to implement desired changes to the server infrastructure or might take _extremely_ long to do so. Often the setup of a scheduler is also significantly more complicated in this context, for example due to authentication via a University LDAP or oddities of the server setup.
  - bonoboTP 7y ago
    Luckily in our academic research group we administer our own workstations and servers ourselves. Has downsides as well of course, but at least we can fix or change anything we want hardware or software-wise. I can imagine it would be excruciating to explain our requirements to an external IT department and then go back and forth clarifying what we want, make them learn stuff they'd otherwise not need, convince them that we really need it, wait until they have time... The downside is that things are not "professionally set up" (we're computer scientists but not pro admins), but at least if there's a deadline and something isn't working, we'll surely fix it because we have skin in the game.
- rgovostes 7y ago
  Slurm is not great. I found recently that when submitting an array of jobs, it will reject you if the upper bound is too high, regardless of the size of the array. I filed a bug but received no response.
  - bonoboTP 7y ago
    That's according to documentation (https://slurm.schedmd.com/job_array.html https://slurm.schedmd.com/job_array.html). > Note that the minimum index value is zero and the maximum value is a Slurm configuration parameter (MaxArraySize minus one). Note that they say "the maximum value is a Slurm configuration parameter", not "the maximum value minus minimum value is a Slurm configuration parameter". The name of the parameter is confusing though, I admit. Also, they prioritize paying customers when responding to bug tickets. Slurm is free software but if you want to interact with them, you need to pay. Nevertheless, for simple use cases (just scheduling a bunch of jobs) it's very comfortable to use. If you get into more complex territory (like array jobs), you'll need to read more docs. And it scales very nicely from just one workstation up to the largest HPC clusters on the planet with complex accounting, multi-node jobs and so on.

Topolomancer 7y ago

Did you actually _look_ at the repo? The author does not claim that this is the only way of doing it; and in fact, they seem to be quite aware of the existence of other schedulers. This is just a very simple one that does not require a lot of overhead and installation time. In fact, I would go as far as to say that this scheduler is written in the spirit of Unix: make a task easy and provide a simple, chainable interface.

deleted 7y ago

[deleted]

dang 7y ago

"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html