4 ms·
Another place to look for similar workflows is the HPC world. HPC schedulers have been doing resource requirement based scheduling for forever. Many of those th
by mbreese 3y ago
Another place to look for similar workflows is the HPC world. HPC schedulers have been doing resource requirement based scheduling for forever. Many of those those keep the scheduling state centralized though. Instead of asking the workers “who can fulfill these requirements”, the coordinator process keeps track of the state of what resources are still available for each worker. If you have a task that needs 12GB of RAM, 4 processors, and is in us-west, the coordinator already knows the answer. So instead of N^2 messages, you just have two (submit work and ack).
The major reason why HPC keeps this centrally is that it makes time based scheduling easier too. In general, you have more work than nodes and each job only has a finite run time. This lets the scheduler do fun things like backfilling jobs and more coordinated scheduling (such as keeping an entire rack idle for power savings unless necessary).
- dangoodmanUT 3y agoYeah, the problem with central is that it can get out of sync, but for tracking time it's definitely superior
- mbreese 3y agoTrue… state can get out of sync. But in general, I think this can be (and is?) handled with a heartbeat process. The compute nodes notify the central server of their status every X minutes (or get polled by the central node). Or even when the central server sends a job out, the state of the compute node could be queried as opposed to just assumed. In general, you’re going to have something like this running anyway just to keep track of nodes that die unexpectedly.
- gregod 3y agoSlurm [1] is a open-source HPC scheduler thats a good example. Its featureful and (surprisingly) easy to run if no user accounting (i.e., accounting for individual computation budgets) is needed. [1]: https://slurm.schedmd.com/documentation.html https://slurm.schedmd.com/documentation.html