Its interesting to see how other industries tackle the same problem. VFX has essentially the same problem to google: a huge bunch of tasks that need to perform

Its interesting to see how other industries tackle the same problem. VFX has essentially the same problem to google: a huge bunch of tasks that need to perform all at once. However VFX only tend to have one data center, so they don;t need or want clustered scheduler. https://github.com/mikrosimage/openrendermanagement https://github.com/mikrosimage/openrendermanagement, Alfred and tractor from pixar, and framestore's FQ (which is faster and more efficient than Borg at job dispatch. ) Are a few good example of task management.