There's probably a more unix-centric / idiomatic way to do it, but using GNU Parallel along with Redis made automating tasks to process tens of thousands of sys

There's probably a more unix-centric / idiomatic way to do it, but using GNU Parallel along with Redis made automating tasks to process tens of thousands of systems extremely easy, repeatable, pausable, restartable, etc. And the fact you can do atomic operations on queues prevented any potential worker collisions.