Personally, I've cobbled something together using this: https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/ https://www.2ndqua

Personally, I've cobbled something together using this: https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/ https://www.2ndquadrant.com/en/blog/what-is-select-skip-lock... Storing the queue state and task results in postgres makes it easy to integrate with workers in different languages, but you need to write the library code to query and lock a free task to process. I'm not sure how well this would scale for 10 million tasks in "a short period". It works fine for me running the database and multiple workers on a single machine with around 100k tasks that are scheduled and processed every week or two. > Features most important to me are multiple retries, restarting workers that are not responding, ability to monitor status of the queue and workers. Some of these concerns might not be the responsibility of the job processing system: you might just need to set up some monitoring and health checks to restart services or machines if they stop responding