it might be relevant to say how much throughput you need or other factors, such as if your problem in inherently concurrent (reacting to events outside your con

it might be relevant to say how much throughput you need or other factors, such as if your problem in inherently concurrent (reacting to events outside your control) or if you actually just want to do parallel processing. for example, "10k+ jobs" sounds like a large number, but depending on throughout perhaps it is trivial. i have a hobby project to fetch data from external sources and store the results in a database. this has about 70k different jobs defined. each job is scheduled to be run at some frequency. i run everything on a single physical box with 4gb of ram and a low energy CPU. The worker processes are python scripts, the job queue state is stored in the same postgres database i use to store results. My throughout is low, i only need to process a job every few seconds. The workers run on the same box as the database as I am too lazy to maintain more machines and too cheap to rent cloud servers. Running costs are about $15 / year for energy. The queue implementation is based on this: https://blog.2ndquadrant.com/what-is-select-skip-locked-for-in-postgresql-9-5/ https://blog.2ndquadrant.com/what-is-select-skip-locked-for-... from memory i think I am using primitives offered by database to prevent multiple workers from acquiring the same job (transactions, transaction isolation). this might not be very scalable but i only have two worker processes and each job takes seconds to process. do you want to do what I'm doing? Probably not. but perhaps what you are doing is easy.