I built something similar when I was working on SocialGrapple (processing hundreds of social graph aggregation and processing jobs every hour), called turnip: h

I built something similar when I was working on SocialGrapple (processing hundreds of social graph aggregation and processing jobs every hour), called turnip: https://github.com/shazow/turnip https://github.com/shazow/turnip It uses SQLAlchemy to generate a relational schema for the queue, so it works on most relational databases. I used it with PostgreSQL. One additional twist is that it supports scheduling and recurring jobs using cron-like syntax. Otherwise it's the same idea, minus the automatic worker scaling. The schema stores a reference to the job to call with desired parameters, then you can spawn as many turnip workers you want to consume. Sadly I've stopped using it since I stopped working on SocialGrapple, so it's not maintained anymore. But the code is simple enough and I'm happy to add more documentation if anyone is interested.