Celery is best on RabbitMQ, but supports tons of backing stores, not that you necessarily should[1]. How is RabbitMQ a lot of machinery? Compared to Redis, AMQ

Celery is best on RabbitMQ, but supports tons of backing stores, not that you necessarily should[1]. How is RabbitMQ a lot of machinery? Compared to Redis, AMQP and Rabbit have the complexity of a bologna sandwich. It's actually a quite simple system, and very easy to install. There is almost always no configuration necessary. Developers can install it in one command with MacPorts or Homebrew or apt or yum and don't even have to create users or "vhosts" to get started. Create a named exchange, subscribe to it, you're now AMQPing. It's well-packaged and well-documented. It's also so light on it's feet that it rarely actually needs a dedicated box; just stick it on your database server to get started. I think Resque is a good solution for Rails, for sure. I've built stuff around it, and it works well, and definitely battle tested. It certainly beats the existing AMQP-based solutions. The idea that your application code probably sucks and/or is in a constant state of flux, so handle failure gracefully is rather excellent. The web interface is particularly nice and useful. While I don't necessarily agree 100% with Redis as the backing store, the actual mechanics of the worker queue implementation (plus tools like resque-pool) work so well that for most projects, it's perfectly fine. I actually had a problem specific to using Redis materialize when it ran out of memory while queueing a large batch of per-user-generated weekly emails. A cron job would spin up a script that would cache a large amount of database data in RAM, using it to generate per-user content. The rate at which email could be sent became a major bottleneck, and we had Resque, so we just enqueue'd the entire email and had a small job that connected to the email server and send them. This worked reasonably well and for many months, the high watermark for RAM usage was well below what the box could hold. Unexpectedly, the amount of content contained within these emails quadrupled. Redis ate through all the RAM on the box and it started going to disk using Redis' VM mechanism. This was configured to "theoretically" prevent an unexpected OOM from making the queues unavailable. The unfortunate part is that the Resque stores the entire queue in a single key, and when the key gets swapped onto disk, Redis must read this entire key (and therefore the entire queue) into RAM before it's able to read any data from it. This wasn't Resque's fault really. It was my fault. I did not take the time to understand how Resque and Redis worked well enough to make a truly informed decision. I did not test the OOM condition. I should have tried to fill up the queue on a test box with a billion items just to see what happened. All lessons learned. [1] http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/ http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-sto...