4 ms·

I'm currently trying to decide if I should implement Resque again, or opt for RabbitMQ instead due to this and similar compromises which stem from the ruby/redi

I'm currently trying to decide if I should implement Resque again, or opt for RabbitMQ instead due to this and similar compromises which stem from the ruby/redis combination. What would people say are the major differences, the major pros and cons between the two systems? I'm dealing with simple intermittent, longish running jobs which seem well suited to resque, but I can't shake the feeling that I might be better off with rabbit/0mq/etc. Obviously resque is closer to a "turnkey" solution and so forth, but what are the real fundamental differences?

davetron5000 12y ago
The underlying technology doesn't change the fact that Heroku will send SIGTERM to all processes. Resque has good tools for automatic retry, so if you can make your jobs idempotent, configure them for retry and forget about it.
- JonnieCache 12y ago
  Sorry, I should have been more clear: I'm running this stuff on a dedicated box, not on heroku. I asked because the thread concerns resque and queueing generally.
cheald 12y ago
RabbitMQ has its own set of durability issues (see the recent Jepsen writeup on it), but if the data store itself is stable, then it's really very good. The primary difference you'll notice is that RMQ has an explicit-ack mode. It will send a message to a client, the client processes it and sends an explicit ack (message consumed), at which point RMQ will send the next message. The client can also send a nack (push the job back onto the queue and redeliver it), and if the connection is dropped without the job being ack'd, then RMQ will requeue it and send it to another client. If you're performing all your state mutations in a transaction or something similar that rolls back when a worker terminates, then you can avoid losing jobs and ending up in invalid state even during non-clean shutdowns. As far as other notable changes go, you can have multi-queue routing (one message can be routed into multiple queues) and dead letter exchanges (so that TTL expired messages can be sent to a different queue rather than just being dropped). There's a lot more to it, as well; as a message queue, I do think that RMQ is flatly superior to Redis, but Redis has drop-dead simplicity going for it that is really nice if you don't need the extra features RMQ offers.
- deleted 12y ago
  [deleted]

davetron5000 12y ago

The underlying technology doesn't change the fact that Heroku will send SIGTERM to all processes. Resque has good tools for automatic retry, so if you can make your jobs idempotent, configure them for retry and forget about it.

JonnieCache 12y ago
Sorry, I should have been more clear: I'm running this stuff on a dedicated box, not on heroku. I asked because the thread concerns resque and queueing generally.

cheald 12y ago

RabbitMQ has its own set of durability issues (see the recent Jepsen writeup on it), but if the data store itself is stable, then it's really very good. The primary difference you'll notice is that RMQ has an explicit-ack mode. It will send a message to a client, the client processes it and sends an explicit ack (message consumed), at which point RMQ will send the next message. The client can also send a nack (push the job back onto the queue and redeliver it), and if the connection is dropped without the job being ack'd, then RMQ will requeue it and send it to another client. If you're performing all your state mutations in a transaction or something similar that rolls back when a worker terminates, then you can avoid losing jobs and ending up in invalid state even during non-clean shutdowns. As far as other notable changes go, you can have multi-queue routing (one message can be routed into multiple queues) and dead letter exchanges (so that TTL expired messages can be sent to a different queue rather than just being dropped). There's a lot more to it, as well; as a message queue, I do think that RMQ is flatly superior to Redis, but Redis has drop-dead simplicity going for it that is really nice if you don't need the extra features RMQ offers.

deleted 12y ago
[deleted]