3 ms·
Not exactly. Tasks can last for weeks. It can run fine for several days and then die, and it needs to be requeued, until it is explicitly finished. In fact, we
by adrinavarro 9y ago
Not exactly. Tasks can last for weeks. It can run fine for several days and then die, and it needs to be requeued, until it is explicitly finished. In fact, we do use RabbitMQ to emit "status updates".
With RabbitMQ, I'd need to ack rightaway, otherwise it would re-send the message again after a while.
- Bogdanp 9y ago> With RabbitMQ, I'd need to ack rightaway, otherwise it would re-send the message again after a while. This is not exactly the case. RMQ will only re-enqueue the message when the consumer disconnects. If you're able to keep the consumer connection alive (this is easy to do with the heartbeat mechanism) for the processing duration, even if it takes a long time, RMQ should handle it fine. That said, if the connection between your consumers and RMQ is flaky, you'll have to make your tasks re-entrant.
- adrinavarro 9y agoGood point. I'd rather have something explicit going on. In other places where we do use RabbitMQ (for short-lived, non-critical tasks), the listening processes log reconnects every once in a while, even with heartbeat.
- spyspy 9y agoVery curious what type of work you're doing where atomic tasks can run for weeks at a time.
- softawre 9y agoWe have something like this, not weeks but days. Linear programs, integer math, using IBM Cplex to schedule "people" to do "things" at the ideal time.
- adrinavarro 9y agoIn our case, data migration. Resumable, sometimes dies because of various reasons (or just infra rescaling happening), but takes very long to run.
- jacques_chester 9y agoI've seen folk talk about progress messages for long-running tasks and jobs. If you have checkpointing then they play well together.