4 ms·
My experience has been the durability of jobs is much more important than the execution speed. I've worked on extremely large Rails apps that would not use Redi
by bradly 1y ago
My experience has been the durability of jobs is much more important than the execution speed. I've worked on extremely large Rails apps that would not use Redis for a queue store and preferred data integrity over jobs per sec. I've also seen large, scale financial job queues built on Redis.
Teams should know and be comfortable with what happens to a long running job during a deployment, what happens when a worker disappears (at all points in the jobs lifecycle), is there a non-zero chance the job will be ran more than once? do jobs run in the order they are queued? can your jobs be re-ran? can your jobs be ran simultaneously? Do you know if you are dropping jobs? How are you catching them?
- shayonj 1y agoLikewise - that’s why I mentioned that orchestration backed by PG and treating sidekiq + Redis message bus goes a long way. You get the durability, and the ability to re-enqueue jobs in case of availability issues with Redis. Sidekiq’s reliable fetch provides other kind of durability, esp when recovering from process crash extra. Agree on all points re: job lifecycle