3 ms·
Transactional consistency is nice. Maybe worth it for some low-volume jobs. The downside that comes with it is using your primary DB as your queue is that the
by scosman 5y ago
Transactional consistency is nice. Maybe worth it for some low-volume jobs.
The downside that comes with it is using your primary DB as your queue is that the queues in an app tend to absorb all the exceptional conditions (queue grows to millions during worker outage, backfill which needs to process millions of items). Using your primary DB means all that disk usage, memory usage, and IO hit your primary DB in exceptional ways and impacts your app in more ways than just queue delays. It also scales much less linearly than other queue options.
A nice middle ground is to store all your data and state in your DB transactionally, and only queue tiny lightweight jobs with database primary keys (database: sentWelcomeMailDate=nil, queue: sentWelcomeMailForUser:1234).
This is another case for neither DJ or Sidekiq. A fully hosted option like SQS addresses the durability concerns of Sidekiq, and the scaling concerns of both. I’ve queued 1B+ jobs in SQS and watched the workers slowly burn through them over the course of a month.
DJ and Sidekiq can go quite far. I’ve used both and patched some perf issues on DJ. The lesson learned was queues at scale are hard distributed systems problems - you probably don’t want to have to know how they work, or host the distributed system yourself.
- berkes 5y ago> Maybe worth it for some low-volume jobs. > ... > The lesson learned was queues at scale are hard distributed systems problems I'd go so far to say that when you need such scale, Rails or Job Queues like Rails' are not the solution. Event-based architectures (event-sourced, event-buses, actors etc) are far better suited. Rather than trying to solve all the problems that those architectures were explicitly designed for in redis/sidekiq/rails, just move to those architectures. You won't have the nice "rails generate CreditCard" anymore /s, but you'll gain a lot of consistency, simplicity, performance and architectural sanity.
- scosman 5y agoTotally. Or use rails for your web app, and something else for your queue processing. Rails is nice for rails things. Rails is scales for serving web apps much better than DJ/Sidekiq do at serving queues.