3 ms·
> Maybe worth it for some low-volume jobs. > ... > The lesson learned was queues at scale are hard distributed systems problems I'd go so far to say that w
by berkes 5y ago
> Maybe worth it for some low-volume jobs.
> ...
> The lesson learned was queues at scale are hard distributed systems problems
I'd go so far to say that when you need such scale, Rails or Job Queues like Rails' are not the solution. Event-based architectures (event-sourced, event-buses, actors etc) are far better suited.
Rather than trying to solve all the problems that those architectures were explicitly designed for in redis/sidekiq/rails, just move to those architectures.
You won't have the nice "rails generate CreditCard" anymore /s, but you'll gain a lot of consistency, simplicity, performance and architectural sanity.
- scosman 5y agoTotally. Or use rails for your web app, and something else for your queue processing. Rails is nice for rails things. Rails is scales for serving web apps much better than DJ/Sidekiq do at serving queues.