3 ms·

> A system that does tens to hundreds of thousands of jobs/s I mean - yes - but how many systems really do tens of thousands of jobs a second? Some, obviously.

> A system that does tens to hundreds of thousands of jobs/s I mean - yes - but how many systems really do tens of thousands of jobs a second? Some, obviously. But that's a buttload of jobs for the vast majority of verticals. Like imagine a hypothetical dental SaaS that has 100.0% market penetration in the US and needs to handle the 500M dentist visits per year that happen in the US. That's 16/s, let's say they all run during working hours (x3), there are 10 async jobs on average per dentist visit (x10), that gets us up to 500 jobs/s. You could even add 100% market penetration for this unicorn SaaS on all veterinary visits (+200M visits/yr), surgeries (+50M), emergency department visits (+130M), chiropractic visits (+35M), and hair salon visits (1B/yr)... and only get to 2000 jobs/s.

mperham 1y ago
I have dozens of customers doing 1b+ jobs/day. The world is bigger than you think.
- mattbessey 1y ago
  I'm not sure that's a particularly fair conclusion to make Mike. 1bn a day is order 10k/s. if you the creator of Sidekiq know "dozens" of people hitting that rates, if anything you're supporting arcticfox's point that these systems are quite rare. No shade intended. I have been a very grateful Sidekiq Enterprise customer in past Ruby jobs.
  - zanellato19 1y ago
    It's a piece of mind thing. Everywhere I worked did a lot of stuff on the background. The db is already so full on a normal Rails application that adding more reads doesn't make sense to me when Sidekiq works so well. You never have to worry about Sidekiq throughput, or when you do is to be careful to not overload the db because Sidekiq is so fast. Rails is already on a slow language, I don't understand why we want to overload even more with slow things.
    - pythonaut_16 1y ago
      Having both available is a huge win for the Rails ecosystem. If you’re just starting or have a small project Solid Queue is invaluable and reduces costs and friction. If and when you need more performance or power Sidekiq is still available. Both run in the context of your Rails app so moving jobs from one to the other shouldn’t be terribly hard.
      - ksec 1y ago
        Yes. But as with all things Rails I do wish the default performance could be pushed a bit. So say Solid Queue could work for 90% of project instead of 70 - 80%. May be ZJIT could help.
      - gommm 1y ago
        Exactly, it's not an either/or situation. I've worked on a codebase that did 3-4k/s jobs through sidekiq and in this case sidekiq is obviously the best option (although we also used a go sidekiq client for certain tasks that didn't need the rails app context) but using sidekiq from the beginning is premature optimization. You can easily start with Solid Queue and switch over to sidekiq when you actually have data that tells you that it'll potentially become a bottleneck. Premature optimization is a complexity cost.
    - deleted 1y ago
      [deleted]
  - mperham 1y ago
    Dozens is just the number I've talked to who have told me their scale. I'm sure there's a lot more. And ultimately the precise number is not as important as the fact that it *is normal* for midsized Ruby apps to operate at this level of scale. The more you start to use the background job pattern, the more you tend to create. "A Job for every ProductVariant record" can easily create 10-20m jobs. You want tools that scale to the size of "success". That would be a good marketing tagline for Sidekiq.
ndriscoll 1y ago
That's easy to do in an IoT setting. Less common if human action is involved. But if you need that kind of throughput, just use a more suitable technology like Scala that can work with in memory queues and background tasks and can handle that much without any trickiness.
- deleted 1y ago
  [deleted]
nickjj 1y ago
That is a good example, but don't forget your DB will also be doing many other things besides managing jobs. In the Rails world of using the DB for everything you have your primary DB, solid queue, solid cache and solid cable. If you lean into modern Rails features (using Turbo Streams which involves jobs, async destroying dependents, etc.) it doesn't seem that hard to rack up a lot of DB activity. Combine this with pretty spotty SSD I/O performance for most VPS servers, I don't know about hosting everything on 1 box on a $20-40 / month server but I feel very comfortable doing that with Redis backing queue + cache + cable while PG focuses as being a primary DB. I haven't seen any success stories yet of anyone using the DB for everything on an entry level VPS. It's mainly DHH mentioning it works great with splitting things into different databases and running it on massive dedicated hardware systems with the best SSDs you can buy. I like the idea on paper but I don't think I'll switch away from Redis until I see a bunch of social proof of people showing how they are using all of these DB backed tools on a VPS to serve a reasonable amount of traffic. We have 10+ years of proof that Redis works great in these cases.
- jprosevear 1y ago
  Solid Queue (by default actually) uses a second, segregated database for the queue. You can scale it independently of your app database.
  - nickjj 1y ago
    Yep but using 2 DBs within the same PG instance won't improve performance since it's using the same disk. Managing multiple DB instances across multiple hosts requires a lot more effort, configuration and doubles your hosting costs vs running Redis on the same host with Sidekiq IMO.

mperham 1y ago

I have dozens of customers doing 1b+ jobs/day. The world is bigger than you think.

mattbessey 1y ago
I'm not sure that's a particularly fair conclusion to make Mike. 1bn a day is order 10k/s. if you the creator of Sidekiq know "dozens" of people hitting that rates, if anything you're supporting arcticfox's point that these systems are quite rare. No shade intended. I have been a very grateful Sidekiq Enterprise customer in past Ruby jobs.
- zanellato19 1y ago
  It's a piece of mind thing. Everywhere I worked did a lot of stuff on the background. The db is already so full on a normal Rails application that adding more reads doesn't make sense to me when Sidekiq works so well. You never have to worry about Sidekiq throughput, or when you do is to be careful to not overload the db because Sidekiq is so fast. Rails is already on a slow language, I don't understand why we want to overload even more with slow things.
  - pythonaut_16 1y ago
    Having both available is a huge win for the Rails ecosystem. If you’re just starting or have a small project Solid Queue is invaluable and reduces costs and friction. If and when you need more performance or power Sidekiq is still available. Both run in the context of your Rails app so moving jobs from one to the other shouldn’t be terribly hard.
    - ksec 1y ago
      Yes. But as with all things Rails I do wish the default performance could be pushed a bit. So say Solid Queue could work for 90% of project instead of 70 - 80%. May be ZJIT could help.
    - gommm 1y ago
      Exactly, it's not an either/or situation. I've worked on a codebase that did 3-4k/s jobs through sidekiq and in this case sidekiq is obviously the best option (although we also used a go sidekiq client for certain tasks that didn't need the rails app context) but using sidekiq from the beginning is premature optimization. You can easily start with Solid Queue and switch over to sidekiq when you actually have data that tells you that it'll potentially become a bottleneck. Premature optimization is a complexity cost.
  - deleted 1y ago
    [deleted]
- mperham 1y ago
  Dozens is just the number I've talked to who have told me their scale. I'm sure there's a lot more. And ultimately the precise number is not as important as the fact that it *is normal* for midsized Ruby apps to operate at this level of scale. The more you start to use the background job pattern, the more you tend to create. "A Job for every ProductVariant record" can easily create 10-20m jobs. You want tools that scale to the size of "success". That would be a good marketing tagline for Sidekiq.

ndriscoll 1y ago

That's easy to do in an IoT setting. Less common if human action is involved. But if you need that kind of throughput, just use a more suitable technology like Scala that can work with in memory queues and background tasks and can handle that much without any trickiness.

deleted 1y ago
[deleted]

nickjj 1y ago

That is a good example, but don't forget your DB will also be doing many other things besides managing jobs. In the Rails world of using the DB for everything you have your primary DB, solid queue, solid cache and solid cable. If you lean into modern Rails features (using Turbo Streams which involves jobs, async destroying dependents, etc.) it doesn't seem that hard to rack up a lot of DB activity. Combine this with pretty spotty SSD I/O performance for most VPS servers, I don't know about hosting everything on 1 box on a $20-40 / month server but I feel very comfortable doing that with Redis backing queue + cache + cable while PG focuses as being a primary DB. I haven't seen any success stories yet of anyone using the DB for everything on an entry level VPS. It's mainly DHH mentioning it works great with splitting things into different databases and running it on massive dedicated hardware systems with the best SSDs you can buy. I like the idea on paper but I don't think I'll switch away from Redis until I see a bunch of social proof of people showing how they are using all of these DB backed tools on a VPS to serve a reasonable amount of traffic. We have 10+ years of proof that Redis works great in these cases.

jprosevear 1y ago
Solid Queue (by default actually) uses a second, segregated database for the queue. You can scale it independently of your app database.
- nickjj 1y ago
  Yep but using 2 DBs within the same PG instance won't improve performance since it's using the same disk. Managing multiple DB instances across multiple hosts requires a lot more effort, configuration and doubles your hosting costs vs running Redis on the same host with Sidekiq IMO.