5 ms·

When i started my project, I used Resque but almost immediately switched to Sidekiq a few months later. I understand it's almost impossible to run into thread s

When i started my project, I used Resque but almost immediately switched to Sidekiq a few months later. I understand it's almost impossible to run into thread safety issues with Resque, but if you're doing anything CPU intensive, Sidekiq really offers much higher performance. The maintainer also makes a living off Sidekiq, selling it to businesses and enterprises too so you can expect more frequent updates.

otterley 11y ago
> but if you're doing anything CPU intensive, Sidekiq really offers much higher performance The threading model (LWPs vs. forking) should have no impact whatsoever on how quickly a CPU-intensive task completes.
- andrewvc 11y ago
  Unless it's a short-lived CPU intensive task, in which case in matters a great deal.
  - otterley 11y ago
    The overhead of fork() is negligible, well under 0.1ms.
    - andrewvc 11y ago
      There's also the issue, in ruby, that CoW doesn't really work due to the fact that the GC dirties all pages.
      - mperham 11y ago
        This was fixed in Ruby 2.0.
- AznHisoka 11y ago
  It matters how many of these tasks can be completed in a fixed amount of time. Creating 50 processes requires more resources than 1 process with 50 threads.
  - sanderjd 11y ago
    Yeah, Sidekiq is an obvious switch once you find yourself running 20 500MB Rails processes just because you want to be able to handle a few simple tasks concurrently. If you have a few big, long-running, CPU-intensive background jobs, then the benefits are less clear, but if you have lots of small, short running jobs where you're mostly interested in throughput, then Sidekiq makes a lot more sense.
    - otterley 11y ago
      Resque-pool (https://github.com/nevans/resque-pool https://github.com/nevans/resque-pool) -- which every Resque user should be using -- loads your process first, then forks the workers. Forking uses copy-on-write semantics, so as long as you're loading everything before forking, you're conserving a great deal of otherwise wasted memory on what would be shareable pages.

otterley 11y ago

> but if you're doing anything CPU intensive, Sidekiq really offers much higher performance The threading model (LWPs vs. forking) should have no impact whatsoever on how quickly a CPU-intensive task completes.

andrewvc 11y ago
Unless it's a short-lived CPU intensive task, in which case in matters a great deal.
- otterley 11y ago
  The overhead of fork() is negligible, well under 0.1ms.
  - andrewvc 11y ago
    There's also the issue, in ruby, that CoW doesn't really work due to the fact that the GC dirties all pages.
    - mperham 11y ago
      This was fixed in Ruby 2.0.
AznHisoka 11y ago
It matters how many of these tasks can be completed in a fixed amount of time. Creating 50 processes requires more resources than 1 process with 50 threads.
- sanderjd 11y ago
  Yeah, Sidekiq is an obvious switch once you find yourself running 20 500MB Rails processes just because you want to be able to handle a few simple tasks concurrently. If you have a few big, long-running, CPU-intensive background jobs, then the benefits are less clear, but if you have lots of small, short running jobs where you're mostly interested in throughput, then Sidekiq makes a lot more sense.
  - otterley 11y ago
    Resque-pool (https://github.com/nevans/resque-pool https://github.com/nevans/resque-pool) -- which every Resque user should be using -- loads your process first, then forks the workers. Forking uses copy-on-write semantics, so as long as you're loading everything before forking, you're conserving a great deal of otherwise wasted memory on what would be shareable pages.