10 ms·

Rescuing Resque again (2015)

47 points by polysaturate 11y ago

fphilipe 11y ago
The "again" in the title confused me. I knew that I read an article along these lines a while back so I thought this is yet another one. Turns out it was this post, which is dated October 23, 2015.
- polysaturate 11y ago
  Aw poo...missed the date when I read it, and haven't seen it before. I'll update the title.
- steveklabnik 11y ago
  It's again because it's calling back to this one from 2012! http://blog.steveklabnik.com/posts/2012-09-22-resque--let-s-do-this http://blog.steveklabnik.com/posts/2012-09-22-resque--let-s-...
AznHisoka 11y ago
When i started my project, I used Resque but almost immediately switched to Sidekiq a few months later. I understand it's almost impossible to run into thread safety issues with Resque, but if you're doing anything CPU intensive, Sidekiq really offers much higher performance. The maintainer also makes a living off Sidekiq, selling it to businesses and enterprises too so you can expect more frequent updates.
- otterley 11y ago
  > but if you're doing anything CPU intensive, Sidekiq really offers much higher performance The threading model (LWPs vs. forking) should have no impact whatsoever on how quickly a CPU-intensive task completes.
  - andrewvc 11y ago
    Unless it's a short-lived CPU intensive task, in which case in matters a great deal.
    - otterley 11y ago
      The overhead of fork() is negligible, well under 0.1ms.
      - andrewvc 11y ago
        There's also the issue, in ruby, that CoW doesn't really work due to the fact that the GC dirties all pages.
        mperham 11y ago
        This was fixed in Ruby 2.0.
  - AznHisoka 11y ago
    It matters how many of these tasks can be completed in a fixed amount of time. Creating 50 processes requires more resources than 1 process with 50 threads.
    - sanderjd 11y ago
      Yeah, Sidekiq is an obvious switch once you find yourself running 20 500MB Rails processes just because you want to be able to handle a few simple tasks concurrently. If you have a few big, long-running, CPU-intensive background jobs, then the benefits are less clear, but if you have lots of small, short running jobs where you're mostly interested in throughput, then Sidekiq makes a lot more sense.
      - otterley 11y ago
        Resque-pool (https://github.com/nevans/resque-pool https://github.com/nevans/resque-pool) -- which every Resque user should be using -- loads your process first, then forks the workers. Forking uses copy-on-write semantics, so as long as you're loading everything before forking, you're conserving a great deal of otherwise wasted memory on what would be shareable pages.
whatupdave 11y ago
I almost alway use Que [1] now. It uses postgres advisory locks instead of Redis so one less dependency to worry about. You probably only need Redis if you're running a tonne of jobs. [1] https://github.com/chanks/que https://github.com/chanks/que
- bdcravens 11y ago
  If you're already using Resque (the audience for the blog post), you're already using Redis, so Sidekiq would make more sense. Additionally, a quick glance at the syntax for a que job appears that you'd need to change more code (I could be wrong), and the available UI seems to lag Sidekiq's. Reading this: "Que's job table undergoes a lot of churn when it is under high load, and like any heavily-written table, is susceptible to bloat and slowness if Postgres isn't able to clean it up. The most common cause of this is long-running transactions, so it's recommended to try to keep all transactions against the database housing Que's job table as short as possible." I wonder what consistutes "high load". I throw hundreds of thousands of jobs atop Sidekiq often; if you're using the same database to manage your jobs, it seems that would dogpile your database (since each of those jobs is likely making db calls, and if it's a Rails app, you're typically using the same database)
- callmeed 11y ago
  Rails 5 will require Redis so in the future it will be a dependency anyway ...
  - djur 11y ago
    Rails 5 does not require Redis unless you want to use Action Cable.
bdcravens 11y ago
Interesting to note that the current stable branch has commits a couple of days old.
- steveklabnik 11y ago
  Yes, I've been doing some work, and so have some other contributors. Shopify is upstreaming some patches so they can get back on master.
steveklabnik 11y ago
Hi all! Happy to answer any questions about this effort. I'm targeting 1.26.0 sometime in the next couple of weeks.
- sanderjd 11y ago
  Thanks for all your hard work on this over the years! Something I've wondered for awhile: what are the benefits of continuing Resque development, rather than focusing on making the migration path to Sidekiq as painless as possible? Put another way, what do you hear from users of Resque (like Shopify, as you mentioned in another comment) as the reason that they don't want to migrate off of it? I was curious about this way back in 2012 when you announced work on Resque 2.0, and I had already made the switch to Sidekiq. I thought there must be a good reason to keep working on it, but I wasn't sure what it was, and I'm still not sure, so I figured I'd just ask.
  - steveklabnik 11y ago
    Sidekiq has a lot of attention amongst new developers, but has at least an order of magnitude less users, at least, if gem downloads are something to go by. So from my standpoint, those people deserve bugfixes, and "throw away what works just to transition to something new" isn't really a great response. Threading vs process-based jobs is a tradeoff. Threads are not universally better.
    - sanderjd 11y ago
      Thanks for the quick answer. Makes sense.
    - mperham 11y ago
      As of today, Resque has 6,308,611 downloads. Sidekiq has 5,669,265 downloads. Can you explain why you think "order of magnitude"?
      - steveklabnik 11y ago
        Hey Mike! Sorry, I 1) mis-read that number, 2) when we used to release more often, the download counts for latest versions used to be 10x yours. I don't think that's true at this point. I will update that cache in my brain and not say this again, I apologize.
avitzurel 11y ago
I have to be honest here. My initial thought when reading this a couple of weeks back was that some things are better left to dead. Resque has been an example of how not to run an open source project for a while now. (3+ years). We were working with Resque for a long while and had a lot of issues with performance. Sidekiq is awesome and it's being well managed and releases are frequent. But, after thinking about it, I think that Resque has it's place in the community, I definitely think that more than a single project doing this task is needed. Many people are already familiar with Resque and trust it, those people should be taken care of. I wish everyone involved a lot of luck with it.
- sanderjd 11y ago
  This is another good answer to the question[0] I just posed to Steve. You're right that competition is generally good. On the other hand, it would (arguably?) be much more "Rails-y" to have a single conventional solution that works well and gets a lot of eyeballs. Neither Resque nor Sidekiq are that thing though. It has always struck me as a bit odd that Rails outsources the decision on this component, which nearly every Rails project needs. I think previously it was probably to avoid an external dependency (on Redis), but perhaps that could be reconsidered now that >5.0 has eschewed that avoidance. Having said all that, one of the things that is most exciting to me about Elixir/Phoenix is that this sort of thing doesn't even come up, because a great and mature solution is built directly into the runtime. [0]: https://news.ycombinator.com/item?id=10911600 https://news.ycombinator.com/item?id=10911600
  - steveklabnik 11y ago
    Newer Rails has ActiveJob: https://github.com/rails/rails/tree/master/activejob https://github.com/rails/rails/tree/master/activejob
- steveklabnik 11y ago
  Thanks! > Resque has been an example of how not to run an open source project > for a while now. (3+ years). Resque is the story of a gem that _everyone_ uses, yet nobody invests in. So the maintainers burn out, and burn out, and burn out. Chris, Terence, Me... trying not to do that again ;)
thibaut_barrere 11y ago
In my opinion, Resque so far failed because like a lot of OSS projects (and unlike Sidekiq), it isn't built on sustainable economical grounds - from the maintainer POV. Bringing outside "funding" can work for a while (see JRuby), but when the funding stops, it only underlines that the financial engine should really be carefully "designed" at the core of the OSS project, for things to be maintainable on the very long run. Otherwise history is bound to repeat itself. Large projects can enjoy a large pool of contributors, sometimes with less risks of burn-out (often seen on small OSS projects around me at least), each contributing marginally, with a few paid core members (either directly or indirectly via by-products of a well paid activity), but for smaller projects like Resque, it's more likely a single individual full-time, paid by the project. Again Sidekiq is really a great inspiration here for me (disclaimer: I plan to launch a "pro" offering for an OSS project I manage at http://kiba-etl.org/ http://kiba-etl.org/, and maintained another open-source for a while before which ultimately went to the cesspit).
- steveklabnik 11y ago
  Yup, this is exactly it. The only reason I was able to do all the open source work I did was because of my job at Jumpstart Labs, and once that ended, well... That said, as you mentioned, this is pretty much the default state for the large, large part of open source projects. Something's got to give somewhere, eventually.
  - thibaut_barrere 11y ago
    What is your take on the Sidekiq model? I'm really curious to hear more about that :-) I see a number of people around me currently thinking about putting a "fair salary" at the core of their future open-source work. Often it completely changes what the delivered value should ultimately be etc (something I think Mike did think about - I believe). Do you have thoughts on that approach?
    - steveklabnik 11y ago
      My thoughts are mostly "I'm glad it's working out for Mike, and I hope others can follow his path." I personally have taken pursued the patron model, but that also is because with Rust, that's the only real way to do it.

47 points by polysaturate 11y ago

The "again" in the title confused me. I knew that I read an article along these lines a while back so I thought this is yet another one. Turns out it was this post, which is dated October 23, 2015.

polysaturate 11y ago
Aw poo...missed the date when I read it, and haven't seen it before. I'll update the title.
steveklabnik 11y ago
It's again because it's calling back to this one from 2012! http://blog.steveklabnik.com/posts/2012-09-22-resque--let-s-do-this http://blog.steveklabnik.com/posts/2012-09-22-resque--let-s-...

AznHisoka 11y ago

When i started my project, I used Resque but almost immediately switched to Sidekiq a few months later. I understand it's almost impossible to run into thread safety issues with Resque, but if you're doing anything CPU intensive, Sidekiq really offers much higher performance. The maintainer also makes a living off Sidekiq, selling it to businesses and enterprises too so you can expect more frequent updates.

otterley 11y ago
> but if you're doing anything CPU intensive, Sidekiq really offers much higher performance The threading model (LWPs vs. forking) should have no impact whatsoever on how quickly a CPU-intensive task completes.
- andrewvc 11y ago
  Unless it's a short-lived CPU intensive task, in which case in matters a great deal.
  - otterley 11y ago
    The overhead of fork() is negligible, well under 0.1ms.
    - andrewvc 11y ago
      There's also the issue, in ruby, that CoW doesn't really work due to the fact that the GC dirties all pages.
      - mperham 11y ago
        This was fixed in Ruby 2.0.
- AznHisoka 11y ago
  It matters how many of these tasks can be completed in a fixed amount of time. Creating 50 processes requires more resources than 1 process with 50 threads.
  - sanderjd 11y ago
    Yeah, Sidekiq is an obvious switch once you find yourself running 20 500MB Rails processes just because you want to be able to handle a few simple tasks concurrently. If you have a few big, long-running, CPU-intensive background jobs, then the benefits are less clear, but if you have lots of small, short running jobs where you're mostly interested in throughput, then Sidekiq makes a lot more sense.
    - otterley 11y ago
      Resque-pool (https://github.com/nevans/resque-pool https://github.com/nevans/resque-pool) -- which every Resque user should be using -- loads your process first, then forks the workers. Forking uses copy-on-write semantics, so as long as you're loading everything before forking, you're conserving a great deal of otherwise wasted memory on what would be shareable pages.

whatupdave 11y ago

I almost alway use Que [1] now. It uses postgres advisory locks instead of Redis so one less dependency to worry about. You probably only need Redis if you're running a tonne of jobs. [1] https://github.com/chanks/que https://github.com/chanks/que

bdcravens 11y ago
If you're already using Resque (the audience for the blog post), you're already using Redis, so Sidekiq would make more sense. Additionally, a quick glance at the syntax for a que job appears that you'd need to change more code (I could be wrong), and the available UI seems to lag Sidekiq's. Reading this: "Que's job table undergoes a lot of churn when it is under high load, and like any heavily-written table, is susceptible to bloat and slowness if Postgres isn't able to clean it up. The most common cause of this is long-running transactions, so it's recommended to try to keep all transactions against the database housing Que's job table as short as possible." I wonder what consistutes "high load". I throw hundreds of thousands of jobs atop Sidekiq often; if you're using the same database to manage your jobs, it seems that would dogpile your database (since each of those jobs is likely making db calls, and if it's a Rails app, you're typically using the same database)
callmeed 11y ago
Rails 5 will require Redis so in the future it will be a dependency anyway ...
- djur 11y ago
  Rails 5 does not require Redis unless you want to use Action Cable.

bdcravens 11y ago

Interesting to note that the current stable branch has commits a couple of days old.

steveklabnik 11y ago
Yes, I've been doing some work, and so have some other contributors. Shopify is upstreaming some patches so they can get back on master.

steveklabnik 11y ago

Hi all! Happy to answer any questions about this effort. I'm targeting 1.26.0 sometime in the next couple of weeks.

sanderjd 11y ago
Thanks for all your hard work on this over the years! Something I've wondered for awhile: what are the benefits of continuing Resque development, rather than focusing on making the migration path to Sidekiq as painless as possible? Put another way, what do you hear from users of Resque (like Shopify, as you mentioned in another comment) as the reason that they don't want to migrate off of it? I was curious about this way back in 2012 when you announced work on Resque 2.0, and I had already made the switch to Sidekiq. I thought there must be a good reason to keep working on it, but I wasn't sure what it was, and I'm still not sure, so I figured I'd just ask.
- steveklabnik 11y ago
  Sidekiq has a lot of attention amongst new developers, but has at least an order of magnitude less users, at least, if gem downloads are something to go by. So from my standpoint, those people deserve bugfixes, and "throw away what works just to transition to something new" isn't really a great response. Threading vs process-based jobs is a tradeoff. Threads are not universally better.
  - sanderjd 11y ago
    Thanks for the quick answer. Makes sense.
  - mperham 11y ago
    As of today, Resque has 6,308,611 downloads. Sidekiq has 5,669,265 downloads. Can you explain why you think "order of magnitude"?
    - steveklabnik 11y ago
      Hey Mike! Sorry, I 1) mis-read that number, 2) when we used to release more often, the download counts for latest versions used to be 10x yours. I don't think that's true at this point. I will update that cache in my brain and not say this again, I apologize.

avitzurel 11y ago

I have to be honest here. My initial thought when reading this a couple of weeks back was that some things are better left to dead. Resque has been an example of how not to run an open source project for a while now. (3+ years). We were working with Resque for a long while and had a lot of issues with performance. Sidekiq is awesome and it's being well managed and releases are frequent. But, after thinking about it, I think that Resque has it's place in the community, I definitely think that more than a single project doing this task is needed. Many people are already familiar with Resque and trust it, those people should be taken care of. I wish everyone involved a lot of luck with it.

sanderjd 11y ago
This is another good answer to the question[0] I just posed to Steve. You're right that competition is generally good. On the other hand, it would (arguably?) be much more "Rails-y" to have a single conventional solution that works well and gets a lot of eyeballs. Neither Resque nor Sidekiq are that thing though. It has always struck me as a bit odd that Rails outsources the decision on this component, which nearly every Rails project needs. I think previously it was probably to avoid an external dependency (on Redis), but perhaps that could be reconsidered now that >5.0 has eschewed that avoidance. Having said all that, one of the things that is most exciting to me about Elixir/Phoenix is that this sort of thing doesn't even come up, because a great and mature solution is built directly into the runtime. [0]: https://news.ycombinator.com/item?id=10911600 https://news.ycombinator.com/item?id=10911600
- steveklabnik 11y ago
  Newer Rails has ActiveJob: https://github.com/rails/rails/tree/master/activejob https://github.com/rails/rails/tree/master/activejob
steveklabnik 11y ago
Thanks! > Resque has been an example of how not to run an open source project > for a while now. (3+ years). Resque is the story of a gem that _everyone_ uses, yet nobody invests in. So the maintainers burn out, and burn out, and burn out. Chris, Terence, Me... trying not to do that again ;)

thibaut_barrere 11y ago

In my opinion, Resque so far failed because like a lot of OSS projects (and unlike Sidekiq), it isn't built on sustainable economical grounds - from the maintainer POV. Bringing outside "funding" can work for a while (see JRuby), but when the funding stops, it only underlines that the financial engine should really be carefully "designed" at the core of the OSS project, for things to be maintainable on the very long run. Otherwise history is bound to repeat itself. Large projects can enjoy a large pool of contributors, sometimes with less risks of burn-out (often seen on small OSS projects around me at least), each contributing marginally, with a few paid core members (either directly or indirectly via by-products of a well paid activity), but for smaller projects like Resque, it's more likely a single individual full-time, paid by the project. Again Sidekiq is really a great inspiration here for me (disclaimer: I plan to launch a "pro" offering for an OSS project I manage at http://kiba-etl.org/ http://kiba-etl.org/, and maintained another open-source for a while before which ultimately went to the cesspit).

steveklabnik 11y ago
Yup, this is exactly it. The only reason I was able to do all the open source work I did was because of my job at Jumpstart Labs, and once that ended, well... That said, as you mentioned, this is pretty much the default state for the large, large part of open source projects. Something's got to give somewhere, eventually.
- thibaut_barrere 11y ago
  What is your take on the Sidekiq model? I'm really curious to hear more about that :-) I see a number of people around me currently thinking about putting a "fair salary" at the core of their future open-source work. Often it completely changes what the delivered value should ultimately be etc (something I think Mike did think about - I believe). Do you have thoughts on that approach?
  - steveklabnik 11y ago
    My thoughts are mostly "I'm glad it's working out for Mike, and I hope others can follow his path." I personally have taken pursued the patron model, but that also is because with Rust, that's the only real way to do it.