9 ms·

Rails Performance Needs an Overhaul

64 points by igrigorik 16y ago

stanleydrew 16y ago
" It just baffles me that [Rails scalability] is not a solved problem already." Maybe if they open-sourced this "Goliath" framework they created we'd be a little closer.
- trebor 16y ago
  Rails is released under the MIT license, Stanley. It's been open-source for a long time. Fork them and make whatever improvements you want and start a push request. Instead of complain about how slow it is, why not attempt making an "evented" version of Rails? Enable thread-safety for Rails to get a speedy application, even on a shared server. Frankly, I can't write code for EventMachine. Just looking at its style makes me cringe: there's just something about the design of EM apps that I can't compromise with (and it ain't the fact that it's evented).
  - igrigorik 16y ago
    Yep, here's the async activerecord mysql driver: http://www.igvita.com/2010/04/15/non-blocking-activerecord-rails/ http://www.igvita.com/2010/04/15/non-blocking-activerecord-r... It's a start.
    - stanleydrew 16y ago
      Sweet. Checking this out now.
  - stanleydrew 16y ago
    This completely misses the point of what I wrote. I didn't complain about how slow Rails is. The original post does. I'm saying that we could all use some help in improving Rails performance so why not release Goliath as open-source? Sounds like that is in the pipeline though based on other comments in this thread.
- igrigorik 16y ago
  Agreed, that's a sweeping statement to make on Rails scalability. Just trying to make a point though. :-) As for Goliath - we will. Time is the scarcest resource in a startup, and that's why haven't opened it up already. We need to get some documentation in place, and remove a few dependencies. Point is though, there is nothing inherently novel in our stack! All off the shelf pieces: thin, async-rank, async middleware, em drivers + em-synchrony.
  - stanleydrew 16y ago
    To be clear, I understand fully that time is limited in a startup. I'm not sure you should spend the time to clean up and generalize your Goliath stuff. It may not be worth it. But in the context of this post, it just came off as a little bit "holier-than-thou" to hear you say more-or-less: jeez everybody, what's your problem? We've solved this so why is it taking everyone else so long to catch up? Now I believe you didn't mean it that way, but it seems like a helpful way to push things forward is to release bits and pieces of what you have, or at least write up a bit more detailed descriptions of the various pieces. That would be very helpful :)
    - grandalf 16y ago
      I think his point was that Postrank needed performance characteristics that are not consistent with Rails' strengths. While the article came off as a rant (probably to help with publicity) it was more of an ad for Postrank and possibly for Goliath.
allertonm 16y ago
Good piece - having built a big Rails application, I love the speed of development but your comments about scaling totally resonate. Actually it was even worse in my case due to us using jruby (add VM overhead to the mix.) RAM is often the most expensive component of the cost to run an application and as things stand now Ruby/Rails makes it hard to keep those costs down. I really like the idea of using Ruby's continuations to avoid the need to write all of your logic in CPS (ala node.js.) From the memory standpoint, doesn't this just move your memory overhead from the machine stack to continuations on the heap? I do realize there are some advantages to heap over stack, especially on 32-bit architectures. And in addition, what kind of overhead do you get when creating a continuation for the kind of deep call stacks you're dealing with?
- jules 16y ago
  Continuations are not the best solution for not writing asynchronous code in CPS, not in terms of ease of use and not in terms of performance. Coroutines are cheaper as they don't involve stack copying or elaborate compilation schemes, and they map more directly to the problem. That said continuations don't necessarily involve stack copying. The implementations of continuations that don't do make normal procedure calls slower as far as I know.
  - barrkel 16y ago
    If you consider the stack as an implicit closure-like argument, coroutines are continuations for the purpose of cps (no resuming from an earlier activation record).
  - allertonm 16y ago
    Perhaps I'm being a bit dense, but it's hard to see how one could take an existing asynchronous API and adapt it to use co-routines, since the point where you'd ideally want to "yield" is behind the API - and it may not be desirable or even possible to change this (for example, changing a C-based driver to use Ruby-level co-routines would be a challenge.) OTOH it's fairly straightforward to store a continuation and resume it in a callback. So while I think you're correct, pragmatically I suspect this approach has more chance of being implemented.
deleted 16y ago
[deleted]
bradgessler 16y ago
http://www.espace.com.eg/neverblock/ http://www.espace.com.eg/neverblock/ is the closest thing I've heard of to an async Rails framework.
patio11 16y ago
I run Rails -- minimally six processes of memory chewing lovin' at any time, plus the assorted DBs and cache stores -- and spend, oh, call it $2,000 a year more on my 1.5GB VPS than I would need to for equivalent functionality for Java. I get a metric truckload more than $2k of developer productivity out of using Rails instead of Java, though. If my traffic increased such that I needed 40 Mongrels and 20 Delayed Job workers and I was spending $20k a year on hosting, I'd... I'd... I'd really, really like if that happened, actually.
- jackowayed 16y ago
  But he's not saying "we should all switch to Java to save on hosting costs." He's saying he wants it to be fast and still be Rails. As much as the extra $2k of hosting is worth the productivity boost, wouldn't you love it if you could have the Rails productivity and save that $2k/year compared to current Rails performance?
- jamesbritt 16y ago
  Have you tried running Rails under JRuby + Glassfish?
  - tritowntim 16y ago
    have you? How does it run?
    - jamesbritt 16y ago
      Fast. I initially had some set-up issues making sure the MySQL jars were found and tweaking Glassfish settings. Also, development of Ruby + Glassfish almost always meant Rails + Glassfish. I prefer Ramaze, and had to hack a few things (in warbler, for example, and the glassfish3 gem) to get around some things that that assumed you were using Rails. If you are using one of the anointed frameworks (i.e. Rails or Sinatra) you're probably OK. If you are coloring outside the lines you may have issues. But I've not looked into it much since I got my site running (though now I'm just running it as a rackup app through jruby) At this point though I'm looking at using Haskell when speed is a concern. But that's a whole other thread ...
grandalf 16y ago
Is this article applicable to Rails 3? Why not just handle expensive parts of one's app with a simple rack handler? Do you really need form helpers, etc. in every part of the app?
- aarongough 16y ago
  Using a Rack endpoint only reduces the overhead for that specific expensive call. For example if you have a Net:Http.get_request in a Rack endpoint that takes 3 seconds to complete then that Rack endpoint isn't going to be any faster than writing a Rails controller action that does the same thing. If the whole thing were handled asynchronously though you would be able to handle dozens of requests (if not hundreds) in the same time that blocking NET:Http.get_request would take to complete.
  - grandalf 16y ago
    True.. not sure Rails necessarily needs to be the right tool for every job, though. If I had a workflow like that I would probably not use Rails.
    - aarongough 16y ago
      That's totally true. And as I said in another comment I have come up against this exact problem and had considered switching to Node.js to fix it. But when it comes down to it, hacking Rails to do async requests is going to teach me a lot about how Rack and Thin work, so that's why I'm continuing to push in that direction.
aarongough 16y ago
This is actually a very timely article. I built a web-service in Rails over the weekend that has turned out to be much slower in production than I had anticipated... The catch is that it spends 96% of it's time waiting for network responses. 10ms in the application code, 289ms waiting. So now I either re-write the application in a language more suited to the task (node.js) or I work to build async capability into the current app. Personally, I'll take the second if it's not stupidly hard.
- ryanhuff 16y ago
  Did you use Rails 3 and Ruby 1.9.1? One of the benefits of upgrading to the latest version of Rails and Ruby is supposedly performance. As many people haven't yet converted to the new version, I wonder how much of this discussion is a byproduct of running the older software. Thoughts?
  - aarongough 16y ago
    I didn't go with Rails 3, I like to keep at least a minor version behind so I don't have to deal with framework bugs in production :-p In my case the performance issues really have nothing to do with Rails and everything to do with blocking I/O. Rails 3 and Ruby 1.9 may be faster, but that doesn't help you at all if you do something like this: class ExampleController < ActionController::Base def index sleep(1) # contrived example, imagine a slow API call here render :text => "Completed!" end end Because Rails is setup to be run in a blocking fashion my example will only be able to serve 1 request per second. The fix here is non-blocking languages (like Node.js) or programming in an event-driven fashion (by using EventMachine or the like). For example (in pseudocode): when( request.received ) do register slow_api_request, :callback => :answer end def answer do response.send( api_result ) end Ideally the framework should do this by itself. Standard libraries that are blocking, like Net:Http, should continue to appear to the developer like they are blocking, but in reality would pass back control to other instructions while they're just sitting around waiting for the remote server to respond. This is totally possible, and getting more feasible by the day. Thin (the application server) and Rack both allow applications to run asynchronously, Neverblock provides a cleaner way for providing pools of Ruby Fibers to an application to allow implicit scheduling, in time Rails will hopefully build upon this. I've spent several hours this morning doing research into how I can fix this problem for my application, some of the stuff I've dug-up is here: http://github.com/igrigorik/em-http-request http://github.com/rkh/async-rack http://www.espace.com.eg/neverblock/ http://macournoyer.com/blog/2009/06/04/pusher-and-async-with-thin/ It looks like the solution in my case is going to be something like creating a Rails Metal endpoint for Rack, that completes the Net:Http requests using em-http-request. I'm not sure that this is actually possible yet, but it'll be something close to that.

64 points by igrigorik 16y ago

" It just baffles me that [Rails scalability] is not a solved problem already." Maybe if they open-sourced this "Goliath" framework they created we'd be a little closer.

trebor 16y ago
Rails is released under the MIT license, Stanley. It's been open-source for a long time. Fork them and make whatever improvements you want and start a push request. Instead of complain about how slow it is, why not attempt making an "evented" version of Rails? Enable thread-safety for Rails to get a speedy application, even on a shared server. Frankly, I can't write code for EventMachine. Just looking at its style makes me cringe: there's just something about the design of EM apps that I can't compromise with (and it ain't the fact that it's evented).
- igrigorik 16y ago
  Yep, here's the async activerecord mysql driver: http://www.igvita.com/2010/04/15/non-blocking-activerecord-rails/ http://www.igvita.com/2010/04/15/non-blocking-activerecord-r... It's a start.
  - stanleydrew 16y ago
    Sweet. Checking this out now.
- stanleydrew 16y ago
  This completely misses the point of what I wrote. I didn't complain about how slow Rails is. The original post does. I'm saying that we could all use some help in improving Rails performance so why not release Goliath as open-source? Sounds like that is in the pipeline though based on other comments in this thread.
igrigorik 16y ago
Agreed, that's a sweeping statement to make on Rails scalability. Just trying to make a point though. :-) As for Goliath - we will. Time is the scarcest resource in a startup, and that's why haven't opened it up already. We need to get some documentation in place, and remove a few dependencies. Point is though, there is nothing inherently novel in our stack! All off the shelf pieces: thin, async-rank, async middleware, em drivers + em-synchrony.
- stanleydrew 16y ago
  To be clear, I understand fully that time is limited in a startup. I'm not sure you should spend the time to clean up and generalize your Goliath stuff. It may not be worth it. But in the context of this post, it just came off as a little bit "holier-than-thou" to hear you say more-or-less: jeez everybody, what's your problem? We've solved this so why is it taking everyone else so long to catch up? Now I believe you didn't mean it that way, but it seems like a helpful way to push things forward is to release bits and pieces of what you have, or at least write up a bit more detailed descriptions of the various pieces. That would be very helpful :)
  - grandalf 16y ago
    I think his point was that Postrank needed performance characteristics that are not consistent with Rails' strengths. While the article came off as a rant (probably to help with publicity) it was more of an ad for Postrank and possibly for Goliath.

allertonm 16y ago

Good piece - having built a big Rails application, I love the speed of development but your comments about scaling totally resonate. Actually it was even worse in my case due to us using jruby (add VM overhead to the mix.) RAM is often the most expensive component of the cost to run an application and as things stand now Ruby/Rails makes it hard to keep those costs down. I really like the idea of using Ruby's continuations to avoid the need to write all of your logic in CPS (ala node.js.) From the memory standpoint, doesn't this just move your memory overhead from the machine stack to continuations on the heap? I do realize there are some advantages to heap over stack, especially on 32-bit architectures. And in addition, what kind of overhead do you get when creating a continuation for the kind of deep call stacks you're dealing with?

jules 16y ago
Continuations are not the best solution for not writing asynchronous code in CPS, not in terms of ease of use and not in terms of performance. Coroutines are cheaper as they don't involve stack copying or elaborate compilation schemes, and they map more directly to the problem. That said continuations don't necessarily involve stack copying. The implementations of continuations that don't do make normal procedure calls slower as far as I know.
- barrkel 16y ago
  If you consider the stack as an implicit closure-like argument, coroutines are continuations for the purpose of cps (no resuming from an earlier activation record).
- allertonm 16y ago
  Perhaps I'm being a bit dense, but it's hard to see how one could take an existing asynchronous API and adapt it to use co-routines, since the point where you'd ideally want to "yield" is behind the API - and it may not be desirable or even possible to change this (for example, changing a C-based driver to use Ruby-level co-routines would be a challenge.) OTOH it's fairly straightforward to store a continuation and resume it in a callback. So while I think you're correct, pragmatically I suspect this approach has more chance of being implemented.

deleted 16y ago

[deleted]

bradgessler 16y ago

http://www.espace.com.eg/neverblock/ http://www.espace.com.eg/neverblock/ is the closest thing I've heard of to an async Rails framework.

patio11 16y ago

I run Rails -- minimally six processes of memory chewing lovin' at any time, plus the assorted DBs and cache stores -- and spend, oh, call it $2,000 a year more on my 1.5GB VPS than I would need to for equivalent functionality for Java. I get a metric truckload more than $2k of developer productivity out of using Rails instead of Java, though. If my traffic increased such that I needed 40 Mongrels and 20 Delayed Job workers and I was spending $20k a year on hosting, I'd... I'd... I'd really, really like if that happened, actually.

jackowayed 16y ago
But he's not saying "we should all switch to Java to save on hosting costs." He's saying he wants it to be fast and still be Rails. As much as the extra $2k of hosting is worth the productivity boost, wouldn't you love it if you could have the Rails productivity and save that $2k/year compared to current Rails performance?
jamesbritt 16y ago
Have you tried running Rails under JRuby + Glassfish?
- tritowntim 16y ago
  have you? How does it run?
  - jamesbritt 16y ago
    Fast. I initially had some set-up issues making sure the MySQL jars were found and tweaking Glassfish settings. Also, development of Ruby + Glassfish almost always meant Rails + Glassfish. I prefer Ramaze, and had to hack a few things (in warbler, for example, and the glassfish3 gem) to get around some things that that assumed you were using Rails. If you are using one of the anointed frameworks (i.e. Rails or Sinatra) you're probably OK. If you are coloring outside the lines you may have issues. But I've not looked into it much since I got my site running (though now I'm just running it as a rackup app through jruby) At this point though I'm looking at using Haskell when speed is a concern. But that's a whole other thread ...

grandalf 16y ago

Is this article applicable to Rails 3? Why not just handle expensive parts of one's app with a simple rack handler? Do you really need form helpers, etc. in every part of the app?

aarongough 16y ago
Using a Rack endpoint only reduces the overhead for that specific expensive call. For example if you have a Net:Http.get_request in a Rack endpoint that takes 3 seconds to complete then that Rack endpoint isn't going to be any faster than writing a Rails controller action that does the same thing. If the whole thing were handled asynchronously though you would be able to handle dozens of requests (if not hundreds) in the same time that blocking NET:Http.get_request would take to complete.
- grandalf 16y ago
  True.. not sure Rails necessarily needs to be the right tool for every job, though. If I had a workflow like that I would probably not use Rails.
  - aarongough 16y ago
    That's totally true. And as I said in another comment I have come up against this exact problem and had considered switching to Node.js to fix it. But when it comes down to it, hacking Rails to do async requests is going to teach me a lot about how Rack and Thin work, so that's why I'm continuing to push in that direction.

aarongough 16y ago

This is actually a very timely article. I built a web-service in Rails over the weekend that has turned out to be much slower in production than I had anticipated... The catch is that it spends 96% of it's time waiting for network responses. 10ms in the application code, 289ms waiting. So now I either re-write the application in a language more suited to the task (node.js) or I work to build async capability into the current app. Personally, I'll take the second if it's not stupidly hard.

ryanhuff 16y ago
Did you use Rails 3 and Ruby 1.9.1? One of the benefits of upgrading to the latest version of Rails and Ruby is supposedly performance. As many people haven't yet converted to the new version, I wonder how much of this discussion is a byproduct of running the older software. Thoughts?
- aarongough 16y ago
  I didn't go with Rails 3, I like to keep at least a minor version behind so I don't have to deal with framework bugs in production :-p In my case the performance issues really have nothing to do with Rails and everything to do with blocking I/O. Rails 3 and Ruby 1.9 may be faster, but that doesn't help you at all if you do something like this: class ExampleController < ActionController::Base def index sleep(1) # contrived example, imagine a slow API call here render :text => "Completed!" end end Because Rails is setup to be run in a blocking fashion my example will only be able to serve 1 request per second. The fix here is non-blocking languages (like Node.js) or programming in an event-driven fashion (by using EventMachine or the like). For example (in pseudocode): when( request.received ) do register slow_api_request, :callback => :answer end def answer do response.send( api_result ) end Ideally the framework should do this by itself. Standard libraries that are blocking, like Net:Http, should continue to appear to the developer like they are blocking, but in reality would pass back control to other instructions while they're just sitting around waiting for the remote server to respond. This is totally possible, and getting more feasible by the day. Thin (the application server) and Rack both allow applications to run asynchronously, Neverblock provides a cleaner way for providing pools of Ruby Fibers to an application to allow implicit scheduling, in time Rails will hopefully build upon this. I've spent several hours this morning doing research into how I can fix this problem for my application, some of the stuff I've dug-up is here: http://github.com/igrigorik/em-http-request http://github.com/rkh/async-rack http://www.espace.com.eg/neverblock/ http://macournoyer.com/blog/2009/06/04/pusher-and-async-with-thin/ It looks like the solution in my case is going to be something like creating a Rails Metal endpoint for Rack, that completes the Net:Http requests using em-http-request. I'm not sure that this is actually possible yet, but it'll be something close to that.