5 ms·

> What kind of speedup is available for big Rails applications? If 90% of the time in an application is spent in database calls, then there’s little opportunity

> What kind of speedup is available for big Rails applications? If 90% of the time in an application is spent in database calls, then there’s little opportunity for improvement via JIT technologies. This is written as speculation of course, but it matches some long-time "conventional wisdom", that most Rails apps spend most of their time on I/O waiting for something external. That may have been true once, but i don't believe it's true anymore. My Rails apps now, when things like "n+1 database query problems" are properly eliminated, all spend more of their time on CPU than I/O, 60-80%. (I think most of that CPU time is ActionView rendering. Unsure about CPU parts of ActiveRecord, that aren't waiting on DB, but converting db results to in-memory AR objects etc). When this comes up and I ask around on social media who else has actually looked at their app and can confirm or deny, almost all who respond agree. I would be curious to see here too. Definitely some more systematic rather than anecdotal investigation is called for. But I think it's time to stop repeating that it's likely that a Rails app is I/O-bound rather than CPU-bound, my experience leads me to think the reverse is currently likely. [*edit* after I wrote this comment, I noticed multiple other comments in this thread saying basically the same thing. OK, so. Can we put this rumor to rest?]

technion 4y ago
My preference at this point is rail's in api mode with a separate javascript fronted. I haven't had had to scale enough to really see it, but I often see rendering discussed as CPU heavy and I do wonder how much it helps that all my rendering is the OJ gem.
- aantix 4y ago
  With proper nested catching, you can see response times in 50ms. No need to over complicate the views with a separate js front end.
- xoac 4y ago
  If you go this route you end up having a drastically more complicated front end, and usually without the speed benefit. Unless you really need a SPA I would stay away.
treis 4y ago
Views are definitely one of the places that can get slow. Each partial is an IO read and they happen sequentially. Looping and rendering a partial is a common mistake that kills performance.
- byroot 4y ago
  > Each partial is an IO read No it's not. Once a partial has been compiled into bytecode, Rails doesn't reach to disk anymore, it's all in memory.
  - treis 4y ago
    This article says otherwise: https://scoutapm.com/blog/performance-impact-of-using-ruby-on-rails-view-partials https://scoutapm.com/blog/performance-impact-of-using-ruby-o... Maybe it has changed but either way they still can be a bottle neck
    - byroot 4y ago
      Yeah, that article is incorrect about the IO part, unless somehow they've been testing in development or something like that. It's however correct that partial rendering is quite more costly than it should, I explain why elsewhere in this thread.
    - e12e 4y ago
      That gives me a 500 error. At any rate, I think many rails developers should refresh their knowledge of rails caching: https://guides.rubyonrails.org/caching_with_rails.html https://guides.rubyonrails.org/caching_with_rails.html. For example using the data from the db (model instance) as cache key is quite effective solution for being able to deliver most view/fragments from cache. The downside is that some care must be taken with keys, which parts are cached (eg: logged in pages).
jbverschoor 4y ago
The whole I/O is the bottleneck thing is the same as saying “memory is cheap”. It’s true in a way, but that doesn’t mean we should all ignore it. It’s usually not a big problem, until it is. I love Ruby, but I love crystal in terms of performance, except for compilation times.
- nobleach 4y ago
  Crystal is just awesome. I doubt I'll have a chance to use it in a real app any time soon... but I've kept my eye on it for the past few years.
JamesSwift 4y ago
Agreed. With proper DB modeling and AR configuration I usually find a large majority of time spent in generating JSON for the apps I've worked on.
breckenedge 4y ago
The abundance of great APM suites (Datadog, NewRelic, etc) has been a huge help finding and fixing IO-related performance issues.
- ryanbrunner 4y ago
  For Rails specifically, I can't say enough good things about AppSignal. Being Rails specific gives it really great performance analytics right out of the box (it hooks into Rails standard instrumentation), and it handles error reporting for you as a bonus.
notpachet 4y ago
> I think it's time to stop repeating that it's likely that a Rails app is I/O-bound rather than CPU-bound, my experience leads me to think the reverse is currently likely. I'd like to offer an anecdotal counterpoint. I worked at Shopify for several years. The big problems were always due to I/O. Usually that was in the form of long-running network calls. You're calling the FedEx API for shipping rates in the cart. All of a sudden FedEx starts taking 5 seconds instead of 1 second, and all of your worker capacity evaporates. That kind of thing. Or, if databases are more your speed, we faced another problem with the GraphQL API needing to rattle off a bunch of requests to the DB, without being able to do so in parallel. So the more different slices of data you needed to fetch, the longer the single GraphQL call would take. We spent significant amounts of time finding ways to work around Rail's inability to do these sorts of things in parallel (or at least, the inability to do it without lots and lots of Spell-o-Tape). Writing proxy services in other languages that could parallelize an incoming request and fan it out to the actual Ruby boxes, etc. I like Ruby but I feel like it's pretty unequipped to deal with the problem of long-running I/O and worker saturation. There's work happening there but it will be a long time before you have the same ease-of-use as languages with better concurrency support out of the box. You could argue that these sorts of long-running calls are not the norm, but in my experience it's way more common for a web app to need to perform some call to an external service at some point during its lifetime than not, and that's not a use case that Ruby/Rails are built to elegantly handle.
- deleted 4y ago
  [deleted]
- neon_electro 4y ago
  I'm curious if you could share more on the GraphQL part of the situation you describe here, and if being architected on your more traditional "RESTful" (however you want to define that, but usually Rails defaults get you a wonderful implementation of "RESTful") API convention would have made any difference in the performance issues you encountered at Shopify.
  - notpachet 4y ago
    We didn't have the same problem with the old REST API, because the requests were naturally sliced by whatever resource type people were requesting. Whereas the GraphQL API allowed clients to request big slices of disparate data all in the same request. If we had (for whatever reason) exposed a REST API that fetched lots of different slices of data and combined them like GraphQL does, those requests would have faced the same problem. So it was really due to the nature of GraphQL that the problem materialized, rather than being a GraphQL vs REST issue per se.
jb3689 4y ago
> This is written as speculation of course, but it matches some long-time "conventional wisdom", that most Rails apps spend most of their time on I/O waiting for something external. > Can we put this rumor to rest? You have only given one example though. What rumor is there to put to rest? I think the statement is still accurate: "most Rails apps spend most of their time on I/O waiting for something external" To your specific concerns though - I have also see ActionView and ActiveRecord casting take up non-trivial amounts of time (I recall a situation in a past job where the mysql driver being notoriously bad with allocations of enums - probably our own doing however).
mrinterweb 4y ago
Of course mileage varies. I think it is fair to say that it is still the case that for many rails apps most of the request/response time is spent waiting on I/O. The DB I/O performance can be greatly improved by minimizing number of queries, indexing, query optimizations, ETC. DB I/O performance bottleneck is not a problem unique to rails, but maybe felt by rails devs more because the abstraction ActiveRecord provides doesn't make it obvious when you're misusing the DB.
rco8786 4y ago
> most Rails apps spend most of their time on I/O waiting for something external. Not just Rails apps. My experience has been that every "typical" business app falls into this bucket.

technion 4y ago

My preference at this point is rail's in api mode with a separate javascript fronted. I haven't had had to scale enough to really see it, but I often see rendering discussed as CPU heavy and I do wonder how much it helps that all my rendering is the OJ gem.

aantix 4y ago
With proper nested catching, you can see response times in 50ms. No need to over complicate the views with a separate js front end.
xoac 4y ago
If you go this route you end up having a drastically more complicated front end, and usually without the speed benefit. Unless you really need a SPA I would stay away.

treis 4y ago

Views are definitely one of the places that can get slow. Each partial is an IO read and they happen sequentially. Looping and rendering a partial is a common mistake that kills performance.

byroot 4y ago
> Each partial is an IO read No it's not. Once a partial has been compiled into bytecode, Rails doesn't reach to disk anymore, it's all in memory.
- treis 4y ago
  This article says otherwise: https://scoutapm.com/blog/performance-impact-of-using-ruby-on-rails-view-partials https://scoutapm.com/blog/performance-impact-of-using-ruby-o... Maybe it has changed but either way they still can be a bottle neck
  - byroot 4y ago
    Yeah, that article is incorrect about the IO part, unless somehow they've been testing in development or something like that. It's however correct that partial rendering is quite more costly than it should, I explain why elsewhere in this thread.
  - e12e 4y ago
    That gives me a 500 error. At any rate, I think many rails developers should refresh their knowledge of rails caching: https://guides.rubyonrails.org/caching_with_rails.html https://guides.rubyonrails.org/caching_with_rails.html. For example using the data from the db (model instance) as cache key is quite effective solution for being able to deliver most view/fragments from cache. The downside is that some care must be taken with keys, which parts are cached (eg: logged in pages).

jbverschoor 4y ago

The whole I/O is the bottleneck thing is the same as saying “memory is cheap”. It’s true in a way, but that doesn’t mean we should all ignore it. It’s usually not a big problem, until it is. I love Ruby, but I love crystal in terms of performance, except for compilation times.

nobleach 4y ago
Crystal is just awesome. I doubt I'll have a chance to use it in a real app any time soon... but I've kept my eye on it for the past few years.

JamesSwift 4y ago

Agreed. With proper DB modeling and AR configuration I usually find a large majority of time spent in generating JSON for the apps I've worked on.

breckenedge 4y ago

The abundance of great APM suites (Datadog, NewRelic, etc) has been a huge help finding and fixing IO-related performance issues.

ryanbrunner 4y ago
For Rails specifically, I can't say enough good things about AppSignal. Being Rails specific gives it really great performance analytics right out of the box (it hooks into Rails standard instrumentation), and it handles error reporting for you as a bonus.

notpachet 4y ago

> I think it's time to stop repeating that it's likely that a Rails app is I/O-bound rather than CPU-bound, my experience leads me to think the reverse is currently likely. I'd like to offer an anecdotal counterpoint. I worked at Shopify for several years. The big problems were always due to I/O. Usually that was in the form of long-running network calls. You're calling the FedEx API for shipping rates in the cart. All of a sudden FedEx starts taking 5 seconds instead of 1 second, and all of your worker capacity evaporates. That kind of thing. Or, if databases are more your speed, we faced another problem with the GraphQL API needing to rattle off a bunch of requests to the DB, without being able to do so in parallel. So the more different slices of data you needed to fetch, the longer the single GraphQL call would take. We spent significant amounts of time finding ways to work around Rail's inability to do these sorts of things in parallel (or at least, the inability to do it without lots and lots of Spell-o-Tape). Writing proxy services in other languages that could parallelize an incoming request and fan it out to the actual Ruby boxes, etc. I like Ruby but I feel like it's pretty unequipped to deal with the problem of long-running I/O and worker saturation. There's work happening there but it will be a long time before you have the same ease-of-use as languages with better concurrency support out of the box. You could argue that these sorts of long-running calls are not the norm, but in my experience it's way more common for a web app to need to perform some call to an external service at some point during its lifetime than not, and that's not a use case that Ruby/Rails are built to elegantly handle.

deleted 4y ago
[deleted]
neon_electro 4y ago
I'm curious if you could share more on the GraphQL part of the situation you describe here, and if being architected on your more traditional "RESTful" (however you want to define that, but usually Rails defaults get you a wonderful implementation of "RESTful") API convention would have made any difference in the performance issues you encountered at Shopify.
- notpachet 4y ago
  We didn't have the same problem with the old REST API, because the requests were naturally sliced by whatever resource type people were requesting. Whereas the GraphQL API allowed clients to request big slices of disparate data all in the same request. If we had (for whatever reason) exposed a REST API that fetched lots of different slices of data and combined them like GraphQL does, those requests would have faced the same problem. So it was really due to the nature of GraphQL that the problem materialized, rather than being a GraphQL vs REST issue per se.

jb3689 4y ago

> This is written as speculation of course, but it matches some long-time "conventional wisdom", that most Rails apps spend most of their time on I/O waiting for something external. > Can we put this rumor to rest? You have only given one example though. What rumor is there to put to rest? I think the statement is still accurate: "most Rails apps spend most of their time on I/O waiting for something external" To your specific concerns though - I have also see ActionView and ActiveRecord casting take up non-trivial amounts of time (I recall a situation in a past job where the mysql driver being notoriously bad with allocations of enums - probably our own doing however).

mrinterweb 4y ago

Of course mileage varies. I think it is fair to say that it is still the case that for many rails apps most of the request/response time is spent waiting on I/O. The DB I/O performance can be greatly improved by minimizing number of queries, indexing, query optimizations, ETC. DB I/O performance bottleneck is not a problem unique to rails, but maybe felt by rails devs more because the abstraction ActiveRecord provides doesn't make it obvious when you're misusing the DB.

rco8786 4y ago

> most Rails apps spend most of their time on I/O waiting for something external. Not just Rails apps. My experience has been that every "typical" business app falls into this bucket.