6 ms·

Ruby on Rails load testing habits

128 points by a12b 3y ago

rdoherty 3y ago
These are great tips for load testing any web service. Don't forget that you can also saturate memory, disk or network too, so watch those graphs too. I've seen load tests unable to saturate CPU because another resource was limited. Also don't forget that you are load testing all the dependencies of your service. Database, caching tier, external services, etc. Make sure other teams are aware! Also nothing beats real world traffic. Users' connections will stay open longer than a synthetic tool may hold them open due to bandwidth, they make very random, sporadic requests too. Your service will behave very differently under large amounts of real world traffic vs synthetic. Other options if you are running multiple web servers is to shift traffic around to increase traffic to 1 host and see where it fails. That is usually a very reliable signal for peak load. And don't forget to do this on a schedule as your codebase (and your dependencies codebases) changes!
- wdfx 3y ago
  > external services, etc. Make sure other teams are aware! Guilty. I've had one of our partners call me one time because I'd caused a huge load on their end. Lots of apologies and embarrassment followed. Later I mocked out those external calls with stubs which behaved similarly, in that I could specify the min/max/average wait times and error rates.
Hendrikto 3y ago
> If the application can’t saturate the CPU, there’s a fundamental problem. It’s a shame because it makes adding more servers less efficient. Money is being wasted on hosting costs, and this should be a priority to address. Talking about efficiency being a priority, but using RoR. I guess that is one way of saturating the CPU.
- stouset 3y ago
  If your application can’t saturate the CPU, Rails probably isn’t your bottleneck.
  - resonious 3y ago
    [flagged]
- block_dagger 3y ago
  The author means efficiency in terms of the software that is running, not efficiency in terms of all possible optimizations such as switching to a different lang.
- irjustin 3y ago
  While you missed the point of the saturation comment, this is why we love AWS Lambda over ECS+Fargate. Rails has really poor startup time due to loading all codepaths. We switched to Django and it runs beautifuly on AWS Lambda where our CI is more expensive than actual server costs. We're a b2b application so traffic is quite low so we REALLY don't saturate the CPU in a normal Fargate setup.
  - cocoflunchy 3y ago
    I'm surprised to see a mention of Django when talking about fast startup times! This is one of my main issues with Django at the moment. How big is your project? Our ~500k lines app takes multiple seconds to start, which is why I'm not really investigating a lambda-style setup... Do you have specific strategies to make startup fast?
    - irjustin 3y ago
      Apologies on the lay reply. Your app is significantly bigger than ours, so grain of salt. We play very close attention to what's loaded on startup. There are two key tricks. 1. Heavy libraries/packages load at runtime and are only in the "background job" codepath. def my_heavy_func(): from heavy_library import sum_heavy_function sum_heavy_function() vs the import at top of file. 2. Limit which apps are loaded via `INSTALLED_APPS`, again no heavy packages. Lambda is SUPER nice for us. The bottleneck becomes the DB. Webserver can basically never go down on its own as you can create 1000x by default. Best of Luck!
- eek2121 3y ago
  I have never seen Ruby on Rails be a bottleneck, and I have been using it since version 2. Most bottlenecks are either that database choices or poor code/design choices by developers. That is especially true today.
  - ndriscoll 3y ago
    Doesn't rails not have multithreading? How do you gather requests to batch your database calls? A coworker made similar claims to me about Laravel, but the framework really encourages you to do half a dozen database queries in even a pretty minimal request, and for example implemented bulk inserts as a for loop that did single inserts. If you didn't know better with an access pattern like that, you might think the database is the bottleneck long before it actually should be. Is Rails different? My sense was they are very similar.
    - _chap 3y ago
      Rails supports multiple threads: https://guides.rubyonrails.org/threading_and_code_execution.html https://guides.rubyonrails.org/threading_and_code_execution.... Rails has some tooling to help with query bloat: https://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations https://guides.rubyonrails.org/active_record_querying.html#e...
      - ndriscoll 3y ago
        Interesting. My understanding is that part of why mastodon is so slow/resource hungry is that it serializes background tasks to redis for a sidecar process to pick up, and that that's the normal way to do things. If rails has a concurrent runtime, why don't they just run background work directly?
        jmuguy 3y ago
        Rails isn't super opinionated about database writes, its mostly left up to developers to discover that for relational DBs you do not want to be doing a bunch of small writes all at once. That said it specifically has tools to address this that started appearing a few years ago https://github.com/rails/rails/pull/35077 https://github.com/rails/rails/pull/35077 The way my team handles it is to stick Kafka in between whats generating the records (for us, a bunch of web scraping workers) and and a consumer that pulls off the Kafka queue and runs an insert when its internal buffer reaches around 50k rows. Rails is also looking to add some more direct background type work with https://github.com/basecamp/solid_queue https://github.com/basecamp/solid_queue but this is still very new - most larger Rails shops are going to be running a second system and a gem called Sidekiq that pulls jobs out of Redis. In terms of read queries, again I think that comes down to the individual team realizing (hopefully very early in their careers) that's something that needs to be considered. Rails isn't going to stop you from doing N+1 type mistakes or hammering your DB with 30 separate queries for each page load. But it has plenty of tools and documentation on how to do that better.
pistoriusp 3y ago
Having your test database mirror production as closely as possible is also an important habit, however I’m biased since that’s part of the offering that I’m building.
- e12e 3y ago
  You might want to try and maintain a synthetic dataset for testing and staging that has the same "shape" as your production data - to avoid exposing sensitive data. We're currently trying to have each rails model implement a #new_example method that builds a valid subgraph filled in by Faker, ready to save. Ie a user = User.new_example will come with a Company.new_example if every user needs a company relationship. Still early, we'll see how it goes.
  - pistoriusp 3y ago
    We're doing the same, but for the TypeScript world with "Snaplet Seed." We use generative AI to generate deterministic values + the required relational data: https://www.snaplet.dev/seed https://www.snaplet.dev/seed We generate data based off of your database schema and your production data (if you give us access.) Since you've kinda already built something like this I would be curious to hear what you think!
  - bdcravens 3y ago
    Wouldn't it be easier to do the same with FactoryBot? It'll similarly cascade creation of associated records.
    - e12e 3y ago
      I don't enjoy documenting the graph in two places, first in models, then in factories. But yes, the pattern is essentially the same, just our example methods and Faker - without Factory Bot.
      - bdcravens 3y ago
        I can understand that. I prefer keeping my models clean without environment-specific implementation details, which is why I've settled on the FactoryBot approach for testing, seeding, etc.
ericb 3y ago
If you'd like to write your load tests in Ruby (plain ruby or browser based), and use your own libraries (internal libraries, specs, ets) browserup does that: Ruby with browser: https://browserup.com/docs/en/load/ruby-load-test.html https://browserup.com/docs/en/load/ruby-load-test.html Command-line installation: https://www.npmjs.com/package/browserup https://www.npmjs.com/package/browserup
sfc32 3y ago
RoRvsWild is a pretty cool product, and built by 2 of the nicest guys in the Ruby ecosystem.
- antoinem 3y ago
  Aww, thank you! Who are you, lovely anonymous friend?
mplewis 3y ago
If you're looking to run programmable load test scenarios as described in this blog post, consider checking out https://k6.io/ https://k6.io/, which I've found to be extremely easy to use and powerful.
anonacct37 3y ago
> My initial requirement was to send requests with unique parameters. To the best of my knowledge, no tool could do this. wrk does this with lua. https://github.com/wg/wrk/blob/master/src/wrk.lua https://github.com/wg/wrk/blob/master/src/wrk.lua Also even things like the venerable jmeter supported pulling parameters from a csv file.
- nomilk 3y ago
  I've never done load testing before, but would it be hard to write a script in pure ruby (maybe with a few libraries) that makes a lot of concurrent requests to whatever endpoints and using whatever params you like?
  - paulryanrogers 3y ago
    Depends on needs and familiarity. I used Bash and Curl, then PHP, then settled on Jmeter for a tricky bug due to a memory leak from a race under load. All three could reproduce the problem. Jmeter had quite a learning curve, and preparing data for it was a pain. Ultimately though it was more reliable, can do distributed testing, and has lots of nice built in features and analytics.
  - toast0 3y ago
    Load testing can be harder on the client side than the server side at high loads, which might be surprising, but consider how many servers need to handle large numbers of clients connecting and how few clients need to connect to large number of servers --- if you do it yourself, you're going to have to relearn all the techniques to make large number of connections possible. If your desired load for testing is small, it's not a big deal, of course.
  - vidarh 3y ago
    It is simple enough to do a script doing that in Ruby, but there also enough off the shelf tools specifically for load testing that encodes a lot of experience and avoids a lot of obvious and not so obvious mistakes, so it's usually not worth writing your own.
  - 3y ago
rubyissimo 3y ago
fwiw, I know it's the right thing to run tests from a different computer. But it's more annoying. And I hereby tell you that 90% of the time it probably doesn't matter. Definitely times it isn't true. But if you're not doing a load test bc it's a pita, do it locally. Most of the time I've wanted to do this, all the action is inside the app. Just be careful to acknowledge that there could be limitations / surprises.

128 points by a12b 3y ago

These are great tips for load testing any web service. Don't forget that you can also saturate memory, disk or network too, so watch those graphs too. I've seen load tests unable to saturate CPU because another resource was limited. Also don't forget that you are load testing all the dependencies of your service. Database, caching tier, external services, etc. Make sure other teams are aware! Also nothing beats real world traffic. Users' connections will stay open longer than a synthetic tool may hold them open due to bandwidth, they make very random, sporadic requests too. Your service will behave very differently under large amounts of real world traffic vs synthetic. Other options if you are running multiple web servers is to shift traffic around to increase traffic to 1 host and see where it fails. That is usually a very reliable signal for peak load. And don't forget to do this on a schedule as your codebase (and your dependencies codebases) changes!

wdfx 3y ago
> external services, etc. Make sure other teams are aware! Guilty. I've had one of our partners call me one time because I'd caused a huge load on their end. Lots of apologies and embarrassment followed. Later I mocked out those external calls with stubs which behaved similarly, in that I could specify the min/max/average wait times and error rates.

Hendrikto 3y ago

> If the application can’t saturate the CPU, there’s a fundamental problem. It’s a shame because it makes adding more servers less efficient. Money is being wasted on hosting costs, and this should be a priority to address. Talking about efficiency being a priority, but using RoR. I guess that is one way of saturating the CPU.

stouset 3y ago
If your application can’t saturate the CPU, Rails probably isn’t your bottleneck.
- resonious 3y ago
  [flagged]
block_dagger 3y ago
The author means efficiency in terms of the software that is running, not efficiency in terms of all possible optimizations such as switching to a different lang.
irjustin 3y ago
While you missed the point of the saturation comment, this is why we love AWS Lambda over ECS+Fargate. Rails has really poor startup time due to loading all codepaths. We switched to Django and it runs beautifuly on AWS Lambda where our CI is more expensive than actual server costs. We're a b2b application so traffic is quite low so we REALLY don't saturate the CPU in a normal Fargate setup.
- cocoflunchy 3y ago
  I'm surprised to see a mention of Django when talking about fast startup times! This is one of my main issues with Django at the moment. How big is your project? Our ~500k lines app takes multiple seconds to start, which is why I'm not really investigating a lambda-style setup... Do you have specific strategies to make startup fast?
  - irjustin 3y ago
    Apologies on the lay reply. Your app is significantly bigger than ours, so grain of salt. We play very close attention to what's loaded on startup. There are two key tricks. 1. Heavy libraries/packages load at runtime and are only in the "background job" codepath. def my_heavy_func(): from heavy_library import sum_heavy_function sum_heavy_function() vs the import at top of file. 2. Limit which apps are loaded via `INSTALLED_APPS`, again no heavy packages. Lambda is SUPER nice for us. The bottleneck becomes the DB. Webserver can basically never go down on its own as you can create 1000x by default. Best of Luck!
eek2121 3y ago
I have never seen Ruby on Rails be a bottleneck, and I have been using it since version 2. Most bottlenecks are either that database choices or poor code/design choices by developers. That is especially true today.
- ndriscoll 3y ago
  Doesn't rails not have multithreading? How do you gather requests to batch your database calls? A coworker made similar claims to me about Laravel, but the framework really encourages you to do half a dozen database queries in even a pretty minimal request, and for example implemented bulk inserts as a for loop that did single inserts. If you didn't know better with an access pattern like that, you might think the database is the bottleneck long before it actually should be. Is Rails different? My sense was they are very similar.
  - _chap 3y ago
    Rails supports multiple threads: https://guides.rubyonrails.org/threading_and_code_execution.html https://guides.rubyonrails.org/threading_and_code_execution.... Rails has some tooling to help with query bloat: https://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations https://guides.rubyonrails.org/active_record_querying.html#e...
    - ndriscoll 3y ago
      Interesting. My understanding is that part of why mastodon is so slow/resource hungry is that it serializes background tasks to redis for a sidecar process to pick up, and that that's the normal way to do things. If rails has a concurrent runtime, why don't they just run background work directly?
      - jmuguy 3y ago
        Rails isn't super opinionated about database writes, its mostly left up to developers to discover that for relational DBs you do not want to be doing a bunch of small writes all at once. That said it specifically has tools to address this that started appearing a few years ago https://github.com/rails/rails/pull/35077 https://github.com/rails/rails/pull/35077 The way my team handles it is to stick Kafka in between whats generating the records (for us, a bunch of web scraping workers) and and a consumer that pulls off the Kafka queue and runs an insert when its internal buffer reaches around 50k rows. Rails is also looking to add some more direct background type work with https://github.com/basecamp/solid_queue https://github.com/basecamp/solid_queue but this is still very new - most larger Rails shops are going to be running a second system and a gem called Sidekiq that pulls jobs out of Redis. In terms of read queries, again I think that comes down to the individual team realizing (hopefully very early in their careers) that's something that needs to be considered. Rails isn't going to stop you from doing N+1 type mistakes or hammering your DB with 30 separate queries for each page load. But it has plenty of tools and documentation on how to do that better.

pistoriusp 3y ago

Having your test database mirror production as closely as possible is also an important habit, however I’m biased since that’s part of the offering that I’m building.

e12e 3y ago
You might want to try and maintain a synthetic dataset for testing and staging that has the same "shape" as your production data - to avoid exposing sensitive data. We're currently trying to have each rails model implement a #new_example method that builds a valid subgraph filled in by Faker, ready to save. Ie a user = User.new_example will come with a Company.new_example if every user needs a company relationship. Still early, we'll see how it goes.
- pistoriusp 3y ago
  We're doing the same, but for the TypeScript world with "Snaplet Seed." We use generative AI to generate deterministic values + the required relational data: https://www.snaplet.dev/seed https://www.snaplet.dev/seed We generate data based off of your database schema and your production data (if you give us access.) Since you've kinda already built something like this I would be curious to hear what you think!
- bdcravens 3y ago
  Wouldn't it be easier to do the same with FactoryBot? It'll similarly cascade creation of associated records.
  - e12e 3y ago
    I don't enjoy documenting the graph in two places, first in models, then in factories. But yes, the pattern is essentially the same, just our example methods and Faker - without Factory Bot.
    - bdcravens 3y ago
      I can understand that. I prefer keeping my models clean without environment-specific implementation details, which is why I've settled on the FactoryBot approach for testing, seeding, etc.

ericb 3y ago

If you'd like to write your load tests in Ruby (plain ruby or browser based), and use your own libraries (internal libraries, specs, ets) browserup does that: Ruby with browser: https://browserup.com/docs/en/load/ruby-load-test.html https://browserup.com/docs/en/load/ruby-load-test.html Command-line installation: https://www.npmjs.com/package/browserup https://www.npmjs.com/package/browserup

sfc32 3y ago

RoRvsWild is a pretty cool product, and built by 2 of the nicest guys in the Ruby ecosystem.

antoinem 3y ago
Aww, thank you! Who are you, lovely anonymous friend?

mplewis 3y ago

If you're looking to run programmable load test scenarios as described in this blog post, consider checking out https://k6.io/ https://k6.io/, which I've found to be extremely easy to use and powerful.

anonacct37 3y ago

> My initial requirement was to send requests with unique parameters. To the best of my knowledge, no tool could do this. wrk does this with lua. https://github.com/wg/wrk/blob/master/src/wrk.lua https://github.com/wg/wrk/blob/master/src/wrk.lua Also even things like the venerable jmeter supported pulling parameters from a csv file.

nomilk 3y ago
I've never done load testing before, but would it be hard to write a script in pure ruby (maybe with a few libraries) that makes a lot of concurrent requests to whatever endpoints and using whatever params you like?
- paulryanrogers 3y ago
  Depends on needs and familiarity. I used Bash and Curl, then PHP, then settled on Jmeter for a tricky bug due to a memory leak from a race under load. All three could reproduce the problem. Jmeter had quite a learning curve, and preparing data for it was a pain. Ultimately though it was more reliable, can do distributed testing, and has lots of nice built in features and analytics.
- toast0 3y ago
  Load testing can be harder on the client side than the server side at high loads, which might be surprising, but consider how many servers need to handle large numbers of clients connecting and how few clients need to connect to large number of servers --- if you do it yourself, you're going to have to relearn all the techniques to make large number of connections possible. If your desired load for testing is small, it's not a big deal, of course.
- vidarh 3y ago
  It is simple enough to do a script doing that in Ruby, but there also enough off the shelf tools specifically for load testing that encodes a lot of experience and avoids a lot of obvious and not so obvious mistakes, so it's usually not worth writing your own.
- 3y ago

rubyissimo 3y ago

fwiw, I know it's the right thing to run tests from a different computer. But it's more annoying. And I hereby tell you that 90% of the time it probably doesn't matter. Definitely times it isn't true. But if you're not doing a load test bc it's a pita, do it locally. Most of the time I've wanted to do this, all the action is inside the app. Just be careful to acknowledge that there could be limitations / surprises.