5 ms·

Load Testing: An Unorthodox Guide

90 points by fagnerbrack 4y ago

cvwright 4y ago
Agreed with a lot of the points here, like starting small with a single piece of your API, then slowly expanding your tests once you’re comfortable that you know what you’re doing. Note that if you use the Locust framework to write your load tests in Python, it takes care of measuring and reporting the latency and throughput for you. It’s really nice. https://locust.io/ https://locust.io/
- sparsely 4y ago
  This seems to encourage you to run it with a target number of concurrent users, rather than a target RPS from the loaders, which IME can result in difficulty interpreting the results (it also doesn't reflect most use cases, except in some heavily controlled scenarios). When latencies increase, the fixed number of virtual users will necessarily be sending fewer new requests, leading to a constant or decreasing number of RPS actually being served. Other load testing tools (Gatling, newer versions of k6) will let you set a target RPS instead.
  - ericb 4y ago
    Targeting RPS without also targeting concurrent users is a huge mistake! Here's a metaphor I like. Imagine I am trying to open a small high-volume restaurant. I call the chef in for a trial run. I sit at a table, and while I watch, she is able to make 200 dishes an hour. I'm excited, at 200 dishes an hour, we'll be rich! I tell the investors. Opening day comes! I have 10 tables and people eat for an hour on average. We serve 10 dishes an hour. Our restaurant fails. Throughput, without targeting concurrent users, is a bad test.
    - bombcar 4y ago
      Part of it is identifying the bottleneck. You verified the chef wouldn’t be it, but the tables ended up being the limiting factor. And say you installed 200 tables, but only get ten customers an hour - now the bottleneck is the input and that likely has to be solved by marketing or something exterior. And maybe you get that fixed and now have 200 tables an hour, wnd discover your dishwasher can only handle 50 tables an hour. So you need more dishes (buffer) but that will only help until the poor dishwasher is washing dishes 24/7 and at that point the buffer will eventually run out. Successfully modeling the system and identifying the point at which it will fail is useful, because then you can keep an eye on that point AND know if something unexpected is causing it to fail earlier.
    - sparsely 4y ago
      I don't really understand how this analogy works in terms of the actual practice of running load tests. You should of course test the full flow, the important part is not implicitly limiting the input volume based on the system performance. I think in your analogy it would be something about people queuing outside without you noticing but am unsure. Also, to be clear, the bad thing is normally doing closed workload tests, not having a multistep workflow or whatever (https://gatling.io/docs/gatling/reference/current/core/injection/ https://gatling.io/docs/gatling/reference/current/core/injec...)
      - ericb 4y ago
        To complete the analogy, you need to have one concurrent session per "virtual user." This is the "concurrency" of the test. In my bad test, I had one concurrent user driving all the throughput--me. If I had targeted 200 concurrent users, I would have seen a failure in my test and found what the real throughput would be. Each concurrent user uses many resources that could be limited. In the restaurant, it is tables, plates, silverware. In a web app, it is sessions, connections, memory, database connections, and many other resources that can be associated with each session, and each may be a potential bottleneck that limits your actual throughput. If we target throughput and ignore concurrency, we set ourselves up for failure. In practicality, the better load testing tools let you create "concurrent users." In JMeter, this is the "threads." In other tools, it will be called "virtual users", "vus" or "sessions." In locust, the setting is -u NUM_USERS. The caution is not to fool yourself with high RPS if these settings aren't right for your test, and to normally target both.
sparsely 4y ago
It's very easy to misinterpret the results of load tests. A common mistake is to run the wrong sort of load test (explained here[1]) and have your loaders automatically back off when latency starts increasing. As the author of the OP touches on in a different context, you're now running a pseudo-limit test instead! I like the author's emphasis on only proceeding if the server's metrics are clearly within normal ranges. [1] https://gatling.io/docs/gatling/reference/current/core/injection/ https://gatling.io/docs/gatling/reference/current/core/injec...
kqr 4y ago
Very good article, I think! Reveals many of the subtleties that are easy to get wrong. ---- > A common shortcut is to generate the load on the same machine (i.e. the developer’s laptop), that the server is running on. What’s problematic about that? Generating load needs CPU/Memory/Network Traffic/IO and that will naturally skew your test results, as to what capacity your server can handle requests. This fear is overplayed, I think. A lot of production software is slow enough that load generation will not be a significant CPU/memory/network load. By worrying about this people miss other things that they should worry about much more. ---- > Run the actual load test for 1-5 minutes. (great numbers, huh?) This is another common problem: for many garbage collected platforms (and platforms with other types of occasional delayed work), 1–5 minutes is not enough to trigger the pathological behaviour you want to test for. Some systems need to be loaded for 4–6 hours or more to see the full spectrum of behaviour.
- drewcoo 4y ago
  > This fear is overplayed, I think. It can be difficult to separate the effects on the system from the loader and the server. I think if it's overplayed it's because it's trying to convince people to not make it hard. Even doing them both on the same box is valuable, though. Functional tests aren't going to catch concurrency issues but load testing however you want to do it just might.
AzzieElbab 4y ago
Good article. One thing I would add is the necessity to include failures/unhappy path scenarios in your load scripts. See if all that logging/tracing really is free :) Slightly off topic: do people still run any load/performance tests on cloud hosted sites? it seems like spiking your aws load is becoming prohibitively expensive. Also, in the real world replacing third party services with some intelligent mocks(simulating failures and latency) can be super complex.
- noltncy 4y ago
  Yes cloud hosted apps are still tested. You will need to let your cloud provide know about your tests or you (IPs) might get banned.
  - AzzieElbab 4y ago
    I am talking about monetary costs resulting from actually pushing your load balanced elastic cluster to its limits.
vinay_ys 4y ago
Building systems that can scale efficiently requires careful design and implementation. I have seen teams relax the efficiency requirement and manage to handle high load by pre-scaling-up the provisioned capacity of their systems many hours in advance of an anticipated traffic load. This is obviously very wasteful of resources, but also it makes your system brittle, non-resilient. An unanticipated spike in load that stresses the system beyond its provisioned capacity will cause the system to overload and crash/jam-up. So, rather than testing if your system can handle high load when your system is pre-scaled, testing if your system can continue to provide good throughput even when under heavy load is a better test. Then, if your system is designed to auto-scale in an elastic cloud, testing fast auto-scaling of your systems to see if it increases the good throughput dynamically is a better strategy. Load testing should be about ascertaining the resiliency, scalability and efficiency characteristics of your system architecture and deployment setup. Some definitions: 1. A system is said to be resilient if it can sustain the good throughput even when under load excess relative to available system resources. 2. A system is said to be scalable if it can quickly respond to increasing load by scaling up its provisioned resources and scale-up its good throughput. 3. Good throughput is the throughput (request/sec, concurrent user sessions etc) of useful work done with acceptable performance characteristics – ie., latency curve at different percentiles is acceptable. 4. Efficiency of scalability is the ratio of increase in good throughput to increase in provisioned resources. 5. Efficiency in general is the concurrent user sessions or requests/sec per node of the provisioned system.
i_like_apis 4y ago
I used to do this a lot. It may look a little clunky, but there is a piece of software called JMeter (https://jmeter.apache.org/ https://jmeter.apache.org/) that is capable of doing just about anything you would want to do in a load test - modeling any request behavior, distributed traffic, awesome reports, etc. 10/10 recommend.
- drewcoo 4y ago
  JMeter is old and crusty and not at all friendly to work with. But I used it for years because it was really about the best we had. Today I don't wish it on anyone. Ruby JMeter finally made JMeter easier to manage, but I haven't worked in a Ruby shop for years, and I'm not going to force everyone to learn Ruby just to do some load testing. https://github.com/flood-io/ruby-jmeter https://github.com/flood-io/ruby-jmeter Then along came k6. It's developer-friendly and I've seen people actually enjoy using it. I recommend anyone considering JMeter also take a look at k6. They do a better job of selling it than I do: https://k6.io https://k6.io I am also Gatling-curious. Seems like an option for anyone in the JVM ecosystem. https://gatling.io https://gatling.io
  - brabel 4y ago
    Looks like K6 is basically Gatling but with a JS-based DSL while Gatling uses Scala (or anything JVM-based like Java/Groovy/Kotlin)... there's also Locust[1] which is Python-based. What I really wanted was an interactive tool, like old LoadUI used to do (Wikipedia, nicely, still has the old screenshots which show how cool it used to look: https://en.wikipedia.org/wiki/LoadUI https://en.wikipedia.org/wiki/LoadUI)... because until you run tests, you just don't know what kind of load you can throw at the server (and whether you're hitting CPU/Memory/Bandwitdth limits instead of server limits). Visual Components were written in a scripting DSL (Groovy based) and dynamically loaded so you could change the code even while the test ran... really awesome stuff, a bit like coding in Lisp with a visual facade on top. Based on existing tools, it should be relatively easy to build something like that and I am surprised there seems to be nothing of the kind. I've always wanted to do it myself, maybe one day if no one else finally tackles the problem. [1] https://docs.locust.io/en/stable/ https://docs.locust.io/en/stable/
arein3 4y ago
Did anyone use hedless chrome with puppeteer for load tests?
- dev_throw 4y ago
  None of the utility libraries helped with this, instead wrote a shell script to orchestrate concurrent puppeteer load tests with jest and collect and parse reports. It is very resource intensive and spiky. I'm looking to see how we can make it more efficient.
- hassy 4y ago
  This lets you load test with Playwright (very similar to Puppeteer): https://github.com/artilleryio/artillery-engine-playwright https://github.com/artilleryio/artillery-engine-playwright
deleted 4y ago
[deleted]
aymar_99 4y ago
https://gatling.io/ https://gatling.io/ Gatling is one among the load test tools people need to give a shot. It uses akka for generating asynchronous loads. And it's good from a programmer standpoint since load test scripts can be programmed and maintained as any other script.
mikessoft_gmail 4y ago
MikeYasnev007 4y ago
LA is marketing cinema and drive) Load testing is more for Arizona server farms
fizwhiz 4y ago
> It’s also important that the loader generates those requests at a constant rate, best done asynchronously, so that response processing doesn’t get in the way of sending out new requests. Coordinated Omission is subtle but deserves more color: https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHfE https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHf...

90 points by fagnerbrack 4y ago

Agreed with a lot of the points here, like starting small with a single piece of your API, then slowly expanding your tests once you’re comfortable that you know what you’re doing. Note that if you use the Locust framework to write your load tests in Python, it takes care of measuring and reporting the latency and throughput for you. It’s really nice. https://locust.io/ https://locust.io/

sparsely 4y ago
This seems to encourage you to run it with a target number of concurrent users, rather than a target RPS from the loaders, which IME can result in difficulty interpreting the results (it also doesn't reflect most use cases, except in some heavily controlled scenarios). When latencies increase, the fixed number of virtual users will necessarily be sending fewer new requests, leading to a constant or decreasing number of RPS actually being served. Other load testing tools (Gatling, newer versions of k6) will let you set a target RPS instead.
- ericb 4y ago
  Targeting RPS without also targeting concurrent users is a huge mistake! Here's a metaphor I like. Imagine I am trying to open a small high-volume restaurant. I call the chef in for a trial run. I sit at a table, and while I watch, she is able to make 200 dishes an hour. I'm excited, at 200 dishes an hour, we'll be rich! I tell the investors. Opening day comes! I have 10 tables and people eat for an hour on average. We serve 10 dishes an hour. Our restaurant fails. Throughput, without targeting concurrent users, is a bad test.
  - bombcar 4y ago
    Part of it is identifying the bottleneck. You verified the chef wouldn’t be it, but the tables ended up being the limiting factor. And say you installed 200 tables, but only get ten customers an hour - now the bottleneck is the input and that likely has to be solved by marketing or something exterior. And maybe you get that fixed and now have 200 tables an hour, wnd discover your dishwasher can only handle 50 tables an hour. So you need more dishes (buffer) but that will only help until the poor dishwasher is washing dishes 24/7 and at that point the buffer will eventually run out. Successfully modeling the system and identifying the point at which it will fail is useful, because then you can keep an eye on that point AND know if something unexpected is causing it to fail earlier.
  - sparsely 4y ago
    I don't really understand how this analogy works in terms of the actual practice of running load tests. You should of course test the full flow, the important part is not implicitly limiting the input volume based on the system performance. I think in your analogy it would be something about people queuing outside without you noticing but am unsure. Also, to be clear, the bad thing is normally doing closed workload tests, not having a multistep workflow or whatever (https://gatling.io/docs/gatling/reference/current/core/injection/ https://gatling.io/docs/gatling/reference/current/core/injec...)
    - ericb 4y ago
      To complete the analogy, you need to have one concurrent session per "virtual user." This is the "concurrency" of the test. In my bad test, I had one concurrent user driving all the throughput--me. If I had targeted 200 concurrent users, I would have seen a failure in my test and found what the real throughput would be. Each concurrent user uses many resources that could be limited. In the restaurant, it is tables, plates, silverware. In a web app, it is sessions, connections, memory, database connections, and many other resources that can be associated with each session, and each may be a potential bottleneck that limits your actual throughput. If we target throughput and ignore concurrency, we set ourselves up for failure. In practicality, the better load testing tools let you create "concurrent users." In JMeter, this is the "threads." In other tools, it will be called "virtual users", "vus" or "sessions." In locust, the setting is -u NUM_USERS. The caution is not to fool yourself with high RPS if these settings aren't right for your test, and to normally target both.

sparsely 4y ago

It's very easy to misinterpret the results of load tests. A common mistake is to run the wrong sort of load test (explained here[1]) and have your loaders automatically back off when latency starts increasing. As the author of the OP touches on in a different context, you're now running a pseudo-limit test instead! I like the author's emphasis on only proceeding if the server's metrics are clearly within normal ranges. [1] https://gatling.io/docs/gatling/reference/current/core/injection/ https://gatling.io/docs/gatling/reference/current/core/injec...

kqr 4y ago

Very good article, I think! Reveals many of the subtleties that are easy to get wrong. ---- > A common shortcut is to generate the load on the same machine (i.e. the developer’s laptop), that the server is running on. What’s problematic about that? Generating load needs CPU/Memory/Network Traffic/IO and that will naturally skew your test results, as to what capacity your server can handle requests. This fear is overplayed, I think. A lot of production software is slow enough that load generation will not be a significant CPU/memory/network load. By worrying about this people miss other things that they should worry about much more. ---- > Run the actual load test for 1-5 minutes. (great numbers, huh?) This is another common problem: for many garbage collected platforms (and platforms with other types of occasional delayed work), 1–5 minutes is not enough to trigger the pathological behaviour you want to test for. Some systems need to be loaded for 4–6 hours or more to see the full spectrum of behaviour.

drewcoo 4y ago
> This fear is overplayed, I think. It can be difficult to separate the effects on the system from the loader and the server. I think if it's overplayed it's because it's trying to convince people to not make it hard. Even doing them both on the same box is valuable, though. Functional tests aren't going to catch concurrency issues but load testing however you want to do it just might.

AzzieElbab 4y ago

Good article. One thing I would add is the necessity to include failures/unhappy path scenarios in your load scripts. See if all that logging/tracing really is free :) Slightly off topic: do people still run any load/performance tests on cloud hosted sites? it seems like spiking your aws load is becoming prohibitively expensive. Also, in the real world replacing third party services with some intelligent mocks(simulating failures and latency) can be super complex.

noltncy 4y ago
Yes cloud hosted apps are still tested. You will need to let your cloud provide know about your tests or you (IPs) might get banned.
- AzzieElbab 4y ago
  I am talking about monetary costs resulting from actually pushing your load balanced elastic cluster to its limits.

vinay_ys 4y ago

Building systems that can scale efficiently requires careful design and implementation. I have seen teams relax the efficiency requirement and manage to handle high load by pre-scaling-up the provisioned capacity of their systems many hours in advance of an anticipated traffic load. This is obviously very wasteful of resources, but also it makes your system brittle, non-resilient. An unanticipated spike in load that stresses the system beyond its provisioned capacity will cause the system to overload and crash/jam-up. So, rather than testing if your system can handle high load when your system is pre-scaled, testing if your system can continue to provide good throughput even when under heavy load is a better test. Then, if your system is designed to auto-scale in an elastic cloud, testing fast auto-scaling of your systems to see if it increases the good throughput dynamically is a better strategy. Load testing should be about ascertaining the resiliency, scalability and efficiency characteristics of your system architecture and deployment setup. Some definitions: 1. A system is said to be resilient if it can sustain the good throughput even when under load excess relative to available system resources. 2. A system is said to be scalable if it can quickly respond to increasing load by scaling up its provisioned resources and scale-up its good throughput. 3. Good throughput is the throughput (request/sec, concurrent user sessions etc) of useful work done with acceptable performance characteristics – ie., latency curve at different percentiles is acceptable. 4. Efficiency of scalability is the ratio of increase in good throughput to increase in provisioned resources. 5. Efficiency in general is the concurrent user sessions or requests/sec per node of the provisioned system.

i_like_apis 4y ago

I used to do this a lot. It may look a little clunky, but there is a piece of software called JMeter (https://jmeter.apache.org/ https://jmeter.apache.org/) that is capable of doing just about anything you would want to do in a load test - modeling any request behavior, distributed traffic, awesome reports, etc. 10/10 recommend.

drewcoo 4y ago
JMeter is old and crusty and not at all friendly to work with. But I used it for years because it was really about the best we had. Today I don't wish it on anyone. Ruby JMeter finally made JMeter easier to manage, but I haven't worked in a Ruby shop for years, and I'm not going to force everyone to learn Ruby just to do some load testing. https://github.com/flood-io/ruby-jmeter https://github.com/flood-io/ruby-jmeter Then along came k6. It's developer-friendly and I've seen people actually enjoy using it. I recommend anyone considering JMeter also take a look at k6. They do a better job of selling it than I do: https://k6.io https://k6.io I am also Gatling-curious. Seems like an option for anyone in the JVM ecosystem. https://gatling.io https://gatling.io
- brabel 4y ago
  Looks like K6 is basically Gatling but with a JS-based DSL while Gatling uses Scala (or anything JVM-based like Java/Groovy/Kotlin)... there's also Locust[1] which is Python-based. What I really wanted was an interactive tool, like old LoadUI used to do (Wikipedia, nicely, still has the old screenshots which show how cool it used to look: https://en.wikipedia.org/wiki/LoadUI https://en.wikipedia.org/wiki/LoadUI)... because until you run tests, you just don't know what kind of load you can throw at the server (and whether you're hitting CPU/Memory/Bandwitdth limits instead of server limits). Visual Components were written in a scripting DSL (Groovy based) and dynamically loaded so you could change the code even while the test ran... really awesome stuff, a bit like coding in Lisp with a visual facade on top. Based on existing tools, it should be relatively easy to build something like that and I am surprised there seems to be nothing of the kind. I've always wanted to do it myself, maybe one day if no one else finally tackles the problem. [1] https://docs.locust.io/en/stable/ https://docs.locust.io/en/stable/

arein3 4y ago

Did anyone use hedless chrome with puppeteer for load tests?

dev_throw 4y ago
None of the utility libraries helped with this, instead wrote a shell script to orchestrate concurrent puppeteer load tests with jest and collect and parse reports. It is very resource intensive and spiky. I'm looking to see how we can make it more efficient.
hassy 4y ago
This lets you load test with Playwright (very similar to Puppeteer): https://github.com/artilleryio/artillery-engine-playwright https://github.com/artilleryio/artillery-engine-playwright

deleted 4y ago

[deleted]

aymar_99 4y ago

https://gatling.io/ https://gatling.io/ Gatling is one among the load test tools people need to give a shot. It uses akka for generating asynchronous loads. And it's good from a programmer standpoint since load test scripts can be programmed and maintained as any other script.

mikessoft_gmail 4y ago

MikeYasnev007 4y ago

LA is marketing cinema and drive) Load testing is more for Arizona server farms

fizwhiz 4y ago

> It’s also important that the loader generates those requests at a constant rate, best done asynchronously, so that response processing doesn’t get in the way of sending out new requests. Coordinated Omission is subtle but deserves more color: https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHfE https://groups.google.com/g/mechanical-sympathy/c/icNZJejUHf...