4 ms·

This seems to encourage you to run it with a target number of concurrent users, rather than a target RPS from the loaders, which IME can result in difficulty in

This seems to encourage you to run it with a target number of concurrent users, rather than a target RPS from the loaders, which IME can result in difficulty interpreting the results (it also doesn't reflect most use cases, except in some heavily controlled scenarios). When latencies increase, the fixed number of virtual users will necessarily be sending fewer new requests, leading to a constant or decreasing number of RPS actually being served. Other load testing tools (Gatling, newer versions of k6) will let you set a target RPS instead.

ericb 4y ago
Targeting RPS without also targeting concurrent users is a huge mistake! Here's a metaphor I like. Imagine I am trying to open a small high-volume restaurant. I call the chef in for a trial run. I sit at a table, and while I watch, she is able to make 200 dishes an hour. I'm excited, at 200 dishes an hour, we'll be rich! I tell the investors. Opening day comes! I have 10 tables and people eat for an hour on average. We serve 10 dishes an hour. Our restaurant fails. Throughput, without targeting concurrent users, is a bad test.
- bombcar 4y ago
  Part of it is identifying the bottleneck. You verified the chef wouldn’t be it, but the tables ended up being the limiting factor. And say you installed 200 tables, but only get ten customers an hour - now the bottleneck is the input and that likely has to be solved by marketing or something exterior. And maybe you get that fixed and now have 200 tables an hour, wnd discover your dishwasher can only handle 50 tables an hour. So you need more dishes (buffer) but that will only help until the poor dishwasher is washing dishes 24/7 and at that point the buffer will eventually run out. Successfully modeling the system and identifying the point at which it will fail is useful, because then you can keep an eye on that point AND know if something unexpected is causing it to fail earlier.
- sparsely 4y ago
  I don't really understand how this analogy works in terms of the actual practice of running load tests. You should of course test the full flow, the important part is not implicitly limiting the input volume based on the system performance. I think in your analogy it would be something about people queuing outside without you noticing but am unsure. Also, to be clear, the bad thing is normally doing closed workload tests, not having a multistep workflow or whatever (https://gatling.io/docs/gatling/reference/current/core/injection/ https://gatling.io/docs/gatling/reference/current/core/injec...)
  - ericb 4y ago
    To complete the analogy, you need to have one concurrent session per "virtual user." This is the "concurrency" of the test. In my bad test, I had one concurrent user driving all the throughput--me. If I had targeted 200 concurrent users, I would have seen a failure in my test and found what the real throughput would be. Each concurrent user uses many resources that could be limited. In the restaurant, it is tables, plates, silverware. In a web app, it is sessions, connections, memory, database connections, and many other resources that can be associated with each session, and each may be a potential bottleneck that limits your actual throughput. If we target throughput and ignore concurrency, we set ourselves up for failure. In practicality, the better load testing tools let you create "concurrent users." In JMeter, this is the "threads." In other tools, it will be called "virtual users", "vus" or "sessions." In locust, the setting is -u NUM_USERS. The caution is not to fool yourself with high RPS if these settings aren't right for your test, and to normally target both.

ericb 4y ago

Targeting RPS without also targeting concurrent users is a huge mistake! Here's a metaphor I like. Imagine I am trying to open a small high-volume restaurant. I call the chef in for a trial run. I sit at a table, and while I watch, she is able to make 200 dishes an hour. I'm excited, at 200 dishes an hour, we'll be rich! I tell the investors. Opening day comes! I have 10 tables and people eat for an hour on average. We serve 10 dishes an hour. Our restaurant fails. Throughput, without targeting concurrent users, is a bad test.

bombcar 4y ago
Part of it is identifying the bottleneck. You verified the chef wouldn’t be it, but the tables ended up being the limiting factor. And say you installed 200 tables, but only get ten customers an hour - now the bottleneck is the input and that likely has to be solved by marketing or something exterior. And maybe you get that fixed and now have 200 tables an hour, wnd discover your dishwasher can only handle 50 tables an hour. So you need more dishes (buffer) but that will only help until the poor dishwasher is washing dishes 24/7 and at that point the buffer will eventually run out. Successfully modeling the system and identifying the point at which it will fail is useful, because then you can keep an eye on that point AND know if something unexpected is causing it to fail earlier.
sparsely 4y ago
I don't really understand how this analogy works in terms of the actual practice of running load tests. You should of course test the full flow, the important part is not implicitly limiting the input volume based on the system performance. I think in your analogy it would be something about people queuing outside without you noticing but am unsure. Also, to be clear, the bad thing is normally doing closed workload tests, not having a multistep workflow or whatever (https://gatling.io/docs/gatling/reference/current/core/injection/ https://gatling.io/docs/gatling/reference/current/core/injec...)
- ericb 4y ago
  To complete the analogy, you need to have one concurrent session per "virtual user." This is the "concurrency" of the test. In my bad test, I had one concurrent user driving all the throughput--me. If I had targeted 200 concurrent users, I would have seen a failure in my test and found what the real throughput would be. Each concurrent user uses many resources that could be limited. In the restaurant, it is tables, plates, silverware. In a web app, it is sessions, connections, memory, database connections, and many other resources that can be associated with each session, and each may be a potential bottleneck that limits your actual throughput. If we target throughput and ignore concurrency, we set ourselves up for failure. In practicality, the better load testing tools let you create "concurrent users." In JMeter, this is the "threads." In other tools, it will be called "virtual users", "vus" or "sessions." In locust, the setting is -u NUM_USERS. The caution is not to fool yourself with high RPS if these settings aren't right for your test, and to normally target both.