4 ms·

Locust – A modern load testing framework

127 points by danielmiessler 11y ago

the100rabh 11y ago
What is the QPS this can achieve, any clue? I have not seen anything do better than tsung as of now.
TheBranca18 11y ago
A bit tongue in cheek, but is a testimonial from someone at EA/Dice something you'd want to highlight? I know Battlefield 4 is almost two years old at this point, but man, I still have the scars from that game... In all seriousness, definitely going to check this out, not enough load testing is done at the company I work for, plus I want to learn more python.
- thomasrossi 11y ago
  I have exactly the same point:) More python, balancing needs to be done, Battlefield 4.. ouuuuch.
eatonphil 11y ago
Took a look at Locust before and did not find it as useful or generic as ApacheBench or Httperf (by HP). Locust felt a little too cutesy and high-level for most of the work I wanted to do - which was pretty simple. Furthermore there were a lot of timing inaccuracies between the same test runs. It certainly could have been a fault in our code, but ab and Httperf seemed to give a lot more reliable and useful timing information. And they are just simpler to use. For one, not being browserbased can be nice when you are trying to iterate quickly. Not trying to sell anyone against Locust, but at the least, I recommend you take a look at ab and Httperf to make sure you don't need something a whole lot simpler.
- zodvik 11y ago
  If you're using ab or httpperf, you should also take a look at wrk (https://github.com/wg/wrk https://github.com/wg/wrk). It can generate significant amount of load and has support for Lua scripting for request generation, response processing and reporting. There is another fork of it (https://github.com/giltene/wrk2 https://github.com/giltene/wrk2) which can be used if you want test at a consistent RPS.
- jonatanheyman 11y ago
  That isn't a very good comparison, because Locust's use-case is very different. Locust is a tool/framework for writing code that simulates real user behaviour in a fairly realistic way. For example, it's very common to store state for each simulated user. Once you have written your "user behaviour code", you can then simulate a lot of simultaneous users by running it distributed across multiple machines, and hopefully get realistic load sent to you system. If I wanted to just send a lot of requests/s to one or very few URL endpoints, I would also use something like ApacheBench, and I'm author of Locust.
dalyons 11y ago
Used it for some substantial loadtesting, loved it. It's real advantage compared to AB, siege, etc, is how easy it makes it to simulate real user interactions, not just mash on a few endpoints. Only flaw in my books is the crazy high ulimit you need for simulating high numbers of unique actors. The easy clustering mitigates this though.
therealmarv 11y ago
Locust is a great tool. We use it to loadtest our websites and even a single Mac can make huge traffic. If somebody is interested in a highly optimized Vagrant VM for loadtesting and Locust (it's possible through a VM, but not exorbitant high loadtesting) check out this url https://github.com/PyBossa/loadtesting https://github.com/PyBossa/loadtesting
krat0sprakhar 11y ago
So from what I understand this stands somewhere between as complex as Apache Bench and something as simple as Seige? I've had very little success with the former since its scripts are massive XML files and the clunky UI also doesn't help much. As a python developer, I'm glad to see this and will surely give this a try shortly. Thanks!
tuna 11y ago
My fave locust setup is spinning up aws instances to tests from time to time. I've created ansible automation to help on that (https://github.com/gleicon/locust-swarm https://github.com/gleicon/locust-swarm) but it can be done manually. I still look on how to make it run casper.js tests but I think it would interfere with locust's performance.
arnoooooo 11y ago
Does anyone know how it compares to Gatling (http://gatling.io http://gatling.io) ?
- therealmarv 11y ago
  Gatling is far more advanced but both tools share similar architectural designs regarding how they handle lots of traffic. Locust is more leightweight (and I would say also more easy, but depends on your needs) and is missing good graphical presentations of results and you will not get a browser recorder with it.
- rdsubhas 11y ago
  I used Locust for a good long time around a year back, and here are some observations. Locust is awesome to start with, and has master/slave for spawning multiple servers, great for testing real concurrency with large loads. Gatling doesn't have master/slave mode yet. Main problems with Locust are: * Lack of good presentation of results (compare Gatling and Locust results, and Locust is very primitive) * Lack of Pipelines/Scenarios. This IMO was a big deal breaker for me. You can't say: "100% users hit the home screen, out of which 70% add a product, out of which 25% actually pay" etc. This is easily modeled with Gatling, and the same request session with same data falls through each scenario. I'm still following https://github.com/locustio/locust/issues/171 https://github.com/locustio/locust/issues/171 though, but I don't think they have any idea of implementing it.
ponytech 11y ago
Does it support testing a websocket app ?
- jonatanheyman 11y ago
  Not out of the box. However, as long as it's "request/response based" it's quite simple to write and use a custom client. See: http://docs.locust.io/en/latest/testing-other-systems.html http://docs.locust.io/en/latest/testing-other-systems.html
rmc 11y ago
I used it to test a Django app once. It's great because you can test the entire "flow" of an app, clicking on various forms.
evantahler 11y ago
Do we have an equivalent for WS yet?
codingvelocity 11y ago
I used locust to supplement testing that third party was doing (they used some ibm product that I didn't have a license too). I liked locust because it was easy to start using, but you write a decent amount of code if you have a complex application. I suggest using pyquery (https://pypi.python.org/pypi/pyquery https://pypi.python.org/pypi/pyquery) with it if you need to inspect any html, it was quite useful
crcsmnky 11y ago
Shameless self promotion - I just wrote up a little doc [1] on using Locust and Kubernetes to run and scale distributed load tests along with some sample code [2]. I looked at JMeter and Gatling as alternatives but they were either a bit clunky (JMeter) or not distributed out of the box (Gatling). Late in the game I did discover Tsung (Erlang) but didn't get time to try it out - looks pretty powerful though. [1] https://cloud.google.com/solutions/distributed-load-testing-using-kubernetes https://cloud.google.com/solutions/distributed-load-testing-... [2] https://github.com/GoogleCloudPlatform/distributed-load-testing-using-kubernetes https://github.com/GoogleCloudPlatform/distributed-load-test...
- coolrhymes 11y ago
  Great writeup. It is so close to what we built at tradewave.net (except for Locust). I am glad I am not the only one that saw load testing using scalable containers. At some point, we will be going towards HFT and having a true load testing suite is handy.
meakaakka 11y ago
Does it provide graphical data representation or any kind of dashboard/intergation with graphite or something else?
robert_nsu 11y ago
I've been using siege for load testing. I'll give this a go at some point. I don't use python as much as I'd like and this seems like a good work related use-case for me.

127 points by danielmiessler 11y ago

What is the QPS this can achieve, any clue? I have not seen anything do better than tsung as of now.

A bit tongue in cheek, but is a testimonial from someone at EA/Dice something you'd want to highlight? I know Battlefield 4 is almost two years old at this point, but man, I still have the scars from that game... In all seriousness, definitely going to check this out, not enough load testing is done at the company I work for, plus I want to learn more python.

thomasrossi 11y ago
I have exactly the same point:) More python, balancing needs to be done, Battlefield 4.. ouuuuch.

eatonphil 11y ago

Took a look at Locust before and did not find it as useful or generic as ApacheBench or Httperf (by HP). Locust felt a little too cutesy and high-level for most of the work I wanted to do - which was pretty simple. Furthermore there were a lot of timing inaccuracies between the same test runs. It certainly could have been a fault in our code, but ab and Httperf seemed to give a lot more reliable and useful timing information. And they are just simpler to use. For one, not being browserbased can be nice when you are trying to iterate quickly. Not trying to sell anyone against Locust, but at the least, I recommend you take a look at ab and Httperf to make sure you don't need something a whole lot simpler.

zodvik 11y ago
If you're using ab or httpperf, you should also take a look at wrk (https://github.com/wg/wrk https://github.com/wg/wrk). It can generate significant amount of load and has support for Lua scripting for request generation, response processing and reporting. There is another fork of it (https://github.com/giltene/wrk2 https://github.com/giltene/wrk2) which can be used if you want test at a consistent RPS.
jonatanheyman 11y ago
That isn't a very good comparison, because Locust's use-case is very different. Locust is a tool/framework for writing code that simulates real user behaviour in a fairly realistic way. For example, it's very common to store state for each simulated user. Once you have written your "user behaviour code", you can then simulate a lot of simultaneous users by running it distributed across multiple machines, and hopefully get realistic load sent to you system. If I wanted to just send a lot of requests/s to one or very few URL endpoints, I would also use something like ApacheBench, and I'm author of Locust.

dalyons 11y ago

Used it for some substantial loadtesting, loved it. It's real advantage compared to AB, siege, etc, is how easy it makes it to simulate real user interactions, not just mash on a few endpoints. Only flaw in my books is the crazy high ulimit you need for simulating high numbers of unique actors. The easy clustering mitigates this though.

therealmarv 11y ago

Locust is a great tool. We use it to loadtest our websites and even a single Mac can make huge traffic. If somebody is interested in a highly optimized Vagrant VM for loadtesting and Locust (it's possible through a VM, but not exorbitant high loadtesting) check out this url https://github.com/PyBossa/loadtesting https://github.com/PyBossa/loadtesting

krat0sprakhar 11y ago

So from what I understand this stands somewhere between as complex as Apache Bench and something as simple as Seige? I've had very little success with the former since its scripts are massive XML files and the clunky UI also doesn't help much. As a python developer, I'm glad to see this and will surely give this a try shortly. Thanks!

tuna 11y ago

My fave locust setup is spinning up aws instances to tests from time to time. I've created ansible automation to help on that (https://github.com/gleicon/locust-swarm https://github.com/gleicon/locust-swarm) but it can be done manually. I still look on how to make it run casper.js tests but I think it would interfere with locust's performance.

arnoooooo 11y ago

Does anyone know how it compares to Gatling (http://gatling.io http://gatling.io) ?

therealmarv 11y ago
Gatling is far more advanced but both tools share similar architectural designs regarding how they handle lots of traffic. Locust is more leightweight (and I would say also more easy, but depends on your needs) and is missing good graphical presentations of results and you will not get a browser recorder with it.
rdsubhas 11y ago
I used Locust for a good long time around a year back, and here are some observations. Locust is awesome to start with, and has master/slave for spawning multiple servers, great for testing real concurrency with large loads. Gatling doesn't have master/slave mode yet. Main problems with Locust are: * Lack of good presentation of results (compare Gatling and Locust results, and Locust is very primitive) * Lack of Pipelines/Scenarios. This IMO was a big deal breaker for me. You can't say: "100% users hit the home screen, out of which 70% add a product, out of which 25% actually pay" etc. This is easily modeled with Gatling, and the same request session with same data falls through each scenario. I'm still following https://github.com/locustio/locust/issues/171 https://github.com/locustio/locust/issues/171 though, but I don't think they have any idea of implementing it.

ponytech 11y ago

Does it support testing a websocket app ?

jonatanheyman 11y ago
Not out of the box. However, as long as it's "request/response based" it's quite simple to write and use a custom client. See: http://docs.locust.io/en/latest/testing-other-systems.html http://docs.locust.io/en/latest/testing-other-systems.html

rmc 11y ago

I used it to test a Django app once. It's great because you can test the entire "flow" of an app, clicking on various forms.

evantahler 11y ago

Do we have an equivalent for WS yet?

codingvelocity 11y ago

I used locust to supplement testing that third party was doing (they used some ibm product that I didn't have a license too). I liked locust because it was easy to start using, but you write a decent amount of code if you have a complex application. I suggest using pyquery (https://pypi.python.org/pypi/pyquery https://pypi.python.org/pypi/pyquery) with it if you need to inspect any html, it was quite useful

crcsmnky 11y ago

Shameless self promotion - I just wrote up a little doc [1] on using Locust and Kubernetes to run and scale distributed load tests along with some sample code [2]. I looked at JMeter and Gatling as alternatives but they were either a bit clunky (JMeter) or not distributed out of the box (Gatling). Late in the game I did discover Tsung (Erlang) but didn't get time to try it out - looks pretty powerful though. [1] https://cloud.google.com/solutions/distributed-load-testing-using-kubernetes https://cloud.google.com/solutions/distributed-load-testing-... [2] https://github.com/GoogleCloudPlatform/distributed-load-testing-using-kubernetes https://github.com/GoogleCloudPlatform/distributed-load-test...

coolrhymes 11y ago
Great writeup. It is so close to what we built at tradewave.net (except for Locust). I am glad I am not the only one that saw load testing using scalable containers. At some point, we will be going towards HFT and having a true load testing suite is handy.

meakaakka 11y ago

Does it provide graphical data representation or any kind of dashboard/intergation with graphite or something else?

robert_nsu 11y ago

I've been using siege for load testing. I'll give this a go at some point. I don't use python as much as I'd like and this seems like a good work related use-case for me.