5 ms·

Why Cloud Based Load Testing Is A Killer App

16 points by keltecp11 18y ago

moe 18y ago
What a nonsense article. Makes it sound as if the script-playback was the difficult or expensive part of a load-test. In reality that is the trivial part.
- mustpax 18y ago
  That's not how I read it, I think this actually makes a lot of sense. In a regular (non-cloud) deployment, there are unavoidable differences between the staging and real production configurations. No one can afford to double their hardware costs for the sake of uber-realistic load testing results. So people usually opt to create a closely matched staging environment where testing is performed. Every once in a while something slips by that's only reproducible on production, and you're stuck debugging a critical and hard to reproduce bug. Not the best combination. With cloud computing, you're supposed to have lots of redundant computing capacity. So it's trivial to test in an environment that's identical to production by just asking for more processing power for the purposes of testing. As an added bonus you also have the distributed resources to generate considerable parallel load, which also helps. That last part like you said is not that hard. But that's only the icing. The cake is being able to test on a 1-1 replica of production. That's precious.
  - moe 18y ago
    Hmm indeed, from that angle it makes more sense. I guess I skimmed it a bit too lightly. But I'm still not overly impressed by the article as this is really just stating the obvious - basically a rehash of the standard cloud-propaganda that has been raining down on us for years. Nonetheless I take back my "nonsense" statement. That was too quick a judgement...
- Lightbody 18y ago
  (Note: I'm the founder of a cloud-based load testing provider...) Well, there are a lot of reasons why cloud-based load testing makes sense beyond scripting. But specifically about scripting, I believe that the cloud actually _does_ make script creation MUCH easier and script playback MUCH more realistic. Here's why: Traditional load testing tools emulate HTTP traffic. Their recorders watch the protocol-level traffic a browser makes, and then poor engineers have to disassemble the traffic to make it appear as if thousands of users are coming from one machine. They end up writing regex expressions and other such logic which effectively emulates the more expensive parsing a browser does. Often times, especially with rich AJAX apps, they end up duplicate much of the same code written as JavaScript to run in the browser, but this time inside of their load tool. For an example, see my article on why load testing is hard when it comes to AJAX: http://ajaxian.com/archives/why-load-testing-ajax-is-hard http://ajaxian.com/archives/why-load-testing-ajax-is-hard With cloud-based scalability, a new opportunity is here: to ditch the old ways of load testing and actually use real browsers. Scripts become way simpler because you're no longer scripting complex protocol-level traffic, but instead scripting simpler user behavior. And the script playback becomes more authenticate because you are guaranteed not to have a bug in the protocol-level playback, error reporting can show you what the user actually saw with screenshots, and the traffic pattern is realistic (two connections per host, etc). Of course, not many providers are doing this (yet). My company, BrowserMob, is. Thanks to the cloud, instead of allocating 2GB and 2 CPU cores to a load test, we might allocate 2TB and 2000 CPU cores. But because it's elastic usage, our cost is still low, so our prices stay low too. All of the scripting is done with Selenium, an open source browser automation technology I am a contributor for. If you're interested in trying it out, we provide $100 worth of free trial credits: http://browsermob.com http://browsermob.com Besides the browser vs virtual user stuff, there are definitely plenty of other reasons why the cloud is a great way to do load testing, but I think those have already been touched in the comments.
  - moe 18y ago
    Sorry, but this is exactly the nonsense that I falsely smelled in the original post. Emulating a full fledged browser for a load test is akin to killing a flea with a sledgehammer - a mindboggling waste of ressources. The webserver you are testing couldn't care less whether the requested pages are actually rendered or not. What I'd really want is a smart HTTP session recorder, smarter than Selenium and the ilk at least. The test-creator should perform a few average user-sessions in his browser and then be assisted with inserting smart tokens into the recorded HTTP traffic. The content that passes between browser and server is highly structured anyways, so things like "user clicked on first item" could even largely be auto-determined with a bit of pattern-analysis. I have constructed such tests in the past through ngrep and a bit of scripting magic - so I know it's possible and would merely need to be wrapped up in a shiny GUI. You make the "poor engineer" part sound harder than it has to be here. The place where it becomes really nasty is persistent connections via flash or comet. But plain old HTTP requests (AJAX or not) would be rather easy, if the existing tools didn't suck so hard.
    - rlonn 18y ago
      Session recording is something we are very interested in. We currently have a session recorder that is run as a zero-configuration proxy. I.e. you don't need to install any plugin or anything, and you don't need to change your browser configuration. The recorder rewrites all URLs it sees to go through itself rather than directly to the destination. There are other similar proxies that do this - I think one is called phproxy or similar - but ours can also handle javascript, which is a bit unusual. Anyway, if there are people here with bright ideas they feel like sharing about how to construct the greatest recorder of all, don't hesitate to get in touch with us (Load Impact). We have some experience with this and are always interested in collaboration with others.
      - moe 18y ago
        Why did you not implement a real proxy instead of that rewriting hack? Zero-configuration doesn't sound like an argument to me in that context.
    - 18y ago
dmv 18y ago
"Virtual Stress-free Testing in the Cloud" http://aws.typepad.com/aws/2009/02/virtual-stressfree-testing-in-the-cloud.html http://aws.typepad.com/aws/2009/02/virtual-stressfree-testin... Has links to several load testing products using EC2.
ncarlson 18y ago
Did he link to Apple's Keynote software as a "load testing managed service provider"?
- wmf 18y ago
  He must have confused Keynote with Keynote. http://www.keynote.com/ http://www.keynote.com/
  - jgannon 18y ago
    Thanks for the catch - I use Zemanta to speed up the linking and tagging of posts and I didn't check to make sure that the proper Keynote was linked :) In any case, it should be fixed now.

16 points by keltecp11 18y ago

moe 18y ago

What a nonsense article. Makes it sound as if the script-playback was the difficult or expensive part of a load-test. In reality that is the trivial part.

mustpax 18y ago
That's not how I read it, I think this actually makes a lot of sense. In a regular (non-cloud) deployment, there are unavoidable differences between the staging and real production configurations. No one can afford to double their hardware costs for the sake of uber-realistic load testing results. So people usually opt to create a closely matched staging environment where testing is performed. Every once in a while something slips by that's only reproducible on production, and you're stuck debugging a critical and hard to reproduce bug. Not the best combination. With cloud computing, you're supposed to have lots of redundant computing capacity. So it's trivial to test in an environment that's identical to production by just asking for more processing power for the purposes of testing. As an added bonus you also have the distributed resources to generate considerable parallel load, which also helps. That last part like you said is not that hard. But that's only the icing. The cake is being able to test on a 1-1 replica of production. That's precious.
- moe 18y ago
  Hmm indeed, from that angle it makes more sense. I guess I skimmed it a bit too lightly. But I'm still not overly impressed by the article as this is really just stating the obvious - basically a rehash of the standard cloud-propaganda that has been raining down on us for years. Nonetheless I take back my "nonsense" statement. That was too quick a judgement...
Lightbody 18y ago
(Note: I'm the founder of a cloud-based load testing provider...) Well, there are a lot of reasons why cloud-based load testing makes sense beyond scripting. But specifically about scripting, I believe that the cloud actually _does_ make script creation MUCH easier and script playback MUCH more realistic. Here's why: Traditional load testing tools emulate HTTP traffic. Their recorders watch the protocol-level traffic a browser makes, and then poor engineers have to disassemble the traffic to make it appear as if thousands of users are coming from one machine. They end up writing regex expressions and other such logic which effectively emulates the more expensive parsing a browser does. Often times, especially with rich AJAX apps, they end up duplicate much of the same code written as JavaScript to run in the browser, but this time inside of their load tool. For an example, see my article on why load testing is hard when it comes to AJAX: http://ajaxian.com/archives/why-load-testing-ajax-is-hard http://ajaxian.com/archives/why-load-testing-ajax-is-hard With cloud-based scalability, a new opportunity is here: to ditch the old ways of load testing and actually use real browsers. Scripts become way simpler because you're no longer scripting complex protocol-level traffic, but instead scripting simpler user behavior. And the script playback becomes more authenticate because you are guaranteed not to have a bug in the protocol-level playback, error reporting can show you what the user actually saw with screenshots, and the traffic pattern is realistic (two connections per host, etc). Of course, not many providers are doing this (yet). My company, BrowserMob, is. Thanks to the cloud, instead of allocating 2GB and 2 CPU cores to a load test, we might allocate 2TB and 2000 CPU cores. But because it's elastic usage, our cost is still low, so our prices stay low too. All of the scripting is done with Selenium, an open source browser automation technology I am a contributor for. If you're interested in trying it out, we provide $100 worth of free trial credits: http://browsermob.com http://browsermob.com Besides the browser vs virtual user stuff, there are definitely plenty of other reasons why the cloud is a great way to do load testing, but I think those have already been touched in the comments.
- moe 18y ago
  Sorry, but this is exactly the nonsense that I falsely smelled in the original post. Emulating a full fledged browser for a load test is akin to killing a flea with a sledgehammer - a mindboggling waste of ressources. The webserver you are testing couldn't care less whether the requested pages are actually rendered or not. What I'd really want is a smart HTTP session recorder, smarter than Selenium and the ilk at least. The test-creator should perform a few average user-sessions in his browser and then be assisted with inserting smart tokens into the recorded HTTP traffic. The content that passes between browser and server is highly structured anyways, so things like "user clicked on first item" could even largely be auto-determined with a bit of pattern-analysis. I have constructed such tests in the past through ngrep and a bit of scripting magic - so I know it's possible and would merely need to be wrapped up in a shiny GUI. You make the "poor engineer" part sound harder than it has to be here. The place where it becomes really nasty is persistent connections via flash or comet. But plain old HTTP requests (AJAX or not) would be rather easy, if the existing tools didn't suck so hard.
  - rlonn 18y ago
    Session recording is something we are very interested in. We currently have a session recorder that is run as a zero-configuration proxy. I.e. you don't need to install any plugin or anything, and you don't need to change your browser configuration. The recorder rewrites all URLs it sees to go through itself rather than directly to the destination. There are other similar proxies that do this - I think one is called phproxy or similar - but ours can also handle javascript, which is a bit unusual. Anyway, if there are people here with bright ideas they feel like sharing about how to construct the greatest recorder of all, don't hesitate to get in touch with us (Load Impact). We have some experience with this and are always interested in collaboration with others.
    - moe 18y ago
      Why did you not implement a real proxy instead of that rewriting hack? Zero-configuration doesn't sound like an argument to me in that context.
  - 18y ago

dmv 18y ago

"Virtual Stress-free Testing in the Cloud" http://aws.typepad.com/aws/2009/02/virtual-stressfree-testing-in-the-cloud.html http://aws.typepad.com/aws/2009/02/virtual-stressfree-testin... Has links to several load testing products using EC2.

ncarlson 18y ago

Did he link to Apple's Keynote software as a "load testing managed service provider"?

wmf 18y ago
He must have confused Keynote with Keynote. http://www.keynote.com/ http://www.keynote.com/
- jgannon 18y ago
  Thanks for the catch - I use Zemanta to speed up the linking and tagging of posts and I didn't check to make sure that the proper Keynote was linked :) In any case, it should be fixed now.