5 ms·

Sorry, but this is exactly the nonsense that I falsely smelled in the original post. Emulating a full fledged browser for a load test is akin to killing a flea

by moe 18y ago

Sorry, but this is exactly the nonsense that I falsely smelled in the original post. Emulating a full fledged browser for a load test is akin to killing a flea with a sledgehammer - a mindboggling waste of ressources. The webserver you are testing couldn't care less whether the requested pages are actually rendered or not. What I'd really want is a smart HTTP session recorder, smarter than Selenium and the ilk at least. The test-creator should perform a few average user-sessions in his browser and then be assisted with inserting smart tokens into the recorded HTTP traffic. The content that passes between browser and server is highly structured anyways, so things like "user clicked on first item" could even largely be auto-determined with a bit of pattern-analysis. I have constructed such tests in the past through ngrep and a bit of scripting magic - so I know it's possible and would merely need to be wrapped up in a shiny GUI. You make the "poor engineer" part sound harder than it has to be here. The place where it becomes really nasty is persistent connections via flash or comet. But plain old HTTP requests (AJAX or not) would be rather easy, if the existing tools didn't suck so hard.

rlonn 18y ago
Session recording is something we are very interested in. We currently have a session recorder that is run as a zero-configuration proxy. I.e. you don't need to install any plugin or anything, and you don't need to change your browser configuration. The recorder rewrites all URLs it sees to go through itself rather than directly to the destination. There are other similar proxies that do this - I think one is called phproxy or similar - but ours can also handle javascript, which is a bit unusual. Anyway, if there are people here with bright ideas they feel like sharing about how to construct the greatest recorder of all, don't hesitate to get in touch with us (Load Impact). We have some experience with this and are always interested in collaboration with others.
- moe 18y ago
  Why did you not implement a real proxy instead of that rewriting hack? Zero-configuration doesn't sound like an argument to me in that context.
Lightbody 18y ago
Moe, Sorry, but you haven't convinced me. Yes, a very smart engineer with knowledge of all the internal workings of a web app can get by just fine. You seem to be one of those people. Congratulations to you! But not everyone is in the same boat you're in, so don't be so quick to write off new ideas, and certainly no need to impolite about it. I pasted a link about Ajax load testing which, if you read it, explains why your "smart token" idea often won't work and will fail at a growing rate as apps get more complex (ie: as Google evolves to include Google Suggest). I'd love to have a substantial discussion about how one could write a smart recorder (esp. since I'm 99.9% sure it's impossible - I've thought a REALLY long time about it). You say this approach is a "waste of ressources [sic]", but I disagree. If the cost is lower than traditional load testing services (thanks to the cloud), what's the harm in using a higher fidelity approach? To me, the "nonsense" would be avoiding such an approach! Your point about persistent connections is another reason why BrowserMob exists. Why spend hours or days working around an issue like that when you could be focussing on tuning the system performance? Again - if the cost is low enough, why should you care that terabytes of RAM are being used in the process? I appreciate your passion about this, but please understand that not everyone has the knowledge set you do, and not everyone tests the same apps you do. It's too bad you don't appreciate a unique approach to the problem even if it doesn't apply directly to you. Hopefully it doesn't turn off others from thinking outside the box! Patrick
- moe 18y ago
  Well, I read your link and that one, in turn, doesn't convince me. The hangup seems to be incremental search and similar operations that trigger multiple requests in the background (although I can't think of another common one from the top of my head). If that appears as a hard problem then I can only assume the author is looking at it at the wrong level of abstraction. In fact he is - he's judging it by what selenium can do. If you take a step back from selenium for a moment and instead consider presenting the user with a meaningful visualization of all requests that happened during his sample-session, then this and similar problems suddenly become more of a challenge in UI design than anything else. A few basic control-primitives in the UI to re-arrange the recorded script would go a long way here. Primitives such as "Clone-this-request", "Clone-with-parameter-X-from-dict", "Simulate-Incremental-Search-By-Making-n-Substr-Clones" etc. - you get the idea. This whole "edit my script"-step seems to be missing from browsermob, just like it is missing in most similar offerings. Hence I have to ask: How would I go about modifying my recorded selenium script to login with random users? To enter random data into forms? To actually perform random incremental searches (to evade caching)? To enter randomly chosen data into forms (e.g. to post a message to a random but existing other user)? To repeat requests first with random, then with chosen data (to hit form validation)? To react on special AJAX poll results or even server push via comet? To have multiple concurrent sessions interact with each other? These are the common challenges that I have faced during the construction of load-tests and in my expirience that is the time-consuming part. It seems with browsermob I'm down to editing selenium scripts by hand again? Sorry if I come across as disrespectful but the lack of usable, on-demand load-testing solutions [that don't make you pay through the nose] has indeed nagged me for a while, too. If there's one thing that I've learned during the implementation of such tests then it would be that bad testing is worse than no testing. Nothing easier than giving yourself a false sense of security. The incremental search example you cited is actually a textbook example for how relying on purely recorded sessions will lead to a bad test. Playing back the same recorded search a zillion times is nowhere near a realistic workload, no matter how much you crank up the concurrency.

by moe 18y ago

rlonn 18y ago

Session recording is something we are very interested in. We currently have a session recorder that is run as a zero-configuration proxy. I.e. you don't need to install any plugin or anything, and you don't need to change your browser configuration. The recorder rewrites all URLs it sees to go through itself rather than directly to the destination. There are other similar proxies that do this - I think one is called phproxy or similar - but ours can also handle javascript, which is a bit unusual. Anyway, if there are people here with bright ideas they feel like sharing about how to construct the greatest recorder of all, don't hesitate to get in touch with us (Load Impact). We have some experience with this and are always interested in collaboration with others.

moe 18y ago
Why did you not implement a real proxy instead of that rewriting hack? Zero-configuration doesn't sound like an argument to me in that context.

Lightbody 18y ago

Moe, Sorry, but you haven't convinced me. Yes, a very smart engineer with knowledge of all the internal workings of a web app can get by just fine. You seem to be one of those people. Congratulations to you! But not everyone is in the same boat you're in, so don't be so quick to write off new ideas, and certainly no need to impolite about it. I pasted a link about Ajax load testing which, if you read it, explains why your "smart token" idea often won't work and will fail at a growing rate as apps get more complex (ie: as Google evolves to include Google Suggest). I'd love to have a substantial discussion about how one could write a smart recorder (esp. since I'm 99.9% sure it's impossible - I've thought a REALLY long time about it). You say this approach is a "waste of ressources [sic]", but I disagree. If the cost is lower than traditional load testing services (thanks to the cloud), what's the harm in using a higher fidelity approach? To me, the "nonsense" would be avoiding such an approach! Your point about persistent connections is another reason why BrowserMob exists. Why spend hours or days working around an issue like that when you could be focussing on tuning the system performance? Again - if the cost is low enough, why should you care that terabytes of RAM are being used in the process? I appreciate your passion about this, but please understand that not everyone has the knowledge set you do, and not everyone tests the same apps you do. It's too bad you don't appreciate a unique approach to the problem even if it doesn't apply directly to you. Hopefully it doesn't turn off others from thinking outside the box! Patrick

moe 18y ago
Well, I read your link and that one, in turn, doesn't convince me. The hangup seems to be incremental search and similar operations that trigger multiple requests in the background (although I can't think of another common one from the top of my head). If that appears as a hard problem then I can only assume the author is looking at it at the wrong level of abstraction. In fact he is - he's judging it by what selenium can do. If you take a step back from selenium for a moment and instead consider presenting the user with a meaningful visualization of all requests that happened during his sample-session, then this and similar problems suddenly become more of a challenge in UI design than anything else. A few basic control-primitives in the UI to re-arrange the recorded script would go a long way here. Primitives such as "Clone-this-request", "Clone-with-parameter-X-from-dict", "Simulate-Incremental-Search-By-Making-n-Substr-Clones" etc. - you get the idea. This whole "edit my script"-step seems to be missing from browsermob, just like it is missing in most similar offerings. Hence I have to ask: How would I go about modifying my recorded selenium script to login with random users? To enter random data into forms? To actually perform random incremental searches (to evade caching)? To enter randomly chosen data into forms (e.g. to post a message to a random but existing other user)? To repeat requests first with random, then with chosen data (to hit form validation)? To react on special AJAX poll results or even server push via comet? To have multiple concurrent sessions interact with each other? These are the common challenges that I have faced during the construction of load-tests and in my expirience that is the time-consuming part. It seems with browsermob I'm down to editing selenium scripts by hand again? Sorry if I come across as disrespectful but the lack of usable, on-demand load-testing solutions [that don't make you pay through the nose] has indeed nagged me for a while, too. If there's one thing that I've learned during the implementation of such tests then it would be that bad testing is worse than no testing. Nothing easier than giving yourself a false sense of security. The incremental search example you cited is actually a textbook example for how relying on purely recorded sessions will lead to a bad test. Playing back the same recorded search a zillion times is nowhere near a realistic workload, no matter how much you crank up the concurrency.