3 ms·

Well, I read your link and that one, in turn, doesn't convince me. The hangup seems to be incremental search and similar operations that trigger multiple reque

by moe 18y ago

Well, I read your link and that one, in turn, doesn't convince me. The hangup seems to be incremental search and similar operations that trigger multiple requests in the background (although I can't think of another common one from the top of my head). If that appears as a hard problem then I can only assume the author is looking at it at the wrong level of abstraction. In fact he is - he's judging it by what selenium can do. If you take a step back from selenium for a moment and instead consider presenting the user with a meaningful visualization of all requests that happened during his sample-session, then this and similar problems suddenly become more of a challenge in UI design than anything else. A few basic control-primitives in the UI to re-arrange the recorded script would go a long way here. Primitives such as "Clone-this-request", "Clone-with-parameter-X-from-dict", "Simulate-Incremental-Search-By-Making-n-Substr-Clones" etc. - you get the idea. This whole "edit my script"-step seems to be missing from browsermob, just like it is missing in most similar offerings. Hence I have to ask: How would I go about modifying my recorded selenium script to login with random users? To enter random data into forms? To actually perform random incremental searches (to evade caching)? To enter randomly chosen data into forms (e.g. to post a message to a random but existing other user)? To repeat requests first with random, then with chosen data (to hit form validation)? To react on special AJAX poll results or even server push via comet? To have multiple concurrent sessions interact with each other? These are the common challenges that I have faced during the construction of load-tests and in my expirience that is the time-consuming part. It seems with browsermob I'm down to editing selenium scripts by hand again? Sorry if I come across as disrespectful but the lack of usable, on-demand load-testing solutions [that don't make you pay through the nose] has indeed nagged me for a while, too. If there's one thing that I've learned during the implementation of such tests then it would be that bad testing is worse than no testing. Nothing easier than giving yourself a false sense of security. The incremental search example you cited is actually a textbook example for how relying on purely recorded sessions will lead to a bad test. Playing back the same recorded search a zillion times is nowhere near a realistic workload, no matter how much you crank up the concurrency.

Lightbody 18y ago
Moe, Not sure if you noticed, but I was the author of that link :P It's not just about incremental search, that's just one example. Anything in which there is logic in the client that reads in user-entered data and mutates it in a way in which simulating at the protocol level becomes non-trivial and effectively require duplicating the logic that was first written in JavaScript. For another example: I'm working with a large online invitation website that is doing a big redesign of their site. The new UI is effectively one "page" and everything is done via AJAX calls to a JSON service layer, including login. Authentication to that service layer is done via a custom HTTP header that is a SHA encoding of the username + password + timestamp (among other things). If you wanted to test with 1000's of user accounts with traditional load testing, you end up needing to do the SHA encoding on the fly in whatever tool you're using (VBScript, LoadRunner's crazy C-like language, etc). I really just can't think of a UI that would allow you easily record a Google search for one term and then abstract the test out to work with N search terms. Because the AJAX requests become data dependent ("foo" causes 3 requests, "foobar" causes 6), you ultimately end up writing control structures (loops, branches, etc) that are better off written in a programming language. Generally, if the browser isn't doing simple "passthrough" of the user data, I find protocol simulation becomes much more difficult. For these reasons, I'm not sure it's just a UI design issue, though I'd love to be proven wrong. Unless there was an Iterate-over-characters-of-data-X-and-issue-http-requests-with-pattern-Y command as well as a SHA-encode-data-X-and-data-Y command (which of course is way too domain specific), you'll likely end up writing your tests with a Real Programming Language. As for your questions about data parameterization/randomization, that was the entire point of my article. I specifically called attention to data randomization as the point where it really gets tough. That's what load testing is all about, and that's why traditional approaches are breaking down. BrowserMob lets you import a Selenium script that has static data in it (ie: "type q google_search"). The script is converted to a JavaScript, which you can then edit and replace the data (ie: "google_search") with a random/parameterized data. The ensuing AJAX HTTP requests are generated automatically and correctly since the browser owns that. All of that kind of stuff ultimately still has to get done with some scripting. I don't pretend that BrowserMob load testing is "script free". Far from it - we embrace JavaScript as the way to get things done. But if you're scripting a browser, your scripts end up dealing with user behavior emulation, which is almost always simpler than protocol layer emulation. Understand: I'm not saying this stuff is impossible today. I'm saying it's harder than it needs to be and can be greatly simplified by just using a real browser. All that said, I appreciate your frustration with the lack of tools. I hope you'll give our approach a closer look. I'd certainly appreciate hands-on feedback. We do offer more traditional virtual user scripting support as well (at 1/10th the price of our real browser service), so if you're good with JavaScript you can get some pretty affordable on-demand testing! Patrick

by moe 18y ago

Lightbody 18y ago

Moe, Not sure if you noticed, but I was the author of that link :P It's not just about incremental search, that's just one example. Anything in which there is logic in the client that reads in user-entered data and mutates it in a way in which simulating at the protocol level becomes non-trivial and effectively require duplicating the logic that was first written in JavaScript. For another example: I'm working with a large online invitation website that is doing a big redesign of their site. The new UI is effectively one "page" and everything is done via AJAX calls to a JSON service layer, including login. Authentication to that service layer is done via a custom HTTP header that is a SHA encoding of the username + password + timestamp (among other things). If you wanted to test with 1000's of user accounts with traditional load testing, you end up needing to do the SHA encoding on the fly in whatever tool you're using (VBScript, LoadRunner's crazy C-like language, etc). I really just can't think of a UI that would allow you easily record a Google search for one term and then abstract the test out to work with N search terms. Because the AJAX requests become data dependent ("foo" causes 3 requests, "foobar" causes 6), you ultimately end up writing control structures (loops, branches, etc) that are better off written in a programming language. Generally, if the browser isn't doing simple "passthrough" of the user data, I find protocol simulation becomes much more difficult. For these reasons, I'm not sure it's just a UI design issue, though I'd love to be proven wrong. Unless there was an Iterate-over-characters-of-data-X-and-issue-http-requests-with-pattern-Y command as well as a SHA-encode-data-X-and-data-Y command (which of course is way too domain specific), you'll likely end up writing your tests with a Real Programming Language. As for your questions about data parameterization/randomization, that was the entire point of my article. I specifically called attention to data randomization as the point where it really gets tough. That's what load testing is all about, and that's why traditional approaches are breaking down. BrowserMob lets you import a Selenium script that has static data in it (ie: "type q google_search"). The script is converted to a JavaScript, which you can then edit and replace the data (ie: "google_search") with a random/parameterized data. The ensuing AJAX HTTP requests are generated automatically and correctly since the browser owns that. All of that kind of stuff ultimately still has to get done with some scripting. I don't pretend that BrowserMob load testing is "script free". Far from it - we embrace JavaScript as the way to get things done. But if you're scripting a browser, your scripts end up dealing with user behavior emulation, which is almost always simpler than protocol layer emulation. Understand: I'm not saying this stuff is impossible today. I'm saying it's harder than it needs to be and can be greatly simplified by just using a real browser. All that said, I appreciate your frustration with the lack of tools. I hope you'll give our approach a closer look. I'd certainly appreciate hands-on feedback. We do offer more traditional virtual user scripting support as well (at 1/10th the price of our real browser service), so if you're good with JavaScript you can get some pretty affordable on-demand testing! Patrick