6 ms·

Selenium tests are inherently slow, unreliable and flappy. They have been the bane of developers for every employer I've had. Do yourself a favor and write Reac

Selenium tests are inherently slow, unreliable and flappy. They have been the bane of developers for every employer I've had. Do yourself a favor and write React and test your components without a browser driver in good ol' JS with the occasional JSDom shim. It removes almost the entire need for Selenium, which should be reserved for only the faintest of smoke tests. And please, if you have to use Selenium, use headless Firefox, because PhantomJS is very bad software.

patio11 11y ago
I had a Rails consultancy (Makandra) recently work on a JS-heavy application that I happen to own, and they got Selenium singing on it, which had been beyond my capabilities for years. One of their tricks, which you can inspect the implementation of in their (public) utilities library [+], is using basically a vendored Firefox per project and VNCing into that Firefox to drive things around. It is thus off-screen and out of the way when you're using it, but apparently is more true-to-reality than headless. The test suite they wrote has about ~600 tests and while they're slower than I'd like (2~3 minutes) they've been bulletproof since we got my dev environment configured properly. It includes some fairly complicated interactions, most relevantly around our calendar interface. [+] https://github.com/makandra/geordi https://github.com/makandra/geordi
- logn 11y ago
  I had been using Firefox driver in Xvfb but wasn't happy with the performance/stability. So I built a Selenium driver out of Java only (using JavaFX's embedded WebKit) and used a headless JRE windowing toolkit (Monocle). My project is still a pre-release but the headless capability, Java-only system requirement, and its ajax handling might make it useful to some people currently: https://github.com/MachinePublishers/jBrowserDriver https://github.com/MachinePublishers/jBrowserDriver
  - boundlessdreamz 11y ago
    This looks very interesting! 1. How does it compare with phantomJS? 2. What's the current webkit version? 3. How often does the javaFX webkit update?
    - logn 11y ago
      1. Not quite sure. I've only used PhantomJS via Selenium Ghost Driver. From that usage they're similar. The main difference is that my driver uses only Java so under the hood the JRE is launching WebKit through JNI and everything runs in the same JRE process. 2. Current WebKit version depends on the JRE used. Oracle Java 1.8.0_45 has WebKit version 537.44. 3. Java maintainers will update WebKit periodically, including within a major version. E.g., here they update WebKit for the 1.8.0_60 JRE: http://openjdk.java.net/jeps/239 http://openjdk.java.net/jeps/239 ... Other than that I'm not sure.
  - ericb 11y ago
    Neat, but Affero public license? Ick.
    - ericb 11y ago
      To clarify--this was something I was interested in helping with, until I saw the license.
      - logn 11y ago
        Is there any flavor of GPL license you would prefer more? I don't plan on BSD or Apache license in the foreseeable future.
        ericb 11y ago
        Honestly, if it was useful, I would probably use this my side project which is commercial, but in no way competes with what you're doing (load testing). I might make changes or improvements, and I would generally contribute those back. Affero doesn't "mix well" so it is pretty much a non-starter for me. I've found this is true for a lot of projects and it seems like restrictive licenses prevent projects from going mainstream.
- tkinom 11y ago
  I have trouble with Selenium's failure rate too. End up writing a test engine in Javascript. It handles async js func call with js function callback when finished to get rid of all the sleep-wait-retry type logic in Selenium. Works very well. I can run the 100+ test cases in all IE/FF/Chrome/Safari, ios/android browser without change one line of JS/Test code. Runs fine with desktop with wire connect to cellphone browser on cell connection. It tests out all the app backend db logic also. The time/pass/fail info are submitted back to the test backend db.
  - boundlessdreamz 11y ago
    Are you planning to open source it? Can you give an example of how it works? Say navigate to a page, fill in a form, click submit and verify that some text is present after submitting the form
jmickey 11y ago
This has been our experience as well. We invested a lot of time and money in making sure Selenium tests run reliably for our clients. Despite this, the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable. We have compiled a few tips we learned along the way in our blog post - http://novoit.eu/blog/05-5-tips-when-writing-Selenium-browser-tests http://novoit.eu/blog/05-5-tips-when-writing-Selenium-browse...
- crdoconnor 11y ago
  >the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable. What was actually going wrong during that 10%? I get something closer to 100% reliability, so I'm feeling a little perplexed by all of this. Do you make heavy use of sleeps?
  - jmickey 11y ago
    Mostly these would be cases where the browser would, seemingly at random, end up in an unpredictable state and all proceeding test scenarios would fail because of this. (Page is white, or a completely unrelated website gets opened. We have seen lots of weird situations so far) This might be exacerbated by the fact that we use the remote Browserstack Selenium hosting service so that the tests can be executed automatically as a part of our deployment process.
    - crdoconnor 11y ago
      White page and randomly ending up on an unrelated website both sound like bugs.
- omouse 11y ago
  >tests that run for 40 minutes This is pretty good actually. It sucks if you're relying on Selenium testing for verifying your code as you're writing it, but before and after deploys to staging and production? This isn't bad at all.
technion 11y ago
Can you expand on this? The extent of the testing I'm currently interested in is "load a page, does the JS on that page run without error"? It won't execute from a CLI and everyone I talked to pointed me at Selenium.
- stevebmark 11y ago
  Testing for a page load without JS error is a fine use case, and is an example of what I meant by the "faintest of smoke tests." It's a test that has very little chance to flap, fail, or force you to write hacky commands around Selenium's unreliable API.
rtpg 11y ago
Does it really? Does it test for whether a button you thought was present isn't actually clickable? If you're going to write tests, I think it makes an insane amount of sense to emulate real world conditions as much as feasibly possible (making judgement calls on things that don't matter like speed of the mouse).
- masklinn 11y ago
  Most Selenium tests don't test that a button is actually clickable though, they find things through the DOM and if the button is offscreen or hidden they won't realise it.
  - xirdstl 11y ago
    In my experience, an exception is thrown in my Selenium test if a button is hidden or not clickable and I try to click it.
discodave 11y ago
I pretty much agree with your thoughts but what makes you say PhantomJS is bad software?
lhorie 11y ago
For testing Mithril.js, I wrote a mock window object, which allow you to do things like simulate requestAnimationFrame clicks, JSON-P calls and browser quirks from non-browser environments (e.g. from a Node.js script). So to test, you simply swap `window` with the mock and you can drive your fake browser however you wish. http://lhorie.github.io/mithril/mithril.deps.html http://lhorie.github.io/mithril/mithril.deps.html You can cover a lot of ground with that approach and make an extremely fast test suite that is suitable for a save-refresh-test workflow and then you can put trickier tests in a secondary test suite that you only run once in a while (e.g. before a commit)

patio11 11y ago

I had a Rails consultancy (Makandra) recently work on a JS-heavy application that I happen to own, and they got Selenium singing on it, which had been beyond my capabilities for years. One of their tricks, which you can inspect the implementation of in their (public) utilities library [+], is using basically a vendored Firefox per project and VNCing into that Firefox to drive things around. It is thus off-screen and out of the way when you're using it, but apparently is more true-to-reality than headless. The test suite they wrote has about ~600 tests and while they're slower than I'd like (2~3 minutes) they've been bulletproof since we got my dev environment configured properly. It includes some fairly complicated interactions, most relevantly around our calendar interface. [+] https://github.com/makandra/geordi https://github.com/makandra/geordi

logn 11y ago
I had been using Firefox driver in Xvfb but wasn't happy with the performance/stability. So I built a Selenium driver out of Java only (using JavaFX's embedded WebKit) and used a headless JRE windowing toolkit (Monocle). My project is still a pre-release but the headless capability, Java-only system requirement, and its ajax handling might make it useful to some people currently: https://github.com/MachinePublishers/jBrowserDriver https://github.com/MachinePublishers/jBrowserDriver
- boundlessdreamz 11y ago
  This looks very interesting! 1. How does it compare with phantomJS? 2. What's the current webkit version? 3. How often does the javaFX webkit update?
  - logn 11y ago
    1. Not quite sure. I've only used PhantomJS via Selenium Ghost Driver. From that usage they're similar. The main difference is that my driver uses only Java so under the hood the JRE is launching WebKit through JNI and everything runs in the same JRE process. 2. Current WebKit version depends on the JRE used. Oracle Java 1.8.0_45 has WebKit version 537.44. 3. Java maintainers will update WebKit periodically, including within a major version. E.g., here they update WebKit for the 1.8.0_60 JRE: http://openjdk.java.net/jeps/239 http://openjdk.java.net/jeps/239 ... Other than that I'm not sure.
- ericb 11y ago
  Neat, but Affero public license? Ick.
  - ericb 11y ago
    To clarify--this was something I was interested in helping with, until I saw the license.
    - logn 11y ago
      Is there any flavor of GPL license you would prefer more? I don't plan on BSD or Apache license in the foreseeable future.
      - ericb 11y ago
        Honestly, if it was useful, I would probably use this my side project which is commercial, but in no way competes with what you're doing (load testing). I might make changes or improvements, and I would generally contribute those back. Affero doesn't "mix well" so it is pretty much a non-starter for me. I've found this is true for a lot of projects and it seems like restrictive licenses prevent projects from going mainstream.
tkinom 11y ago
I have trouble with Selenium's failure rate too. End up writing a test engine in Javascript. It handles async js func call with js function callback when finished to get rid of all the sleep-wait-retry type logic in Selenium. Works very well. I can run the 100+ test cases in all IE/FF/Chrome/Safari, ios/android browser without change one line of JS/Test code. Runs fine with desktop with wire connect to cellphone browser on cell connection. It tests out all the app backend db logic also. The time/pass/fail info are submitted back to the test backend db.
- boundlessdreamz 11y ago
  Are you planning to open source it? Can you give an example of how it works? Say navigate to a page, fill in a form, click submit and verify that some text is present after submitting the form

jmickey 11y ago

This has been our experience as well. We invested a lot of time and money in making sure Selenium tests run reliably for our clients. Despite this, the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable. We have compiled a few tips we learned along the way in our blog post - http://novoit.eu/blog/05-5-tips-when-writing-Selenium-browser-tests http://novoit.eu/blog/05-5-tips-when-writing-Selenium-browse...

crdoconnor 11y ago
>the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable. What was actually going wrong during that 10%? I get something closer to 100% reliability, so I'm feeling a little perplexed by all of this. Do you make heavy use of sleeps?
- jmickey 11y ago
  Mostly these would be cases where the browser would, seemingly at random, end up in an unpredictable state and all proceeding test scenarios would fail because of this. (Page is white, or a completely unrelated website gets opened. We have seen lots of weird situations so far) This might be exacerbated by the fact that we use the remote Browserstack Selenium hosting service so that the tests can be executed automatically as a part of our deployment process.
  - crdoconnor 11y ago
    White page and randomly ending up on an unrelated website both sound like bugs.
omouse 11y ago
>tests that run for 40 minutes This is pretty good actually. It sucks if you're relying on Selenium testing for verifying your code as you're writing it, but before and after deploys to staging and production? This isn't bad at all.

technion 11y ago

Can you expand on this? The extent of the testing I'm currently interested in is "load a page, does the JS on that page run without error"? It won't execute from a CLI and everyone I talked to pointed me at Selenium.

stevebmark 11y ago
Testing for a page load without JS error is a fine use case, and is an example of what I meant by the "faintest of smoke tests." It's a test that has very little chance to flap, fail, or force you to write hacky commands around Selenium's unreliable API.

rtpg 11y ago

Does it really? Does it test for whether a button you thought was present isn't actually clickable? If you're going to write tests, I think it makes an insane amount of sense to emulate real world conditions as much as feasibly possible (making judgement calls on things that don't matter like speed of the mouse).

masklinn 11y ago
Most Selenium tests don't test that a button is actually clickable though, they find things through the DOM and if the button is offscreen or hidden they won't realise it.
- xirdstl 11y ago
  In my experience, an exception is thrown in my Selenium test if a button is hidden or not clickable and I try to click it.

discodave 11y ago

I pretty much agree with your thoughts but what makes you say PhantomJS is bad software?

lhorie 11y ago

For testing Mithril.js, I wrote a mock window object, which allow you to do things like simulate requestAnimationFrame clicks, JSON-P calls and browser quirks from non-browser environments (e.g. from a Node.js script). So to test, you simply swap `window` with the mock and you can drive your fake browser however you wish. http://lhorie.github.io/mithril/mithril.deps.html http://lhorie.github.io/mithril/mithril.deps.html You can cover a lot of ground with that approach and make an extremely fast test suite that is suitable for a save-refresh-test workflow and then you can put trickier tests in a secondary test suite that you only run once in a while (e.g. before a commit)