5 ms·

I remember writing tests in Selenium in the past. Writing them was a horrible experience and some tests were not deterministic. The same test could fail or pass

by luhego 7y ago

I remember writing tests in Selenium in the past. Writing them was a horrible experience and some tests were not deterministic. The same test could fail or pass randomly.

bluntfang 7y ago
preface: i think there are tricks that experienced selenium developers use to make test less brittle, and it's annoying that there are tricks for that. At the end of the day, if you can't get stable tests using the WebDriver W3C standard [0], you are doing something weird and overly complex with your web application. End to end testing isn't going to give you an objective right or wrong every time, but it should make you ask the question "Why does this happen?" and the answer is usually "Oh, we are doing something weird". 0: https://www.w3.org/TR/webdriver/ https://www.w3.org/TR/webdriver/
Afton 7y ago
I've worked with selenium for a couple of years. There are a fair number of gotchas, but it's definitely possible to write pretty deterministic tests. For one, retries of selenium actions are just necessary. So necessary, that you build them in to your test framework, instead of making them something that test-writers have to interact with. I don't know what specific problems you were having, so I can't give more specific advice than that.
- seleniumbase 7y ago
  SeleniumBase has those retries built-in. The browser will automatically wait for page elements to fully load before interacting with them. In addition, SeleniumBase includes all the abilities of pytest, which transforms Selenium from a library into a complete test automation framework.
ratbeard 7y ago
We invested a large amount of time in them recently, and I agree they are terrible to write. We dare not even turn on IE or any other browser, we can't even keep the tests green in Firefox. Really wish we'd used cypress instead as its 100x easier to debug and we're not getting cross-browser benefits anyways. The underlying architecture is a bad design for heavy javascript apps in my opinion. The roundtrips between the test runner talking to selenium server talking to selenium driver in a browser and back the other way is slow and so much can change on the page in between steps in your tests. Cypress runs your test code in the javascript process of your browser so I believe theres no or minimal roundtrip lag. We use `waitFor()` for the UI to stabilize but thats been a hard mental model for devs to follow and as a result we have tons of unnecessary waits in the tests which slows them down. Even things like waiting for a loading modal to disappear before trying to interact with the UI is hard since your code: `waitFor('.loading-modal', false) // wait for it NOT to exist` may run BEFORE its even appeared, then fail in the next step when you try clicking on a button and the modal is there now. You can't wait for it to appear first to prevent that, as your code may run after its already come and gone too. Tons of annoyances or strange behavior like chromedriver doesn't support emojis in inputs, ieDriver had select boxes set value broken at one point, setValue('123') on a field actually does something like 'focus, clear, blur, APPEND 123' so blur logic to set a default value of '0' on your field will result in the final value being '0123' in your tests… just the worst.
- 2rsf 7y ago
  > You can't wait for it to appear first to prevent that, as your code may run after its already come and gone too. While valid that's not typical for many sites- what's th e point of a very short lived pop up? and even if it is part of your page, you can skip the "risky" part of the test and verify it otherwise (logs ? side effects ?) or not at all.
  - ratbeard 7y ago
    A 'saving…' or 'loading…' popup or any type of interaction preventing mask is a common UX pattern in my opinion in javascripts heavy apps. We didn't care about testing the popup at all, it was just breaking our other tests in the following way. In our UI you could click a 'save' button, then a 'saving…' popup appears, meanwhile the 'save' button goes away and an 'edit' button appears (behind the popup), and when the response comes back it says 'Saved.' in the ui. A test for `$('div=Saved.').toExist()` in wdio works, it does a waitFor under the covers and polls the UI until that text appears. It doesn't care if theres a popup shown or not. However moving on to the next step in the tests, `$('button=Edit').click()` throws an error 'element is not clickable' if the popup is visible when it happens to runs. Doing multi-command steps like 'check if popup is there, if not click' in general doesn't work as theres so much latency between commands. You can inject javascript in to the page that does both checks in the browsers js process as a hacky workaround. We did upgrade our webdriver library partly to get waitForClickable() which based on the name at least sounded like it handle the above, but there were no volunteers to update the 168 instances where waitForLoader() had spread in the codebase :/
zmmmmm 7y ago
Yes, using bare selenium is a writeoff, especially if you are testing modern frameworks like React or VueJS which dynamically modify the DOM. You need firstly a framework that understands these technologies and then on top of that you want something that offers a structure to capture the common actions and build an automatable view of a page. For these requirements I have used Geb [1] with the Page Object Model approach with reasonable success - but you still have to approach writing your tests as actual software, not ad hoc random scripting. [1] https://gebish.org/ https://gebish.org/
- sumedh 7y ago
  Have you looked at cypress?
  - bluntfang 7y ago
    cypress is a non-starter for end-to-end testing until it supports the WebDriver W3C spec. I hear it's great for unit testing javascript though.
kccqzy 7y ago
I also remember doing so. It was such an effort that we only managed to get tests working for the registration and login flow, and only on Firefox ESR. They are frequent broken by changing CSS classes or IDs or the structure of the HTML. Errors are very opaque like "element not clickable at point" followed by a coordinate. This was all before the W3C standardized browser automation, so I guess the situation may have improved a bit.
- bluntfang 7y ago
  >It was such an effort that we only managed to get tests working for the registration and login flow, and only on Firefox ESR. They are frequent broken by changing CSS classes or IDs or the structure of the HTML. It sounds like you were approaching the end-to-end tests as an add-on and not part of your production code. If your problem is changing selectors, you need to make sure your developers know that changing selectors is going to make the tests fail and you need to equip them to handle it through tooling.
nitwit005 7y ago
I've fixed up tests that randomly failed in the past. It's definitely possible to get them to work reliably, but it does require a paranoid approach of carefully waiting for changes. Humorously enough though, people had never bothered to debug why the tests were failing. One reason was that a backend service crashed. It turned out that quickly deactivating users after doing things in the UI caused stuck threads that ran infinite loops. No one had ever looked at the logs.
mintzworld 7y ago
SeleniumBase (which wraps Selenium) adds reliability by improving existing methods so that the browser waits for page elements to fully load before interacting with them. It might be worth trying some of the examples from the SeleniumBase GitHub page to see if you still feel that way.
pas 7y ago
We used NightwatchJS (went all the way with Page Objects) and it was a joy, because we had very real e2e coverage.
dplgk 7y ago
Same experience here. Invested a lot time rewriting tests after each framework did the same thing: - spent days/weeks refactoring tests to work with new testing framework - got tests 100% passing locally, passing in CI env - tests fail randomly on other people's machine or in CI - pick new framework, rinse repeat. Tried cypress, webdriver, jest, etc. Total waste of time.
- seleniumbase 7y ago
  SeleniumBase gives you more consistent results than just using WebDriver alone, as SeleniumBase wraps WebDriver methods to improve the reliability of browser actions.

by luhego 7y ago

I remember writing tests in Selenium in the past. Writing them was a horrible experience and some tests were not deterministic. The same test could fail or pass randomly.

bluntfang 7y ago

preface: i think there are tricks that experienced selenium developers use to make test less brittle, and it's annoying that there are tricks for that. At the end of the day, if you can't get stable tests using the WebDriver W3C standard [0], you are doing something weird and overly complex with your web application. End to end testing isn't going to give you an objective right or wrong every time, but it should make you ask the question "Why does this happen?" and the answer is usually "Oh, we are doing something weird". 0: https://www.w3.org/TR/webdriver/ https://www.w3.org/TR/webdriver/

Afton 7y ago

I've worked with selenium for a couple of years. There are a fair number of gotchas, but it's definitely possible to write pretty deterministic tests. For one, retries of selenium actions are just necessary. So necessary, that you build them in to your test framework, instead of making them something that test-writers have to interact with. I don't know what specific problems you were having, so I can't give more specific advice than that.

seleniumbase 7y ago
SeleniumBase has those retries built-in. The browser will automatically wait for page elements to fully load before interacting with them. In addition, SeleniumBase includes all the abilities of pytest, which transforms Selenium from a library into a complete test automation framework.

ratbeard 7y ago

We invested a large amount of time in them recently, and I agree they are terrible to write. We dare not even turn on IE or any other browser, we can't even keep the tests green in Firefox. Really wish we'd used cypress instead as its 100x easier to debug and we're not getting cross-browser benefits anyways. The underlying architecture is a bad design for heavy javascript apps in my opinion. The roundtrips between the test runner talking to selenium server talking to selenium driver in a browser and back the other way is slow and so much can change on the page in between steps in your tests. Cypress runs your test code in the javascript process of your browser so I believe theres no or minimal roundtrip lag. We use `waitFor()` for the UI to stabilize but thats been a hard mental model for devs to follow and as a result we have tons of unnecessary waits in the tests which slows them down. Even things like waiting for a loading modal to disappear before trying to interact with the UI is hard since your code: `waitFor('.loading-modal', false) // wait for it NOT to exist` may run BEFORE its even appeared, then fail in the next step when you try clicking on a button and the modal is there now. You can't wait for it to appear first to prevent that, as your code may run after its already come and gone too. Tons of annoyances or strange behavior like chromedriver doesn't support emojis in inputs, ieDriver had select boxes set value broken at one point, setValue('123') on a field actually does something like 'focus, clear, blur, APPEND 123' so blur logic to set a default value of '0' on your field will result in the final value being '0123' in your tests… just the worst.

2rsf 7y ago
> You can't wait for it to appear first to prevent that, as your code may run after its already come and gone too. While valid that's not typical for many sites- what's th e point of a very short lived pop up? and even if it is part of your page, you can skip the "risky" part of the test and verify it otherwise (logs ? side effects ?) or not at all.
- ratbeard 7y ago
  A 'saving…' or 'loading…' popup or any type of interaction preventing mask is a common UX pattern in my opinion in javascripts heavy apps. We didn't care about testing the popup at all, it was just breaking our other tests in the following way. In our UI you could click a 'save' button, then a 'saving…' popup appears, meanwhile the 'save' button goes away and an 'edit' button appears (behind the popup), and when the response comes back it says 'Saved.' in the ui. A test for `$('div=Saved.').toExist()` in wdio works, it does a waitFor under the covers and polls the UI until that text appears. It doesn't care if theres a popup shown or not. However moving on to the next step in the tests, `$('button=Edit').click()` throws an error 'element is not clickable' if the popup is visible when it happens to runs. Doing multi-command steps like 'check if popup is there, if not click' in general doesn't work as theres so much latency between commands. You can inject javascript in to the page that does both checks in the browsers js process as a hacky workaround. We did upgrade our webdriver library partly to get waitForClickable() which based on the name at least sounded like it handle the above, but there were no volunteers to update the 168 instances where waitForLoader() had spread in the codebase :/

zmmmmm 7y ago

Yes, using bare selenium is a writeoff, especially if you are testing modern frameworks like React or VueJS which dynamically modify the DOM. You need firstly a framework that understands these technologies and then on top of that you want something that offers a structure to capture the common actions and build an automatable view of a page. For these requirements I have used Geb [1] with the Page Object Model approach with reasonable success - but you still have to approach writing your tests as actual software, not ad hoc random scripting. [1] https://gebish.org/ https://gebish.org/

sumedh 7y ago
Have you looked at cypress?
- bluntfang 7y ago
  cypress is a non-starter for end-to-end testing until it supports the WebDriver W3C spec. I hear it's great for unit testing javascript though.

kccqzy 7y ago

I also remember doing so. It was such an effort that we only managed to get tests working for the registration and login flow, and only on Firefox ESR. They are frequent broken by changing CSS classes or IDs or the structure of the HTML. Errors are very opaque like "element not clickable at point" followed by a coordinate. This was all before the W3C standardized browser automation, so I guess the situation may have improved a bit.

bluntfang 7y ago
>It was such an effort that we only managed to get tests working for the registration and login flow, and only on Firefox ESR. They are frequent broken by changing CSS classes or IDs or the structure of the HTML. It sounds like you were approaching the end-to-end tests as an add-on and not part of your production code. If your problem is changing selectors, you need to make sure your developers know that changing selectors is going to make the tests fail and you need to equip them to handle it through tooling.

nitwit005 7y ago

I've fixed up tests that randomly failed in the past. It's definitely possible to get them to work reliably, but it does require a paranoid approach of carefully waiting for changes. Humorously enough though, people had never bothered to debug why the tests were failing. One reason was that a backend service crashed. It turned out that quickly deactivating users after doing things in the UI caused stuck threads that ran infinite loops. No one had ever looked at the logs.

mintzworld 7y ago

SeleniumBase (which wraps Selenium) adds reliability by improving existing methods so that the browser waits for page elements to fully load before interacting with them. It might be worth trying some of the examples from the SeleniumBase GitHub page to see if you still feel that way.

pas 7y ago

We used NightwatchJS (went all the way with Page Objects) and it was a joy, because we had very real e2e coverage.

dplgk 7y ago

Same experience here. Invested a lot time rewriting tests after each framework did the same thing: - spent days/weeks refactoring tests to work with new testing framework - got tests 100% passing locally, passing in CI env - tests fail randomly on other people's machine or in CI - pick new framework, rinse repeat. Tried cypress, webdriver, jest, etc. Total waste of time.

seleniumbase 7y ago
SeleniumBase gives you more consistent results than just using WebDriver alone, as SeleniumBase wraps WebDriver methods to improve the reliability of browser actions.