7 ms·

Has anyone had rousing success with automation tests like this? I almost always find the ratio of maintenance to issues caught to be unacceptably high, especial

by ng12 7y ago

Has anyone had rousing success with automation tests like this? I almost always find the ratio of maintenance to issues caught to be unacceptably high, especially when compared to simple unit tests.

l0b0 7y ago
As much as I love unit tests (being simpler and running more than 1000 times faster than web tests) there is a place for basically all types of tests in a CRUD application. 100% unit test coverage isn't going to help you if you can't actually spin up the whole system and serve requests over high-level protocols. It's all about ROI, and ROI is huge for simple high-level smoke tests.
- freeone3000 7y ago
  My experience with selenium is that the tests broke more often than the UI, whether through timing issues, DOM changes, or framework changes. The binding is on a layer that humans don't see, and it makes it fragile.
  - jdc 7y ago
    At this point I might do it for greenfield work.
  - joshstrange 7y ago
    This was my experience to a a T but then we hired a guy to solely manage the tests instead of having the developers do it and his tests caught a few false positives at the start but over time we were able to eliminate a 2-day+ crawl over the site by QA and UAT. My takeaway from this (which was the same argument I made back when we were failing at producing anything even usable let alone reliable) is: It needs to be someone's sole goal and I'll go a step further and say they should be on the outside of the team developing the main app. Being too close to the underlying code makes for a bad QA/Automated Tester IMHO. I have no problem even being the person who is managing/writing the tests but it's very hard to do both in my experiance.
    - ethbro 7y ago
      As someone who's worked in the UI automation field for approaching 10 years, getting it to not break takes a little of (1) knowing what actual user interactions look like, (2) knowing how UIs commonly break, (3) knowing how the underlying UI framework commonly breaks / what shortcuts programmers are apt to take (e.g. simulated async in VB6), (4) knowing how events and code interact in a running system, & (5) striking the proper balance between performance and reliability. Honestly, though, 75% of automation errors are timing missteps -- automation dispatching events at a specific millisecond that a human never would.
  - bryanrasmussen 7y ago
    DOM changes should break the test (although you can write the tests not to be some tightly bound to the DOM), just like API changes should break tests of the API.
  - Ididntdothis 7y ago
    There is a learning curve. Your devs need to learn how to make things easier for the testers and with experience the testers can work around most of the issues you are describing.
  - gregmac 7y ago
    I was recently involved in my first "integration testing" project involving Selenium, Winium, and ssh (doing cross-platform testing of an installer). One of the things that really struck me as I was reading through examples and such was how common it seems to be to have test code like "Click x; Wait 5 seconds; Click y; Type 'abc'; Click Z". My experience with unit testing tells me that as tests break 'randomly' (because something occasionally takes a few milliseconds too long to load) those delays get inserted, and eventually you end up with slow tests that are harder to maintain and still flaky. In my project, we were installing on a variety of OSes and versions, and based on what dependencies had to be downloaded+installed it could take anywhere from a few seconds to a couple minutes. I ended up writing a lot of helper functions like `WaitForFile(filename, absoluteTimeout)` that basically run a retry loop, checking every couple seconds (to allow fast-as-possible pass/fail), but frankly was surprised that this was non-standard -- I kind of expected any "integration test assertion library" to be chock full of helpers like this. I think there's a lot of ground to be covered in improving integration testing in general, and especially bringing in some more modern software design and engineering principles. Unit and integration tests are still code after all -- just because it doesn't ship with the product doesn't mean it shouldn't be maintained at the same quality.
    - danidiaz 7y ago
      > WaitForFile(filename, absoluteTimeout) The Java bindings have support for this in the form of FluentWait#until http://javadox.com/org.seleniumhq.selenium/selenium-support/2.53.0/org/openqa/selenium/support/ui/FluentWait.html#until-com.google.common.base.Function- http://javadox.com/org.seleniumhq.selenium/selenium-support/... Using explicit waits for expected conditions (instead of static waits or implicit waits) helps with making the tests more robust. It's a bit more work though.
    - plorntus 7y ago
      Cypress is pretty good for this. Every command is queued up and sort of deferred. You write tests in a synchronous manner and it has retrryability built in so it will wait for an assertion to be truth before continuing (with a max timeout ofc). It's been working pretty well for me so far.
    - 7y ago
  - maeln 7y ago
    I had the same experience on projects where actual testing was not a priority over "dumb" unit testing (testing dummy function just to get the 100% coverage ...). But, I work on some project where testing with selenium/cypress was a important functionality so the whole code was designed to be testable, and this made all the difference. Once you start to be careful and design your code to be automatically testable (which in any case, is just applying some good practice most of the time), it yield some very good result and it does catch a lot of regression/bugs. Unit testing on the other end, they were letting a lot of problems pass through...
  - cowsandmilk 7y ago
    > The binding is on a layer that humans don't see, and it makes it fragile. And that seems to be part of the point of this project. From the description: > In Selenium, you need to use HTML IDs, XPaths and CSS selectors to identify web page elements. Helium on the other hand lets you refer to elements by their user-visible labels. The example script uses a bunch of user-visible labels, not classes, ids, ancestors, etc. It is things like "click('Sign in')" which you would tell a user you were walking through an interface.
    - bauerd 7y ago
      This works fine until it does not, for example when dealing with custom UI controls, which is when reality sets in and you find yourself wrangling HTML IDs, XPaths and CSS selectors again.
      - ethbro 7y ago
        In worst cases, I always tell people to bind to user-visible elements, then only navigate the local DOM from there. Global XPath type traversal is fragile, unless you're plugged in enough that the dev team lets you know when they break it.
paulryanrogers 7y ago
I've had limited success. IME it's critical to make them as deterministic as possible without being too specific. Non-random test data coupled with 'id' attributes on interacted elements help a lot. Beyond that each webdriver abstraction has quirks one must learn. Animations and rendering races are sticky points too. Usually it's possible to put in some strategic waits to get consistent results. Lastly getting them running in parallel can be tough, but the payoff is usually worth it if the number of tests must grow beyond a few dozen.
- ethbro 7y ago
  If your UX designers can stomach it, simply ensuring async actions have a defined signpost to signify completion is the most helpful. E.g. an element that shows up when results are finished loading on the page. "Spinner that disappears" isn't terrible, but isn't great either. "Everything looks the same, and then some results load, and more results might load at some point, maybe" is sadly not that rare.
pintxo 7y ago
We had them, then stopped maintaining them when the development speed was to high. Now (years later) we have a subset of the original tests to ensure the most crucial processes are regression tested. My experience is that it's a lot harder for these tests to find the right balance and depth than for unit or even other integration tests. As already said, it's good to consistently have ids on relevant elements, as otherwise the test code complexity grows significantly.
- Ididntdothis 7y ago
  "As already said, it's good to consistently have ids on relevant elements, as otherwise the test code complexity grows significantly. " Life is definitely much easier if the devs set up their pages to be automation friendly. I view this as a prerequisite. Otherwise tests can get nightmarish complexity.
  - ethbro 7y ago
    Here's looking at you, Material UI. (And almost any vendor-GUI-framework-in-a-box solution) :/
soperj 7y ago
We have, but I work in banking (basically), and we have a group solely dedicated to testing. So they're consistently maintaining them.
Ididntdothis 7y ago
I work in medical devices and we have to do UI testing if we like it or not. Automated UI testing can be very hard and has a steep learning curve but the payoff is huge. I can see how maintaining them in very fast moving environment would be a pain. Thank God we are much slower.
wpietri 7y ago
Like others, I've had them be very useful as high-level smoke tests. For making sure the details work, I test at lower layers. But without some external-perspective smoke tests, I find that I'll just end up doing them manually to make sure things really still work.
tdumitrescu 7y ago
Yes, we have a Selenium-based UI test suite at Mixpanel that's overall quite stable and runs all tests across several browsers. It tests strictly the front-end code, with API calls stubbed out, which is the biggest factor in maintaining fast deterministic tests. By way of contrast we have an older end-to-end Selenium suite that spins up the entire service cluster, and that is the usual flaming pile of slow flakiness associated with Selenium. One of our engineers did a writeup on it a few years ago (now somewhat outdated but the overall picture is still accurate): https://engineering.mixpanel.com/2018/10/31/the-state-of-ui-testing-at-mixpanel/ https://engineering.mixpanel.com/2018/10/31/the-state-of-ui-...
bryanrasmussen 7y ago
I've always been of the opinion that GUI tests are the best, except for the expensiveness of running the tests means you can't cover as many cases as you can with unit tests. At any rate there are of course issues that can be caught in GUI tests that cannot be caught in unit tests.
inglor 7y ago
I work at Testim.io, we record this sort of thing. It works pretty well but it's pretty expensive to do automation well. We work with Microsoft, Wix, JP Morgan and a bunch of other large companies successfully for years now.
TeeWEE 7y ago
Every heard about the testing pyramid? https://martinfowler.com/articles/practical-test-pyramid.html https://martinfowler.com/articles/practical-test-pyramid.htm... A few UI tests on top of the pyramid does help. We did move from selenium to Cypress because of flakyness in selenium, and because most browser run the same engine under the hood (cypress is chrome only).
- acemarke 7y ago
  Cypress actually _just_ released Firefox and Edge support a couple weeks ago: https://www.cypress.io/blog/2020/02/06/introducing-firefox-and-edge-support-in-cypress-4-0/ https://www.cypress.io/blog/2020/02/06/introducing-firefox-a... The Edge support kinda came for free thanks to its switch to the Blink engine, but the FF support had been under development for a long time.
jfengel 7y ago
I have found that they're especially good at spotting embarrassing mistakes -- where you've broken something really obvious to every user, such as outright crashes in common use cases. Such things make users really angry: "How could you not have spotted this? How can I trust your product at all?" The ratio of maintenance to issues remains very high, but the kinds of issues that it can catch put a thumb on that ratio.

by ng12 7y ago

Has anyone had rousing success with automation tests like this? I almost always find the ratio of maintenance to issues caught to be unacceptably high, especially when compared to simple unit tests.

l0b0 7y ago

As much as I love unit tests (being simpler and running more than 1000 times faster than web tests) there is a place for basically all types of tests in a CRUD application. 100% unit test coverage isn't going to help you if you can't actually spin up the whole system and serve requests over high-level protocols. It's all about ROI, and ROI is huge for simple high-level smoke tests.

freeone3000 7y ago
My experience with selenium is that the tests broke more often than the UI, whether through timing issues, DOM changes, or framework changes. The binding is on a layer that humans don't see, and it makes it fragile.
- jdc 7y ago
  At this point I might do it for greenfield work.
- joshstrange 7y ago
  This was my experience to a a T but then we hired a guy to solely manage the tests instead of having the developers do it and his tests caught a few false positives at the start but over time we were able to eliminate a 2-day+ crawl over the site by QA and UAT. My takeaway from this (which was the same argument I made back when we were failing at producing anything even usable let alone reliable) is: It needs to be someone's sole goal and I'll go a step further and say they should be on the outside of the team developing the main app. Being too close to the underlying code makes for a bad QA/Automated Tester IMHO. I have no problem even being the person who is managing/writing the tests but it's very hard to do both in my experiance.
  - ethbro 7y ago
    As someone who's worked in the UI automation field for approaching 10 years, getting it to not break takes a little of (1) knowing what actual user interactions look like, (2) knowing how UIs commonly break, (3) knowing how the underlying UI framework commonly breaks / what shortcuts programmers are apt to take (e.g. simulated async in VB6), (4) knowing how events and code interact in a running system, & (5) striking the proper balance between performance and reliability. Honestly, though, 75% of automation errors are timing missteps -- automation dispatching events at a specific millisecond that a human never would.
- bryanrasmussen 7y ago
  DOM changes should break the test (although you can write the tests not to be some tightly bound to the DOM), just like API changes should break tests of the API.
- Ididntdothis 7y ago
  There is a learning curve. Your devs need to learn how to make things easier for the testers and with experience the testers can work around most of the issues you are describing.
- gregmac 7y ago
  I was recently involved in my first "integration testing" project involving Selenium, Winium, and ssh (doing cross-platform testing of an installer). One of the things that really struck me as I was reading through examples and such was how common it seems to be to have test code like "Click x; Wait 5 seconds; Click y; Type 'abc'; Click Z". My experience with unit testing tells me that as tests break 'randomly' (because something occasionally takes a few milliseconds too long to load) those delays get inserted, and eventually you end up with slow tests that are harder to maintain and still flaky. In my project, we were installing on a variety of OSes and versions, and based on what dependencies had to be downloaded+installed it could take anywhere from a few seconds to a couple minutes. I ended up writing a lot of helper functions like `WaitForFile(filename, absoluteTimeout)` that basically run a retry loop, checking every couple seconds (to allow fast-as-possible pass/fail), but frankly was surprised that this was non-standard -- I kind of expected any "integration test assertion library" to be chock full of helpers like this. I think there's a lot of ground to be covered in improving integration testing in general, and especially bringing in some more modern software design and engineering principles. Unit and integration tests are still code after all -- just because it doesn't ship with the product doesn't mean it shouldn't be maintained at the same quality.
  - danidiaz 7y ago
    > WaitForFile(filename, absoluteTimeout) The Java bindings have support for this in the form of FluentWait#until http://javadox.com/org.seleniumhq.selenium/selenium-support/2.53.0/org/openqa/selenium/support/ui/FluentWait.html#until-com.google.common.base.Function- http://javadox.com/org.seleniumhq.selenium/selenium-support/... Using explicit waits for expected conditions (instead of static waits or implicit waits) helps with making the tests more robust. It's a bit more work though.
  - plorntus 7y ago
    Cypress is pretty good for this. Every command is queued up and sort of deferred. You write tests in a synchronous manner and it has retrryability built in so it will wait for an assertion to be truth before continuing (with a max timeout ofc). It's been working pretty well for me so far.
  - 7y ago
- maeln 7y ago
  I had the same experience on projects where actual testing was not a priority over "dumb" unit testing (testing dummy function just to get the 100% coverage ...). But, I work on some project where testing with selenium/cypress was a important functionality so the whole code was designed to be testable, and this made all the difference. Once you start to be careful and design your code to be automatically testable (which in any case, is just applying some good practice most of the time), it yield some very good result and it does catch a lot of regression/bugs. Unit testing on the other end, they were letting a lot of problems pass through...
- cowsandmilk 7y ago
  > The binding is on a layer that humans don't see, and it makes it fragile. And that seems to be part of the point of this project. From the description: > In Selenium, you need to use HTML IDs, XPaths and CSS selectors to identify web page elements. Helium on the other hand lets you refer to elements by their user-visible labels. The example script uses a bunch of user-visible labels, not classes, ids, ancestors, etc. It is things like "click('Sign in')" which you would tell a user you were walking through an interface.
  - bauerd 7y ago
    This works fine until it does not, for example when dealing with custom UI controls, which is when reality sets in and you find yourself wrangling HTML IDs, XPaths and CSS selectors again.
    - ethbro 7y ago
      In worst cases, I always tell people to bind to user-visible elements, then only navigate the local DOM from there. Global XPath type traversal is fragile, unless you're plugged in enough that the dev team lets you know when they break it.

paulryanrogers 7y ago

I've had limited success. IME it's critical to make them as deterministic as possible without being too specific. Non-random test data coupled with 'id' attributes on interacted elements help a lot. Beyond that each webdriver abstraction has quirks one must learn. Animations and rendering races are sticky points too. Usually it's possible to put in some strategic waits to get consistent results. Lastly getting them running in parallel can be tough, but the payoff is usually worth it if the number of tests must grow beyond a few dozen.

ethbro 7y ago
If your UX designers can stomach it, simply ensuring async actions have a defined signpost to signify completion is the most helpful. E.g. an element that shows up when results are finished loading on the page. "Spinner that disappears" isn't terrible, but isn't great either. "Everything looks the same, and then some results load, and more results might load at some point, maybe" is sadly not that rare.

pintxo 7y ago

We had them, then stopped maintaining them when the development speed was to high. Now (years later) we have a subset of the original tests to ensure the most crucial processes are regression tested. My experience is that it's a lot harder for these tests to find the right balance and depth than for unit or even other integration tests. As already said, it's good to consistently have ids on relevant elements, as otherwise the test code complexity grows significantly.

Ididntdothis 7y ago
"As already said, it's good to consistently have ids on relevant elements, as otherwise the test code complexity grows significantly. " Life is definitely much easier if the devs set up their pages to be automation friendly. I view this as a prerequisite. Otherwise tests can get nightmarish complexity.
- ethbro 7y ago
  Here's looking at you, Material UI. (And almost any vendor-GUI-framework-in-a-box solution) :/

soperj 7y ago

We have, but I work in banking (basically), and we have a group solely dedicated to testing. So they're consistently maintaining them.

Ididntdothis 7y ago

I work in medical devices and we have to do UI testing if we like it or not. Automated UI testing can be very hard and has a steep learning curve but the payoff is huge. I can see how maintaining them in very fast moving environment would be a pain. Thank God we are much slower.

wpietri 7y ago

Like others, I've had them be very useful as high-level smoke tests. For making sure the details work, I test at lower layers. But without some external-perspective smoke tests, I find that I'll just end up doing them manually to make sure things really still work.

tdumitrescu 7y ago

Yes, we have a Selenium-based UI test suite at Mixpanel that's overall quite stable and runs all tests across several browsers. It tests strictly the front-end code, with API calls stubbed out, which is the biggest factor in maintaining fast deterministic tests. By way of contrast we have an older end-to-end Selenium suite that spins up the entire service cluster, and that is the usual flaming pile of slow flakiness associated with Selenium. One of our engineers did a writeup on it a few years ago (now somewhat outdated but the overall picture is still accurate): https://engineering.mixpanel.com/2018/10/31/the-state-of-ui-testing-at-mixpanel/ https://engineering.mixpanel.com/2018/10/31/the-state-of-ui-...

bryanrasmussen 7y ago

I've always been of the opinion that GUI tests are the best, except for the expensiveness of running the tests means you can't cover as many cases as you can with unit tests. At any rate there are of course issues that can be caught in GUI tests that cannot be caught in unit tests.

inglor 7y ago

I work at Testim.io, we record this sort of thing. It works pretty well but it's pretty expensive to do automation well. We work with Microsoft, Wix, JP Morgan and a bunch of other large companies successfully for years now.

TeeWEE 7y ago

Every heard about the testing pyramid? https://martinfowler.com/articles/practical-test-pyramid.html https://martinfowler.com/articles/practical-test-pyramid.htm... A few UI tests on top of the pyramid does help. We did move from selenium to Cypress because of flakyness in selenium, and because most browser run the same engine under the hood (cypress is chrome only).

acemarke 7y ago
Cypress actually _just_ released Firefox and Edge support a couple weeks ago: https://www.cypress.io/blog/2020/02/06/introducing-firefox-and-edge-support-in-cypress-4-0/ https://www.cypress.io/blog/2020/02/06/introducing-firefox-a... The Edge support kinda came for free thanks to its switch to the Blink engine, but the FF support had been under development for a long time.

jfengel 7y ago

I have found that they're especially good at spotting embarrassing mistakes -- where you've broken something really obvious to every user, such as outright crashes in common use cases. Such things make users really angry: "How could you not have spotted this? How can I trust your product at all?" The ratio of maintenance to issues remains very high, but the kinds of issues that it can catch put a thumb on that ratio.