6 ms·

As much as I love unit tests (being simpler and running more than 1000 times faster than web tests) there is a place for basically all types of tests in a CRUD

by l0b0 7y ago

As much as I love unit tests (being simpler and running more than 1000 times faster than web tests) there is a place for basically all types of tests in a CRUD application. 100% unit test coverage isn't going to help you if you can't actually spin up the whole system and serve requests over high-level protocols. It's all about ROI, and ROI is huge for simple high-level smoke tests.

freeone3000 7y ago
My experience with selenium is that the tests broke more often than the UI, whether through timing issues, DOM changes, or framework changes. The binding is on a layer that humans don't see, and it makes it fragile.
- jdc 7y ago
  At this point I might do it for greenfield work.
- joshstrange 7y ago
  This was my experience to a a T but then we hired a guy to solely manage the tests instead of having the developers do it and his tests caught a few false positives at the start but over time we were able to eliminate a 2-day+ crawl over the site by QA and UAT. My takeaway from this (which was the same argument I made back when we were failing at producing anything even usable let alone reliable) is: It needs to be someone's sole goal and I'll go a step further and say they should be on the outside of the team developing the main app. Being too close to the underlying code makes for a bad QA/Automated Tester IMHO. I have no problem even being the person who is managing/writing the tests but it's very hard to do both in my experiance.
  - ethbro 7y ago
    As someone who's worked in the UI automation field for approaching 10 years, getting it to not break takes a little of (1) knowing what actual user interactions look like, (2) knowing how UIs commonly break, (3) knowing how the underlying UI framework commonly breaks / what shortcuts programmers are apt to take (e.g. simulated async in VB6), (4) knowing how events and code interact in a running system, & (5) striking the proper balance between performance and reliability. Honestly, though, 75% of automation errors are timing missteps -- automation dispatching events at a specific millisecond that a human never would.
- bryanrasmussen 7y ago
  DOM changes should break the test (although you can write the tests not to be some tightly bound to the DOM), just like API changes should break tests of the API.
- Ididntdothis 7y ago
  There is a learning curve. Your devs need to learn how to make things easier for the testers and with experience the testers can work around most of the issues you are describing.
- gregmac 7y ago
  I was recently involved in my first "integration testing" project involving Selenium, Winium, and ssh (doing cross-platform testing of an installer). One of the things that really struck me as I was reading through examples and such was how common it seems to be to have test code like "Click x; Wait 5 seconds; Click y; Type 'abc'; Click Z". My experience with unit testing tells me that as tests break 'randomly' (because something occasionally takes a few milliseconds too long to load) those delays get inserted, and eventually you end up with slow tests that are harder to maintain and still flaky. In my project, we were installing on a variety of OSes and versions, and based on what dependencies had to be downloaded+installed it could take anywhere from a few seconds to a couple minutes. I ended up writing a lot of helper functions like `WaitForFile(filename, absoluteTimeout)` that basically run a retry loop, checking every couple seconds (to allow fast-as-possible pass/fail), but frankly was surprised that this was non-standard -- I kind of expected any "integration test assertion library" to be chock full of helpers like this. I think there's a lot of ground to be covered in improving integration testing in general, and especially bringing in some more modern software design and engineering principles. Unit and integration tests are still code after all -- just because it doesn't ship with the product doesn't mean it shouldn't be maintained at the same quality.
  - danidiaz 7y ago
    > WaitForFile(filename, absoluteTimeout) The Java bindings have support for this in the form of FluentWait#until http://javadox.com/org.seleniumhq.selenium/selenium-support/2.53.0/org/openqa/selenium/support/ui/FluentWait.html#until-com.google.common.base.Function- http://javadox.com/org.seleniumhq.selenium/selenium-support/... Using explicit waits for expected conditions (instead of static waits or implicit waits) helps with making the tests more robust. It's a bit more work though.
  - plorntus 7y ago
    Cypress is pretty good for this. Every command is queued up and sort of deferred. You write tests in a synchronous manner and it has retrryability built in so it will wait for an assertion to be truth before continuing (with a max timeout ofc). It's been working pretty well for me so far.
  - 7y ago
- maeln 7y ago
  I had the same experience on projects where actual testing was not a priority over "dumb" unit testing (testing dummy function just to get the 100% coverage ...). But, I work on some project where testing with selenium/cypress was a important functionality so the whole code was designed to be testable, and this made all the difference. Once you start to be careful and design your code to be automatically testable (which in any case, is just applying some good practice most of the time), it yield some very good result and it does catch a lot of regression/bugs. Unit testing on the other end, they were letting a lot of problems pass through...
- cowsandmilk 7y ago
  > The binding is on a layer that humans don't see, and it makes it fragile. And that seems to be part of the point of this project. From the description: > In Selenium, you need to use HTML IDs, XPaths and CSS selectors to identify web page elements. Helium on the other hand lets you refer to elements by their user-visible labels. The example script uses a bunch of user-visible labels, not classes, ids, ancestors, etc. It is things like "click('Sign in')" which you would tell a user you were walking through an interface.
  - bauerd 7y ago
    This works fine until it does not, for example when dealing with custom UI controls, which is when reality sets in and you find yourself wrangling HTML IDs, XPaths and CSS selectors again.
    - ethbro 7y ago
      In worst cases, I always tell people to bind to user-visible elements, then only navigate the local DOM from there. Global XPath type traversal is fragile, unless you're plugged in enough that the dev team lets you know when they break it.

by l0b0 7y ago

freeone3000 7y ago

My experience with selenium is that the tests broke more often than the UI, whether through timing issues, DOM changes, or framework changes. The binding is on a layer that humans don't see, and it makes it fragile.

jdc 7y ago
At this point I might do it for greenfield work.
joshstrange 7y ago
This was my experience to a a T but then we hired a guy to solely manage the tests instead of having the developers do it and his tests caught a few false positives at the start but over time we were able to eliminate a 2-day+ crawl over the site by QA and UAT. My takeaway from this (which was the same argument I made back when we were failing at producing anything even usable let alone reliable) is: It needs to be someone's sole goal and I'll go a step further and say they should be on the outside of the team developing the main app. Being too close to the underlying code makes for a bad QA/Automated Tester IMHO. I have no problem even being the person who is managing/writing the tests but it's very hard to do both in my experiance.
- ethbro 7y ago
  As someone who's worked in the UI automation field for approaching 10 years, getting it to not break takes a little of (1) knowing what actual user interactions look like, (2) knowing how UIs commonly break, (3) knowing how the underlying UI framework commonly breaks / what shortcuts programmers are apt to take (e.g. simulated async in VB6), (4) knowing how events and code interact in a running system, & (5) striking the proper balance between performance and reliability. Honestly, though, 75% of automation errors are timing missteps -- automation dispatching events at a specific millisecond that a human never would.
bryanrasmussen 7y ago
DOM changes should break the test (although you can write the tests not to be some tightly bound to the DOM), just like API changes should break tests of the API.
Ididntdothis 7y ago
There is a learning curve. Your devs need to learn how to make things easier for the testers and with experience the testers can work around most of the issues you are describing.
gregmac 7y ago
I was recently involved in my first "integration testing" project involving Selenium, Winium, and ssh (doing cross-platform testing of an installer). One of the things that really struck me as I was reading through examples and such was how common it seems to be to have test code like "Click x; Wait 5 seconds; Click y; Type 'abc'; Click Z". My experience with unit testing tells me that as tests break 'randomly' (because something occasionally takes a few milliseconds too long to load) those delays get inserted, and eventually you end up with slow tests that are harder to maintain and still flaky. In my project, we were installing on a variety of OSes and versions, and based on what dependencies had to be downloaded+installed it could take anywhere from a few seconds to a couple minutes. I ended up writing a lot of helper functions like `WaitForFile(filename, absoluteTimeout)` that basically run a retry loop, checking every couple seconds (to allow fast-as-possible pass/fail), but frankly was surprised that this was non-standard -- I kind of expected any "integration test assertion library" to be chock full of helpers like this. I think there's a lot of ground to be covered in improving integration testing in general, and especially bringing in some more modern software design and engineering principles. Unit and integration tests are still code after all -- just because it doesn't ship with the product doesn't mean it shouldn't be maintained at the same quality.
- danidiaz 7y ago
  > WaitForFile(filename, absoluteTimeout) The Java bindings have support for this in the form of FluentWait#until http://javadox.com/org.seleniumhq.selenium/selenium-support/2.53.0/org/openqa/selenium/support/ui/FluentWait.html#until-com.google.common.base.Function- http://javadox.com/org.seleniumhq.selenium/selenium-support/... Using explicit waits for expected conditions (instead of static waits or implicit waits) helps with making the tests more robust. It's a bit more work though.
- plorntus 7y ago
  Cypress is pretty good for this. Every command is queued up and sort of deferred. You write tests in a synchronous manner and it has retrryability built in so it will wait for an assertion to be truth before continuing (with a max timeout ofc). It's been working pretty well for me so far.
- 7y ago
maeln 7y ago
I had the same experience on projects where actual testing was not a priority over "dumb" unit testing (testing dummy function just to get the 100% coverage ...). But, I work on some project where testing with selenium/cypress was a important functionality so the whole code was designed to be testable, and this made all the difference. Once you start to be careful and design your code to be automatically testable (which in any case, is just applying some good practice most of the time), it yield some very good result and it does catch a lot of regression/bugs. Unit testing on the other end, they were letting a lot of problems pass through...
cowsandmilk 7y ago
> The binding is on a layer that humans don't see, and it makes it fragile. And that seems to be part of the point of this project. From the description: > In Selenium, you need to use HTML IDs, XPaths and CSS selectors to identify web page elements. Helium on the other hand lets you refer to elements by their user-visible labels. The example script uses a bunch of user-visible labels, not classes, ids, ancestors, etc. It is things like "click('Sign in')" which you would tell a user you were walking through an interface.
- bauerd 7y ago
  This works fine until it does not, for example when dealing with custom UI controls, which is when reality sets in and you find yourself wrangling HTML IDs, XPaths and CSS selectors again.
  - ethbro 7y ago
    In worst cases, I always tell people to bind to user-visible elements, then only navigate the local DOM from there. Global XPath type traversal is fragile, unless you're plugged in enough that the dev team lets you know when they break it.