4 ms·
It is heartening to know that the great and mighty Google spends a real amount of time and effort dealing with flaky Selenium tests, just like the rest of us!
by kyoob 9y ago
It is heartening to know that the great and mighty Google spends a real amount of time and effort dealing with flaky Selenium tests, just like the rest of us!
- paulddraper 9y agoAnd it prompts the question: Is this really a challenge so difficult that even the Gods struggle?
- billsmithaustin 9y agoYes, it really is. Automated UI tests are notoriously fragile.
- paulddraper 9y agoI'm curious: Why? It's bits and bytes. What stops it from being reliably deterministic like any other kind of computation?
- odonnellryan 9y agoNot sure if this paper helps answer your question: http://alandix.com/academic/papers/nond90/ http://alandix.com/academic/papers/nond90/
- nunez 9y agoSome examples I've seen: Changes in browser code or DOM handling can break the test. Memory issues from having too many instances of Chrome can break the test. There are a lot of factors at play that are slightly difficult to control for
- Buge 9y agoIf multiple threads are running, the order that the threads are scheduled is nondeterministic. And if there are any timeouts anywhere in the code, the result is nondeterministic due to the nondeterministic method that the kernel will schedule the process and environmental effects of whatever other programs are running at the same time or recently ran.
- nitwit005 9y agoYou only need to make one mistaken assumption to introduce non-determinism. Say you wait for the page to load, then click a button, is that valid? Sometimes it is, but some UI frameworks might render the button after the page reports being loaded. Your test will generally pass regardless, but mysteriously fail periodically. You also tend to never achieve full reliability because the environments aren't entirely stable. Almost everyone sees more "random" failures on old IE versions. Poor over-stressed IE9 running in a VM will have a bad day and your test will fail.
- flukus 9y agoI'm surprised that google can't manage it, they have proper test engineers. But automated testing is a discipline that very few learn to do well. Even most unit test you see out in the wild are absolutely woeful. I've got a long list of horror stories about testing if you'd like? I used to believe that devs should write the tests, but now I'm not so sure, all to often they just produce more technical debt and fail at improving quality. The industry considers knowing the api of a test framework as an experience automated tester.