3 ms·
And it prompts the question: Is this really a challenge so difficult that even the Gods struggle?
by paulddraper 10y ago
And it prompts the question: Is this really a challenge so difficult that even the Gods struggle?
- billsmithaustin 10y agoYes, it really is. Automated UI tests are notoriously fragile.
- paulddraper 10y agoI'm curious: Why? It's bits and bytes. What stops it from being reliably deterministic like any other kind of computation?
- odonnellryan 10y agoNot sure if this paper helps answer your question: http://alandix.com/academic/papers/nond90/ http://alandix.com/academic/papers/nond90/
- nunez 10y agoSome examples I've seen: Changes in browser code or DOM handling can break the test. Memory issues from having too many instances of Chrome can break the test. There are a lot of factors at play that are slightly difficult to control for
- Buge 10y agoIf multiple threads are running, the order that the threads are scheduled is nondeterministic. And if there are any timeouts anywhere in the code, the result is nondeterministic due to the nondeterministic method that the kernel will schedule the process and environmental effects of whatever other programs are running at the same time or recently ran.
- nitwit005 10y agoYou only need to make one mistaken assumption to introduce non-determinism. Say you wait for the page to load, then click a button, is that valid? Sometimes it is, but some UI frameworks might render the button after the page reports being loaded. Your test will generally pass regardless, but mysteriously fail periodically. You also tend to never achieve full reliability because the environments aren't entirely stable. Almost everyone sees more "random" failures on old IE versions. Poor over-stressed IE9 running in a VM will have a bad day and your test will fail.
- flukus 10y agoI'm surprised that google can't manage it, they have proper test engineers. But automated testing is a discipline that very few learn to do well. Even most unit test you see out in the wild are absolutely woeful. I've got a long list of horror stories about testing if you'd like? I used to believe that devs should write the tests, but now I'm not so sure, all to often they just produce more technical debt and fail at improving quality. The industry considers knowing the api of a test framework as an experience automated tester.