3 ms·
No they're not. But yes, Selenium is brittle. That said, Google engineers actually did some investigation into this, and although I think their methods were pr
by Vinnl 9y ago
No they're not.
But yes, Selenium is brittle. That said, Google engineers actually did some investigation into this, and although I think their methods were probably a bit heavyweight, they did conclude that it's mostly RAM use that leads to brittleness.
[1] https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html https://testing.googleblog.com/2017/04/where-do-our-flaky-te...
- breatheoften 9y agoInteresting thanks for the link! I’m curious how many tests were in the small size range for that chart which provides evidence to show the size-flakiness correlation holds in tests that use tools associated with higher than average flakiness... I’m also feeling like I want to have more clarity around the mechanism for measuring flakiness — the definition they use is that a test is flakey if it shows both failing and success runs with the “same code” — does “same code” refer to a freeze of only the codebase under test or also a statement about change to the tools in the testing environment ...? I wonder what the test suites for tools like selenium/WebDriver look like ... do they track a concept of “meta-flakiness” to try and observe changes to test flakiness results caused by changes to the test tooling ...?
- Vinnl 9y agoYeah, good questions, the post leaves some to be desired. And meta-flakiness tooling actually sounds like it could be really useful!