4 ms·

Brittle tests seem not useful in general though aren't they? I'm not sure its necessarily true that brittleness must correlate with height in pyramid or execut

Brittle tests seem not useful in general though aren't they? I'm not sure its necessarily true that brittleness must correlate with height in pyramid or execution time -- in my experience brittleness correlates with selenium more than it does pyramid height (that's a statement about selenium more than it is a statement about any particular category of testing pyramid). Its possible to write very useful non-brittle tests using something like headless chrome ...

Vinnl 9y ago
No they're not. But yes, Selenium is brittle. That said, Google engineers actually did some investigation into this, and although I think their methods were probably a bit heavyweight, they did conclude that it's mostly RAM use that leads to brittleness. [1] https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html https://testing.googleblog.com/2017/04/where-do-our-flaky-te...
- breatheoften 9y ago
  Interesting thanks for the link! I’m curious how many tests were in the small size range for that chart which provides evidence to show the size-flakiness correlation holds in tests that use tools associated with higher than average flakiness... I’m also feeling like I want to have more clarity around the mechanism for measuring flakiness — the definition they use is that a test is flakey if it shows both failing and success runs with the “same code” — does “same code” refer to a freeze of only the codebase under test or also a statement about change to the tools in the testing environment ...? I wonder what the test suites for tools like selenium/WebDriver look like ... do they track a concept of “meta-flakiness” to try and observe changes to test flakiness results caused by changes to the test tooling ...?
  - Vinnl 9y ago
    Yeah, good questions, the post leaves some to be desired. And meta-flakiness tooling actually sounds like it could be really useful!

Vinnl 9y ago

No they're not. But yes, Selenium is brittle. That said, Google engineers actually did some investigation into this, and although I think their methods were probably a bit heavyweight, they did conclude that it's mostly RAM use that leads to brittleness. [1] https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html https://testing.googleblog.com/2017/04/where-do-our-flaky-te...

breatheoften 9y ago
Interesting thanks for the link! I’m curious how many tests were in the small size range for that chart which provides evidence to show the size-flakiness correlation holds in tests that use tools associated with higher than average flakiness... I’m also feeling like I want to have more clarity around the mechanism for measuring flakiness — the definition they use is that a test is flakey if it shows both failing and success runs with the “same code” — does “same code” refer to a freeze of only the codebase under test or also a statement about change to the tools in the testing environment ...? I wonder what the test suites for tools like selenium/WebDriver look like ... do they track a concept of “meta-flakiness” to try and observe changes to test flakiness results caused by changes to the test tooling ...?
- Vinnl 9y ago
  Yeah, good questions, the post leaves some to be desired. And meta-flakiness tooling actually sounds like it could be really useful!