3 ms·

> As a side note flaky tests is such an idiotic concept. There are tests that pass and ones that don't. How good is button which works 35% of time? It's not a g

by _hilro 5y ago

> As a side note flaky tests is such an idiotic concept. There are tests that pass and ones that don't. How good is button which works 35% of time? It's not a good button, period Or more like the test runner succumbs to non deterministic flaky behaviour. If something failed 65% of the time, it would be one of the easiest thing in the world to fix. If it fails .001% of the time, that's what the industry refers to as flaky. > Forbit flakiness, there is no such thing as passed flaky test - those are just shitty tests. Have you ever written and monitored e2e tests over a year? It's industry wide. Selenium/selenium grid always works great until it doesn't. Ditto with the new kids on the block. e2e outside of a browser is 100% fine unless there's an actual bug somewhere.

mirekrusin 5y ago
Let me clarify, I think I haven't expressed myself well. What I meant to bash is the culture where you wrap e2e tests with retries and consider flaky tests green. This arrangement is a pathology. Industry did not put 1 out of 100000 (.001%) or less as threshold for calling something flaky. From my experience I've seen 20% success rate tests passing due to retries and teams are living with it as normal. > Have you ever written and monitored e2e tests over a year? It's industry wide. Yes, on high profile projects. I find myself repeating how important determinism is. Flaky tests, no mater how frequent, are indistinguishable from bugs, which implies they can't ever be considered green. Architecting testing environment in such a way that it is deterministic is fundamental. Sometimes it doesn't require big reshuffles, it just means the test has to be rephrased in deterministic terms that matter without asserting intermediate, timing based, racing middle states. As an example testing random failures by killing services doesn't have to assert intermediate client states, it has to assert that final state is eventually correct, which implies reconnects did happen and eventually state is correct, regardless of intermediate client states (ie. retry logic on the client auto healing itself on idempotent actions or erroring notifying client the service is offline, in which case user itself has to retry action - both are ok depending on how long service was offline, both can be progressed from test PoV, assertion that specific one happened is irrelevant).

by _hilro 5y ago

mirekrusin 5y ago

Let me clarify, I think I haven't expressed myself well. What I meant to bash is the culture where you wrap e2e tests with retries and consider flaky tests green. This arrangement is a pathology. Industry did not put 1 out of 100000 (.001%) or less as threshold for calling something flaky. From my experience I've seen 20% success rate tests passing due to retries and teams are living with it as normal. > Have you ever written and monitored e2e tests over a year? It's industry wide. Yes, on high profile projects. I find myself repeating how important determinism is. Flaky tests, no mater how frequent, are indistinguishable from bugs, which implies they can't ever be considered green. Architecting testing environment in such a way that it is deterministic is fundamental. Sometimes it doesn't require big reshuffles, it just means the test has to be rephrased in deterministic terms that matter without asserting intermediate, timing based, racing middle states. As an example testing random failures by killing services doesn't have to assert intermediate client states, it has to assert that final state is eventually correct, which implies reconnects did happen and eventually state is correct, regardless of intermediate client states (ie. retry logic on the client auto healing itself on idempotent actions or erroring notifying client the service is offline, in which case user itself has to retry action - both are ok depending on how long service was offline, both can be progressed from test PoV, assertion that specific one happened is irrelevant).