3 ms·

> At Google, for example, 16% of tests exhibited flakiness This really surprised me. In my experience, usually a flaky test indicates some kind of race conditi

by gwbas1c 9mo ago

> At Google, for example, 16% of tests exhibited flakiness This really surprised me. In my experience, usually a flaky test indicates some kind of race condition, and often a difficult-to-reproduce bug. In the past year, we had a flaky unit test that caused about 1-2% of builds to fail. Upon fixing it, we learned it was what caused a deadlock in a production service every 5-6 months. As a result of fixing this one "flaky" test, we eliminated our biggest cause of manual intervention in our production environments.

inetknght 9mo ago
> > At Google, for example, 16% of tests exhibited flakiness > This really surprised me. It doesn't surprise me. Close to 80% of Google code is actually quite terrible. > In my experience, usually a flaky test indicates some kind of race condition, and often a difficult-to-reproduce bug. Yup, that's my experience too. I flip shit when I start seeing flaky tests because it means a lot of actual work. Luckily there's generally a lot of tools available (sanitizers are a godsend) to fix it, and a lot of experience I rely on to look for smells when those tools come up empty. It makes me really sad when I work on a project with few (or, gasp, zero) tests. I've spent the last 5 months just adding tests to my current project at $employer and... those tests have revealed all kinds of problems in dependencies created by other teams at $employer. Ugh.
zmj 9mo ago
At scale, every test is flaky.
- gwbas1c 9mo ago
  Oh no, a well-written unit test is 100% deterministic. It's once you start doing tests in real environments with real databases and real networks that the unpredictability of the real world creeps in!
- m3047 9mo ago
  Well, strictly anecdata but there was a time when I worked on code which for $reasons had no dedicated test environment, and externalities were not our problem according to management. So we were reaching out to the big bad world for testing, but we didn't own the network between it and us. I ended up writing tests for the network. Caught a lot of problems with the network. Didn't make any friends. DISCLAIMER: I've been around IT for probably the majority of y'all's lifetimes, so I'm not saying this happens often. But just because something is fundamentally wrong, doesn't mean that all fundamentally wrong things are the same. In my experience they differ more from each other than the possible good ways of doing the same thing. Don't conflate things without a good reason.

by gwbas1c 9mo ago

inetknght 9mo ago

> > At Google, for example, 16% of tests exhibited flakiness > This really surprised me. It doesn't surprise me. Close to 80% of Google code is actually quite terrible. > In my experience, usually a flaky test indicates some kind of race condition, and often a difficult-to-reproduce bug. Yup, that's my experience too. I flip shit when I start seeing flaky tests because it means a lot of actual work. Luckily there's generally a lot of tools available (sanitizers are a godsend) to fix it, and a lot of experience I rely on to look for smells when those tools come up empty. It makes me really sad when I work on a project with few (or, gasp, zero) tests. I've spent the last 5 months just adding tests to my current project at $employer and... those tests have revealed all kinds of problems in dependencies created by other teams at $employer. Ugh.

zmj 9mo ago

At scale, every test is flaky.

gwbas1c 9mo ago
Oh no, a well-written unit test is 100% deterministic. It's once you start doing tests in real environments with real databases and real networks that the unpredictability of the real world creeps in!
m3047 9mo ago
Well, strictly anecdata but there was a time when I worked on code which for $reasons had no dedicated test environment, and externalities were not our problem according to management. So we were reaching out to the big bad world for testing, but we didn't own the network between it and us. I ended up writing tests for the network. Caught a lot of problems with the network. Didn't make any friends. DISCLAIMER: I've been around IT for probably the majority of y'all's lifetimes, so I'm not saying this happens often. But just because something is fundamentally wrong, doesn't mean that all fundamentally wrong things are the same. In my experience they differ more from each other than the possible good ways of doing the same thing. Don't conflate things without a good reason.