8 ms·

> Unfortunately, one can’t fully eradicate intermittently failing tests, Oh boy do I disagree with this. I have a zero tolerance policy for flaky tests. In cod

• tick_tock_tick 6y ago
  The biggest thing is anything time based the effort to mock out of fake time is normally way larger than just dealing with a test that flakes from time to time. That being said I make a concentrated effort to take in time rather then randomly calling now() in the middle of my code but most code bases aren't green fields.
  • TallGuyShort 6y ago
    It is my personal opinion that in that first statement you are severely underestimating the time and product quality that is eventually lost because of a flaky test that isn't dealt with quickly.
• quicklime 6y ago
  I generally agree, but there's one thing that bothers me about the practice: Before disabling the test, does anyone even consider the possibility that maybe it's not the test that's flaky, but that the software under test is itself buggy in a non-deterministic way? I don't have a good answer for how to deal with this. It would be an enormous amount of effort to go through every flaky result and reason about whether the problem is with the test or the code under test, especially in larger codebases. Maybe the heuristic of assuming that it's the test's fault is good enough, and this is what I've done in every team I've worked in. But it does bother me.
  • wyldfire 6y ago
    > It would be an enormous amount of effort to go through every flaky result and reason about whether the problem is with the test or the code under test, especially in larger codebases. IMO a "flaky test" is one for which the investigation has been performed and the design of the test assessed to be "flaky." i.e. not a test that passes some times and not others, but one whose design cannot reliably expect a given result or one whose implementation can be impacted by factors outside of the control of the test suite. Timeouts are a classic source of flaky tests. Once a test is determined to be flaky it should be evicted from the test suite, or at least demoted from the "reliable/frequently run" rank to a lower one that can tolerate some hand holding.
    • nogabebop23 6y ago
      I'm not sure if it's exhaustive, but any test with a dynamic dependency is one I would likely classify as "flaky", like anything hitting another system. Draw your own boundaries as fit around what "dynamic" means (ex: some might say static files are OK, others not). Also, I'm talking every-commit, unit-level testing for context.
  • keegancsmith 6y ago
    This is tricky but when you reframe it it's clear the heuristic you mention is the correct one. The value of a test is giving signal to a developer that the code they wrote is correct. If a test is flaky due to the test or the code, the signal the test provides has now become negative. The parent mentions filing it on the team that owns the test. This sounds correct and the onus is then on that team. At larger companies this process is automated. You may enjoy the flake detection section in this blog post https://dropbox.tech/infrastructure/athena-our-automated-build-health-management-system https://dropbox.tech/infrastructure/athena-our-automated-bui...
  • hinkley 6y ago
    This is why your CI system should record statistics on which tests failed and when. A test with a pattern of failing intermittently over its entire lifetime is most likely a flaky test. Disabling the test modifies a bunch of metrics, like skipped tests and code coverage (woe be unto you if some idiot set it up so you can't commit code that lowers test coverage, instead of just warning you). There are much bigger and more likely sources of regression in the code than a new concurrency bug in code covered by a test that already has a concurrency bug in it. Like someone modifying a test to match the regression they just created. If there is a concurrency or randomness bug in the untested code, that will probably surface when trying to write a robust test to replace the old one. If not by the author, then by the person the author asks for help when they can't figure it out.
• hinkley 6y ago
  In talking to people about this problem, a common misconception I hear is that "well it only fails 1% of the time (so why should I care?)" Developers don't think in percentages. Our experiences are based on how frequently something happens, not the percent of the times it happens. In a CI build, where we are agressively building many times a day, your one bad test is going to fail every week or two. Now add two dozen more tests that do the same, and now builds are failing 2x a day on average and more frequently during crunch times and every few months the build will fail 4, 5 times in a row. And for some reason this happens right when someone actually needs a build now because they broke something and everyone is waiting for them to fix it. Talk about stressful. Every day is "all the time" to most people, and 5 builds in a row might be "hours" (because you have to keep babysitting it). These are serious broken windows and that is how you should treat them.
  • l0b0 6y ago
    The worst thing is you end up normalizing failure - nobody even bothers checking whether it's a "real" failure until it's failed a few times in a row. At that point you might have introduced other test failures which nobody noticed, or someone might have built on top of the broken code, making a fix harder, and so on and so on.
    • hinkley 6y ago
      Indeed. This pattern though is often the point at which an intervention is possible. Almost like it has to happen repeatedly before you can get consensus to stop it.
• nogabebop23 6y ago
  We just implemented a rule where tests taking > 100ms to execute are considered a failure. You can experiment with the parameters or limits but this sort of rule has paid huge dividends. The majority of the tests it has disqualified sucked and didn't really "test" anything. They also tended to be the most brittle. You don't write tests that hit APIs and databases when you have strict time limits and this eliminates a lot of "flaky" tests.
  • patrec 6y ago
    If you can't hit your database in CI within 100ms or talking to it makes your tests flaky, you got problems.
    • nogabebop23 6y ago
      Tests that hit dynamic data stores like DBs don't fail very often due to network speed constraints compared to configuration, system updates, security, schema changes, and a myriad of other time-consuming hard to diagnose problems. It's easier to just prevent these situations. In the abstract I'm not sure what a test that talks to the database is actually testing.
      • patrec 6y ago
        How do you effectively develop if you can't reliably talk to the DB in dev due to configuration, system updates, security, schema changes, and a myriad of other time-consuming hard to diagnose problems? > In the abstract I'm not sure what a test that talks to the database is actually testing. Well, presumably the reason that you have a DB in the first place is that your application requires a DB for some of its functionality. So how do you make sure this functionality works if you don't talk to the DB? Don't get me wrong -- I love that you refuse to accept ridiculously slow tests and enforce a 100ms deadline; I wish everything I was working on had the same approach. But unless you have some specific requirements (like talking to a proprietary cloud DB), you should maybe investigate if you might not have to accept slow and flaky DBs in dev or CI either. My experience is you don't.
    • 6y ago
• gen220 6y ago
  I totally agree with you. We have a monorepo with maybe a hundred thousand tests, and we have a nightly build that runs against all of them, and publishes the results in an email to all engineering. 3/5 days we come out clean. 2/5 days we discover and fix a flaky test or broken build target. If there’s an intensity of flakiness that can withstand the gaze of a couple hundred engineers for more than an hour or so, we haven’t found one yet. Edit: looks like our numbers are quite close to Shopify’s. We don’t need anything close to hundreds of docker containers running in parallel. But we’re mostly a python shop, maybe they’re being a ruby shop slows things down here.
  • dudzik 6y ago
    Ruby is not really slowing us down, the bigger issue is that we have a lot of integration tests and few isolated unit tests. So we are massively parallelizing our builds so we can get p50 build times of 23 minutes when running the whole test suite. The time mentioned in the article is a little bit closer to the p95 time. > If there’s an intensity of flakiness that can withstand the gaze of a couple hundred engineers for more than an hour or so, we haven’t found one yet. That is a good approach and we also used to disable tests that would fail on full master builds. When it comes to fixing intermittently failing tests we are now storing every test run so we can analyze our test suite. This way we are able to find the slowest and flakiest tests in the codebase, disable them, and notify the owners of the code to investigate.
    • gen220 6y ago
      That’s interesting and definitely helps make the numbers make more sense! :) In your experience, do you find high value in having your tests be 80/20 (rough numbers) integration/unit vs the other way around? I guess it depends on the programming style, but here it’s the other way around. Some minority of engineers here clamor for end to end tests every once in a while, but we’ve rarely found the difficulty of establishing the test harness to be worth the marginal gains in reliability. We’re also wary of the temptation to write e2e tests to test unit-test-level correctness, because of the implications on test performance, like you describe. On our side, the closest thing we have to ubiquitous e2e testing is that we spin up and spin down one ephemeral database instance per test module (could be used by many test functions).
• dudzik 6y ago
  I've worked on the project in the article so I can give some perspective on our handling of intermittently failing tests. Generally, I agree with your sentiment. Intermittently failing test are eroding trust in the codebase. In our case there are practical challenges with the size of our test base. Not everybody has context on every part of the codebase. This makes it really hard for people to decide if a test is intermittently failing and decide to remove it. The other issue is that at the size of our test base, we are introducing more intermittently failing tests than we can eradicate effectively. To combat this problem we build a test-onboarding system that tests test for common issues. Once a test enters the codebase we are disabling the tests if they are failing and then passing for the same commit too many times so that an individual developer doesn't have to make the decision.
• viraptor 6y ago
  How does your zero tolerance policy work when you can't find a reason the rest fails? I.e. you have critical functionality you test, but fail one test run in 1000. You can spend 2 days on the issue and not find a solution. Sinking more time into it doesn't really help anyone much and won't offset any time saving in a realistic future. On the other hand, if you actually remove the test, you risk messing up a critical transaction.
  • maccard 6y ago
    If you have a critical function failing testing, you need to figure out the root cause, whether it takes 2 days or 2 months. 2 days is nothing - I've spent more than that on UI polish bugs, to not be able to dedicate 5x that time to fixing critical functionality, something is desperately wrong.
    • majkinetor 6y ago
      Viraptor is right. Your "solution" is far from unversally acceptable. If you came to me as developer telling me that you did 2 months debug on test that fails randmoly once in a week and you didn't do much else, I would probably show you the dor as a team lead. We need to put things into perspective. I hate the flaky tests just as much as a next dude but seriously, you need to factor in everything - deadlines, features, CI/CD, metrics etc, while still having constant time frame. So lets not get into such polar opinions, its just not realistic at all. Life is never that ordered.
      • mpweiher 6y ago
        If you came to me with that attitude, I would fire you immediately. This is, per definition, critical functionality, and we can't figure out whether it is working or not, and you are saying you don't care.
      • maccard 6y ago
        > If you came to me as developer telling me that you did 2 months debug on test that fails randmoly once in a week and you didn't do much else, I would probably show you the dor as a team lead. We need to put things into perspective. This is a pretty polar opinion right here - it's not a flaky test that fails once a week, it's unreliable critical functionality. If I were _your_ lead and you told me you fired someone because they spent two months on something, I'd probably fire you for not knowing what your team are doing for two months.
• quest88 6y ago
  Demote it to a group of flaky tests that don't block pushes. Or, demote them and block the push when a flaky test fails 100% of the time.
  • tcbasche 6y ago
    Then what’s the point of them?
    • FridgeSeal 6y ago
      That gives me an idea: what if you had a pool of flaky/failing tests, but it was limited in size, and if you add a test to that pool that takes it over the maximum size, you must either fix it, or evict another flaky rest by fixing it. You could use the pool capacity and size as a proxy for technical debt. Starting a new, large project? You can increase the pool size temporarily until the project reaches mvp. Periodically the size of the test pool should be evaluated, tests fixed and the pool size shrunk. Over time the overall pool size should shrink or at least stabilise.
      • tcbasche 6y ago
        I imagine without any discipline though you just get a consistent set of tests that no-one has the bandwidth to go and fix or make unflaky. I wonder if flakiness of tests correlates with discipline to paying down tech debt and improving testing
    • quest88 6y ago
      They still detect real failures. You want to know when they go from failing 10% of the time to 100%. If your team wants, block pushes at 100% failure.
      • tcbasche 6y ago
        But how is it a reliable indicator of any sort of failure if it doesn’t always fail? Doesn’t it erode trust?
• deleted 6y ago
  [deleted]