4 ms·
Yeah...I mostly have experience only running test suites for popular libraries and for my own software...but the existence of this library is a massive WTF...su
by danso 11y ago
Yeah...I mostly have experience only running test suites for popular libraries and for my own software...but the existence of this library is a massive WTF...such that had I read its official introductory post, I would have thought it to be really good satire:
https://www.box.com/blog/introducing-flaky-a-nose-test-plugin-for-automatically-rerunning-flaky-tests/ https://www.box.com/blog/introducing-flaky-a-nose-test-plugi...
Can someone describe a real life production scenario in which this flaky behavior is desirable? That is, preferable to these other generally accepted practices:
- flag the test and mark the bug as an issue and at some point, attempt to fix it
- delete the test, if it happens to relate to dead code or was poorly conceived in the first place
- use fixtures and other libraries to mock dependencies, e.g. Webmock and/or vcr to intercept http request and respond with a pre-recorded fixture.
I understand that there are scenarios in which production most go on even when a test fails. But to throw on another testing layer that tells you, "hey, it kind of works, for some unknown reason", instead of just marking the test as a failure to be investigated...what possible value or insight could outweigh the additional noise generated? I guess one possibility is that it lets you know that something is truly fucked up...but that is not at all the tone of the Box blog announcement:
> When testing Sync 4, Box's desktop sync application, we also ran into this issue, but we also didn't want to simply remove our flaky tests. When we noticed that most flaky tests would pass when rerun, we realized we could make doing so automatic. Flaky is a nose plugin that can rerun flaky tests without interrupting your test run. Using it is as easy as decorating your test methods with @flaky
- usaar333 11y ago> flag the test and mark the bug as an issue and at some point, attempt to fix it But what do you do with the test in the meantime? If you disable it you lose coverage. The best temporary state might be to re-run the test until it passes. If it fails after allowed re-runs, you have a regression.
- ghettoimp 11y ago"Can someone describe a real life production scenario in which this flaky behavior is desirable?" I think so? My company develops a processor that is meant to be compatible with processors from other vendors. To try to ensure compatibility, we have a test suite that compares our "golden model" of intended behavior against the observed behavior of competitors' chips. We sometimes have run into cases where a competitor's product "randomly" gets wrong results, but when we run the instruction again, it gets the right answer. This happens frequently enough that we've arranged the test suite to automatically try again to see if a failure is reproducible before bothering a human with it.
- danso 11y agoYeah I figured these kind of errors fall into the system interoperability/integration category...but I also figured they would fall into their own kind of test suite, one that is much more geared toward the measurements of thresholds and probabilities. Adding a "flaky" plugin to the standard test framework to do this kind of non-standard testing...feels like designing the workflow in a slightly backwards way, like monkeypatching a basic data object for very specific behavior needed in a few niche libraries.
- rifung 11y agoAt AWS we had some tests which involved adding stuff to DynamoDB. Since it's eventually consistent, sometimes the tests would fail because some assertions wouldn't be true until some time later. Of course we tried to make them succeed every time with retries, but we didn't want the tests to take forever either. So, it was expected that the tests would fail once in a while. Maybe there could be a better way to handle this, but what would happen is if that test failed we would just check the data manually, waiting for it to come up.
- charleslmunger 11y agoTh nuance of flakiness (if you work at a very large company with one continuous build system and one big repository for everyone) is that in order to not break dependent teams with your checkins, you run all their tests before checkin. Sometimes this catches major, subtle issues - out of a run with a million+ test targets, one test fails and it reveals a critical security bug in a widely used open source library. But most of the time, large numbers of poorly written (or neglected) tests fail, and it's not because of your change. The important thing is to show a distinction in the UI between a flaky test and a passing test, so that people don't look at a wall of green and see success, and push broken code to production.