5 ms·

This post seems so good and so self-evidently true that I'm surprised at the amount of pushback it's getting here. Not sure what else to say about it. Well, I'

This post seems so good and so self-evidently true that I'm surprised at the amount of pushback it's getting here. Not sure what else to say about it. Well, I'll say this- the "@flaky" thing is pretty mind-blowing. In my own company I have noticed many engineers have a disturbing level of comfort with deciding something is a "mystery". There are no mysteries in what we do. The test fails because something is fucked up. Flappy tests are annoying, but the right thing to do is to address the situation.

lmm 11y ago
I think it corresponds to a lot of normal practice - even if you don't use @flaky, if one build fails and you run it again and it passes, what do you do? It's very possible to build tests that will pass sometimes if the code is correct and fail always if the code is wrong (e.g. test where there are race conditions in mock initialization, test that asserts that rows come out of a database in the same order they were put in). IME that comes up much more often than a test that will pass always if the code is correct and fail sometimes if the code is wrong. It's not pretty and it usually indicates underlying poor design but it may be the pragmatic option.
- draw_down 11y ago
  > even if you don't use @flaky, if one build fails and you run it again and it passes, what do you do? One's answer to this question is a look into one's soul, at least in terms of engineering. The question is exactly what do you do. In my opinion, good engineers find out why the first one failed and why the second one passed. Either could be a false result. Sometimes the tests failed because the build box ran out of hard drive space, that's a false result. Sometimes a test passes because its condition for passing is incorrect, that's a false result too. Tests passing is not necessarily a good thing in itself. That is why we have the notion of "testing the test"- a green icon means nothing if it's a lie.
- mod 11y ago
  > if one build fails and you run it again and it passes, what do you do? I run it a third time, to start.
  - lmm 11y ago
    What do you do differently based on the outcome of that? The only scenario I can think of where that makes sense as a first step is if you're happy to ignore a 1-out-of-3 failure but feel the need to investigate a 2-out-of-3, which seems like an odd position to take.
    - mod 11y ago
      I just think it's the best diagnostic step to take first. If it fails again, perhaps you've got a timing issue, or something that's switching back and forth. Of course you're not happy to ignore the failure.
danso 11y ago
Yeah...I mostly have experience only running test suites for popular libraries and for my own software...but the existence of this library is a massive WTF...such that had I read its official introductory post, I would have thought it to be really good satire: https://www.box.com/blog/introducing-flaky-a-nose-test-plugin-for-automatically-rerunning-flaky-tests/ https://www.box.com/blog/introducing-flaky-a-nose-test-plugi... Can someone describe a real life production scenario in which this flaky behavior is desirable? That is, preferable to these other generally accepted practices: - flag the test and mark the bug as an issue and at some point, attempt to fix it - delete the test, if it happens to relate to dead code or was poorly conceived in the first place - use fixtures and other libraries to mock dependencies, e.g. Webmock and/or vcr to intercept http request and respond with a pre-recorded fixture. I understand that there are scenarios in which production most go on even when a test fails. But to throw on another testing layer that tells you, "hey, it kind of works, for some unknown reason", instead of just marking the test as a failure to be investigated...what possible value or insight could outweigh the additional noise generated? I guess one possibility is that it lets you know that something is truly fucked up...but that is not at all the tone of the Box blog announcement: > When testing Sync 4, Box's desktop sync application, we also ran into this issue, but we also didn't want to simply remove our flaky tests. When we noticed that most flaky tests would pass when rerun, we realized we could make doing so automatic. Flaky is a nose plugin that can rerun flaky tests without interrupting your test run. Using it is as easy as decorating your test methods with @flaky
- usaar333 11y ago
  > flag the test and mark the bug as an issue and at some point, attempt to fix it But what do you do with the test in the meantime? If you disable it you lose coverage. The best temporary state might be to re-run the test until it passes. If it fails after allowed re-runs, you have a regression.
- ghettoimp 11y ago
  "Can someone describe a real life production scenario in which this flaky behavior is desirable?" I think so? My company develops a processor that is meant to be compatible with processors from other vendors. To try to ensure compatibility, we have a test suite that compares our "golden model" of intended behavior against the observed behavior of competitors' chips. We sometimes have run into cases where a competitor's product "randomly" gets wrong results, but when we run the instruction again, it gets the right answer. This happens frequently enough that we've arranged the test suite to automatically try again to see if a failure is reproducible before bothering a human with it.
pjungwir 11y ago
It depends. I recall one project with flaky tests, and shortly after joining I decided to investigate. It turned out to be a problem with floating-point values in the shopping cart logic. Tests were using random item prices, so sometimes it appeared and sometimes not. I'm glad I found that one. :-) On the other hand I've seen something like @flaky used in the Ruby world to run Capybara tests with a lot of Ajax, where you get failures because of timing issues deep down inside Selenium. It's not a problem with production or your app, but your testing tools. In the second case I still am not totally comfortable, since it makes it easier to overlook flaky tests that really do need attention, but I can understand it there.
- draw_down 11y ago
  Yeah, I'm with you. Of course things can never be perfect but we should be striving to make them as perfect as we can. I've felt Selenium pain too. But, and this is crucial-- notice how you understand why the tests were flappy. As opposed to just grumbling like "oh that dumb old thing again. I wish it would shut up, the thing totally works."
rincebrain 11y ago
I agree, but with a caveat - there's an important time/value tradeoff to going down any particular rabbit hole where you have more tasks than time to expend on them, particularly if you're in a billable or result-oriented area, and it lacks obvious benefit. There are plenty of times I've seen people think of things as "well, it just does X" because of a lack of depth of knowledge, but you also can't necessarily go down every rabbit hole to the bottom, unfortunately.
- draw_down 11y ago
  I disagree because things like flappy tests are a "smell" that could lead to discovering a deeper problem. We simply don't know ahead of time that the payoff won't be worth the effort in such a situation. If you can't explain what's wrong, that's bad.
  - rincebrain 11y ago
    Certainly, but given that the value of the test can be nonzero even if unreliable, and you may not have time to run down why it's unreliable, it makes sense to have a mechanism to utilize it. Not ever having engineering time to investigate why it's unreliable is a (mostly) unrelated problem.

lmm 11y ago

I think it corresponds to a lot of normal practice - even if you don't use @flaky, if one build fails and you run it again and it passes, what do you do? It's very possible to build tests that will pass sometimes if the code is correct and fail always if the code is wrong (e.g. test where there are race conditions in mock initialization, test that asserts that rows come out of a database in the same order they were put in). IME that comes up much more often than a test that will pass always if the code is correct and fail sometimes if the code is wrong. It's not pretty and it usually indicates underlying poor design but it may be the pragmatic option.

draw_down 11y ago
> even if you don't use @flaky, if one build fails and you run it again and it passes, what do you do? One's answer to this question is a look into one's soul, at least in terms of engineering. The question is exactly what do you do. In my opinion, good engineers find out why the first one failed and why the second one passed. Either could be a false result. Sometimes the tests failed because the build box ran out of hard drive space, that's a false result. Sometimes a test passes because its condition for passing is incorrect, that's a false result too. Tests passing is not necessarily a good thing in itself. That is why we have the notion of "testing the test"- a green icon means nothing if it's a lie.
mod 11y ago
> if one build fails and you run it again and it passes, what do you do? I run it a third time, to start.
- lmm 11y ago
  What do you do differently based on the outcome of that? The only scenario I can think of where that makes sense as a first step is if you're happy to ignore a 1-out-of-3 failure but feel the need to investigate a 2-out-of-3, which seems like an odd position to take.
  - mod 11y ago
    I just think it's the best diagnostic step to take first. If it fails again, perhaps you've got a timing issue, or something that's switching back and forth. Of course you're not happy to ignore the failure.

danso 11y ago

Yeah...I mostly have experience only running test suites for popular libraries and for my own software...but the existence of this library is a massive WTF...such that had I read its official introductory post, I would have thought it to be really good satire: https://www.box.com/blog/introducing-flaky-a-nose-test-plugin-for-automatically-rerunning-flaky-tests/ https://www.box.com/blog/introducing-flaky-a-nose-test-plugi... Can someone describe a real life production scenario in which this flaky behavior is desirable? That is, preferable to these other generally accepted practices: - flag the test and mark the bug as an issue and at some point, attempt to fix it - delete the test, if it happens to relate to dead code or was poorly conceived in the first place - use fixtures and other libraries to mock dependencies, e.g. Webmock and/or vcr to intercept http request and respond with a pre-recorded fixture. I understand that there are scenarios in which production most go on even when a test fails. But to throw on another testing layer that tells you, "hey, it kind of works, for some unknown reason", instead of just marking the test as a failure to be investigated...what possible value or insight could outweigh the additional noise generated? I guess one possibility is that it lets you know that something is truly fucked up...but that is not at all the tone of the Box blog announcement: > When testing Sync 4, Box's desktop sync application, we also ran into this issue, but we also didn't want to simply remove our flaky tests. When we noticed that most flaky tests would pass when rerun, we realized we could make doing so automatic. Flaky is a nose plugin that can rerun flaky tests without interrupting your test run. Using it is as easy as decorating your test methods with @flaky

usaar333 11y ago
> flag the test and mark the bug as an issue and at some point, attempt to fix it But what do you do with the test in the meantime? If you disable it you lose coverage. The best temporary state might be to re-run the test until it passes. If it fails after allowed re-runs, you have a regression.
ghettoimp 11y ago
"Can someone describe a real life production scenario in which this flaky behavior is desirable?" I think so? My company develops a processor that is meant to be compatible with processors from other vendors. To try to ensure compatibility, we have a test suite that compares our "golden model" of intended behavior against the observed behavior of competitors' chips. We sometimes have run into cases where a competitor's product "randomly" gets wrong results, but when we run the instruction again, it gets the right answer. This happens frequently enough that we've arranged the test suite to automatically try again to see if a failure is reproducible before bothering a human with it.

pjungwir 11y ago

It depends. I recall one project with flaky tests, and shortly after joining I decided to investigate. It turned out to be a problem with floating-point values in the shopping cart logic. Tests were using random item prices, so sometimes it appeared and sometimes not. I'm glad I found that one. :-) On the other hand I've seen something like @flaky used in the Ruby world to run Capybara tests with a lot of Ajax, where you get failures because of timing issues deep down inside Selenium. It's not a problem with production or your app, but your testing tools. In the second case I still am not totally comfortable, since it makes it easier to overlook flaky tests that really do need attention, but I can understand it there.

draw_down 11y ago
Yeah, I'm with you. Of course things can never be perfect but we should be striving to make them as perfect as we can. I've felt Selenium pain too. But, and this is crucial-- notice how you understand why the tests were flappy. As opposed to just grumbling like "oh that dumb old thing again. I wish it would shut up, the thing totally works."

rincebrain 11y ago

I agree, but with a caveat - there's an important time/value tradeoff to going down any particular rabbit hole where you have more tasks than time to expend on them, particularly if you're in a billable or result-oriented area, and it lacks obvious benefit. There are plenty of times I've seen people think of things as "well, it just does X" because of a lack of depth of knowledge, but you also can't necessarily go down every rabbit hole to the bottom, unfortunately.

draw_down 11y ago
I disagree because things like flappy tests are a "smell" that could lead to discovering a deeper problem. We simply don't know ahead of time that the payoff won't be worth the effort in such a situation. If you can't explain what's wrong, that's bad.
- rincebrain 11y ago
  Certainly, but given that the value of the test can be nonzero even if unreliable, and you may not have time to run down why it's unreliable, it makes sense to have a mechanism to utilize it. Not ever having engineering time to investigate why it's unreliable is a (mostly) unrelated problem.