4 ms·

But of course larger tests are flaky. They have more lines of code. That’s practically the null hypothesis, where you assume every line of code has a certain ch

by johnfn 8y ago

But of course larger tests are flaky. They have more lines of code. That’s practically the null hypothesis, where you assume every line of code has a certain chance of flaking, and then you just multiply out all the probabilities. I was expecting some in depth analysis, like “tests that use time libraries are flaky” or “tests that have callbacks are flaky” or something. This doesn’t really make any headway towards understanding the problem.

joshuamorton 8y ago
>where you assume every line of code has a certain chance of flaking, and then you just multiply out all the probabilities. This assumption doesn't make much sense though. `y = x + 1` shouldn't flake. Nor is your comment here a correct reflection of "size". Size in the article means time the test runs or amount of memory used (ie. constructing a larger SUT), not LoC. Edit: I'm partially mistaken, one measure of size is binary size, which should be generally correlated with LoC, though not absolutely. Webdriver tests which use image/video goldens are an example where test size may not correlate with LoC at all. >This doesn’t really make any headway towards understanding the problem. The second part of the article does address exactly this (webdriver tests are correlated with flakyness, although they're also larger by virtue of needing to spin up a headless browser, which has a memory cost). More edit: As someone who has written and heavily deflaked some (very large) tests using, I believe, the "internal testing framework" mentioned in the article, flakyness always appeared as a tradeoff with speed. Parallelizing tasks introduces a level of randomness which can lead to flakes. Short(er) timeouts on things that can have long 95 or 99th percentile latencies, etc. It can sometimes be worth it to have a test flake some small percentage of the time instead of waiting longer (granted this is also often a flaw in test design somewhere). But when you're testing a big system with many moving parts, it's not uncommon to have something that takes 30 seconds 95 or 99% of the time, and every once in a while 5 minutes. The question is if its worth it to keep extending your timeouts for n-sigma events, especially since that means that when your tests are actually broken, they'll potentially take much, much longer to complete. You can lower the false false negative rate, but the cost is that true negatives become much more expensive to compute. Is that worth it? It depends.
- andrewprock 8y ago
  It's probably worth noting that y = x + 1 will flake out if x is an int, and it's value is INT_MAX.
  - joshuamorton 8y ago
    I'm pretty sure it will still act consistently. Under the same compiled binary (which a hermetic environment enforces), the undefined behavior will be consistent from run to run, since the same optimizations will be used. That's not true in a theoretical sense (undefined behavior allows me to rewrite signed overflow to "return 7 with some probability, and `rm ~/` otherwise"), but for any modern compiler, I'd assume that the optimizations chosen where signed overflow is undefined don't involve randomness.
    - P_I_Staker 8y ago
      It may be fun to point out that the complier could technically launch a game of Tetris or rm -rf /, but is that what any mainstream compiler does? I guess the behavior could be inconsistent, but who'd design a compiler like that? Seems like it'll roll over or it won't (ie. stay stuck at INT_MAX). Good unit tests would catch this if you changed compiler, but really you should handle it explicitly (look before you leap).
    - andrewprock 8y ago
      It will act consistently if the value of x is consistent. If it's not, then it could flake out on that line.
      - joshuamorton 8y ago
        If the value of x is inconsistent, you had flakyness somewhere else. That line will always act consistently, modulo solar radiation.
    - voidmain 8y ago
      Not at all. In a code path where the compiler reasons that x is definitely INT_MAX, it could simply omit this code, leaving y uninitialized and nondeterministic.

by johnfn 8y ago

joshuamorton 8y ago

>where you assume every line of code has a certain chance of flaking, and then you just multiply out all the probabilities. This assumption doesn't make much sense though. `y = x + 1` shouldn't flake. Nor is your comment here a correct reflection of "size". Size in the article means time the test runs or amount of memory used (ie. constructing a larger SUT), not LoC. Edit: I'm partially mistaken, one measure of size is binary size, which should be generally correlated with LoC, though not absolutely. Webdriver tests which use image/video goldens are an example where test size may not correlate with LoC at all. >This doesn’t really make any headway towards understanding the problem. The second part of the article does address exactly this (webdriver tests are correlated with flakyness, although they're also larger by virtue of needing to spin up a headless browser, which has a memory cost). More edit: As someone who has written and heavily deflaked some (very large) tests using, I believe, the "internal testing framework" mentioned in the article, flakyness always appeared as a tradeoff with speed. Parallelizing tasks introduces a level of randomness which can lead to flakes. Short(er) timeouts on things that can have long 95 or 99th percentile latencies, etc. It can sometimes be worth it to have a test flake some small percentage of the time instead of waiting longer (granted this is also often a flaw in test design somewhere). But when you're testing a big system with many moving parts, it's not uncommon to have something that takes 30 seconds 95 or 99% of the time, and every once in a while 5 minutes. The question is if its worth it to keep extending your timeouts for n-sigma events, especially since that means that when your tests are actually broken, they'll potentially take much, much longer to complete. You can lower the false false negative rate, but the cost is that true negatives become much more expensive to compute. Is that worth it? It depends.

andrewprock 8y ago
It's probably worth noting that y = x + 1 will flake out if x is an int, and it's value is INT_MAX.
- joshuamorton 8y ago
  I'm pretty sure it will still act consistently. Under the same compiled binary (which a hermetic environment enforces), the undefined behavior will be consistent from run to run, since the same optimizations will be used. That's not true in a theoretical sense (undefined behavior allows me to rewrite signed overflow to "return 7 with some probability, and `rm ~/` otherwise"), but for any modern compiler, I'd assume that the optimizations chosen where signed overflow is undefined don't involve randomness.
  - P_I_Staker 8y ago
    It may be fun to point out that the complier could technically launch a game of Tetris or rm -rf /, but is that what any mainstream compiler does? I guess the behavior could be inconsistent, but who'd design a compiler like that? Seems like it'll roll over or it won't (ie. stay stuck at INT_MAX). Good unit tests would catch this if you changed compiler, but really you should handle it explicitly (look before you leap).
  - andrewprock 8y ago
    It will act consistently if the value of x is consistent. If it's not, then it could flake out on that line.
    - joshuamorton 8y ago
      If the value of x is inconsistent, you had flakyness somewhere else. That line will always act consistently, modulo solar radiation.
  - voidmain 8y ago
    Not at all. In a code path where the compiler reasons that x is definitely INT_MAX, it could simply omit this code, leaving y uninitialized and nondeterministic.