4 ms·

> My controversial opinion is that most software tests should be integration tests that use randomized inputs. I doubt I'll get very many upvotes for that idea.

> My controversial opinion is that most software tests should be integration tests that use randomized inputs. I doubt I'll get very many upvotes for that idea. Probably not, but largely because you used the word "randomized". One of the most annoying things to deal with in a CI workflow is flaky tests (where something unchanged will randomly fail, or re-running will result in a different outcome). The most common way I've seen this happen is with an external dependency like using a database server or calling a 3rd party API. The failure is usually temporary (eg: timeout) and so it's a false negative. This trains developers to do a lazy workaround: re-run the build until it passes. True "randomized" data has the possibility that you get all kinds of flaky tests, but that they cause false positives: the failures are legitimate, passing is not. Pseudo-randomized [1] data is a different story (see: fuzzing [2]), because you still have predictable tests -- the difference is a human doesn't have to think up thousands of test cases. I'd say it's also important to add specific test cases when you do find a specific type of failure, just to be sure that always gets tested. I think if you call it "pseudo-random" or "fuzz testing" you'll find it's much less controversial. [1] https://en.wikipedia.org/wiki/Pseudorandomness https://en.wikipedia.org/wiki/Pseudorandomness [2] https://en.wikipedia.org/wiki/Fuzzing https://en.wikipedia.org/wiki/Fuzzing

loup-vaillant 6y ago
> One of the most annoying things to deal with in a CI workflow is flaky tests (where something unchanged will randomly fail, or re-running will result in a different outcome). Just records the input whenever the test fails, and have an option to re-run a test, with those inputs. The tests are still flaky but it doesn't matter if we can reproduce the results. You don't even have to record all inputs. You could just use a random seed, and record the seed at each run. Test become deterministic again simply by fixing the seed (typically to a value that caused the previous run to fail).
- gregmac 6y ago
  Using a seed with the "random" is exactly how you get to pseudo-random, which gives the same sequence everytime, and is absolutely fine for tests. If you want to have the build (test) result feedback to save the input, that's an ok goal but I think it's hard to pull off: now you have your build making commits to source control (causing another build), or you have test data stored somewhere else not in source (when really, it should be part of source).
whack 6y ago
I'm not intimately familiar with "fuzzing" techniques, and by "randomized inputs", I am indeed referring to tests that could be flaky in the presence of bugs. Organizational discipline is indeed needed here, in order to not ignore the occasional flaky failure, and to investigate the cause of the flaky error. I believe the benefits outweigh the drawbacks. Here's a link that talks about the above in more detail - https://software.rajivprab.com/2019/04/28/rethinking-software-testing-perspectives-from-the-world-of-hardware/ https://software.rajivprab.com/2019/04/28/rethinking-softwar...
- josephg 6y ago
  I use fuzzers extensively in software I write. I find it quite enjoyably humbling how quickly a newly minted fuzzer finds bugs. I use a seeded random number generator. During development I seed it randomly. When crashes happen I output both the seed and iteration so I can hardcode that seed and reproduce the problem. And then in the CI I’ll often fix the seed so the test suite is stable. But it’s also good practice to run fuzzers outside the context of a CI. You want your CI tests to be fast and stable. Fuzzer tests can also be slow and thorough. Foundationdb has dedicated hardware that runs their fuzzer 24/7 and spits out any errors it finds. If that fuzzer finds problems it doesn’t break the build. But someone will take a look, reproduce the problem, hopefully convert the bug into a unit test and fix the issue.
karmakaze 6y ago
This is completely fine, have a separate test set for these not in the deploy pipeline which is about preventing regressions not finding bugs that were previously missed.
- thiht 6y ago
  Agree, this is the way to go. On a separate note, if the fuzzy tests find something wrong, the data should definitely be integrated in the stable test suite to ensure non regression.
matfil 6y ago
One of the most annoying things to deal with in a CI workflow is flaky tests I guess another controversial opinion: this is a problem with the idea of CI (or at least how we work with it and what we expect to get out of it), rather than the idea of randomized tests.
- gregmac 6y ago
  In what way? My personal expectation of CI is predictable, repeatable builds, that give me some assurance the software is working as designed. I also like that it forces everything to be scripted: no "only Bob knows how to build the release file." Flaky tests are an indicator of poor code: maybe it's your actual code, maybe it's a bug in the test code, or maybe it's external dependency + lack of error handling in the test code but there's a problem somewhere.
  - matfil 6y ago
    The presence of bugs doesn't necessarily indicate useless software. If tests are failing (or flaky...), that's probably something to look at at some point, but that doesn't necessarily mean it's the highest priority to look at. In most places where CI gets deployed -- at least in commercial environments -- there seems to be a goal of making test failures a non-maskable interrupt. I did admit this was controversial! But it fits in with a more general view that there are a lot of tools which make good servants but poor masters. Auto-builders are a good thing, partly because (as you say) they can help to clarify what is required to make a build, and partly because (especially for dependency-heavy software, which seems to be the norm nowadays) they can help catch things quickly when the dependencies shift beneath you. Making them a hard gate on releases seems a little too close to making the tooling your master, though. (Somewhat separately, I also worry about CI acts as a hiding place for complexity. Sometimes it reaches that point where nobody knows how to make a build without the CI tool any more. Then local testing and debugging becomes difficult.)
    - gregmac 6y ago
      You have a good point. I think the danger comes from the potential for abuse. Once you ship software that has failing tests, you've established a precident and are likely to face pressure to do so again even when the tests are more critical. That's why I'd resist that, at least. My experience is if there's a real problem and later everything in production is on fire, no one cares about the caveats or risks you pointed out. I do also agree that CI can hide complexity, but that's true of any tool as well as non-CI builds. Compared to a human running things, it's at least somewhat self-documenting due to being scripted.

loup-vaillant 6y ago

> One of the most annoying things to deal with in a CI workflow is flaky tests (where something unchanged will randomly fail, or re-running will result in a different outcome). Just records the input whenever the test fails, and have an option to re-run a test, with those inputs. The tests are still flaky but it doesn't matter if we can reproduce the results. You don't even have to record all inputs. You could just use a random seed, and record the seed at each run. Test become deterministic again simply by fixing the seed (typically to a value that caused the previous run to fail).

gregmac 6y ago
Using a seed with the "random" is exactly how you get to pseudo-random, which gives the same sequence everytime, and is absolutely fine for tests. If you want to have the build (test) result feedback to save the input, that's an ok goal but I think it's hard to pull off: now you have your build making commits to source control (causing another build), or you have test data stored somewhere else not in source (when really, it should be part of source).

whack 6y ago

I'm not intimately familiar with "fuzzing" techniques, and by "randomized inputs", I am indeed referring to tests that could be flaky in the presence of bugs. Organizational discipline is indeed needed here, in order to not ignore the occasional flaky failure, and to investigate the cause of the flaky error. I believe the benefits outweigh the drawbacks. Here's a link that talks about the above in more detail - https://software.rajivprab.com/2019/04/28/rethinking-software-testing-perspectives-from-the-world-of-hardware/ https://software.rajivprab.com/2019/04/28/rethinking-softwar...

josephg 6y ago
I use fuzzers extensively in software I write. I find it quite enjoyably humbling how quickly a newly minted fuzzer finds bugs. I use a seeded random number generator. During development I seed it randomly. When crashes happen I output both the seed and iteration so I can hardcode that seed and reproduce the problem. And then in the CI I’ll often fix the seed so the test suite is stable. But it’s also good practice to run fuzzers outside the context of a CI. You want your CI tests to be fast and stable. Fuzzer tests can also be slow and thorough. Foundationdb has dedicated hardware that runs their fuzzer 24/7 and spits out any errors it finds. If that fuzzer finds problems it doesn’t break the build. But someone will take a look, reproduce the problem, hopefully convert the bug into a unit test and fix the issue.

karmakaze 6y ago

This is completely fine, have a separate test set for these not in the deploy pipeline which is about preventing regressions not finding bugs that were previously missed.

thiht 6y ago
Agree, this is the way to go. On a separate note, if the fuzzy tests find something wrong, the data should definitely be integrated in the stable test suite to ensure non regression.

matfil 6y ago

One of the most annoying things to deal with in a CI workflow is flaky tests I guess another controversial opinion: this is a problem with the idea of CI (or at least how we work with it and what we expect to get out of it), rather than the idea of randomized tests.

gregmac 6y ago
In what way? My personal expectation of CI is predictable, repeatable builds, that give me some assurance the software is working as designed. I also like that it forces everything to be scripted: no "only Bob knows how to build the release file." Flaky tests are an indicator of poor code: maybe it's your actual code, maybe it's a bug in the test code, or maybe it's external dependency + lack of error handling in the test code but there's a problem somewhere.
- matfil 6y ago
  The presence of bugs doesn't necessarily indicate useless software. If tests are failing (or flaky...), that's probably something to look at at some point, but that doesn't necessarily mean it's the highest priority to look at. In most places where CI gets deployed -- at least in commercial environments -- there seems to be a goal of making test failures a non-maskable interrupt. I did admit this was controversial! But it fits in with a more general view that there are a lot of tools which make good servants but poor masters. Auto-builders are a good thing, partly because (as you say) they can help to clarify what is required to make a build, and partly because (especially for dependency-heavy software, which seems to be the norm nowadays) they can help catch things quickly when the dependencies shift beneath you. Making them a hard gate on releases seems a little too close to making the tooling your master, though. (Somewhat separately, I also worry about CI acts as a hiding place for complexity. Sometimes it reaches that point where nobody knows how to make a build without the CI tool any more. Then local testing and debugging becomes difficult.)
  - gregmac 6y ago
    You have a good point. I think the danger comes from the potential for abuse. Once you ship software that has failing tests, you've established a precident and are likely to face pressure to do so again even when the tests are more critical. That's why I'd resist that, at least. My experience is if there's a real problem and later everything in production is on fire, no one cares about the caveats or risks you pointed out. I do also agree that CI can hide complexity, but that's true of any tool as well as non-CI builds. Compared to a human running things, it's at least somewhat self-documenting due to being scripted.