4 ms·

While agreeing with the results of this article, I’ve found that convincing other developers of writing test with properties isn’t very easy: coming up with goo

While agreeing with the results of this article, I’ve found that convincing other developers of writing test with properties isn’t very easy: coming up with good properties is not always trivial. Here is an informal testing maturity ladder in increasing order: - code can only be tested with an integration test - code is tested by comparing the stdout with an earlier version (hopefully it’s deterministic!) - code is retrofitted with tests in mind - code is written to be testable first, maybe with lots of mocks - code is testable, and pure functions are tested with unit tests, leaving the rest to integration tests. Fewer mocks, some stubs. - property based tests, assuming the unit tests are fast in the first place - fuzzing - mutation based testing Not speaking of formal specs or performance testing or anything else.

pfdietz 3y ago
Where does automatic failing test input reduction fall into that ladder? Or is that assumed as part of PBT.
IshKebab 3y ago
I've never seen anyone do mutation testing with software (it's pretty common for hardware though). Does it require language level support?
- boxed 3y ago
  I'm the author of mutmut, the primary mutation tester for python, so I think I can speak a bit on this. It's quite straight forward to do MT in software. I've done quite a bit of it for specific libraries that I've built (iommi, tri.struct, and internal code). A big advantage in my book of MT over PBT is the much lower cognitive overhead for MT and that you can know you have done it completely. The second may be just a false sense of security or an emotional blanket, but still. I have written about mutation testing a few times: https://kodare.net/2019/04/10/mutation-vs-property-based-testing.html https://kodare.net/2019/04/10/mutation-vs-property-based-tes... https://kodare.net/2018/11/18/the-missing-mutant-a-performance-bug-we-missed.html https://kodare.net/2018/11/18/the-missing-mutant-a-performan... https://kodare.net/2016/12/12/mutation-testing-in-practice.html https://kodare.net/2016/12/12/mutation-testing-in-practice.h... and my talk on mutmut from pycon sweden is on youtube: https://www.youtube.com/watch?v=fZwB1gQBwnU https://www.youtube.com/watch?v=fZwB1gQBwnU
  - IshKebab 3y ago
    Does that only work in Python though? What about compiled languages? You don't really want to have to recompile your whole project again for every line you change...
    - boxed 3y ago
      mutmut right now is implemented by writing changes to disk and starting new processes. But you can implement it via "mutation schemata" where you functionally compile all possible mutants ahead of time, plus the original function, and replace the original function with a trampoline that either calls the original or one of the mutants depending on some external state. I have a prototype of mutmut that does this and it's 10 to 100x faster. It does have the downside of not being able to mutate stuff like static global variables and such though.
- Smaug123 3y ago
  [I've never used any mutation testing tools either.] In general it doesn't require language-level support, of course - you can just make a change and rebuild it, à la Stryker https://stryker-mutator.io/docs/stryker-net/technical-reference/research/ https://stryker-mutator.io/docs/stryker-net/technical-refere... . PITest operates on JVM bytecode (https://pitest.org/ https://pitest.org/) for orders of magnitude speedup.
  - IshKebab 3y ago
    Yeah I meant language level support to make it viable, which rebuilding a gazillion times doesn't really sound like.
boxed 3y ago
I don't understand why PBT is above mutation testing. It seems like it's more of a popularity contest kind of thing, and not a matter of engineering tradeoffs or how useful it is.
- jiehong 3y ago
  PBT is below mutation testing, the list is ordered with lower tier up.
  - boxed 3y ago
    Sorry, I put that badly. What I meant was that I don't understand why PBT is something you apply before MT.
- MrJohz 3y ago
  In my experience, adding PBT once you've got a codebase amenable to unit testing in general is a relatively easy step: you can add property tests to your existing unit tests without changing much about your setup, and assuming your tests in general are quick, PBT won't add much overall time to your testing process. But adding mutation testing tends to be harder: it's not just an extra thing you can add in without changing any existing code, it's testing whether the tests are useful in the first place. Which means when you introduce it, you'll probably need to spend some time fixing everything that's currently wrong. This makes it a next step in the testing process beyond just adding a new technique to your existing repertoire. That said, I've used PBT for a while and not had as much success with mutation testing, so maybe this is a personal bias.
  - boxed 3y ago
    The setup for MT should be none. You just start it and see what you get. > But adding mutation testing tends to be harder: it's not just an extra thing you can add in without changing any existing code But it is... > it's testing whether the tests are useful in the first place. Not useful. Complete. MT checks that you have tests for all the behavior of the code.
blowski 3y ago
I really like this list, and it's a great idea to explain testing this way. Perhaps there is also a level -1, when the tests actually make things worse. I see this when tests are extremely brittle, flaky, don't test the most complex or valuable bits of code, are very slow to run, or unmaintained with a list of "tests we know fail but haven't fixed".
- Lutger 3y ago
  There might even be a whole ladder going further down. You, something about the value of everything and the cost of nothing. I've seen tests with bugs in them, hiding bugs in the code and giving a false sense of robustness. because, you know, code coverage is 100% so we cannot have any bugs, right? I had to work with tests that test many trivial things, tightly coupled with implementation details. Those would discourage small refactorings, because it took a lot of time to understand and fix the failed tests as changes in details of implementation would lead to failures in unittests. It also slowed down the pace of development for no good reason. Unless a test has a value that exceeds the cost, it is a net negative.
pydry 3y ago
>- code can only be tested with an integration test Some code only makes sense to test with integration tests. It's not more effective with a code base where somebody has decided to fatten up the SLOC with dependency inversion just so that they can write some unit tests which test that x = x. >- code is tested by comparing the stdout with an earlier version (hopefully it’s deterministic!) Making the code deterministic or adapting the tests to accommodate that it isn't should be next on the ladder, not hoping that it is.
troupo 3y ago
Nope. It's this disparaging of integration tests that gets us into hundreds of useless unit tests that in reality test nothing. Integration tests must be much higher on the list. Also: nothing is stopping you from running your prop tests in integration tests, too
- agentultra 3y ago
  Nothing stopping you. I don’t recommend it. Tests need to be fast or they don’t get written. Adding property tests into integration tests will make things slower and flakier.
  - troupo 3y ago
    > will make things slower and flakier. If running tests with different data makes them flaky, your system is bad. Regardless of property based testing, I've seen too many systems where there were hundreds of unit tests and which wold fail for the most trivial reasons because the actual integration of those units was never tested properly.
    - agentultra 3y ago
      I love PBT and use it frequently. So integration tests are generally IO-bound. They'll be slow. With PBT you will generate 100, 1000 slow cases. Integration tests are flaky in the sense that your components might not generate just the right inputs to trigger an error. They may not be configured under test the same way as they are in production. And so on. PBT can be flaky in the sense that, even with carefully selected distributions, you can get a test that ran fine for months to fail on a run. Add that on top of integration test flakiness. There's no silver bullet for writing bad specifications.
  - pydry 3y ago
    Test speed fetishism is a bad idea. It leads people to write unrealistic tests which end up passing when there is a bug and failing when there isn't. Treating flakiness in integration tests as a fait accompli is also a bad idea. It's a bad idea - a bit like treating flakiness in the app itself as a fait accompli.
naasking 3y ago
The problem with your hierarchy is that there's no empirical evidence supporting it. Small unit tests have not empirically been shown to have benefits over integration tests, and test-driven design has failed to show a benefit over tests written after the fact. The only thing that seems to matter is that tests are written, and the more tests, the more chances of finding a bug. That's it. So your list is actually: * integration and unit tests: since these are manually written, they scale poorly but are simple. * property tests: since these are semi-automatic they scale better but are a bit more complicated to setup. * fuzzing: almost fully automatic, although I don't differentiate this much from property-based testing. * mutation based testing
- pfdietz 3y ago
  Is mutation based testing only better because it forces more tests to be written to kill the mutants? Also, mutation based testing is really orthogonal to the others, since it's a way of evaluating the adequacy of the tests, not of actually testing. One could easily imagine using PBT/fuzzing to generate (and then simplify) tests with the express goal of killing mutants.
- eru 3y ago
  > Small unit tests have not empirically been shown to have benefits over integration tests, [...] Do you have any links to these studies? > * property tests: since these are semi-automatic they scale better but are a bit more complicated to setup. Once you are in the groove, I find property based tests to be simpler (or not harder) than example based tests. But that's when I am writing tests as I am developing the system, ie I take testability into account in the design.
jcgrillo 3y ago
> coming up with good properties is not always trivial This is difficult, but one technique that (might) make it easier for real-world applications beyond simple invariants is to take the approach of building a simple model of the system under test and, in the PBT, checking that your system's behavior matches the model's [1]. [1] https://dl.acm.org/doi/10.1145/3477132.3483540 https://dl.acm.org/doi/10.1145/3477132.3483540
- eru 3y ago
  Testing behaviour against an 'oracle' is a great class of properties to check. Especially useful when you want to test an optimized version against a simpler (but slower) baseline version. Or when you have a class of special cases, that you can solve in a simpler way. Testing the system against itself, but under symmetry, is also useful. But that goes close to general properties. A symmetry could be flipping labels around, or shuffling input order etc; it depends on your system.
stouset 3y ago
> code is written to be testable first, maybe with lots of mocks If you mean what I think I mean, this is the bottom rung of the ladder. Code that is only testable with lots of mocks is in practice worse than code with no tests. Tests should do two things: catch undiscovered bugs and enable refactoring. Test mocked to high heaven do one thing: confirm that the code is written the way it’s currently written. This is diametrically opposed to and completely incompatible with those two stated goals. Most importantly, the code can’t be changed without breaking and rewriting tests. Mocks are okay for modeling owners of external state. Even better are dummy/fake implementations that look and behave like the real thing, (but with highly simplified logic).
eru 3y ago
> While agreeing with the results of this article, I’ve found that convincing other developers of writing test with properties isn’t very easy: coming up with good properties is not always trivial. Yes, but there's some easy targets: Your example based tests often have some values that are supposed not to matter. You can replace those with 'arbitrary' values from your property based testing library. Another easy test that's surprisingly powerful: just chuck 'arbitrary' input at your functions, and check that they don't crash. (Or at least, only throw the expected errors.) You can refine what 'arbitrary' means. The implied property you are testing is that the system doesn't crash. With a lot of asserts etc in your code, that's surprisingly powerful.

pfdietz 3y ago

Where does automatic failing test input reduction fall into that ladder? Or is that assumed as part of PBT.

IshKebab 3y ago

I've never seen anyone do mutation testing with software (it's pretty common for hardware though). Does it require language level support?

boxed 3y ago
I'm the author of mutmut, the primary mutation tester for python, so I think I can speak a bit on this. It's quite straight forward to do MT in software. I've done quite a bit of it for specific libraries that I've built (iommi, tri.struct, and internal code). A big advantage in my book of MT over PBT is the much lower cognitive overhead for MT and that you can know you have done it completely. The second may be just a false sense of security or an emotional blanket, but still. I have written about mutation testing a few times: https://kodare.net/2019/04/10/mutation-vs-property-based-testing.html https://kodare.net/2019/04/10/mutation-vs-property-based-tes... https://kodare.net/2018/11/18/the-missing-mutant-a-performance-bug-we-missed.html https://kodare.net/2018/11/18/the-missing-mutant-a-performan... https://kodare.net/2016/12/12/mutation-testing-in-practice.html https://kodare.net/2016/12/12/mutation-testing-in-practice.h... and my talk on mutmut from pycon sweden is on youtube: https://www.youtube.com/watch?v=fZwB1gQBwnU https://www.youtube.com/watch?v=fZwB1gQBwnU
- IshKebab 3y ago
  Does that only work in Python though? What about compiled languages? You don't really want to have to recompile your whole project again for every line you change...
  - boxed 3y ago
    mutmut right now is implemented by writing changes to disk and starting new processes. But you can implement it via "mutation schemata" where you functionally compile all possible mutants ahead of time, plus the original function, and replace the original function with a trampoline that either calls the original or one of the mutants depending on some external state. I have a prototype of mutmut that does this and it's 10 to 100x faster. It does have the downside of not being able to mutate stuff like static global variables and such though.
Smaug123 3y ago
[I've never used any mutation testing tools either.] In general it doesn't require language-level support, of course - you can just make a change and rebuild it, à la Stryker https://stryker-mutator.io/docs/stryker-net/technical-reference/research/ https://stryker-mutator.io/docs/stryker-net/technical-refere... . PITest operates on JVM bytecode (https://pitest.org/ https://pitest.org/) for orders of magnitude speedup.
- IshKebab 3y ago
  Yeah I meant language level support to make it viable, which rebuilding a gazillion times doesn't really sound like.

boxed 3y ago

I don't understand why PBT is above mutation testing. It seems like it's more of a popularity contest kind of thing, and not a matter of engineering tradeoffs or how useful it is.

jiehong 3y ago
PBT is below mutation testing, the list is ordered with lower tier up.
- boxed 3y ago
  Sorry, I put that badly. What I meant was that I don't understand why PBT is something you apply before MT.
MrJohz 3y ago
In my experience, adding PBT once you've got a codebase amenable to unit testing in general is a relatively easy step: you can add property tests to your existing unit tests without changing much about your setup, and assuming your tests in general are quick, PBT won't add much overall time to your testing process. But adding mutation testing tends to be harder: it's not just an extra thing you can add in without changing any existing code, it's testing whether the tests are useful in the first place. Which means when you introduce it, you'll probably need to spend some time fixing everything that's currently wrong. This makes it a next step in the testing process beyond just adding a new technique to your existing repertoire. That said, I've used PBT for a while and not had as much success with mutation testing, so maybe this is a personal bias.
- boxed 3y ago
  The setup for MT should be none. You just start it and see what you get. > But adding mutation testing tends to be harder: it's not just an extra thing you can add in without changing any existing code But it is... > it's testing whether the tests are useful in the first place. Not useful. Complete. MT checks that you have tests for all the behavior of the code.

blowski 3y ago

I really like this list, and it's a great idea to explain testing this way. Perhaps there is also a level -1, when the tests actually make things worse. I see this when tests are extremely brittle, flaky, don't test the most complex or valuable bits of code, are very slow to run, or unmaintained with a list of "tests we know fail but haven't fixed".

Lutger 3y ago
There might even be a whole ladder going further down. You, something about the value of everything and the cost of nothing. I've seen tests with bugs in them, hiding bugs in the code and giving a false sense of robustness. because, you know, code coverage is 100% so we cannot have any bugs, right? I had to work with tests that test many trivial things, tightly coupled with implementation details. Those would discourage small refactorings, because it took a lot of time to understand and fix the failed tests as changes in details of implementation would lead to failures in unittests. It also slowed down the pace of development for no good reason. Unless a test has a value that exceeds the cost, it is a net negative.

pydry 3y ago

>- code can only be tested with an integration test Some code only makes sense to test with integration tests. It's not more effective with a code base where somebody has decided to fatten up the SLOC with dependency inversion just so that they can write some unit tests which test that x = x. >- code is tested by comparing the stdout with an earlier version (hopefully it’s deterministic!) Making the code deterministic or adapting the tests to accommodate that it isn't should be next on the ladder, not hoping that it is.

troupo 3y ago

Nope. It's this disparaging of integration tests that gets us into hundreds of useless unit tests that in reality test nothing. Integration tests must be much higher on the list. Also: nothing is stopping you from running your prop tests in integration tests, too

agentultra 3y ago
Nothing stopping you. I don’t recommend it. Tests need to be fast or they don’t get written. Adding property tests into integration tests will make things slower and flakier.
- troupo 3y ago
  > will make things slower and flakier. If running tests with different data makes them flaky, your system is bad. Regardless of property based testing, I've seen too many systems where there were hundreds of unit tests and which wold fail for the most trivial reasons because the actual integration of those units was never tested properly.
  - agentultra 3y ago
    I love PBT and use it frequently. So integration tests are generally IO-bound. They'll be slow. With PBT you will generate 100, 1000 slow cases. Integration tests are flaky in the sense that your components might not generate just the right inputs to trigger an error. They may not be configured under test the same way as they are in production. And so on. PBT can be flaky in the sense that, even with carefully selected distributions, you can get a test that ran fine for months to fail on a run. Add that on top of integration test flakiness. There's no silver bullet for writing bad specifications.
- pydry 3y ago
  Test speed fetishism is a bad idea. It leads people to write unrealistic tests which end up passing when there is a bug and failing when there isn't. Treating flakiness in integration tests as a fait accompli is also a bad idea. It's a bad idea - a bit like treating flakiness in the app itself as a fait accompli.

naasking 3y ago

The problem with your hierarchy is that there's no empirical evidence supporting it. Small unit tests have not empirically been shown to have benefits over integration tests, and test-driven design has failed to show a benefit over tests written after the fact. The only thing that seems to matter is that tests are written, and the more tests, the more chances of finding a bug. That's it. So your list is actually: * integration and unit tests: since these are manually written, they scale poorly but are simple. * property tests: since these are semi-automatic they scale better but are a bit more complicated to setup. * fuzzing: almost fully automatic, although I don't differentiate this much from property-based testing. * mutation based testing

pfdietz 3y ago
Is mutation based testing only better because it forces more tests to be written to kill the mutants? Also, mutation based testing is really orthogonal to the others, since it's a way of evaluating the adequacy of the tests, not of actually testing. One could easily imagine using PBT/fuzzing to generate (and then simplify) tests with the express goal of killing mutants.
eru 3y ago
> Small unit tests have not empirically been shown to have benefits over integration tests, [...] Do you have any links to these studies? > * property tests: since these are semi-automatic they scale better but are a bit more complicated to setup. Once you are in the groove, I find property based tests to be simpler (or not harder) than example based tests. But that's when I am writing tests as I am developing the system, ie I take testability into account in the design.

jcgrillo 3y ago

> coming up with good properties is not always trivial This is difficult, but one technique that (might) make it easier for real-world applications beyond simple invariants is to take the approach of building a simple model of the system under test and, in the PBT, checking that your system's behavior matches the model's [1]. [1] https://dl.acm.org/doi/10.1145/3477132.3483540 https://dl.acm.org/doi/10.1145/3477132.3483540

eru 3y ago
Testing behaviour against an 'oracle' is a great class of properties to check. Especially useful when you want to test an optimized version against a simpler (but slower) baseline version. Or when you have a class of special cases, that you can solve in a simpler way. Testing the system against itself, but under symmetry, is also useful. But that goes close to general properties. A symmetry could be flipping labels around, or shuffling input order etc; it depends on your system.

stouset 3y ago

> code is written to be testable first, maybe with lots of mocks If you mean what I think I mean, this is the bottom rung of the ladder. Code that is only testable with lots of mocks is in practice worse than code with no tests. Tests should do two things: catch undiscovered bugs and enable refactoring. Test mocked to high heaven do one thing: confirm that the code is written the way it’s currently written. This is diametrically opposed to and completely incompatible with those two stated goals. Most importantly, the code can’t be changed without breaking and rewriting tests. Mocks are okay for modeling owners of external state. Even better are dummy/fake implementations that look and behave like the real thing, (but with highly simplified logic).

eru 3y ago

> While agreeing with the results of this article, I’ve found that convincing other developers of writing test with properties isn’t very easy: coming up with good properties is not always trivial. Yes, but there's some easy targets: Your example based tests often have some values that are supposed not to matter. You can replace those with 'arbitrary' values from your property based testing library. Another easy test that's surprisingly powerful: just chuck 'arbitrary' input at your functions, and check that they don't crash. (Or at least, only throw the expected errors.) You can refine what 'arbitrary' means. The implied property you are testing is that the system doesn't crash. With a lot of asserts etc in your code, that's surprisingly powerful.