3 ms·

There are a lot of words without a strong definition, but it seems to summarize to: “Property testing is where you vary the parameters, perhaps within certain

by dfee 6y ago

There are a lot of words without a strong definition, but it seems to summarize to: “Property testing is where you vary the parameters, perhaps within certain well defined constraints, and through randomization ensure certain coverage.” I think there are other ways of handling this, beyond the way described, but I’m not sure if that’s because I just read a field guide to Hypothesis (again, without a firm definition), or if other approaches - such as test parameterization at key boundaries is more effective. My hunch is that property based testing provides less value on unit tests - where it could be used to reduce coverage - and more value in integrating those units together - where the number of permutations can grow very quickly.

BerislavLopac 6y ago
> Property testing is where you vary the parameters, perhaps within certain well defined constraints, and through randomization ensure certain coverage. No - this would be akin to saying "unit testing is when you use assert statements". Varying the parameters -- and using far more than simple randomisation -- is just a technique; the purpose of property testing is that the behaviour of the unit (or system) under test fulfills certain properties, in ma mathematical sense of the word. For a simple example: if you're writing a function that is adding two integer numbers, you want to test that it satisfies all the properties of addition [0]. So your tests need to validate against a number of combinations of input values, many of which are not random (e.g. to test for the "successor" property one of them needs to be 1). [0] https://en.wikipedia.org/wiki/Addition#Properties https://en.wikipedia.org/wiki/Addition#Properties
Jtsummers 6y ago
PBT is nice in place of (some) unit tests in that you can describe immediately the properties you expect without needing to produce a collection of examples (or write a custom generator that produces a limited set of examples, at which point you're halfway to PBT anyways). It's also helpful to use it in a piece wise fashion if you're doing TDD. An illustrative example (though perhaps not stellar as it is a synthetic, non-real-world example) uses the diamond kata, TDD, and PBT together [0]. None of the tests on their own fully specify the system, but in total they do. If you're doing TDD (or attempting to) I think this is an interesting case. Many TDD methods have you start off with an example case like (to stick with this kata, and using Pythonesque pseudocode because I'm still not awake this Saturday morning): diamond-kata-test-a: assert(diamond('A') == 'A') Great, so now someone makes that absolute simplest solution: diamond(c): return 'A' Now repeat with a second test case: diamond-kata-test-b: assert(diamond('B') == ' A \nB B\n A ') And the function is duly complicated: diamond(c): switch c: case 'A': return 'A' case 'B': return ' A \nB B\n A ' default: return 'blah' // or error, doesn't matter it's not tested But not actually generalized to reflect the intent of the system. By focusing on properties, I've found, the progression of the UUT is a bit better/more natural. Another interesting thing to do with PBT is model-based testing [1]. The useful thing here is that sometimes the errors are triggered by a peculiar, though plausible, sequence of commands to your system. We've all worked with that one guy who somehow manages to find exactly the right sequence that triggers weird edge cases and errors, but unless we're him having a system which will generate arbitrary execution traces for you. (I actually used FsCheck for this last year in trying to sell PBT to my colleagues and was able to identify where a known issue originated as well as several other problems that hadn't been found by users or testers yet.) In the end, when these failures are found you can always turn them into distinct unit tests in order to preserve them and prevent regressions. The two modes of testing fit well together. [0] http://christophethibaut.com/programming/2020/03/18/Diamond-Kata-PBT-TDD.html http://christophethibaut.com/programming/2020/03/18/Diamond-... [1] https://fscheck.github.io/FsCheck/StatefulTesting.html https://fscheck.github.io/FsCheck/StatefulTesting.html

by dfee 6y ago

BerislavLopac 6y ago

> Property testing is where you vary the parameters, perhaps within certain well defined constraints, and through randomization ensure certain coverage. No - this would be akin to saying "unit testing is when you use assert statements". Varying the parameters -- and using far more than simple randomisation -- is just a technique; the purpose of property testing is that the behaviour of the unit (or system) under test fulfills certain properties, in ma mathematical sense of the word. For a simple example: if you're writing a function that is adding two integer numbers, you want to test that it satisfies all the properties of addition [0]. So your tests need to validate against a number of combinations of input values, many of which are not random (e.g. to test for the "successor" property one of them needs to be 1). [0] https://en.wikipedia.org/wiki/Addition#Properties https://en.wikipedia.org/wiki/Addition#Properties

Jtsummers 6y ago

PBT is nice in place of (some) unit tests in that you can describe immediately the properties you expect without needing to produce a collection of examples (or write a custom generator that produces a limited set of examples, at which point you're halfway to PBT anyways). It's also helpful to use it in a piece wise fashion if you're doing TDD. An illustrative example (though perhaps not stellar as it is a synthetic, non-real-world example) uses the diamond kata, TDD, and PBT together [0]. None of the tests on their own fully specify the system, but in total they do. If you're doing TDD (or attempting to) I think this is an interesting case. Many TDD methods have you start off with an example case like (to stick with this kata, and using Pythonesque pseudocode because I'm still not awake this Saturday morning): diamond-kata-test-a: assert(diamond('A') == 'A') Great, so now someone makes that absolute simplest solution: diamond(c): return 'A' Now repeat with a second test case: diamond-kata-test-b: assert(diamond('B') == ' A \nB B\n A ') And the function is duly complicated: diamond(c): switch c: case 'A': return 'A' case 'B': return ' A \nB B\n A ' default: return 'blah' // or error, doesn't matter it's not tested But not actually generalized to reflect the intent of the system. By focusing on properties, I've found, the progression of the UUT is a bit better/more natural. Another interesting thing to do with PBT is model-based testing [1]. The useful thing here is that sometimes the errors are triggered by a peculiar, though plausible, sequence of commands to your system. We've all worked with that one guy who somehow manages to find exactly the right sequence that triggers weird edge cases and errors, but unless we're him having a system which will generate arbitrary execution traces for you. (I actually used FsCheck for this last year in trying to sell PBT to my colleagues and was able to identify where a known issue originated as well as several other problems that hadn't been found by users or testers yet.) In the end, when these failures are found you can always turn them into distinct unit tests in order to preserve them and prevent regressions. The two modes of testing fit well together. [0] http://christophethibaut.com/programming/2020/03/18/Diamond-Kata-PBT-TDD.html http://christophethibaut.com/programming/2020/03/18/Diamond-... [1] https://fscheck.github.io/FsCheck/StatefulTesting.html https://fscheck.github.io/FsCheck/StatefulTesting.html