3 ms·
I think this is actually the better definition of property based testing, written by the same author: https://increment.com/testing/in-praise-of-property-based-
by amw-zero 5y ago
I think this is actually the better definition of property based testing, written by the same author: https://increment.com/testing/in-praise-of-property-based-testing/ https://increment.com/testing/in-praise-of-property-based-te...
My take: don't shy away from the logic perspective of what properties are, because it's the simplest explanation. A property is a logical proposition, e.g. "p implies q". Property based testing randomizes values of p and q as a proxy for all possible values, since exhaustive testing is practically impossible in almost all cases. The hypothesis is that running these tests for long enough will eventually expose a bug, since more and more of the input space gets generated over time.
PBT is an addicting tool. I used it on a personal project involving a lot of date filtering, and it found so many edge cases related to date parsing and timezone handling. In my experience, you just also write way less test code since the test library is generating the test cases for you.
The two major downsides are the difficulty in expressing useful properties, and test runtime. Regarding expressing properties, it is difficult, but I wonder if it can't be improved with practice. Hillel Wayne wrote a bit about properties here: https://www.hillelwayne.com/contract-examples/ https://www.hillelwayne.com/contract-examples/
Regarding the test runtime - it's just something to be wary of. You can easily start running tens of thousands of test cases depending on how you configure the library. It makes things like testing against a real database really impractical, unless you really control which inputs get generated, which defeats the purpose of random data generation.
I think parallelization and non-blocking test suites are the answer here. Locally and in CI, they can be run with a reasonable N so that obvious errors hopefully get caught, but large N's get saved for an async test suite that can benefit from high parallelization.
My last point about them is, even after all their benefit, example-based tests are still easier to write and serve their purpose. They also allow you to test specific edge cases explicitly.
Like everything else, PBT is just another tool in the shed.
- kragen 5y ago> Regarding the test runtime - it's just something to be wary of. You can easily start running tens of thousands of test cases depending on how you configure the library. It makes things like testing against a real database really impractical, unless you really control which inputs get generated, which defeats the purpose of random data generation. Whether that's practical or not depends on your tooling and your resources. If you're willing to spend US$100k on a test machine, and some people are, you can have a 4096-node test cluster.
- KineticLensman 5y agoYes, i prefer your linked article. It actually includes an example, something lacking in the original