8 ms·

Property testing is a lot like formal methods - really cool, but almost entirely useless in ~95% of commercial contexts. They're both extremely useful when, sa

Property testing is a lot like formal methods - really cool, but almost entirely useless in ~95% of commercial contexts. They're both extremely useful when, say, building a parser, but when the kind of code you write involves displaying custom widgets, taking data and pushing it onto a queue, looking up data in a database, etc. integration tests have a lot more bang for the buck.

diggan 2y ago
I've found it effective for anything that handles arbitrary input, especially from end-users. But if that data is coming from within your systems where you have full control over everything, less valuable.
- hitchstory 2y ago
  If the arbitrary input is, say, a text box which takes a name and puts it into a database, it probably won't uncover any bugs. It has some use if you build something like a complex pricing engine, numerical code or a parser for a mini DSL. I find that problems of this type don't crop up a lot though.
  - IanCal 2y ago
    I disagree. I've used it for things like "regardless of where you are on the page, tab n times and shift tab n times leaves you on the original item". I found a bug in our tv ui library which was actually a bug in the spec. Regardless of how you built the ui, if you press a direction and focus moves, pressing the opposite direction takes you back - but we had another rule that broke this. We had tests for both, and it was only when I made the general test (for all ui, items in it and directions) it found the inconsistency. It was also pretty easy to write. I've also found issues around text processing due to lowercasing not always resulting in the same length string and more. I found a bug demoing pbt for a contact gig I was going for that was around some versioning. To be honest I've never implemented it, even for a demo, and not found a bug.
    - josephg 2y ago
      > To be honest I've never implemented it, even for a demo, and not found a bug. Me too. I tend to roll my own property testers / fuzzers per project instead of using a library. But my experience is similar to yours. Out of maybe 25 testers, I think the only times I didn’t find any bugs was when I messed up the tests themselves. It’s incredibly humbling work.
      - IanCal 2y ago
        I used to build these too, my example was one in actionscript (always fun on restricted devices) and it was a bug in a library I wrote. I was actually testing the property tester and found the bug then, after realizing it wasn't a bug in the new test/tester. > It’s incredibly humbling work Absolutely. Glad others feel it too. I almost feel like you need a decent reason why not to use them in places. Ideally to can describe the behaviour of something in general terms around what stays true. If you can't, is it too hard for people to understand? I will however say I don't usually write pbts but have a strong feeling I should add more.
- jgalt212 2y ago
  Very true. For me, fuzzers and property-based tests are two sides of the same coin. I'd just use whichever feels more natural.
  - pfdietz 2y ago
    Thinking about the difference, I wonder if there is a PBT framework that would exploit feedback like fuzzing does, to improve coverage.
    - chriswarbo 2y ago
      The Choice Gradient Sampling algorithm shown in the paper could be used to integrate/exploit feedback (coverage, etc.). Hypothesis can use a fuzzer as its input source, but it's marked as experimental. It doesn't seem like "proper" fuzzers are used very often with Haskell (e.g. there's an example at https://www.tweag.io/blog/2023-06-15-ghc-libfuzzer/ https://www.tweag.io/blog/2023-06-15-ghc-libfuzzer/ ), though many people seem to have hooked up the HPC code coverage tool to property checkers, e.g. https://www.cse.iitk.ac.in/users/karkare/MTP/2011-12/subhash2012quickeval.pdf https://www.cse.iitk.ac.in/users/karkare/MTP/2011-12/subhash... https://www.doc.ic.ac.uk/~tora/irulan/issta.pdf https://www.doc.ic.ac.uk/~tora/irulan/issta.pdf https://github.com/shapr/kudzu https://github.com/shapr/kudzu
- chriswarbo 2y ago
  It's useful for things that would be too painful to enforce via types. For example, "this list should be non-empty" is pretty easy to enforce via types (use a pair of head element and possibly-empty tail list). On the other hand, something like "map of non-overlapping date ranges to counts, with automatic splitting on insertion" would be pretty tricky. It's easier to state the desired properties as universally-quantified functions rather than types, at which point we might as well throw them into a few property checkers to see if they find anything we didn't think of.
boscillator 2y ago
It's very useful when you're working on numerical software. Often, it's hard to figure out exactly what output your code should return (because if you knew the answer you wouldn't have to write the code), but you can easily list properties you expect.
- matt_d 2y ago
  Right, metamorphic testing in particular (which would be a special case of PBT, with metamorphic relations being properties), https://en.wikipedia.org/wiki/Metamorphic_testing https://en.wikipedia.org/wiki/Metamorphic_testing, https://github.com/MattPD/cpplinks/blob/master/testing.md#property-based-testing-metamorphic https://github.com/MattPD/cpplinks/blob/master/testing.md#pr... One simple example (from the above) is "sin (π − x) = sin x" for the implementation of the sine function not requiring the knowledge of its specific output values. Naturally, instead of the literal equality "=" one can use a more appropriate accuracy specification as in, say, relative ulp (https://en.wikipedia.org/wiki/Unit_in_the_last_place https://en.wikipedia.org/wiki/Unit_in_the_last_place) error bound, cf. https://members.loria.fr/PZimmermann/papers/accuracy.pdf https://members.loria.fr/PZimmermann/papers/accuracy.pdf
thehappyfellow 2y ago
How come e.g. Jane Street uses it so much? It’s the second most common type of test I write.
- hansvm 2y ago
  The same reason Google burns $50M+ in electricity each year using protobufs instead of a more efficient format. An individual company having specific needs isn't at odds with a general statement being broadly true.
  - thehappyfellow 2y ago
    How’s that comparable at all? There are no network effects from writing property based tests, people use them if they are helpful - are they testing enough of the code with reasonable amount of effort. Nobody’s forcing people to write tests, unlike Google forces usage of protobuf on all projects there.
    - hansvm 2y ago
      It's comparable in the way described in sentence #2: > An individual company having specific needs isn't at odds with a general statement being broadly true. Google needs certain things more than reduced carbon emissions, and Jane Street needs certain things more than whatever else they could spend that dev time on.
      - thehappyfellow 2y ago
        Fine but cutting the thought process at "it depends" is not a great way to understand what's happening here. You can explain anything happening at any company by saying "they need certain things more than whatever else they could spend that time on". Why is PBT useful at Jane Street, at least more than in other places? Is it the use of functional language? Average Jane Street dev being more familiar with PBT? Is the domain particularly suited to this style of testing? Explicitly, my claim is that the biggest bottleneck is education on how to use PBT effectively and Jane Street is not using them to get an extra mile in safety, they use it because it's the easiest way to write large chunk of the tests.
choeger 2y ago
Hah! Try to separate your domain logic from your interfaces (e.g., using something like hexagonal architecture) and then say this again. Yes, it's a lot of work coming up with good properties, but it massively helps to find gaps in the domain logic. In my experience, these gaps are what's typically expensive, not the weird problem a junior had with properly using Redis or S3.
- hitchstory 2y ago
  >Hah! Try to separate your domain logic from your interfaces Im not an amateur. The only time I dont do this is when there literally is no domain logic yet (e.g. a CRUD app). >In my experience, these gaps are what's typically expensive, not the weird problem a junior had with properly using Redis or S3. What can I say? Your experience might not be as broad as mine. Redis is a source of almost no bugs because it is very well designed, but most interfaces I couple to have design qualities that are the exact opposite of redis's. Those interfaces (e.g. wonky payment gateway APIs, weird microservice APIs) are the probably source of most bugs in enterprise systems I work on. #2 is probably simple misspecifications (customer said code should do X, it should actually do Y which is almost the same but very slightly different). #3 would be domain logic errors. And even most of those are uncovered and avoided with saner architecture or a couple of unit tests. For the parsers I write at home, sure, property testing kicks ass. For your college degree algorithm coursework, sure, it helps a lot. For 95% of business logic? Pointless, the complexity isnt buried deep in the business logic.
thom 2y ago
I agree, but this is a good thing! My default approach these days is functional tests for everything possible, and property based tests for anything particularly algorithmic or containing lots of edge cases, and no unit tests outside that. This is a great combo, covers all the business value without leaving obscure bugs, and also isn’t a pain every time you refactor.
amw-zero 2y ago
For "business applications," I much prefer the variant of property testing called model-based testing. This is where your property is "does the implementation behave like some simplified model." This is the correct way to test such applications in my opinion, because users don't care about "custom widgets," "queues," or "databases." They care about business functionality, which is quite easily expressible in model form. I don't do much mocking either, because I also prefer integration tests for these types of applications. Here is an example that tests against a database, all the way up to the state management layer of the UI: https://concerningquality.com/model-based-testing/ https://concerningquality.com/model-based-testing/. In fairness, people don't write about doing this a lot. PBT literature seems to revolve around sorting functions or binary tree operations. But PBT much more powerful than just that.
- hitchstory 2y ago
  >in my opinion, because users don't care about "custom widgets," "queues," or "databases." Users care quite a lot when these things break. >They care about business functionality, which is quite easily expressible in model form. They care about reliable applications that do the task they want. Users don't have a concept of "business functionality". If a widget they wanted to use is broken, they'll be pissed - they won't tell you that it's ok because the concept of the application was still good. From your link: >The main thing that I want to highlight here is how simple the model is. This is pure domain logic, and is probably as close to the essential complexity of the application that we can get This is exactly the issue. If I go down my recent list of bugs and categorize them, ~8 out of 10 of them are in areas which the author would have *abstracted away*. The other two probably could have been avoided by having a team member write one test to cover that functionality instead of zero. What does it mean when you abstract something away from a test? Well, it means the test won't catch a bug in that thing. If I went down my recent list of bugs and been able to see that 9 out of 10 of them would have been caught via adding property testing/model testing on top of regular testing/typing, etc. I'd have absolutely no second thoughts about implementing it but that simply isn't the case. I have worked on problems in (not usual) environments where it is the case, and in those rare cases I have advocated for and used property testing but I have more often seen it advocated and used by people who see every bug as a nail and property testing as some kind of super advanced hammer.
LAC-Tech 2y ago
I have to admit that after all these years the term "integration tests" is near meaningless me. But proberty based testing is absolutely real. If you can think of one example based test you can probably think of a way to generalise them. It's a powerful idea, much more powerful than trying to remember what exactly a "unit" means.

diggan 2y ago

I've found it effective for anything that handles arbitrary input, especially from end-users. But if that data is coming from within your systems where you have full control over everything, less valuable.

hitchstory 2y ago
If the arbitrary input is, say, a text box which takes a name and puts it into a database, it probably won't uncover any bugs. It has some use if you build something like a complex pricing engine, numerical code or a parser for a mini DSL. I find that problems of this type don't crop up a lot though.
- IanCal 2y ago
  I disagree. I've used it for things like "regardless of where you are on the page, tab n times and shift tab n times leaves you on the original item". I found a bug in our tv ui library which was actually a bug in the spec. Regardless of how you built the ui, if you press a direction and focus moves, pressing the opposite direction takes you back - but we had another rule that broke this. We had tests for both, and it was only when I made the general test (for all ui, items in it and directions) it found the inconsistency. It was also pretty easy to write. I've also found issues around text processing due to lowercasing not always resulting in the same length string and more. I found a bug demoing pbt for a contact gig I was going for that was around some versioning. To be honest I've never implemented it, even for a demo, and not found a bug.
  - josephg 2y ago
    > To be honest I've never implemented it, even for a demo, and not found a bug. Me too. I tend to roll my own property testers / fuzzers per project instead of using a library. But my experience is similar to yours. Out of maybe 25 testers, I think the only times I didn’t find any bugs was when I messed up the tests themselves. It’s incredibly humbling work.
    - IanCal 2y ago
      I used to build these too, my example was one in actionscript (always fun on restricted devices) and it was a bug in a library I wrote. I was actually testing the property tester and found the bug then, after realizing it wasn't a bug in the new test/tester. > It’s incredibly humbling work Absolutely. Glad others feel it too. I almost feel like you need a decent reason why not to use them in places. Ideally to can describe the behaviour of something in general terms around what stays true. If you can't, is it too hard for people to understand? I will however say I don't usually write pbts but have a strong feeling I should add more.
jgalt212 2y ago
Very true. For me, fuzzers and property-based tests are two sides of the same coin. I'd just use whichever feels more natural.
- pfdietz 2y ago
  Thinking about the difference, I wonder if there is a PBT framework that would exploit feedback like fuzzing does, to improve coverage.
  - chriswarbo 2y ago
    The Choice Gradient Sampling algorithm shown in the paper could be used to integrate/exploit feedback (coverage, etc.). Hypothesis can use a fuzzer as its input source, but it's marked as experimental. It doesn't seem like "proper" fuzzers are used very often with Haskell (e.g. there's an example at https://www.tweag.io/blog/2023-06-15-ghc-libfuzzer/ https://www.tweag.io/blog/2023-06-15-ghc-libfuzzer/ ), though many people seem to have hooked up the HPC code coverage tool to property checkers, e.g. https://www.cse.iitk.ac.in/users/karkare/MTP/2011-12/subhash2012quickeval.pdf https://www.cse.iitk.ac.in/users/karkare/MTP/2011-12/subhash... https://www.doc.ic.ac.uk/~tora/irulan/issta.pdf https://www.doc.ic.ac.uk/~tora/irulan/issta.pdf https://github.com/shapr/kudzu https://github.com/shapr/kudzu
chriswarbo 2y ago
It's useful for things that would be too painful to enforce via types. For example, "this list should be non-empty" is pretty easy to enforce via types (use a pair of head element and possibly-empty tail list). On the other hand, something like "map of non-overlapping date ranges to counts, with automatic splitting on insertion" would be pretty tricky. It's easier to state the desired properties as universally-quantified functions rather than types, at which point we might as well throw them into a few property checkers to see if they find anything we didn't think of.

boscillator 2y ago

It's very useful when you're working on numerical software. Often, it's hard to figure out exactly what output your code should return (because if you knew the answer you wouldn't have to write the code), but you can easily list properties you expect.

matt_d 2y ago
Right, metamorphic testing in particular (which would be a special case of PBT, with metamorphic relations being properties), https://en.wikipedia.org/wiki/Metamorphic_testing https://en.wikipedia.org/wiki/Metamorphic_testing, https://github.com/MattPD/cpplinks/blob/master/testing.md#property-based-testing-metamorphic https://github.com/MattPD/cpplinks/blob/master/testing.md#pr... One simple example (from the above) is "sin (π − x) = sin x" for the implementation of the sine function not requiring the knowledge of its specific output values. Naturally, instead of the literal equality "=" one can use a more appropriate accuracy specification as in, say, relative ulp (https://en.wikipedia.org/wiki/Unit_in_the_last_place https://en.wikipedia.org/wiki/Unit_in_the_last_place) error bound, cf. https://members.loria.fr/PZimmermann/papers/accuracy.pdf https://members.loria.fr/PZimmermann/papers/accuracy.pdf

thehappyfellow 2y ago

How come e.g. Jane Street uses it so much? It’s the second most common type of test I write.

hansvm 2y ago
The same reason Google burns $50M+ in electricity each year using protobufs instead of a more efficient format. An individual company having specific needs isn't at odds with a general statement being broadly true.
- thehappyfellow 2y ago
  How’s that comparable at all? There are no network effects from writing property based tests, people use them if they are helpful - are they testing enough of the code with reasonable amount of effort. Nobody’s forcing people to write tests, unlike Google forces usage of protobuf on all projects there.
  - hansvm 2y ago
    It's comparable in the way described in sentence #2: > An individual company having specific needs isn't at odds with a general statement being broadly true. Google needs certain things more than reduced carbon emissions, and Jane Street needs certain things more than whatever else they could spend that dev time on.
    - thehappyfellow 2y ago
      Fine but cutting the thought process at "it depends" is not a great way to understand what's happening here. You can explain anything happening at any company by saying "they need certain things more than whatever else they could spend that time on". Why is PBT useful at Jane Street, at least more than in other places? Is it the use of functional language? Average Jane Street dev being more familiar with PBT? Is the domain particularly suited to this style of testing? Explicitly, my claim is that the biggest bottleneck is education on how to use PBT effectively and Jane Street is not using them to get an extra mile in safety, they use it because it's the easiest way to write large chunk of the tests.

choeger 2y ago

Hah! Try to separate your domain logic from your interfaces (e.g., using something like hexagonal architecture) and then say this again. Yes, it's a lot of work coming up with good properties, but it massively helps to find gaps in the domain logic. In my experience, these gaps are what's typically expensive, not the weird problem a junior had with properly using Redis or S3.

hitchstory 2y ago
>Hah! Try to separate your domain logic from your interfaces Im not an amateur. The only time I dont do this is when there literally is no domain logic yet (e.g. a CRUD app). >In my experience, these gaps are what's typically expensive, not the weird problem a junior had with properly using Redis or S3. What can I say? Your experience might not be as broad as mine. Redis is a source of almost no bugs because it is very well designed, but most interfaces I couple to have design qualities that are the exact opposite of redis's. Those interfaces (e.g. wonky payment gateway APIs, weird microservice APIs) are the probably source of most bugs in enterprise systems I work on. #2 is probably simple misspecifications (customer said code should do X, it should actually do Y which is almost the same but very slightly different). #3 would be domain logic errors. And even most of those are uncovered and avoided with saner architecture or a couple of unit tests. For the parsers I write at home, sure, property testing kicks ass. For your college degree algorithm coursework, sure, it helps a lot. For 95% of business logic? Pointless, the complexity isnt buried deep in the business logic.

thom 2y ago

I agree, but this is a good thing! My default approach these days is functional tests for everything possible, and property based tests for anything particularly algorithmic or containing lots of edge cases, and no unit tests outside that. This is a great combo, covers all the business value without leaving obscure bugs, and also isn’t a pain every time you refactor.

amw-zero 2y ago

For "business applications," I much prefer the variant of property testing called model-based testing. This is where your property is "does the implementation behave like some simplified model." This is the correct way to test such applications in my opinion, because users don't care about "custom widgets," "queues," or "databases." They care about business functionality, which is quite easily expressible in model form. I don't do much mocking either, because I also prefer integration tests for these types of applications. Here is an example that tests against a database, all the way up to the state management layer of the UI: https://concerningquality.com/model-based-testing/ https://concerningquality.com/model-based-testing/. In fairness, people don't write about doing this a lot. PBT literature seems to revolve around sorting functions or binary tree operations. But PBT much more powerful than just that.

hitchstory 2y ago
>in my opinion, because users don't care about "custom widgets," "queues," or "databases." Users care quite a lot when these things break. >They care about business functionality, which is quite easily expressible in model form. They care about reliable applications that do the task they want. Users don't have a concept of "business functionality". If a widget they wanted to use is broken, they'll be pissed - they won't tell you that it's ok because the concept of the application was still good. From your link: >The main thing that I want to highlight here is how simple the model is. This is pure domain logic, and is probably as close to the essential complexity of the application that we can get This is exactly the issue. If I go down my recent list of bugs and categorize them, ~8 out of 10 of them are in areas which the author would have *abstracted away*. The other two probably could have been avoided by having a team member write one test to cover that functionality instead of zero. What does it mean when you abstract something away from a test? Well, it means the test won't catch a bug in that thing. If I went down my recent list of bugs and been able to see that 9 out of 10 of them would have been caught via adding property testing/model testing on top of regular testing/typing, etc. I'd have absolutely no second thoughts about implementing it but that simply isn't the case. I have worked on problems in (not usual) environments where it is the case, and in those rare cases I have advocated for and used property testing but I have more often seen it advocated and used by people who see every bug as a nail and property testing as some kind of super advanced hammer.

LAC-Tech 2y ago

I have to admit that after all these years the term "integration tests" is near meaningless me. But proberty based testing is absolutely real. If you can think of one example based test you can probably think of a way to generalise them. It's a powerful idea, much more powerful than trying to remember what exactly a "unit" means.