7 ms·

Property-based testing in practice [pdf]

70 points by Smaug123 3y ago

eru 3y ago
Wow, hypothesis seems to have reached 4% of Python users? That's actually pretty impressive.
- throwup238 3y ago
  That's the same percentage of users who have Python package management figured out!
- IshKebab 3y ago
  *of Python users that completed the Python Jetbrains survey.
  - OJFord 3y ago
    I expect there's more data science conda etc. users than users to whom 'Jetbrains survey' even registers. Not to be negative about hypothesis though, I like it, tried to help increase adoption at my previous job after a colleague introduced me, but that was shortly after Pycon UK 2018/19 and I haven't used it since. Not that I don't think we'd benefit from it, just probably lower hanging fruit atm.
    - IshKebab 3y ago
      Yeah most Python users don't even use static type annotations yet... Property testing is ambitious!
eru 3y ago
I found that writing new property based tests was a skill I picked up relatively quickly. But learning how to retro-fit existing tests was a whole 'nother skill that I had to learn almost independently afterwards.
dikei 3y ago
TIL: Jane Street use OCalm in production
- Tomte 3y ago
  They maintain their own standard library for Ocaml, don‘t they?
  - ReleaseCandidat 3y ago
    Not just the standard library, but a whole ecosystem around it. And their own compiler. https://github.com/ocaml-flambda/ocaml-jst https://github.com/ocaml-flambda/ocaml-jst
jiehong 3y ago
While agreeing with the results of this article, I’ve found that convincing other developers of writing test with properties isn’t very easy: coming up with good properties is not always trivial. Here is an informal testing maturity ladder in increasing order: - code can only be tested with an integration test - code is tested by comparing the stdout with an earlier version (hopefully it’s deterministic!) - code is retrofitted with tests in mind - code is written to be testable first, maybe with lots of mocks - code is testable, and pure functions are tested with unit tests, leaving the rest to integration tests. Fewer mocks, some stubs. - property based tests, assuming the unit tests are fast in the first place - fuzzing - mutation based testing Not speaking of formal specs or performance testing or anything else.
- pfdietz 3y ago
  Where does automatic failing test input reduction fall into that ladder? Or is that assumed as part of PBT.
- IshKebab 3y ago
  I've never seen anyone do mutation testing with software (it's pretty common for hardware though). Does it require language level support?
  - boxed 3y ago
    I'm the author of mutmut, the primary mutation tester for python, so I think I can speak a bit on this. It's quite straight forward to do MT in software. I've done quite a bit of it for specific libraries that I've built (iommi, tri.struct, and internal code). A big advantage in my book of MT over PBT is the much lower cognitive overhead for MT and that you can know you have done it completely. The second may be just a false sense of security or an emotional blanket, but still. I have written about mutation testing a few times: https://kodare.net/2019/04/10/mutation-vs-property-based-testing.html https://kodare.net/2019/04/10/mutation-vs-property-based-tes... https://kodare.net/2018/11/18/the-missing-mutant-a-performance-bug-we-missed.html https://kodare.net/2018/11/18/the-missing-mutant-a-performan... https://kodare.net/2016/12/12/mutation-testing-in-practice.html https://kodare.net/2016/12/12/mutation-testing-in-practice.h... and my talk on mutmut from pycon sweden is on youtube: https://www.youtube.com/watch?v=fZwB1gQBwnU https://www.youtube.com/watch?v=fZwB1gQBwnU
mdiep 3y ago
If you're interested in property-based testing, I highly recommend "How to Specify It!" by John Hughes, which is available both as a talk and as a paper. https://research.chalmers.se/publication/517894/file/517894_Fulltext.pdf https://research.chalmers.se/publication/517894/file/517894_... https://www.youtube.com/watch?v=G0NUOst-53U https://www.youtube.com/watch?v=G0NUOst-53U He gives fantastic guidance about how to write PBTs.
sirwhinesalot 3y ago
Can't read the pdf right now but I'm a big fan of property based testing. One thing I find that people struggle with is coming up with "good properties" to test with. That's the wrong way to think about it. The properties you want to test are the function's contract. Checking that contract is the goal. You can be as specific with the contract as you want. The more specific the more bugs you'll find. Property-based tests are just one way to check the contract, as are hand-written unit tests. You could use a static analyzer or a model checker as well, they're all different approaches to do the same thing. EDIT: by contract I mean the guarantees the function imposes on its output. A contract for a sorting function could be as simple as the length of the output being the same as the input. That's one property. Another is that every element in the output is also in the input. You can go all the way and say that for every element at index i, the element at index i+1 (if any) is larger. But you don't need a perfect contract to start with nor to end with. You can add more guarantees/properties as you wish. The more specific, the better (but also slower) the tests.
crabbone 3y ago
Over time of trying to adopt or even reinvent PBT I discovered that there isn't a good language to formally describe software at the level where it becomes interesting to test it. TLA+ gets somewhat close... but it's both too difficult to write and its hard to adapt to eg. interactive systems. I don't mean to say that PBT is a bad idea. I actually think it's very good. I wish there was a way to make it really useful though. As the paper mentions, PBT in their experience is used for "component testing", which is just another name for "unit testing", which is where automation of this kind isn't all that important. Integration and E2E testing is what's a lot more important, but doesn't really have a way to approaching r.n.
- hwayne 3y ago
  Have you looked at P? https://p-org.github.io/P/ https://p-org.github.io/P/
  - crabbone 3y ago
    No, I haven't. Thanks for the link. I'd have to spend some time reading about it. Ugh... > The P compiler is implemented in C# and hence the tool chain requires dotnet. Nah, sorry. I'm not going to read further about it.
    - neonsunset 3y ago
      Why?
      - crabbone 3y ago
        I started my familiarity with programming using a bunch of Micorosoft's products, but later was able to completely migrate away from them. Today, if I'm required to use a Microsoft's product, it's kind of like asking a North Korean who valiantly fought to get away from that cursed country to go back and live there again. You see, my understanding of what's happening with MS is that their sales department is exceptionally successful, which lowers the requirements for everything else in the product. A particular quality of this sales department is that they rarely sell to individual end users: to be efficient, they sell to people who have absolute control over the end users, but are also incompetent when it comes to software. Think governments, hospitals, schools etc. Another aspect of MS software is that they, internally, don't really allow anything that's not made by MS, and they are big enough for this policy to succeed. The unfortunate side effect of this policy is the Stockholm syndrome: MS developers not only get used to the MS tools, they genuinely seem to like the suffering incurred by using them. Another unfortunate side effect of this policy is that MS developers have no idea what normal users actually like. So, they are kind of like Bender (a robot, who has only a very faint idea of how humans eat) in the Futurama episode, where he enters a cooking contest. He then proceeds to put all kind of inedible components into his meals (but eventually wins by adding some LSD to the mix). So... I will never touch any MS products with a ten feet pole, unless forced to by my employer. And if the employer goes as far as requiring me to use MS products a lot, I'll look for another place to work. My job today is quite meaningless, boring and I don't get to work with things I like or people I respect... but having to use MS tools for work would still be a lot, lot worse. It's a kind of a torture... well, of course, it's an exaggeration: it doesn't cause me physical pain, but it's a kind of frustration that makes me want to punch a wall every few seconds. And to top it: using C# for anything is kind of like... starting a cooking recipe with "put a handful of chopped garlic into the pen". You just know it's either a mistake, or a prank, or total incompetence. You just don't keep reading after that line: there are plenty of recipes around, it's not worth your time to try to figure out why this one is defective.

70 points by Smaug123 3y ago

eru 3y ago

Wow, hypothesis seems to have reached 4% of Python users? That's actually pretty impressive.

throwup238 3y ago
That's the same percentage of users who have Python package management figured out!
IshKebab 3y ago
*of Python users that completed the Python Jetbrains survey.
- OJFord 3y ago
  I expect there's more data science conda etc. users than users to whom 'Jetbrains survey' even registers. Not to be negative about hypothesis though, I like it, tried to help increase adoption at my previous job after a colleague introduced me, but that was shortly after Pycon UK 2018/19 and I haven't used it since. Not that I don't think we'd benefit from it, just probably lower hanging fruit atm.
  - IshKebab 3y ago
    Yeah most Python users don't even use static type annotations yet... Property testing is ambitious!

eru 3y ago

I found that writing new property based tests was a skill I picked up relatively quickly. But learning how to retro-fit existing tests was a whole 'nother skill that I had to learn almost independently afterwards.

dikei 3y ago

TIL: Jane Street use OCalm in production

Tomte 3y ago
They maintain their own standard library for Ocaml, don‘t they?
- ReleaseCandidat 3y ago
  Not just the standard library, but a whole ecosystem around it. And their own compiler. https://github.com/ocaml-flambda/ocaml-jst https://github.com/ocaml-flambda/ocaml-jst

jiehong 3y ago

While agreeing with the results of this article, I’ve found that convincing other developers of writing test with properties isn’t very easy: coming up with good properties is not always trivial. Here is an informal testing maturity ladder in increasing order: - code can only be tested with an integration test - code is tested by comparing the stdout with an earlier version (hopefully it’s deterministic!) - code is retrofitted with tests in mind - code is written to be testable first, maybe with lots of mocks - code is testable, and pure functions are tested with unit tests, leaving the rest to integration tests. Fewer mocks, some stubs. - property based tests, assuming the unit tests are fast in the first place - fuzzing - mutation based testing Not speaking of formal specs or performance testing or anything else.

pfdietz 3y ago
Where does automatic failing test input reduction fall into that ladder? Or is that assumed as part of PBT.
IshKebab 3y ago
I've never seen anyone do mutation testing with software (it's pretty common for hardware though). Does it require language level support?
- boxed 3y ago
  I'm the author of mutmut, the primary mutation tester for python, so I think I can speak a bit on this. It's quite straight forward to do MT in software. I've done quite a bit of it for specific libraries that I've built (iommi, tri.struct, and internal code). A big advantage in my book of MT over PBT is the much lower cognitive overhead for MT and that you can know you have done it completely. The second may be just a false sense of security or an emotional blanket, but still. I have written about mutation testing a few times: https://kodare.net/2019/04/10/mutation-vs-property-based-testing.html https://kodare.net/2019/04/10/mutation-vs-property-based-tes... https://kodare.net/2018/11/18/the-missing-mutant-a-performance-bug-we-missed.html https://kodare.net/2018/11/18/the-missing-mutant-a-performan... https://kodare.net/2016/12/12/mutation-testing-in-practice.html https://kodare.net/2016/12/12/mutation-testing-in-practice.h... and my talk on mutmut from pycon sweden is on youtube: https://www.youtube.com/watch?v=fZwB1gQBwnU https://www.youtube.com/watch?v=fZwB1gQBwnU

mdiep 3y ago

If you're interested in property-based testing, I highly recommend "How to Specify It!" by John Hughes, which is available both as a talk and as a paper. https://research.chalmers.se/publication/517894/file/517894_Fulltext.pdf https://research.chalmers.se/publication/517894/file/517894_... https://www.youtube.com/watch?v=G0NUOst-53U https://www.youtube.com/watch?v=G0NUOst-53U He gives fantastic guidance about how to write PBTs.

sirwhinesalot 3y ago

Can't read the pdf right now but I'm a big fan of property based testing. One thing I find that people struggle with is coming up with "good properties" to test with. That's the wrong way to think about it. The properties you want to test are the function's contract. Checking that contract is the goal. You can be as specific with the contract as you want. The more specific the more bugs you'll find. Property-based tests are just one way to check the contract, as are hand-written unit tests. You could use a static analyzer or a model checker as well, they're all different approaches to do the same thing. EDIT: by contract I mean the guarantees the function imposes on its output. A contract for a sorting function could be as simple as the length of the output being the same as the input. That's one property. Another is that every element in the output is also in the input. You can go all the way and say that for every element at index i, the element at index i+1 (if any) is larger. But you don't need a perfect contract to start with nor to end with. You can add more guarantees/properties as you wish. The more specific, the better (but also slower) the tests.

crabbone 3y ago

Over time of trying to adopt or even reinvent PBT I discovered that there isn't a good language to formally describe software at the level where it becomes interesting to test it. TLA+ gets somewhat close... but it's both too difficult to write and its hard to adapt to eg. interactive systems. I don't mean to say that PBT is a bad idea. I actually think it's very good. I wish there was a way to make it really useful though. As the paper mentions, PBT in their experience is used for "component testing", which is just another name for "unit testing", which is where automation of this kind isn't all that important. Integration and E2E testing is what's a lot more important, but doesn't really have a way to approaching r.n.

hwayne 3y ago
Have you looked at P? https://p-org.github.io/P/ https://p-org.github.io/P/
- crabbone 3y ago
  No, I haven't. Thanks for the link. I'd have to spend some time reading about it. Ugh... > The P compiler is implemented in C# and hence the tool chain requires dotnet. Nah, sorry. I'm not going to read further about it.
  - neonsunset 3y ago
    Why?
    - crabbone 3y ago
      I started my familiarity with programming using a bunch of Micorosoft's products, but later was able to completely migrate away from them. Today, if I'm required to use a Microsoft's product, it's kind of like asking a North Korean who valiantly fought to get away from that cursed country to go back and live there again. You see, my understanding of what's happening with MS is that their sales department is exceptionally successful, which lowers the requirements for everything else in the product. A particular quality of this sales department is that they rarely sell to individual end users: to be efficient, they sell to people who have absolute control over the end users, but are also incompetent when it comes to software. Think governments, hospitals, schools etc. Another aspect of MS software is that they, internally, don't really allow anything that's not made by MS, and they are big enough for this policy to succeed. The unfortunate side effect of this policy is the Stockholm syndrome: MS developers not only get used to the MS tools, they genuinely seem to like the suffering incurred by using them. Another unfortunate side effect of this policy is that MS developers have no idea what normal users actually like. So, they are kind of like Bender (a robot, who has only a very faint idea of how humans eat) in the Futurama episode, where he enters a cooking contest. He then proceeds to put all kind of inedible components into his meals (but eventually wins by adding some LSD to the mix). So... I will never touch any MS products with a ten feet pole, unless forced to by my employer. And if the employer goes as far as requiring me to use MS products a lot, I'll look for another place to work. My job today is quite meaningless, boring and I don't get to work with things I like or people I respect... but having to use MS tools for work would still be a lot, lot worse. It's a kind of a torture... well, of course, it's an exaggeration: it doesn't cause me physical pain, but it's a kind of frustration that makes me want to punch a wall every few seconds. And to top it: using C# for anything is kind of like... starting a cooking recipe with "put a handful of chopped garlic into the pen". You just know it's either a mistake, or a prank, or total incompetence. You just don't keep reading after that line: there are plenty of recipes around, it's not worth your time to try to figure out why this one is defective.