6 ms·

What is property based testing?

95 points by hyperpape 10y ago

platz 10y ago
referencing fuzzing so heavily in the definition places a lot of faith in understanding that there are those subtle varieties of fuzzers that can generate structured data/inputs. I didn't know fuzzers did that, so I'd guess that others would miss the nuance there too; and it's a rather large part of property based testing This is a definition that aims to be more "categorically correct" without providing a lot of intuitive explanatory power. (E.g. A monad is a monoid in the category of endofunctors). Maybe that's ok in that was the aim and PBT can't be reduced to a single sentence like a lot of abstract concepts It also assumes a well-defined definition of a fuzzer I was also surprised to not see the word 'invariants' mentioned once. Maybe that's very close to "property", but property has other connotations so it still seems useful
- bbcbasic 10y ago
  To me property based testing is just checking that a function obeys a certain law by throwing lots of random data at it. That means you have to define the law in terms of the input and output. Rather than fix the input like you'd do on a regular test and just assert the output. The hard part is trying to figure out what laws you should expect from your functions.
  - michaelfeathers 10y ago
    I look at property-based testing the same way I look at Design by Contract. It can be used to find errors in existing systems, but it's also valuable when you are designing. Just asking yourself what laws a function should adhere to as you are considering writing it function puts you in a different frame of mind and you can end up with simpler software.
    - nickpsecurity 10y ago
      Yeah, fuzzing is just throwing random data. Property-based is supposed to be more like DbC where it's guided to test the righg things.
  - paulddraper 10y ago
    Maybe you should write a function that does the right thing, and then use it to tell you the correct output :/
    - hyperpape 10y ago
      You may be joking, but this can be a good technique for cases where you have a simple but inefficient solution and a complex optimized solution. Ditto for anything that has a special case that can be easily answered.
      - platz 10y ago
        Also known as an 'oracle'
      - paulddraper 10y ago
        Half-joking. It can be helpful to verify an optimized implementation with a naive one. That's a small minority of testing though. Special casing is effectively traditional testing: input is 1, output is 2.
        nickpsecurity 10y ago
        I did that for high-assurance systems as an equivalence check to ensure optimizing compiler didn't break things. It's also standard in hardware verification where they equivalence check each step from highest-level form to lowest-level form.
    - dbcurtis 10y ago
      Not totally silly. Just last Friday I was writing a Python hypothesis-based test using constrained-random stimulus generation. The target function under test is rather complicated because it is coded to be space and time efficient. It is not so easy to read and reason about, surprise, surprise. But there is a straight-forward, easy to ready, while-loops-and-simple-if's implementation that is slow and inefficient, but it's not too hard to convince yourself it produces the correct answer. So I use that in my test driver to generated expected results. It's certainly possible for both implementations to contain bugs, but extremely unlikely for them both to report the same wrong answer for the same stimulus.
KurtMueller 10y ago
Anybody here do any property-based testing with Ruby? If so, what do you use and what do you test with it?
- hyperpape 10y ago
  I don't do Ruby, but the author of this piece has an article on property testing in many different languages, including Ruby: http://hypothesis.works/articles/quickcheck-in-every-language/ http://hypothesis.works/articles/quickcheck-in-every-languag....
santoriv 10y ago
Here's a question that's tangentially related to the article in that it deals with property based testing. I'm curious what the software veterans thing about this: Should you write property based tests of a third party custom api if you are on a short term contract and pretty sure that no one is going to maintain the testing suite? On my last contract the product quote api and the product order customization UI I was writing were being co-developed at the same time by two different parties. I found a lot of errors in the schema/third party endpoint by randomly generating lots of product orders and tossing them at the endpoint. While this helped a lot in development, it took a fair amount of time to write. Knowing the parties involved I'm 95% certain that the tests were never run again and probably ignored after I left. Is it more cost conscious to just not write the tests and slog through to the hand-off? I realize that this may be a little hard to answer without knowing the project intimately. I just sometimes wonder if I write tests too often.
- deleted 10y ago
  [deleted]
- dorian-graph 10y ago
  I've thought about this same thing before, also in the context of not contractual work, but longterm. At least 1 thing is that if something goes wrong later on, you can point to the tests to show that _your work_ is no the culprit. I'm curious to hear what the more experienced people have to say on your question.
- seanwilson 10y ago
  > Should you write property based tests of a third party custom api if you are on a short term contract and pretty sure that no one is going to maintain the testing suite? That sounds like you're doing their work for them. If I'm accepting risk in a contract, my major assumption would be the third party libraries I'm using are fit for purpose. You surely can't be held liable if the other contractors aren't doing their work to an acceptable standard.
  - nickpsecurity 10y ago
    It's not quite that simple. Much of the problems of Windows reliability was from 3rd party drivers and apps (esp killer apps). They couldn't ditch the hardware or app vendors because their customers demanded them. So, Windows source was littered with all kinds of provisions to basically work around all that. They eventually forced driver analysis and safer language onto the ecosystem when that was too much. But a key part of their path to dominance and backward compatibility was working around known issues in 3rd party code. Avoid this when possible obviously. However, it's sometimes a good idea when it's for tie-in to ecosystem that has benefits worth it.
seanwilson 10y ago
I'm really surprised QuickCheck and similar tools aren't used much outside of languages like Haskell. QuickCheck is easy to implement, simple to learn, the test code is concise and it's really good at finding obscure bugs.
- kmicklas 10y ago
  By the time you discover QuickCheck, you might as well just use Haskell :)
  - seanwilson 10y ago
    Haskell isn't practical for a lot of projects though just based on where your code has to run and what existing libraries you want to use.
    - willtim 10y ago
      Yes. But this is more the problem of vendor lock-in than anything to do with Haskell.
      - seanwilson 10y ago
        Yeah, not knocking the language. It's frustrating how it takes so long for good programming language ideas to defuse into the mainstream. It's crazy how strong type systems have been around for literally decades and we're still using weakly typed dynamic languages for example. TypeScript is a decent compromise for me at the moment despite being a fan of actual functional languages.
vram22 10y ago
Is property-based testing related to equivalence (class) partitioning? https://en.wikipedia.org/wiki/Equivalence_partitioning https://en.wikipedia.org/wiki/Equivalence_partitioning
kristianp 10y ago
The article mentions that smallcheck doesn't qualify as property based testing. However it's hard to see what it is from its description. Any ideas what smallcheck does? > SmallCheck is a testing library that allows to verify properties for all test cases up to some depth. https://hackage.haskell.org/package/smallcheck https://hackage.haskell.org/package/smallcheck
- paradoja 10y ago
  It's like QuickCheck, but instead of testing random values, it tests all values in a certain range. For example, if the function takes an array of 128 bit values, it's test all the possible arrays of length 0, 1, 2... up to the length you want to test for. (Actually, not necessarily all, but all that an exhaustive generator generates; the generator would normally go through all values, but it potentially could only test some values, but would not select them at random).

95 points by hyperpape 10y ago

platz 10y ago

referencing fuzzing so heavily in the definition places a lot of faith in understanding that there are those subtle varieties of fuzzers that can generate structured data/inputs. I didn't know fuzzers did that, so I'd guess that others would miss the nuance there too; and it's a rather large part of property based testing This is a definition that aims to be more "categorically correct" without providing a lot of intuitive explanatory power. (E.g. A monad is a monoid in the category of endofunctors). Maybe that's ok in that was the aim and PBT can't be reduced to a single sentence like a lot of abstract concepts It also assumes a well-defined definition of a fuzzer I was also surprised to not see the word 'invariants' mentioned once. Maybe that's very close to "property", but property has other connotations so it still seems useful

bbcbasic 10y ago
To me property based testing is just checking that a function obeys a certain law by throwing lots of random data at it. That means you have to define the law in terms of the input and output. Rather than fix the input like you'd do on a regular test and just assert the output. The hard part is trying to figure out what laws you should expect from your functions.
- michaelfeathers 10y ago
  I look at property-based testing the same way I look at Design by Contract. It can be used to find errors in existing systems, but it's also valuable when you are designing. Just asking yourself what laws a function should adhere to as you are considering writing it function puts you in a different frame of mind and you can end up with simpler software.
  - nickpsecurity 10y ago
    Yeah, fuzzing is just throwing random data. Property-based is supposed to be more like DbC where it's guided to test the righg things.
- paulddraper 10y ago
  Maybe you should write a function that does the right thing, and then use it to tell you the correct output :/
  - hyperpape 10y ago
    You may be joking, but this can be a good technique for cases where you have a simple but inefficient solution and a complex optimized solution. Ditto for anything that has a special case that can be easily answered.
    - platz 10y ago
      Also known as an 'oracle'
    - paulddraper 10y ago
      Half-joking. It can be helpful to verify an optimized implementation with a naive one. That's a small minority of testing though. Special casing is effectively traditional testing: input is 1, output is 2.
      - nickpsecurity 10y ago
        I did that for high-assurance systems as an equivalence check to ensure optimizing compiler didn't break things. It's also standard in hardware verification where they equivalence check each step from highest-level form to lowest-level form.
  - dbcurtis 10y ago
    Not totally silly. Just last Friday I was writing a Python hypothesis-based test using constrained-random stimulus generation. The target function under test is rather complicated because it is coded to be space and time efficient. It is not so easy to read and reason about, surprise, surprise. But there is a straight-forward, easy to ready, while-loops-and-simple-if's implementation that is slow and inefficient, but it's not too hard to convince yourself it produces the correct answer. So I use that in my test driver to generated expected results. It's certainly possible for both implementations to contain bugs, but extremely unlikely for them both to report the same wrong answer for the same stimulus.

KurtMueller 10y ago

Anybody here do any property-based testing with Ruby? If so, what do you use and what do you test with it?

hyperpape 10y ago
I don't do Ruby, but the author of this piece has an article on property testing in many different languages, including Ruby: http://hypothesis.works/articles/quickcheck-in-every-language/ http://hypothesis.works/articles/quickcheck-in-every-languag....

santoriv 10y ago

Here's a question that's tangentially related to the article in that it deals with property based testing. I'm curious what the software veterans thing about this: Should you write property based tests of a third party custom api if you are on a short term contract and pretty sure that no one is going to maintain the testing suite? On my last contract the product quote api and the product order customization UI I was writing were being co-developed at the same time by two different parties. I found a lot of errors in the schema/third party endpoint by randomly generating lots of product orders and tossing them at the endpoint. While this helped a lot in development, it took a fair amount of time to write. Knowing the parties involved I'm 95% certain that the tests were never run again and probably ignored after I left. Is it more cost conscious to just not write the tests and slog through to the hand-off? I realize that this may be a little hard to answer without knowing the project intimately. I just sometimes wonder if I write tests too often.

deleted 10y ago
[deleted]
dorian-graph 10y ago
I've thought about this same thing before, also in the context of not contractual work, but longterm. At least 1 thing is that if something goes wrong later on, you can point to the tests to show that _your work_ is no the culprit. I'm curious to hear what the more experienced people have to say on your question.
seanwilson 10y ago
> Should you write property based tests of a third party custom api if you are on a short term contract and pretty sure that no one is going to maintain the testing suite? That sounds like you're doing their work for them. If I'm accepting risk in a contract, my major assumption would be the third party libraries I'm using are fit for purpose. You surely can't be held liable if the other contractors aren't doing their work to an acceptable standard.
- nickpsecurity 10y ago
  It's not quite that simple. Much of the problems of Windows reliability was from 3rd party drivers and apps (esp killer apps). They couldn't ditch the hardware or app vendors because their customers demanded them. So, Windows source was littered with all kinds of provisions to basically work around all that. They eventually forced driver analysis and safer language onto the ecosystem when that was too much. But a key part of their path to dominance and backward compatibility was working around known issues in 3rd party code. Avoid this when possible obviously. However, it's sometimes a good idea when it's for tie-in to ecosystem that has benefits worth it.

seanwilson 10y ago

I'm really surprised QuickCheck and similar tools aren't used much outside of languages like Haskell. QuickCheck is easy to implement, simple to learn, the test code is concise and it's really good at finding obscure bugs.

kmicklas 10y ago
By the time you discover QuickCheck, you might as well just use Haskell :)
- seanwilson 10y ago
  Haskell isn't practical for a lot of projects though just based on where your code has to run and what existing libraries you want to use.
  - willtim 10y ago
    Yes. But this is more the problem of vendor lock-in than anything to do with Haskell.
    - seanwilson 10y ago
      Yeah, not knocking the language. It's frustrating how it takes so long for good programming language ideas to defuse into the mainstream. It's crazy how strong type systems have been around for literally decades and we're still using weakly typed dynamic languages for example. TypeScript is a decent compromise for me at the moment despite being a fan of actual functional languages.

vram22 10y ago

Is property-based testing related to equivalence (class) partitioning? https://en.wikipedia.org/wiki/Equivalence_partitioning https://en.wikipedia.org/wiki/Equivalence_partitioning

kristianp 10y ago

The article mentions that smallcheck doesn't qualify as property based testing. However it's hard to see what it is from its description. Any ideas what smallcheck does? > SmallCheck is a testing library that allows to verify properties for all test cases up to some depth. https://hackage.haskell.org/package/smallcheck https://hackage.haskell.org/package/smallcheck

paradoja 10y ago
It's like QuickCheck, but instead of testing random values, it tests all values in a certain range. For example, if the function takes an array of 128 bit values, it's test all the possible arrays of length 0, 1, 2... up to the length you want to test for. (Actually, not necessarily all, but all that an exhaustive generator generates; the generator would normally go through all values, but it potentially could only test some values, but would not select them at random).