4 ms·

I love the concept of hypothesis, but I struggle with finding a real life use case. I just can't formulate my assertions, I don't know what to put in them. uni

I love the concept of hypothesis, but I struggle with finding a real life use case. I just can't formulate my assertions, I don't know what to put in them. unit tests are easy since I know what the code does, and I can just tell it what to do and what I expect. But with hypothesis, I have to find some kind of general property, which I have a hard time to do. Any tips, or materials I can use ? P.S: I read your test.py, and see you use a rule base state machine. I've never seen that in any tutorial on hypothesis I read. What does it do ? How do you use it ?

felixyz 8y ago
You might find these useful: https://hypothesis.works/articles/rule-based-stateful-testing/ https://hypothesis.works/articles/rule-based-stateful-testin... http://propertesting.com/book_stateful_properties.html http://propertesting.com/book_stateful_properties.html For a long time, I felt the same way you describe: all introductions to property-based testing show you things like testing a function that reverses a list and yes it all looks very useful and nifty, but it's hard to imagine how to formulate interesting variants for more complex code. These two posts were the first to really give me a fuller picture and lots of practical strategies: https://fsharpforfunandprofit.com/posts/property-based-testing/ https://fsharpforfunandprofit.com/posts/property-based-testi... https://fsharpforfunandprofit.com/posts/property-based-testing-2/ https://fsharpforfunandprofit.com/posts/property-based-testi... (In general, everything I've ready/watched by Scott Wlaschin has been super helpful, and I've never written a line of F#.)
- sametmax 8y ago
  Thanks a lot. This is why I still use HN :)
  - felixyz 8y ago
    This was my very first comment on HN. You made my day :)
    - sametmax 8y ago
      Just read it. Never heard of fsharpforfunandprofit before. Never touched f# either. But damn, that's good.
    - kragen 8y ago
      Welcome!
    - thefifthsetpin 8y ago
      Welcome!
- curuinor 8y ago
  fred hebert (monocqc) has his actual book out just now on Amazon, iirc https://www.amazon.com/gp/product/1680506218 https://www.amazon.com/gp/product/1680506218
jononor 8y ago
How I often approach this is to 1) write a couple of unit tests 2) refactor unit tests into data-driven tests, using pytest.mark.parametrize 3) use hypothesis to generate the data. I very often get to 2, but not always to 3. The most basic things to check - No valid inputs give an error - All invalid inputs give a error These are enhanced if you sprinkle your code with assertions. Writing assertions inside the code is good training to identify. Something supposed always non-negative, check it. Something always supposed to have more than 0 entries, check it. Something supposed to be sorted, check a[0] <= a[1] and a[-2] <= a[-1]. Often you can test symmetry type properties - deserialize(serialize(in)) == in. - init(); connect(); disconnect(); -> state identical as just init(); Or you can use an 'oracle', something external that tells you whether the output is correct or not. - allTricksOptimized(in) == simpleReferenceImplementation(in) - myImplementation(in) == thirdparty.implementation(in)
- sametmax 8y ago
  Thanks. felixyz links did contains all those, but I'll remember the trick about pytest.mark.parametrize as a step toward hypothesis.
kragen 8y ago
Well, normally Hypothesis just calls a function with some random arguments and checks that it "works", operationalized as "doesn't crash". But that's inadequate for testing stateful APIs, in which calling a single function isn't useful — for stateful APIs, you need to go through a series of calls, not just a single call. So the rule-based state machine thing gives you a way to describe the valid series of calls, and Hypothesis then generates 100 random sequences according to your spec and verifies that they all work. David wrote the article about it in 2017 that felixyz already linked: https://hypothesis.works/articles/rule-based-stateful-testing/ https://hypothesis.works/articles/rule-based-stateful-testin... In many cases, though, it's better if you can refactor your code to support a stateless interface, and then you can write simple single-step function tests. How do you know what to test? There's a whole spectrum. Just testing that valid inputs don't crash is often pretty useful. (If you're not sure what valid inputs would be, maybe try writing the README before the tests.) In the case of Dumpulse, I went to the other extreme; its actual semantics are extremely dumb (thus the name), so the test suite includes a reimplementation in Python of the intended semantics, and the test verifies that the implementation in C has the same behavior. (But in constant space and strictly limited runtime, which the Hypothesis test doesn't check.) I picked a few functions and classes at random to see what kind of testable properties I could find: - sympy.polys.rationaltools.together: this function gives a testable assertion in its docstring: "``apart(together(expr))`` should return expr unchanged." That's an easy property to test. Also, though, its output expression is intended to be equivalent to its input expression, so if you put together some kind of generator of expressions (which is actually the hard part in this context), you could verify that together(expr) always evaluates to the numerically same value as expr. - horizons.world.buildability.settlementcache.SettlementBuildabilityCache: this seems to be a storage system that caches some data and then incrementally updates the cache as the data is modified. The desired property seems to be that adding some data to the cache, then modifying it, gives you the same results as you would have gotten just by adding the modified data to an empty cache. Also I think there's a serialization aspect, so you could check to see if loading the cache from a binary blob gives you the same results as just creating it in memory. - statsmodels.datasets.macrodata.data.load: this function loads a CSV file from disk. Really all you can test about it, without interposition at the filesystem layer, is that it returns some data. It doesn't really take any inputs. A lot of installation problems could cause it to throw an error, though, so that would be a useful test in some circumstances. - sre_constants: if executed as a script, this program should generate a syntactically valid .h file in sre_constants.h. You should be able to feed that to a C compiler to verify that, but if you care about that, your build system is probably already doing it! Do you want to point me at some code of yours so I can see what kind of useful property-based tests could be written for it?

felixyz 8y ago

You might find these useful: https://hypothesis.works/articles/rule-based-stateful-testing/ https://hypothesis.works/articles/rule-based-stateful-testin... http://propertesting.com/book_stateful_properties.html http://propertesting.com/book_stateful_properties.html For a long time, I felt the same way you describe: all introductions to property-based testing show you things like testing a function that reverses a list and yes it all looks very useful and nifty, but it's hard to imagine how to formulate interesting variants for more complex code. These two posts were the first to really give me a fuller picture and lots of practical strategies: https://fsharpforfunandprofit.com/posts/property-based-testing/ https://fsharpforfunandprofit.com/posts/property-based-testi... https://fsharpforfunandprofit.com/posts/property-based-testing-2/ https://fsharpforfunandprofit.com/posts/property-based-testi... (In general, everything I've ready/watched by Scott Wlaschin has been super helpful, and I've never written a line of F#.)

sametmax 8y ago
Thanks a lot. This is why I still use HN :)
- felixyz 8y ago
  This was my very first comment on HN. You made my day :)
  - sametmax 8y ago
    Just read it. Never heard of fsharpforfunandprofit before. Never touched f# either. But damn, that's good.
  - kragen 8y ago
    Welcome!
  - thefifthsetpin 8y ago
    Welcome!
curuinor 8y ago
fred hebert (monocqc) has his actual book out just now on Amazon, iirc https://www.amazon.com/gp/product/1680506218 https://www.amazon.com/gp/product/1680506218

jononor 8y ago

How I often approach this is to 1) write a couple of unit tests 2) refactor unit tests into data-driven tests, using pytest.mark.parametrize 3) use hypothesis to generate the data. I very often get to 2, but not always to 3. The most basic things to check - No valid inputs give an error - All invalid inputs give a error These are enhanced if you sprinkle your code with assertions. Writing assertions inside the code is good training to identify. Something supposed always non-negative, check it. Something always supposed to have more than 0 entries, check it. Something supposed to be sorted, check a[0] <= a[1] and a[-2] <= a[-1]. Often you can test symmetry type properties - deserialize(serialize(in)) == in. - init(); connect(); disconnect(); -> state identical as just init(); Or you can use an 'oracle', something external that tells you whether the output is correct or not. - allTricksOptimized(in) == simpleReferenceImplementation(in) - myImplementation(in) == thirdparty.implementation(in)

sametmax 8y ago
Thanks. felixyz links did contains all those, but I'll remember the trick about pytest.mark.parametrize as a step toward hypothesis.

kragen 8y ago

Well, normally Hypothesis just calls a function with some random arguments and checks that it "works", operationalized as "doesn't crash". But that's inadequate for testing stateful APIs, in which calling a single function isn't useful — for stateful APIs, you need to go through a series of calls, not just a single call. So the rule-based state machine thing gives you a way to describe the valid series of calls, and Hypothesis then generates 100 random sequences according to your spec and verifies that they all work. David wrote the article about it in 2017 that felixyz already linked: https://hypothesis.works/articles/rule-based-stateful-testing/ https://hypothesis.works/articles/rule-based-stateful-testin... In many cases, though, it's better if you can refactor your code to support a stateless interface, and then you can write simple single-step function tests. How do you know what to test? There's a whole spectrum. Just testing that valid inputs don't crash is often pretty useful. (If you're not sure what valid inputs would be, maybe try writing the README before the tests.) In the case of Dumpulse, I went to the other extreme; its actual semantics are extremely dumb (thus the name), so the test suite includes a reimplementation in Python of the intended semantics, and the test verifies that the implementation in C has the same behavior. (But in constant space and strictly limited runtime, which the Hypothesis test doesn't check.) I picked a few functions and classes at random to see what kind of testable properties I could find: - sympy.polys.rationaltools.together: this function gives a testable assertion in its docstring: "``apart(together(expr))`` should return expr unchanged." That's an easy property to test. Also, though, its output expression is intended to be equivalent to its input expression, so if you put together some kind of generator of expressions (which is actually the hard part in this context), you could verify that together(expr) always evaluates to the numerically same value as expr. - horizons.world.buildability.settlementcache.SettlementBuildabilityCache: this seems to be a storage system that caches some data and then incrementally updates the cache as the data is modified. The desired property seems to be that adding some data to the cache, then modifying it, gives you the same results as you would have gotten just by adding the modified data to an empty cache. Also I think there's a serialization aspect, so you could check to see if loading the cache from a binary blob gives you the same results as just creating it in memory. - statsmodels.datasets.macrodata.data.load: this function loads a CSV file from disk. Really all you can test about it, without interposition at the filesystem layer, is that it returns some data. It doesn't really take any inputs. A lot of installation problems could cause it to throw an error, though, so that would be a useful test in some circumstances. - sre_constants: if executed as a script, this program should generate a syntactically valid .h file in sre_constants.h. You should be able to feed that to a C compiler to verify that, but if you care about that, your build system is probably already doing it! Do you want to point me at some code of yours so I can see what kind of useful property-based tests could be written for it?