3 ms·

Very interesting. Previously I've seen two similar ideas: 1. Reduce the test data by property-based tests' auto shrinking a là QuickCheck [1] 2. Reduce the ch

by crvdgc 2y ago

Very interesting. Previously I've seen two similar ideas: 1. Reduce the test data by property-based tests' auto shrinking a là QuickCheck [1] 2. Reduce the change set to a codebase, a là delta debugging [2] The reducer is able to do both at the same time in a pretty generic way, with some restrictions on the expressivity. Seems to be a good fit for end-to-end tests. [1]: https://www.cse.chalmers.se/~rjmh/QuickCheck/ https://www.cse.chalmers.se/~rjmh/QuickCheck/ [2]: https://dl.acm.org/doi/10.1145/318774.318946 https://dl.acm.org/doi/10.1145/318774.318946

eru 2y ago
Btw, Python's Hypothesis is a lot better at shrinking than QuickCheck. Not only do you get shrinking 'for free', Hypothesis's shrinker is also smarter at finding opportunities to shrink.
- one-punch 2y ago
  Do you have examples where Python’s Hypothesis (and its shrinking) works better than Haskell’s QuickCheck? This would let us improve Haskell’s QuickCheck. My understanding is that Haskell’s QuickCheck can shrink functions, and with parametric polymorphism, can work with vector types and more [1], which would be difficult if not impossible in Python. And in general, the base types covered by the shrinking in Python’s Hypothesis should have analogues by the shrinking in Haskell’s QuickCheck. And with deriving, Haskell’s QuickCheck should get shrinking ‘for free’ as in Python’s Hypothesis. Also, Haskell’s QuickCheck extends to stateful and parallel checking, which is not supported in Python [2]. [1]: https://library.mlabs.city/mastering-quickcheck https://library.mlabs.city/mastering-quickcheck "Mastering QuickCheck: Advanced yet Practical Techniques for Property-Based Testing" [2]: https://stevana.github.io/the_sad_state_of_property-based_testing_libraries.html#a-survey-of-property-based-testing-libraries https://stevana.github.io/the_sad_state_of_property-based_te... "The sad state of property-based testing libraries: A survey of property-based testing libraries"
  - eru 2y ago
    I have to admit the last time I seriously worked with QuickCheck was quite a while ago. > My understanding is that Haskell’s QuickCheck can shrink functions [...] Are you talking about shrinking a value of type (a -> b) (for some concrete a and b)? In Hypothesis, if you write a generator that spits out functions, you get a shrinker for free. So that shrinker will shrink functions. As far as I can tell, QuickCheck's shrinking is based on values. I just checked, the type is still `shrink :: a -> [a]`, so that can't have changed. In Hypothesis shrinking is based on the generation process. In Hypothesis generators act a bit like 'Functors' in Haskell, ie you can map over them. (You can also filter and `<>` and do monadic binds etc.) So in Hypothesis you can take a generator that produces integers, and create one that produces only even numbers by (in Haskell terms) mapping (2) over it. Now, the shrinker you get for free will also only produce even numbers. That's not possible (or at least not easily possible) in Haskell. You would have to define at least a newtype wrapper. Have a look at https://hypothesis.works/articles/compositional-shrinking/ https://hypothesis.works/articles/compositional-shrinking/ and https://hypothesis.works/articles/integrated-shrinking/ https://hypothesis.works/articles/integrated-shrinking/ for some more write-up, especially about how Hypothesis can preserve information even when shrinking past a monadic bind. If you follow some links in the two articles above, you land at https://github.com/icicle-lang/disorder.hs-ambiata/tree/master/disorder-jack https://github.com/icicle-lang/disorder.hs-ambiata/tree/mast... which is a Haskell library that shares a few design decisions with Hypothesis. Apart from shrinking, Haskell's QuickCheck can also learn a lot from Hypothesis in terms of floating point number generation. The last time I used QuickCheck seriously, they basically generated floats by converting from a sort-of uniformly sampled fractional number. I see that this commit https://github.com/nick8325/quickcheck/commit/07942642d7987b073a107863b68ac2f75013076b https://github.com/nick8325/quickcheck/commit/07942642d7987b... seems to have made float generation a lot smarter! Hypothesis is especially 'nasty' when it generates floats. It has a good chunk of probability allocated to things like various NaNs, infinities, sub-normal numbers, zero, one float past zero, etc, plus of course some probability mass on uniformly chosen floats. See https://hypothesis.readthedocs.io/en/latest/data.html#hypothesis.strategies.floats https://hypothesis.readthedocs.io/en/latest/data.html#hypoth... for the docs, but you should check out the code, too.
    - 2y ago

by crvdgc 2y ago

eru 2y ago

Btw, Python's Hypothesis is a lot better at shrinking than QuickCheck. Not only do you get shrinking 'for free', Hypothesis's shrinker is also smarter at finding opportunities to shrink.

one-punch 2y ago
Do you have examples where Python’s Hypothesis (and its shrinking) works better than Haskell’s QuickCheck? This would let us improve Haskell’s QuickCheck. My understanding is that Haskell’s QuickCheck can shrink functions, and with parametric polymorphism, can work with vector types and more [1], which would be difficult if not impossible in Python. And in general, the base types covered by the shrinking in Python’s Hypothesis should have analogues by the shrinking in Haskell’s QuickCheck. And with deriving, Haskell’s QuickCheck should get shrinking ‘for free’ as in Python’s Hypothesis. Also, Haskell’s QuickCheck extends to stateful and parallel checking, which is not supported in Python [2]. [1]: https://library.mlabs.city/mastering-quickcheck https://library.mlabs.city/mastering-quickcheck "Mastering QuickCheck: Advanced yet Practical Techniques for Property-Based Testing" [2]: https://stevana.github.io/the_sad_state_of_property-based_testing_libraries.html#a-survey-of-property-based-testing-libraries https://stevana.github.io/the_sad_state_of_property-based_te... "The sad state of property-based testing libraries: A survey of property-based testing libraries"
- eru 2y ago
  I have to admit the last time I seriously worked with QuickCheck was quite a while ago. > My understanding is that Haskell’s QuickCheck can shrink functions [...] Are you talking about shrinking a value of type (a -> b) (for some concrete a and b)? In Hypothesis, if you write a generator that spits out functions, you get a shrinker for free. So that shrinker will shrink functions. As far as I can tell, QuickCheck's shrinking is based on values. I just checked, the type is still `shrink :: a -> [a]`, so that can't have changed. In Hypothesis shrinking is based on the generation process. In Hypothesis generators act a bit like 'Functors' in Haskell, ie you can map over them. (You can also filter and `<>` and do monadic binds etc.) So in Hypothesis you can take a generator that produces integers, and create one that produces only even numbers by (in Haskell terms) mapping (2) over it. Now, the shrinker you get for free will also only produce even numbers. That's not possible (or at least not easily possible) in Haskell. You would have to define at least a newtype wrapper. Have a look at https://hypothesis.works/articles/compositional-shrinking/ https://hypothesis.works/articles/compositional-shrinking/ and https://hypothesis.works/articles/integrated-shrinking/ https://hypothesis.works/articles/integrated-shrinking/ for some more write-up, especially about how Hypothesis can preserve information even when shrinking past a monadic bind. If you follow some links in the two articles above, you land at https://github.com/icicle-lang/disorder.hs-ambiata/tree/master/disorder-jack https://github.com/icicle-lang/disorder.hs-ambiata/tree/mast... which is a Haskell library that shares a few design decisions with Hypothesis. Apart from shrinking, Haskell's QuickCheck can also learn a lot from Hypothesis in terms of floating point number generation. The last time I used QuickCheck seriously, they basically generated floats by converting from a sort-of uniformly sampled fractional number. I see that this commit https://github.com/nick8325/quickcheck/commit/07942642d7987b073a107863b68ac2f75013076b https://github.com/nick8325/quickcheck/commit/07942642d7987b... seems to have made float generation a lot smarter! Hypothesis is especially 'nasty' when it generates floats. It has a good chunk of probability allocated to things like various NaNs, infinities, sub-normal numbers, zero, one float past zero, etc, plus of course some probability mass on uniformly chosen floats. See https://hypothesis.readthedocs.io/en/latest/data.html#hypothesis.strategies.floats https://hypothesis.readthedocs.io/en/latest/data.html#hypoth... for the docs, but you should check out the code, too.
  - 2y ago