3 ms·

Internally, Hypothesis is built on a core engine called Conjecture, which looks more like a byte stream fuzzer than anything else. Hypothesis data generators ar

Internally, Hypothesis is built on a core engine called Conjecture, which looks more like a byte stream fuzzer than anything else. Hypothesis data generators are then effectively parser combinators on top of that fuzzer. This gives Hypothesis a lot more freedom to generically manipulate the data than QuickCheck has, because it has total control of a concrete representation of it. As a result: * Specifying a data generator is a much more declarative process - Hypothesis largely handles size/distribution issues itself. * The fuzzing nature of Hypothesis means that it can do much smarter generation than a normal QuickCheck (though this is more potential than actual right now. The current fuzzer is "pretty good". I'll be starting a PhD soon where I hope to work on making it amazing) * Shrinking is just a built in part of the process and users never have to define their own shrinker. * All examples (both pre and post shrinking) can be serialized because the IR is just bytes, so Hypothesis can replay failing tests automatically without rerunning the shrinking process. This matters both because shrinking tends to be slower than generation in most QuickChecks, but also because in general shrinking is subject to "slippage" where the bugs you find after shrinking are different from the bugs you started with. * Because you can just keep drawing data from the stream, Hypothesis tests can be much more interactive than normal QuickCheck (which does support mixing test execution and generation, but it doesn't work especially well). * Because Hypothesis never needs to touch the values it generates, Hypothesis is much better for mutable data than the approach most QuickCheck takes. * I haven't actually done this yet, but Conjecture is deliberately designed to be quite C like, so "at some point" it will become possible to rewrite the core in C (more likely Rust) and then new Hypotheses will spring up everywhere because 90% of the work can be done by writing bindings to a C library. QuickCheck's approach on the other hand is inherently very tied to the semantics of the host language. The property-based testing libraries for some dynamic languages (test.check for Clojure, the various Erlang ones including QuickCheck) have some of these features, but I think they're more naturally supported in the Hypothesis model, and the Hypothesis model has a lot more room to grow.

feanaro 9y ago
I understand what you mean on most of these, but I'd appreciate some clarification on the following. > The fuzzing nature of Hypothesis means that it can do much smarter generation than a normal QuickCheck (though this is more potential than actual right now. The current fuzzer is "pretty good". I'll be starting a PhD soon where I hope to work on making it amazing) Smarter in what way? > Because you can just keep drawing data from the stream, Hypothesis tests can be much more interactive than normal QuickCheck (which does support mixing test execution and generation, but it doesn't work especially well). What makes QuickCheck's support for this makes it work less than ideal? > Because Hypothesis never needs to touch the values it generates, Hypothesis is much better for mutable data than the approach most QuickCheck takes. What do you mean by not having to "touch" the values it generates?
- DRMacIver 9y ago
  > Smarter in what way? Better at generating values that exhibit interesting behaviour in tests. QuickCheck style testing mostly only works well because it turns out that there are a lot of bugs that are relatively "dense" in the search space of tests, and doesn't do very well at finding hard to reach bugs. This means that some bugs are found with very low probability, which is bad both because you want to find them reliably and because it means that even running your tests for longer is often not enough to find interesting behaviour. (ETA: This mostly applies to Haskell QuickCheck and derivatives. I believe the Quviq Erlang QuickCheck has had a great deal of hand tuning of its generators to get better behaviour, so its data generation is probably strictly better than Hypothesis's in many cases right now. I'm trying to get a generic mechanism for improving things without requiring this hand tuning, so hopefully at some point I'll be able to reverse that situation) Security-oriented fuzzers do a lot of clever things to actually determine the shape of the search space and adapt to it so that they can take advantage of the structure of the program under test so that running for longer gives them more power than just repeatedly trying the same thing over and over again. I'm hoping to incorporate some of those ideas into Hypothesis but don't currently. > What makes QuickCheck's support for this makes it work less than ideal? Mostly that it plays very badly with shrinking (this is better but still not good in test.check and friends) and the API for it is on the clunky side. > What do you mean by not having to "touch" the values it generates? In classic quickcheck, the shrink API is based on taking a value and replacing it with a simpler version of itself. This means that if the value has been mutated (which is mostly not a problem in Haskell, but can be if you're using ioProperty, and is definitely a problem in QuickCheck ports to impure languages) then you run into problems. e.g. a test that appends an element to a variable sized array argument can get the shrinker into an infinite loop. It also means that QuickCheck is limited by the type constraints on the generated values. You can't e.g. do duplicate detection because you aren't constrained to generate values on which that is meaningful. In Hypothesis in comparison everything is based off its IR, so it can do manipulations and comparisons on that, and it doesn't matter what type the generated value is. (Hypothesis also can't do perfect duplicate detection because it has the problem that many IR values may map to the same value, but its duplicate detection still seems to be mostly good enough in practice)

feanaro 9y ago

I understand what you mean on most of these, but I'd appreciate some clarification on the following. > The fuzzing nature of Hypothesis means that it can do much smarter generation than a normal QuickCheck (though this is more potential than actual right now. The current fuzzer is "pretty good". I'll be starting a PhD soon where I hope to work on making it amazing) Smarter in what way? > Because you can just keep drawing data from the stream, Hypothesis tests can be much more interactive than normal QuickCheck (which does support mixing test execution and generation, but it doesn't work especially well). What makes QuickCheck's support for this makes it work less than ideal? > Because Hypothesis never needs to touch the values it generates, Hypothesis is much better for mutable data than the approach most QuickCheck takes. What do you mean by not having to "touch" the values it generates?

DRMacIver 9y ago
> Smarter in what way? Better at generating values that exhibit interesting behaviour in tests. QuickCheck style testing mostly only works well because it turns out that there are a lot of bugs that are relatively "dense" in the search space of tests, and doesn't do very well at finding hard to reach bugs. This means that some bugs are found with very low probability, which is bad both because you want to find them reliably and because it means that even running your tests for longer is often not enough to find interesting behaviour. (ETA: This mostly applies to Haskell QuickCheck and derivatives. I believe the Quviq Erlang QuickCheck has had a great deal of hand tuning of its generators to get better behaviour, so its data generation is probably strictly better than Hypothesis's in many cases right now. I'm trying to get a generic mechanism for improving things without requiring this hand tuning, so hopefully at some point I'll be able to reverse that situation) Security-oriented fuzzers do a lot of clever things to actually determine the shape of the search space and adapt to it so that they can take advantage of the structure of the program under test so that running for longer gives them more power than just repeatedly trying the same thing over and over again. I'm hoping to incorporate some of those ideas into Hypothesis but don't currently. > What makes QuickCheck's support for this makes it work less than ideal? Mostly that it plays very badly with shrinking (this is better but still not good in test.check and friends) and the API for it is on the clunky side. > What do you mean by not having to "touch" the values it generates? In classic quickcheck, the shrink API is based on taking a value and replacing it with a simpler version of itself. This means that if the value has been mutated (which is mostly not a problem in Haskell, but can be if you're using ioProperty, and is definitely a problem in QuickCheck ports to impure languages) then you run into problems. e.g. a test that appends an element to a variable sized array argument can get the shrinker into an infinite loop. It also means that QuickCheck is limited by the type constraints on the generated values. You can't e.g. do duplicate detection because you aren't constrained to generate values on which that is meaningful. In Hypothesis in comparison everything is based off its IR, so it can do manipulations and comparisons on that, and it doesn't matter what type the generated value is. (Hypothesis also can't do perfect duplicate detection because it has the problem that many IR values may map to the same value, but its duplicate detection still seems to be mostly good enough in practice)