3 ms·

For instance, in traditional tests you may need two test cases for some behaviour. One is for usual inputs and one for some edge case. With Hypothesis, you can

For instance, in traditional tests you may need two test cases for some behaviour. One is for usual inputs and one for some edge case. With Hypothesis, you can combine those two tests into one, by letting Hypothesis generate the inputs for you. Hypothesis is pretty smart to find edge cases that you usually don't think of. Overall, it reduce my time to write tests by at least half, maybe 75% even. The caveats are: 1/ Tests may take visibly longer to run. 2/ If you have setup/teardown or fixtures that you want to repeat each generated input, you have to do this manually. This is imo the biggest weak point of Hypothesis right now.

andreareina 6y ago
I like pytest for the ergonomics of its static setup/teardown/fixtures (plus not having to stuff everything into a unittest.TestCase). Unless you mean using generated input in those bits? In which case yeah, not ideal. At $dayjob we do some chonky processing; generating new input for each property we're testing is just barely ok right now, but that's probably not going to hold for very long. [1] https://github.com/pytest-dev/pytest-subtests https://github.com/pytest-dev/pytest-subtests
- nhgiang 6y ago
  Sorry for not being clear. This is what I was talking about: https://github.com/pytest-dev/pytest/issues/916 https://github.com/pytest-dev/pytest/issues/916 It is a known issue. Basically your fixture/setup/teardown, which you expect should run for every example generated by Hypothesis, actually only run once.
  - andreareina 6y ago
    Oh yeah I remember that. Hasn't been an issue for me, thankfully.
carver 6y ago
> Hypothesis is pretty smart to find edge cases that you usually don't think of. This is worth reemphasizing. For certain kinds of hard problems, I've found it really valuable for building confidence in an implementation. If you happen to have a reference implementation, with a small surface API, it can be a very fast way to write tests: tell hypothesis to input anything it wants and validate that both libraries produce the same result. It's not magic, you will probably still have to give hints about how to build interesting inputs, but still a very quick way to get a lot of coverage. (Also be sure to crank up the number of examples to build more confidence, 100 is often not enough to really explore an input space).

andreareina 6y ago

I like pytest for the ergonomics of its static setup/teardown/fixtures (plus not having to stuff everything into a unittest.TestCase). Unless you mean using generated input in those bits? In which case yeah, not ideal. At $dayjob we do some chonky processing; generating new input for each property we're testing is just barely ok right now, but that's probably not going to hold for very long. [1] https://github.com/pytest-dev/pytest-subtests https://github.com/pytest-dev/pytest-subtests

nhgiang 6y ago
Sorry for not being clear. This is what I was talking about: https://github.com/pytest-dev/pytest/issues/916 https://github.com/pytest-dev/pytest/issues/916 It is a known issue. Basically your fixture/setup/teardown, which you expect should run for every example generated by Hypothesis, actually only run once.
- andreareina 6y ago
  Oh yeah I remember that. Hasn't been an issue for me, thankfully.

carver 6y ago

> Hypothesis is pretty smart to find edge cases that you usually don't think of. This is worth reemphasizing. For certain kinds of hard problems, I've found it really valuable for building confidence in an implementation. If you happen to have a reference implementation, with a small surface API, it can be a very fast way to write tests: tell hypothesis to input anything it wants and validate that both libraries produce the same result. It's not magic, you will probably still have to give hints about how to build interesting inputs, but still a very quick way to get a lot of coverage. (Also be sure to crank up the number of examples to build more confidence, 100 is often not enough to really explore an input space).