3 ms·

The tool seems great: It offers a very convenient way to parameterize tests. But the tool is not strictly about property-based testing, right? The only differ

• chriswarbo 4y ago
  The assertions, test suite, etc. are handled by PyTest. Hypothesis is taking care of generating arguments, and supplying them to the test. The '@given' decorator will wrap up the given test, to make a new one which: - Calls the original test multiple times, with randomly generated arguments - If the original test ever fails (throws an exception), the arguments which caused the failure will be "shrunk" to get a list of "smaller" arguments (e.g. smaller numbers; lists with fewer elements, and whose elements have also been shrunk; etc.). The original test will be called with those "smaller" arguments, to see if they also cause a failure: if they all pass (or the "shrunk" list is empty), then the original failing arguments are reported; if the same exception got thrown for some "smaller" arguments, we try again by shrink those into even smaller values (and so on); if a different exception is thrown, both failures are reported. - The wrapped function also keeps a buffer of "notes", to help with debugging failures. Normal 'printf debugging' will be confusing to follow, since the original function will be called many times, so we'll get many copies of the debug output, and it's hard to know which ones came from successes and which ones from failures. If we use the 'note' function, Hypothesis will put them in a buffer, which is reset for each run. When a counterexample/failure is reported, only those notes from that run will be printed. - We can tell Hypothesis to skip certain situations; e.g. the test in the article could have caught ZeroDivisionErrors and told Hypothesis to skip. That doesn't count as a failure, but also doesn't count as a success. The test will just be run again with different arguments. - There is a threshold on the amount of skipped tests; crossing that threshold is treated as a failure, since we aren't seeing enough successful runs to be confident that it's working. - There's also a timeout, to avoid bad interactions between the strategies/generators, tests, skipping, etc. There are a few more things Hypothesis will do, like maintaining a database of previous failures, so it can retry those values as a form of regression test. It's a really nice tool :)
  • pfdietz 4y ago
    Hypothesis also has automatic test input reduction, using a rather clever general purpose mechanism.