3 ms·

I've always liked this concept, but I've only ever seen it demonstrated with simple pure functions like gcd and sort. How practical is generating inputs for pay

• mrkeen 4y ago
  I haven't tried the approach in Python, but example generation is compositional: Generate arbitrary credit cards when you're testing the credit card code. Then later when you're testing a customer code, generate a customer who has 0..n credit cards. "But isn't it hard to set up your input data exactly right to make the test pass?" That's the point! You see what your system does when you give it data you didn't expect to give it.
• imp0cat 4y ago
  Hypothesis will generate a lot of things for you, it can be used for stuff like names, e-mail adresses, passwords, etc... . Instead of having to populate your test data with stuff like John Doe, Doe street 1111, Doe City, you can just use it to generate stuff on the fly. It will then tirelessly try different permutations in the neverending search for more bugs.
  • chriswarbo 4y ago
    Hypothesis also has two different ways to generate data: - One is to write things like in the article: accept data via arguments, and specify a strategy for each up-front. - Another is to be more ad-hoc: use the 'data' strategy to get a data generator which can be called on-the-fly at the point it's needed inside our test. This is useful if our test is complicated, e.g. with a bunch of conditional logic where each branch needs something different; or the data generation depends on some run-time values (e.g. the number of results we got from the database); etc. Hypothesis will still keep track of the random seeds, perform shrinking, etc. as usual. The latter approach is described at https://hypothesis.readthedocs.io/en/latest/data.html#drawing-interactively-in-tests https://hypothesis.readthedocs.io/en/latest/data.html#drawin...
• IanCal 4y ago
  I'll start by just saying I've never introduced property tests without finding a bug in either the specification or implementation. I have repeatedly spent time figuring out where the bug was in my property testing library before realising that it was correct and the obviously correct code I was testing was actually wrong. I often find it easier in tests to start with the default strategies, then in the test construct the world. An example would be instead of creating strategies for user accounts and monetary transfers I made list of (integer, integer, integer) and interpret that as (from_account, to_account, amount). You could have some assertions like (I'm afraid I don't know what benefits adjustment means but I know it's just an example) * This benefits adjustment will always be 0 or positive * It will only ever return a value or raise one of the following explicit exceptions <- "my code doesn't crash" is a good base assertion * Benefits adjustments over longer periods are never less (e.g. adjustment for the time range 2020-2023 >= 2020-2021) * If I create a second organisation with employees, the result of checking the first one never changes. Your tests don't have to check exact results, otherwise you end up rebuilding the logic you're trying to test just to work out what the result should be. You can check general properties that should remain true. You can extend this out as well making it more stateful in a sense. I built a UI library for TVs and programs using it may add and remove elements at any time. A powerful test was 1. Given an empty UI 2. And someone makes a series of API calls where each is [add_element(...), remove_element(...), user_left(), user_right(),...] 3. One and only one element is in focus And another 1. Given an arbitrary UI created with those calls 2. If the user moves right and the focus changes 3. Then when the user moves left they will go back to the item they were previously on It was very hard to write the library such that it didn't do what we wanted and passed these tests. I found a case where we had an explicit unit test and part of the spec that wouldn't pass while these tests were passing, because the spec was inconsistent. These tests I think were actually clearer to write than a series of explicit cases too. > Will it hit the database a hundred extra times to cover combinations that don't matter when I could have just enumerated ones that do? Yes. The benefit is in trying things you think probably work but don't actually know they do, and cases you wouldn't have thought about but are totally possible. They're not a replacement for explicit tests, they're another tool. I have had them failing showing there was a problem in some text extraction, which boiled down to the fact that lowercasing a Turkish I in python at least results in more than one character, so positions in the text after it had been lowercased were not the same as before. I had absolutely not considered that. A finishing thought - if you can't say a general property that should remain true given some arbitrary input, how hard is it to understand what your function will do when you see a call to it?
• epgui 4y ago
  In principle yes it’s worth it even for more complex data. In practice, hypothesis is not good at this (I would argue it is broken currently, as specifications that are valid can fail for… reasons of implementation details) and the maintainers don’t think there’s any problem with this limitation. I’ve had much better luck with generative testing in clojure and haskell.