5 ms·

But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an

But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?

vlovich123 1y ago
Either your code shouldn’t fail or the apostrophe isn’t a valid case. In the former, hypothesis and other similar frameworks are deterministic and will replay the failing test on request or remember the failing tests in a file to rerun in the future to catch regressions. In the latter, you just tell the framework to not generate such values or at least to skip those test cases (better to not generate in terms of testing performance).
- reverius42 1y ago
  I think what they meant is, "won't Hypothesis sometimes fail to generate input with an apostrophe, thus giving you false confidence that your code can handle apostrophes?" I think the answer to this is, in practice, it will not fail to generate such input. My understanding is that it's pretty good at mutating input to cover a large amount of surface area with as few as possible examples.
  - eru 1y ago
    Hypothesis is pretty good, but it's not magic. There's only so many corner cases it can cover in the 200 (or so) cases per tests it's running by default. But by default you also start with a new random seed every time you run the tests, so you can build up more confidence over the older tests and older code, even if you haven't done anything specifically to address this problem. Also, even with Hypothesis you can and should still write specific tests or even just specific generators to cover specific classes of corners cases you are worried about in more detail.
    - thunky 1y ago
      > But by default you also start with a new random seed every time you run the tests, so you can build up more confidence over the older tests and older code Is it common practice to use the same seed and run a ton of tests until you're satisfied it tested it thoroughly? Because I think I would prefer that. With non-deterministic tests I would always wonder if it's going to fail randomly after the code is already in production.
      - bluGill 1y ago
        the more the test runs the less likely it is there is an uncovered case left. So your confidence grows. Remember too anything found before release is something a customer won't find.
      - nyrikki 1y ago
        Saving the seed in the build artifacts/logs has saved a lot of time for me even with tools like faker.
        eru 1y ago
        Yes, you should note the seed you use for a run in the logs, but you should use a new seed each run (unless trying to reproduce a known bug) so you cover more of the search space.
      - vlovich123 1y ago
        You should save the seeds so you can reproduce the issue. But you should let the seed float so that you test as many cases as possible over time.
      - kobebrookskC3 1y ago
        > With non-deterministic tests I would always wonder if it's going to fail randomly after the code is already in production. if you didn't use property-based testing, what are the odds you would've thought of the case?
      - eru 1y ago
jcattle 1y ago
As far as I remember, hypothesis tests smartly. Which means that possibly problematic strings are tested first. It then narrows down which exact part of the tested strings caused the failure. So it might as well just throw the kitchen sink at the function, if it handles that: Great, if not: That string will get narrowed down until you arrive at a minimal set of failing inputs.
- eru 1y ago
  > Which means that possibly problematic strings are tested first. Hypothesis uses a the same probability distribution for all the 200 (or so) random cases it generates for a test. The first case has the same distribution as the 200th. However, Hypothesis gives a pretty large weight in the probability distribution to inputs that are generally 'problematic'. Of course, that's just a heuristic: eg empty lists and 0 and empty strings or strings with apostrophes in them and NaN or infinity often are problematic, but that's just a guess: hypothesis doesn't know anything specific about your code. The heuristics work remarkably well in practice, though. Once Hypothesis has found a failing test case, then it tries to shrink it down.
IanCal 1y ago
Hypothesis keeps a database of failures to use locally and you can add a decorator to mark a specific case that failed. So you run it, see the failure, add it as a specific case and then that’s committed to the codebase. The randomness can bite a little if that test failure happens on an unrelated branch, but it’s not much different to someone just discovering a bug. edit - here's the relevant part of the hypothesis guide https://hypothesis.readthedocs.io/en/latest/tutorial/replaying-failures.html https://hypothesis.readthedocs.io/en/latest/tutorial/replayi...
- theptip 1y ago
  You can also cache the DB across CI runs, which will reduce the randomness (ie failures won’t just disappear between runs).
Balinares 1y ago
No, Hypothesis iterates on test failures to isolate the simplest input that triggers it, so that it can report it to you explicitly.
movpasd 1y ago
You can either use the @example decorator to force Hypothesis to check an edge case you've thought of, or just let Hypothesis uncover the edge cases itself. Hypothesis won't fail a test once and then pass it next time, it keeps track of which examples failed and will re-run them. The generated inputs aren't uniformly randomly distributed and will tend to check pathological cases (complex symbols, NaNs, etc) with priority. You shouldn't think of Hypothesis as a random input generator but as an abstraction over thinking about the input cases. It's not perfect: you'll often need to .map() to get the distribution to reflect the usage of the interface being tested and that requires some knowledge of the shrinking behaviour. However, I was really surprised how easy it was to use.
bluGill 1y ago
If you know it will fail on apostrophe you should have a specific test for that. However if that detail is burried in some function 3 levels deep that you don't even realize is used you wouldn't write the test or handle it even though it matters. This should find those issuses too.
dragonwriter 1y ago
> But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name? If you just naively treat it as a string and let hypothesis generate values, sure. Which is better than if you are doing traditional explicit unit testing and haven’t explicitly defined apostrophes as a concern. If you do have it (or special characters more generally) as a concern, that changes how you specify your test.
chriswarbo 1y ago
Hypothesis is a search algorithm for finding ways that we have misunderstood our code (whether that be in the codebase, or the claims we have made about it in the spec/tests). Conceptually, that's the inverse of "a unit test that sometimes fails": it's a search that sometimes succeeds at finding out that we're wrong. That's infinity% more effective than a search procedure which is hard-coded to check a single example; especially when that example is chosen by the same dev (with the same misconceptions!) that implemented the code-under-test! Now, as for what would actually happen in that situation, when using Hypothesis: - As others have indicated, Hypothesis keeps a database of the failures it's found. Committing that database to your project repo would act like a ratchet: once they've successfully found a problem, they will continue to do so (until it's fixed). - If you don't want to commit your database (to avoid churn/conflicts), then the same would happen per working-copy. This is fine for dev machines, where repo directories can live for a long time; though not so useful for CI, if the working-copy gets nuked after each run. - Even if we disable the failure database, property failures will always show the inputs which caused it (the "counterexample"), along with the random seed used to generate it. Either of those is enough to reproduce that exact run, giving us a ready-made regression test we can copy/paste to make the failure permanent (until it's fixed). As others have said, Hypothesis properties can be decorated with `@example(foo)` to ensure a particular input is always checked. - Even if we disable the failure database, and don't copy the counterexample as a regression test, property checkers like Hypothesis will still "shrink" any counterexamples they find. Your example of apostrophes causing problems is trivial for Hypothesis to shrink, so if/when it happens to find some counterexample, it will always manage to shrink that down to the string "'"; essentially telling us that "employee names fail on apostrophe", which we could either fix right now, or stick in our bug tracker, or whatever.

But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?

vlovich123 1y ago

Either your code shouldn’t fail or the apostrophe isn’t a valid case. In the former, hypothesis and other similar frameworks are deterministic and will replay the failing test on request or remember the failing tests in a file to rerun in the future to catch regressions. In the latter, you just tell the framework to not generate such values or at least to skip those test cases (better to not generate in terms of testing performance).

reverius42 1y ago
I think what they meant is, "won't Hypothesis sometimes fail to generate input with an apostrophe, thus giving you false confidence that your code can handle apostrophes?" I think the answer to this is, in practice, it will not fail to generate such input. My understanding is that it's pretty good at mutating input to cover a large amount of surface area with as few as possible examples.
- eru 1y ago
  Hypothesis is pretty good, but it's not magic. There's only so many corner cases it can cover in the 200 (or so) cases per tests it's running by default. But by default you also start with a new random seed every time you run the tests, so you can build up more confidence over the older tests and older code, even if you haven't done anything specifically to address this problem. Also, even with Hypothesis you can and should still write specific tests or even just specific generators to cover specific classes of corners cases you are worried about in more detail.
  - thunky 1y ago
    > But by default you also start with a new random seed every time you run the tests, so you can build up more confidence over the older tests and older code Is it common practice to use the same seed and run a ton of tests until you're satisfied it tested it thoroughly? Because I think I would prefer that. With non-deterministic tests I would always wonder if it's going to fail randomly after the code is already in production.
    - bluGill 1y ago
      the more the test runs the less likely it is there is an uncovered case left. So your confidence grows. Remember too anything found before release is something a customer won't find.
    - nyrikki 1y ago
      Saving the seed in the build artifacts/logs has saved a lot of time for me even with tools like faker.
      - eru 1y ago
        Yes, you should note the seed you use for a run in the logs, but you should use a new seed each run (unless trying to reproduce a known bug) so you cover more of the search space.
    - vlovich123 1y ago
      You should save the seeds so you can reproduce the issue. But you should let the seed float so that you test as many cases as possible over time.
    - kobebrookskC3 1y ago
      > With non-deterministic tests I would always wonder if it's going to fail randomly after the code is already in production. if you didn't use property-based testing, what are the odds you would've thought of the case?
    - eru 1y ago

jcattle 1y ago

As far as I remember, hypothesis tests smartly. Which means that possibly problematic strings are tested first. It then narrows down which exact part of the tested strings caused the failure. So it might as well just throw the kitchen sink at the function, if it handles that: Great, if not: That string will get narrowed down until you arrive at a minimal set of failing inputs.

eru 1y ago
> Which means that possibly problematic strings are tested first. Hypothesis uses a the same probability distribution for all the 200 (or so) random cases it generates for a test. The first case has the same distribution as the 200th. However, Hypothesis gives a pretty large weight in the probability distribution to inputs that are generally 'problematic'. Of course, that's just a heuristic: eg empty lists and 0 and empty strings or strings with apostrophes in them and NaN or infinity often are problematic, but that's just a guess: hypothesis doesn't know anything specific about your code. The heuristics work remarkably well in practice, though. Once Hypothesis has found a failing test case, then it tries to shrink it down.

IanCal 1y ago

Hypothesis keeps a database of failures to use locally and you can add a decorator to mark a specific case that failed. So you run it, see the failure, add it as a specific case and then that’s committed to the codebase. The randomness can bite a little if that test failure happens on an unrelated branch, but it’s not much different to someone just discovering a bug. edit - here's the relevant part of the hypothesis guide https://hypothesis.readthedocs.io/en/latest/tutorial/replaying-failures.html https://hypothesis.readthedocs.io/en/latest/tutorial/replayi...

theptip 1y ago
You can also cache the DB across CI runs, which will reduce the randomness (ie failures won’t just disappear between runs).

Balinares 1y ago

No, Hypothesis iterates on test failures to isolate the simplest input that triggers it, so that it can report it to you explicitly.

movpasd 1y ago

You can either use the @example decorator to force Hypothesis to check an edge case you've thought of, or just let Hypothesis uncover the edge cases itself. Hypothesis won't fail a test once and then pass it next time, it keeps track of which examples failed and will re-run them. The generated inputs aren't uniformly randomly distributed and will tend to check pathological cases (complex symbols, NaNs, etc) with priority. You shouldn't think of Hypothesis as a random input generator but as an abstraction over thinking about the input cases. It's not perfect: you'll often need to .map() to get the distribution to reflect the usage of the interface being tested and that requires some knowledge of the shrinking behaviour. However, I was really surprised how easy it was to use.

bluGill 1y ago

If you know it will fail on apostrophe you should have a specific test for that. However if that detail is burried in some function 3 levels deep that you don't even realize is used you wouldn't write the test or handle it even though it matters. This should find those issuses too.

dragonwriter 1y ago

> But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name? If you just naively treat it as a string and let hypothesis generate values, sure. Which is better than if you are doing traditional explicit unit testing and haven’t explicitly defined apostrophes as a concern. If you do have it (or special characters more generally) as a concern, that changes how you specify your test.

chriswarbo 1y ago

Hypothesis is a search algorithm for finding ways that we have misunderstood our code (whether that be in the codebase, or the claims we have made about it in the spec/tests). Conceptually, that's the inverse of "a unit test that sometimes fails": it's a search that sometimes succeeds at finding out that we're wrong. That's infinity% more effective than a search procedure which is hard-coded to check a single example; especially when that example is chosen by the same dev (with the same misconceptions!) that implemented the code-under-test! Now, as for what would actually happen in that situation, when using Hypothesis: - As others have indicated, Hypothesis keeps a database of the failures it's found. Committing that database to your project repo would act like a ratchet: once they've successfully found a problem, they will continue to do so (until it's fixed). - If you don't want to commit your database (to avoid churn/conflicts), then the same would happen per working-copy. This is fine for dev machines, where repo directories can live for a long time; though not so useful for CI, if the working-copy gets nuked after each run. - Even if we disable the failure database, property failures will always show the inputs which caused it (the "counterexample"), along with the random seed used to generate it. Either of those is enough to reproduce that exact run, giving us a ready-made regression test we can copy/paste to make the failure permanent (until it's fixed). As others have said, Hypothesis properties can be decorated with `@example(foo)` to ensure a particular input is always checked. - Even if we disable the failure database, and don't copy the counterexample as a regression test, property checkers like Hypothesis will still "shrink" any counterexamples they find. Your example of apostrophes causing problems is trivial for Hypothesis to shrink, so if/when it happens to find some counterexample, it will always manage to shrink that down to the string "'"; essentially telling us that "employee names fail on apostrophe", which we could either fix right now, or stick in our bug tracker, or whatever.