9 ms·

Hypothesis: Property-Based Testing for Python

231 points by lwhsiao 1y ago

NortySpock 1y ago
I keep thinking I have a possible use case for property -based testing, and then I am up to my armpits in trying to understand the on-the-ground problem and don't feel like I have time to learn a DSL for describing all possible inputs and outputs when I already had an existing function (the subject-under-test) that I don't understand. So rather than try to learn to black boxes at the same time , I fall back to "several more unit tests to document more edge cases to defensibly guard against" Is there some simple way to describe this defensive programming iteration pattern in Hypothesis? Normally we just null-check and return early and have to deal with the early-return case. How do I quickly write property tests to check that my code handles the most obvious edge cases?
- fwip 1y ago
  I think the easiest way is to start with general properties and general input, and tighten them up as needed. The property might just be "doesn't throw an exception", in some cases. If you find yourself writing several edge cases manually with a common test logic, I think the @example decorator in Hypothesis is a quick way to do that: https://hypothesis.readthedocs.io/en/latest/reference/api.html#explicit-inputs https://hypothesis.readthedocs.io/en/latest/reference/api.ht...
  - NortySpock 1y ago
    Thanks, the "does not throw an exception" property got my mental gears turning in terms of how to get started on this, and from there I can see how one could add a few more properties as one goes along. Appreciate you taking the time to answer.
    - eru 1y ago
      Yes. One of the first things you have to do when writing any property based tests, no matter what you actually want to test, is defining your input generators. And once you have those generators, you might as well throw together the "doesn't crash unexpectedly" test together. That not only tests your code, but also exercises your just written input generators.
- disambiguation 1y ago
  I've only used it once before, not as unit testing, but as stress testing for a new customer facing api. I wanted to say with confidence "this will never throw an NPE". Also the logic was so complex (and the deadline so short) the only reasonable way to test was to generate large amounts of output data and review it manually for anomalies.
- sunshowers 1y ago
  The simplest practical property-based tests are where you serialize some randomly generated data of a particular shape to JSON, then deserialize it, and ensure that the output is the same. A more complex kind of PBT is if you have two implementations of an algorithm or data structure, one that's fast but tricky and the other one slow but easy to verify. (Say, quick sort vs bubble sort.) Generate data or operations randomly and ensure the results are the same.
  - eru 1y ago
    > The simplest practical property-based tests are where you serialize some randomly generated data of a particular shape to JSON, then deserialize it, and ensure that the output is the same. Testing that f(g(x)) == x for all x and some f and g that are supposed to be inverses of each other is a good test, but it's probably not the simplest. The absolute simplest I can think of is just running your functionality on some randomly generated input and seeing that it doesn't crash unexpectedly. For things like sorting, testing against an oracle is great. But even when you don't have an oracle, there's lots of other possibilities: * Test that sorting twice has the same effect as sorting once. * Start with a known already in-order input like [1, 2, 3, ..., n]; shuffle it, and then check that your sorting algorithm re-creates the original. * Check that the output of your sorting algorithm is in-order. * Check that input and output of your sorting algorithm have the same elements in the same multiplicity. (If you don't already have a datastructure / algorithm that does this efficiently, you can probe it with more randomness: create a random input (say a list of numbers), pick a random number X, count how many times X appears in your list (via a linear scan); then check that you get the same count after sorting. * Check that permuting your input doesn't make a difference. * Etc.
    - sunshowers 1y ago
      Speaking for myself — those are definitely all simpler cases, but for me I never found them compelling enough (beyond the "it doesn't crash" property). For me, the simplest case that truly motivated PBT for me was roundtrip serialization. Now I use PBT quite a lot, and most of them are either serialization roundtrip or oracle/model-based tests.
- meejah 1y ago
  Here are some fairly simple examples: testing port parsing https://github.com/meejah/fowl/blob/e8253467d7072cd05f21de7cd6d94cef3fe5977e/src/fowl/test/test_cli.py#L146 https://github.com/meejah/fowl/blob/e8253467d7072cd05f21de7c... ...and https://github.com/magic-wormhole/magic-wormhole/blob/1b4732cc785dc93b4c50f8679e34fb54e99cc32b/src/wormhole/test/dilate/test_subchannel.py#L303 https://github.com/magic-wormhole/magic-wormhole/blob/1b4732... The simplest ones to get started with are "strings", IMO, and also gives you lots of mileage (because it'll definitely test some weird unicode). So, somewhere in your API where you take some user-entered strings -- even something "open ended" like "a name" -- you can make use of Hypothesis to try a few things. This has definitely uncovered unicode bugs for me. Some more complex things can be made with some custom strategies. The most-Hypothesis-heavy tests I've personally worked with are from Magic Folder strategies: https://github.com/tahoe-lafs/magic-folder/blob/main/src/magic_folder/test/strategies.py#L219 https://github.com/tahoe-lafs/magic-folder/blob/main/src/mag... The only real downside is that a Hypothesis-heavy test-suite like the above can take a while to run (but you can instruct it to only produce one example per test). Obviously, one example per test won't catch everything, but is way faster when developing and Hypothesis remembers "bad" examples so if you occasionally do a longer run it'll remember things that caused errors before.
- akshayshah 1y ago
  Sibling comments have already mentioned some common strategies - but if you have half an hour to spare, the property-based testing series on the F# for Fun and Profit blog is well worth your time. The material isn’t really specific to F#. https://fsharpforfunandprofit.com/series/property-based-testing/ https://fsharpforfunandprofit.com/series/property-based-test...
- eru 1y ago
  In addition to what other people have said: > [...] time to learn a DSL for describing all possible inputs and outputs when I already had an existing function [...] You don't have to describe all possible inputs and outputs. Even just being able to describe some classes of inputs can be useful. As a really simple example: many example-based tests have some values that are arbitrary and the test shouldn't care about them, like eg employees names when you are populating a database or whatever. Instead of just hard-coding 'foo' and 'bar', you can have hypothesis create arbitrary values there. Just like learning how to write (unit) testable code is a skill that needs to be learned, learning how to write property-testable code is also a skill that needs practice. What's less obvious: retro-fitting property-based tests on an exiting codebase with existing example-based tests is almost a separate skill. It's harder than writing your code with property based tests in mind. --- Some common properties to test: * Your code doesn't crash on random inputs (or only throws a short whitelist of allowed exceptions). * Applying a specific functionality should be idempotent, ie doing that operation multiple times should give the same results as applying it only once. * Order of input doesn't matter (for some functionality) * Testing your prod implementation against a simpler implementation, that's perhaps too slow for prod or only works on a restricted subset of the real problem. The reference implementation doesn't even have to be simpler: just having a different approach is often enough.
  - wodenokoto 1y ago
    But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?
    - vlovich123 1y ago
      Either your code shouldn’t fail or the apostrophe isn’t a valid case. In the former, hypothesis and other similar frameworks are deterministic and will replay the failing test on request or remember the failing tests in a file to rerun in the future to catch regressions. In the latter, you just tell the framework to not generate such values or at least to skip those test cases (better to not generate in terms of testing performance).
- bunderbunder 1y ago
  For working with legacy systems I tend to start with Approval Tests (https://approvaltests.com/ https://approvaltests.com/). Because they don't require me to understand the SUT very well before I can get started with them, and because they help me start to understand it more quickly.
- dd82 1y ago
  Essentially this is a good example of parametrized tests, just supercharged with generated inputs. So if you already have parametrized tests, you're already halfway there.
  - eru 1y ago
    Yes, when I saw eg Golang people use table driven tests like this, I was wondering why nobody seems to have told them about generating these tables automatically..
- chriswarbo 1y ago
  When I'm "up to my armpits in trying to understand the on-the-ground problem", I find PBT great for quickly find mistakes in the assumptions/intuitions I'm making about surrounding code and helper functions. Whenever I find myself thinking "WTF? Surely ABC does XYZ?", and the code for ABC isn't immediately obvious, then I'll bang-out an `ABC_does_XYZ` property and see if I'm wrong. This can be much faster than trying to think up "good" examples to check, especially when I'm not familiar with the domain model, and the relevant values would be giant nested things. I'll let the computer have a go first.
dbcurtis 1y ago
It’s been quite some time since I’ve been in the business of writing lots of unit tests, but back in the day, I found hypothesis to be a big force multiplier and it uncovered many subtle/embarrassing bugs for me. Recommend. Also easy and intuitive to use.
- eru 1y ago
  Hypothesis is also a lot better at giving you 'nasty' floats etc than Haskell's QuickCheck or the relevant Rust and OCaml libraries are. (Or at least used to be, I haven't checked on all of them recently.)
- aethor 1y ago
  I concur. Hypothesis saved me many times. It also helped me prove the existence of bugs in third party code, since I was able to generate examples showing that a specific function was not respecting certain properties. Without that I would have spent a lot of time trying to manually find an example, let alone the simplest possible example.
- IanCal 1y ago
  Huge second. I’ve never use pbt and failed to find a new bug. I recommended it in a job interview, they used it and discovered a pretty clear bug on their first test. It’s really powerful.
rmunn 1y ago
I love property-based testing, especially the way it can uncover edge cases you wouldn't have thought about. Haven't used Hypothesis yet, but I once had FsCheck (property-based testing for F#) find a case where the data structure I was writing failed when there were exactly 24 items in the list and you tried to append a 25th. That was a test case I wouldn't have thought to write on my own, but the particular number (it was always the 25th item that failed) quickly led me to find the bug. Once my property tests were running overnight and not finding any failures after thousands and thousands of random cases, I started to feel a lot more confident that I'd nailed down the bugs.
- tombert 1y ago
  I had a similar thing, with F# as well actually. We had some code that used a square root, and in some very rare circumstances, we could get a negative number, which would throw an exception. I don't think i would have even considered that possibility if FsCheck hadn't generated it.
- teiferer 1y ago
  That example caught my attention. What was it in your code that made length 24 special?
  - rmunn 1y ago
    I was trying out an implementation of Relaxed Radix-Balanced Trees, a data structure for immutable lists that allows splitting and joining them in constant* time. In a normal RRB Tree implementation, each leaf in the tree has up to 32 items in it, and a tree node that's not a leaf has up to 32 pointers to leaves or to lower tree nodes. There's also a tail, a single array of up to 32 which, when full, will be promoted to a leaf and a new empty tail array created. (Or the full tail will be promoted the next time an item is appended and a new tail of length 1 will be created). For my implementation, I was using nodes of size 4 rather than size 32, because that let me test much easier; then after testing, I would switch the branching constant to 32 and run final tests. With a branching factor of 4, that means that a tree of size 24 contained one root node with two children, a left child that was a full tree of 16 (a tree node pointing to four leaf nodes, all of them full of 4 items), and a right child that was a leaf node of size 4. That makes 20 items in the tree, plus a full tail array of 4 making 24 items. The failure came when the next item was appended. Now, if you've never implemented an RRB tree you won't have noticed the mistake. But in the previous paragraph, when I said that the right child of the root was a leaf? That was the mistake; in an RRB tree all leaves are supposed to be at the same level, and the right child of the root should have been a one-child node, whose child was a leaf of size 4. My code that promoted the root up one level when it was full (when the tree was size 20, a full 16-item tree plus a tail of 4) was faulty, and had created a new root with the previous root as the left child and the previous tail as the right child, instead of creating a one-child node as the right child of the new root with the 4-item leaf as its child. The tree still functioned for all read-only purposes (traversal, lookups, etc) and even for appends, as long as the appends did no more than fill the tail. Once the tail was full (making a total of 24 items total stored in the structure) and I needed to append a 25th item, the tail-promotion code found an illegal tree state and did the wrong thing. I no longer remember which wrong thing it did, just that once that 25th item was appended, the resulting tree no longer passed all the property checks. * Constant for all practical purposes. In fact it's O(log32 N), but since log32 N is no more than 7 when N is 2^32, the operation is bounded by a constant. So it's O(1) for all practical purposes: most lists of any reasonable size will translate into trees whose height is no more than 2 or 3, which means only 2 or 3 nodes will need to be split when splitting the entire list (and the RRB tree structure that models the list) into two parts. And likewise for joining two RRB structures together (a list append operation): there might be up to O(H) merged nodes where H is the height of the tree, which again no more than log32 N for a tree of branching factor 32.
pyuser583 1y ago
Make sure to read the docs and understand this well. It has its own vocabulary that can be very counterintuitive.
klntsky 1y ago
It seems to only implement a half of QuickCheck idea, because there is no counterexample shrinking. Good effort though! I wonder how hard would it be to derive generators for any custom types in python - probably not too hard, because types are just values
- pfdietz 1y ago
  The way it does counterexample shrinking is the most clever part of Hypothesis.
  - ctenb 1y ago
    Do you have a reference where it is explained? It's not part of the docs as far as I can tell
    - pfdietz 1y ago
      You might try this blog entry (or other blog entries that talk about shrinking): https://hypothesis.works/articles/how-hypothesis-works/ https://hypothesis.works/articles/how-hypothesis-works/
- Jtsummers 1y ago
  > because there is no counterexample shrinking Hypothesis does shrink the examples, though.
  - eru 1y ago
    And Hypothesis is miles ahead of QuickCheck in how it handles shrinking! Not only does it shrink automatically, it has no problem preserving invariants from generation in your shrinking; like only prime numbers or only strings that begin with a vowel etc.
    - lgas 1y ago
      QuickCheck also shrinks automatically and preserves invariants though?
teiferer 1y ago
This approach has two fundamental problems. 1. It requires you to essentially re-implement the business logic of the SUT (subject-under-test) so that you can assert it. Is your function doing a+b? Then instead of asserting that f(1, 2) == 3 you need to do f(a, b) == a+b since the framework provides a and b. You can do a simpler version that's less efficient, but in the end of the day, you somehow need to derive the expected outputs from input arguments, just like your SUT does. Any logical error that might be slipped into your SUT implementation has a high risk of also slipping into your test and will therefore be hidden by the complexity, even though it would be obvious from just looking at a few well thought through examples. 2. Despite some anecdata in the comments here, the chances are slim that this approach will find edge cases that you couldn't think of. You basically just give up and leave edge case finding to chance. Testing for 0 or -1 or 1-more-than-list-length are obvious cases which both you the human test writer and some test framework can easily generate, and they are often actual edge cases. But what really constitutes an edge case depends on your implementation. You as the developer know the implementation and have a chance of coming up with the edge cases. You know the dark corners of your code. Random tests are just playing the lottery, replacing thinking hard.
- robertfw 1y ago
  I feel like this talk by John Hughes showed that there is real value in this approach with production systems of varying levels of complexity, with two different examples of using the approach to find very low level bugs that you'd never think to test for in traditional approaches. https://www.youtube.com/watch?v=zi0rHwfiX1Q https://www.youtube.com/watch?v=zi0rHwfiX1Q
- vlovich123 1y ago
  I have not met anyone that says you should only fuzz/property test, but claiming it can’t possibly find bugs or is unlikely to is silly. I’ve caught numerous non-obvious problems, including a non-fatal but undesirable off-by-1 error in math heavy code due to property testing. It works well when it’s an “np”-hard style problem where the code is harder than the verification. It does not work well for a+b but most problems it’s generally easier to write assertions that have to hold when executing your function. But if it’s not don’t use it - like all testing, it’s an art to determine when it’s useful and how to write it well. Hypothesis in particular does something neat where it tries to generate random inputs that are more likely to execute novel paths within the code under test. That’s not replicated in Rust but is super helpful about reaching more paths of your code and that’s simply not able to be done manually if you have a lot of non obvious boundary conditions.
cat-whisperer 1y ago
A little off topic but theirs’ this https://youtu.be/64t-gPC33cc https://youtu.be/64t-gPC33cc this is a great video by jon about property testing in Rust
__mharrison__ 1y ago
I've taught this in my testing courses. I find that (pytest) fixtures are often good enough for coming up with multiple tests but are simple enough to implement.
thdhhghgbhy 1y ago
Is there something this nice for JS, with the decorators like that?
- ngruhn 1y ago
  No decorators, but fast-check has add-ons to various test frameworks. E.g. if you use Vitest you can write: import { test, fc } from '@fast-check/vitest' test.prop([fc.array(fc.double())])('sort is correct', (lst) => { expect(lst).toEqual(lst.toSorted()) }) https://www.npmjs.com/package/@fast-check/vitest?activeTab=readme https://www.npmjs.com/package/@fast-check/vitest?activeTab=r...
  - chamomeal 1y ago
    Fast check is fantastic!! I found it to be pretty verbose but I think that’s just a typescript limitation. It’s VERY well typed, which was a nice surprise. Such a great library. Shrinking, model based testing, it’s really comprehensive
- aidos 1y ago
  Not decorators (or at least not last time I looked) but we use fast-check. Was already familiar with and using Hypothesis in Python so went in search of something with similar nice ergonomics. Am happy with fast-check in that regard. https://fast-check.dev/ https://fast-check.dev/
- eru 1y ago
  The decorators are a nice approach in Python, but they aren't really core to what Hypothesis does, nor what makes it better than eg Haskell's QuickCheck.
- greener_grass 1y ago
  Decorators are just function application with more syntax.
spooky_deep 1y ago
Property based testing is fantastic. Why is it not more popular? My theory is that only code written in functional languages has complex properties you can actually test. In imperative programs, you might have a few utils that are appropriate for property testing - things like to_title_case(str) - but the bulk of program logic can only be tested imperatively with extensive mocking.
- eru 1y ago
  But wouldn't that apply just as much to example based testing?
- ibizaman 1y ago
  I actually used property testing very successfully to test a DB driver and a migration to another DB driver in Go. I wrote up about it here https://blog.tiserbox.com/posts/2024-02-27-stateful-property-testing-in-go.html https://blog.tiserbox.com/posts/2024-02-27-stateful-property...
  - imiric 1y ago
    Thanks for sharing! Your article illustrates well the benefits of this approach. One drawback I see is that property-based tests inevitably need to be much more complex than example-based ones. This means that bugs are much more likely, they're more difficult to maintain, etc. You do mention that it's a lot of code, but I wonder if the complexity is worth it in the long run. I suppose that since testing these scenarios any other way would be even more tedious and error-prone, the answer is "yes". But it's something to keep in mind.
    - dragonwriter 1y ago
      > One drawback I see is that property-based tests inevitably need to be much more complex than example-based ones. I don’t think that’s true, I just think the complexity is more explicit (in code) rather than implicit (in the process of coming up with examples). Example-based testing usually involves defining conditions and properties to be tested, then involves constructing sets of examples to test them and which attempt to anticipated edge cases from the description of the requirements (black box) or from knowledge of how the code is implemented (white box). Property based testing involves defining the conditiosn and properties, writing code that generates the conditions and for each property writing a bit of code that can refute it by passing if and only if it is true of the subject under test for a particular set of inputs. With a library like Hypothesis which both has good generators for basic types and good abstractions for combining and adapting generators, the latter seems to be less complex overall, as well as moving the complexity into a form where it is explicit and easy to maintain/adapt, whereas adapting example-based tests to requirements changes involves either throwing out examples and starting over or revalidating and updating examples individually.
imiric 1y ago
Coincidentally, I recently stumbled upon a similar library for Go[1]. I haven't used it, or property-based testing, but I can see how it could be useful. [1]: https://github.com/flyingmutant/rapid https://github.com/flyingmutant/rapid
benrutter 1y ago
I love the idea of hypothesis! Haven't found a lot of use cases for it yet, I think the quick start example helps explain why. Essentially, you're testing that "my_sort" returns the same as python's standard "sort". Of course, this means you need a second function that acts the same as the function you wrote. In real life, if you had that you probably wouldn't have written the function "my_sort" at all. Obviously it's just a tiny example, but in general I often find writing a test that will cover every hypothetical case isn't realistic. Anyone here use this testing in the wild? Where's it most useful? Do you have the issue I described? Is there an easy way to overcome it?
- rowanG077 1y ago
  On more complex logic you can often reason about many invariants. I don't think using sorting and then comparing to the existing sort is a good example. Precisely for the reason you mention. A sort itself is a good example though. You can do much better then just compare to an existing sort implementation. Because you can easily check if an array is sorted with a linear scan. If your sort is stable you can do an additional check that for all pairs of adjacent elements that compare as equal the order of them is the same as in the input array.
- thekoma 1y ago
  Indeed. For a sorting algorithm it would be more insightful to show it test the actual property of something that is sorted: every consecutive element is larger than the previous one (or the other way around). You don’t need a sorting function to test the “sorted” property.
- aflukasz 1y ago
  > Anyone here use this testing in the wild? Where's it most useful? Do you have the issue I described? Is there an easy way to overcome it? One example would be when you have a naive implementation of some algorithm and you want to introduce a second one, optimized but with much more complex implementation. Then this naive one will act as a model for comparisons. Another case that comes to mind is when you have rather simple properties to test (like: does it finish without a crash, within a given time?, does not cross some boundaries on the output?), and want to easily run over a sensible set of varying inputs.
ed_blackburn 1y ago
I'm using sqlglot to parse hundreds of old mysql back up files to find diffs of schemas. The joys of legacy code. I've found hypothesis to be super helpful for tightening up my parser. I've identified properties (invariants) and built some strategies. I can now generate many more permutations of DDL than I'd thought of before. And I have absolutely confidence in what I'm running. I started off TDD covered the basics. Learned what I needed to learn about the files I'm dealing with, edge cases, sqlglot and then I moved onto Hypothesis for extra confidence. I'm curious to see if it'll help with commands for APIs. I nothing else it'll help me appreciate how liberal my API is when perhaps I don't want it to be?
- groundhogstate 1y ago
  You might find schemathesis useful for API testing (which IIRC is built on Hypothesis). Certainly helped me find a stack of unhandled edge cases. YMMV, but once I first set up Schemathesis on one of my MVPs, it took several hours to iron out the kinks. But thereafter, if built into CI, I've found it guards against regression quite effectively.
  - IanCal 1y ago
    https://schemathesis.readthedocs.io/en/stable/ https://schemathesis.readthedocs.io/en/stable/ This looks really good. I like using hypothesis for APIs so this is a really nice layer on top, thanks for pointing this one out.
  - agumonkey 1y ago
    pretty nice tool, helped us poke into our app in ways we didn't foresee also pretty happy to see the reports of thousands or api calls that matched the expected responses, that increases the confidence you have in your system
    - dmitry_dygalo 1y ago
      Hi! Author of Schemathesis here, really glad to hear it helped you uncover those edge cases. If any of those bugs were in public or open-source APIs, I’d love to feature them in our new “trophy case”: https://github.com/schemathesis/schemathesis/issues/new?template=trophy-submission.yml https://github.com/schemathesis/schemathesis/issues/new?temp...
vlade11115 1y ago
One cool application of this is schemathesis. I really enjoyed it, and I found more input validation bugs in my code than I can count. Very useful for public APIs.
rtu8lu 1y ago
Great project. I used it a lot, but now I mostly prefer ad hoc generators. Hypothesis combinators quickly become unmaintainable mess for non-trivial objects. Also, shrinking is not such a big deal when you can generate your data in a complexity-sorted order.
- ctenb 1y ago
  What is complexity sorted order?
  - rtu8lu 1y ago
    When you generate randomized data in an order from the simplest to the most complex.
    - ctenb 1y ago
      Isn't that very wasteful or difficult to do in practice? If you consider that shrinkers generally take lower numbers to be 'simpler' than higher numbers, complexity-ordering requires you to generate all the numbers from low to high
      - rtu8lu 1y ago
        Not really. There are many ways depending on your needs. For example, you can partition your space first, then generate randomly inside each of the subspaces. Let's say I need 200 numbers from -1000 to 999. The first range will be 0 to +99, the second -1 to -100, then +100 to +199, and so on. So, to generate a random number I just need an index and the bounds.
widikidiw 1y ago
[dead]
languagehacker 1y ago
My first impression on property-based testing is that it's likely to introduce flakes that distract engineers from their task at hand, forcing them to investigate a new failed test in an area that they weren't even touching.
toast_x 1y ago
this is pretty sick - i imagine this speeds up hardening a lot. will try it out - thank you!
jamesponddotco 1y ago
Never heard of “property-based testing” before. Coming from Go, I mostly use table and fuzzy tests. Is this approach something that makes sense in Go as well? I see at least one package[1][2] for Go, but I wonder if the approach itself makes sense for the language. [1]: https://github.com/flyingmutant/rapid https://github.com/flyingmutant/rapid [2]: Oh, I just found out about testing/quick.
- akshayshah 1y ago
  Rapid is excellent. It also integrates with the standard library's fuzz testing, which is handy to persist a high-priority corpus of inputs that have caused bugs in the past. Testing/quick is adequate for small things and doesn't introduce new dependencies, but it's also frozen. Many years ago, the Go team decided that PBT is complex enough that it shouldn't be in stdlib. Here's a small example of a classic PBT technique used in a very practical Go project: https://github.com/connectrpc/connect-go/blob/cb2e11fb88c9a61804043355a619c12d4a30a1a5/header_test.go#L26 https://github.com/connectrpc/connect-go/blob/cb2e11fb88c9a6...
anordin95 1y ago
I am a huge fan of the "Explanations" page of their docs: "These explanation pages are oriented towards deepening your understanding of Hypothesis, including its design philosophy." I feel much more software documentation could greatly benefit from this approach. Describing the "why" and design tradeoffs helps me grok a system far better than the typical quickstart or tutorials which show snippets but offer little understanding. That is, they rarely help me answer: is this going to solve my problem and how?
- lwhsiao 1y ago
  You would like the Diátaxis framework: https://diataxis.fr/ https://diataxis.fr/ That is the structure they (any many others) are following :).
  - anordin95 1y ago
    I know it and do like it! Even for those that follow Diataxis, in my experience the "Explanation" sections are often lacking, especially compared to the "How-To" or "Tutorial" ones. Python is one example that comes to mind. They do have explanations here: https://docs.python.org/3/howto/index.html https://docs.python.org/3/howto/index.html. And, to be fair, they are generally excellent in my opinion! But they're far from front and center and there's much less overall content compared to the other Diataxis types I think (granted, I haven't rigorously checked).

231 points by lwhsiao 1y ago

I keep thinking I have a possible use case for property -based testing, and then I am up to my armpits in trying to understand the on-the-ground problem and don't feel like I have time to learn a DSL for describing all possible inputs and outputs when I already had an existing function (the subject-under-test) that I don't understand. So rather than try to learn to black boxes at the same time , I fall back to "several more unit tests to document more edge cases to defensibly guard against" Is there some simple way to describe this defensive programming iteration pattern in Hypothesis? Normally we just null-check and return early and have to deal with the early-return case. How do I quickly write property tests to check that my code handles the most obvious edge cases?

fwip 1y ago
I think the easiest way is to start with general properties and general input, and tighten them up as needed. The property might just be "doesn't throw an exception", in some cases. If you find yourself writing several edge cases manually with a common test logic, I think the @example decorator in Hypothesis is a quick way to do that: https://hypothesis.readthedocs.io/en/latest/reference/api.html#explicit-inputs https://hypothesis.readthedocs.io/en/latest/reference/api.ht...
- NortySpock 1y ago
  Thanks, the "does not throw an exception" property got my mental gears turning in terms of how to get started on this, and from there I can see how one could add a few more properties as one goes along. Appreciate you taking the time to answer.
  - eru 1y ago
    Yes. One of the first things you have to do when writing any property based tests, no matter what you actually want to test, is defining your input generators. And once you have those generators, you might as well throw together the "doesn't crash unexpectedly" test together. That not only tests your code, but also exercises your just written input generators.
disambiguation 1y ago
I've only used it once before, not as unit testing, but as stress testing for a new customer facing api. I wanted to say with confidence "this will never throw an NPE". Also the logic was so complex (and the deadline so short) the only reasonable way to test was to generate large amounts of output data and review it manually for anomalies.
sunshowers 1y ago
The simplest practical property-based tests are where you serialize some randomly generated data of a particular shape to JSON, then deserialize it, and ensure that the output is the same. A more complex kind of PBT is if you have two implementations of an algorithm or data structure, one that's fast but tricky and the other one slow but easy to verify. (Say, quick sort vs bubble sort.) Generate data or operations randomly and ensure the results are the same.
- eru 1y ago
  > The simplest practical property-based tests are where you serialize some randomly generated data of a particular shape to JSON, then deserialize it, and ensure that the output is the same. Testing that f(g(x)) == x for all x and some f and g that are supposed to be inverses of each other is a good test, but it's probably not the simplest. The absolute simplest I can think of is just running your functionality on some randomly generated input and seeing that it doesn't crash unexpectedly. For things like sorting, testing against an oracle is great. But even when you don't have an oracle, there's lots of other possibilities: * Test that sorting twice has the same effect as sorting once. * Start with a known already in-order input like [1, 2, 3, ..., n]; shuffle it, and then check that your sorting algorithm re-creates the original. * Check that the output of your sorting algorithm is in-order. * Check that input and output of your sorting algorithm have the same elements in the same multiplicity. (If you don't already have a datastructure / algorithm that does this efficiently, you can probe it with more randomness: create a random input (say a list of numbers), pick a random number X, count how many times X appears in your list (via a linear scan); then check that you get the same count after sorting. * Check that permuting your input doesn't make a difference. * Etc.
  - sunshowers 1y ago
    Speaking for myself — those are definitely all simpler cases, but for me I never found them compelling enough (beyond the "it doesn't crash" property). For me, the simplest case that truly motivated PBT for me was roundtrip serialization. Now I use PBT quite a lot, and most of them are either serialization roundtrip or oracle/model-based tests.
meejah 1y ago
Here are some fairly simple examples: testing port parsing https://github.com/meejah/fowl/blob/e8253467d7072cd05f21de7cd6d94cef3fe5977e/src/fowl/test/test_cli.py#L146 https://github.com/meejah/fowl/blob/e8253467d7072cd05f21de7c... ...and https://github.com/magic-wormhole/magic-wormhole/blob/1b4732cc785dc93b4c50f8679e34fb54e99cc32b/src/wormhole/test/dilate/test_subchannel.py#L303 https://github.com/magic-wormhole/magic-wormhole/blob/1b4732... The simplest ones to get started with are "strings", IMO, and also gives you lots of mileage (because it'll definitely test some weird unicode). So, somewhere in your API where you take some user-entered strings -- even something "open ended" like "a name" -- you can make use of Hypothesis to try a few things. This has definitely uncovered unicode bugs for me. Some more complex things can be made with some custom strategies. The most-Hypothesis-heavy tests I've personally worked with are from Magic Folder strategies: https://github.com/tahoe-lafs/magic-folder/blob/main/src/magic_folder/test/strategies.py#L219 https://github.com/tahoe-lafs/magic-folder/blob/main/src/mag... The only real downside is that a Hypothesis-heavy test-suite like the above can take a while to run (but you can instruct it to only produce one example per test). Obviously, one example per test won't catch everything, but is way faster when developing and Hypothesis remembers "bad" examples so if you occasionally do a longer run it'll remember things that caused errors before.
akshayshah 1y ago
Sibling comments have already mentioned some common strategies - but if you have half an hour to spare, the property-based testing series on the F# for Fun and Profit blog is well worth your time. The material isn’t really specific to F#. https://fsharpforfunandprofit.com/series/property-based-testing/ https://fsharpforfunandprofit.com/series/property-based-test...
eru 1y ago
In addition to what other people have said: > [...] time to learn a DSL for describing all possible inputs and outputs when I already had an existing function [...] You don't have to describe all possible inputs and outputs. Even just being able to describe some classes of inputs can be useful. As a really simple example: many example-based tests have some values that are arbitrary and the test shouldn't care about them, like eg employees names when you are populating a database or whatever. Instead of just hard-coding 'foo' and 'bar', you can have hypothesis create arbitrary values there. Just like learning how to write (unit) testable code is a skill that needs to be learned, learning how to write property-testable code is also a skill that needs practice. What's less obvious: retro-fitting property-based tests on an exiting codebase with existing example-based tests is almost a separate skill. It's harder than writing your code with property based tests in mind. --- Some common properties to test: * Your code doesn't crash on random inputs (or only throws a short whitelist of allowed exceptions). * Applying a specific functionality should be idempotent, ie doing that operation multiple times should give the same results as applying it only once. * Order of input doesn't matter (for some functionality) * Testing your prod implementation against a simpler implementation, that's perhaps too slow for prod or only works on a restricted subset of the real problem. The reference implementation doesn't even have to be simpler: just having a different approach is often enough.
- wodenokoto 1y ago
  But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?
  - vlovich123 1y ago
    Either your code shouldn’t fail or the apostrophe isn’t a valid case. In the former, hypothesis and other similar frameworks are deterministic and will replay the failing test on request or remember the failing tests in a file to rerun in the future to catch regressions. In the latter, you just tell the framework to not generate such values or at least to skip those test cases (better to not generate in terms of testing performance).
bunderbunder 1y ago
For working with legacy systems I tend to start with Approval Tests (https://approvaltests.com/ https://approvaltests.com/). Because they don't require me to understand the SUT very well before I can get started with them, and because they help me start to understand it more quickly.
dd82 1y ago
Essentially this is a good example of parametrized tests, just supercharged with generated inputs. So if you already have parametrized tests, you're already halfway there.
- eru 1y ago
  Yes, when I saw eg Golang people use table driven tests like this, I was wondering why nobody seems to have told them about generating these tables automatically..
chriswarbo 1y ago
When I'm "up to my armpits in trying to understand the on-the-ground problem", I find PBT great for quickly find mistakes in the assumptions/intuitions I'm making about surrounding code and helper functions. Whenever I find myself thinking "WTF? Surely ABC does XYZ?", and the code for ABC isn't immediately obvious, then I'll bang-out an `ABC_does_XYZ` property and see if I'm wrong. This can be much faster than trying to think up "good" examples to check, especially when I'm not familiar with the domain model, and the relevant values would be giant nested things. I'll let the computer have a go first.

dbcurtis 1y ago

It’s been quite some time since I’ve been in the business of writing lots of unit tests, but back in the day, I found hypothesis to be a big force multiplier and it uncovered many subtle/embarrassing bugs for me. Recommend. Also easy and intuitive to use.

eru 1y ago
Hypothesis is also a lot better at giving you 'nasty' floats etc than Haskell's QuickCheck or the relevant Rust and OCaml libraries are. (Or at least used to be, I haven't checked on all of them recently.)
aethor 1y ago
I concur. Hypothesis saved me many times. It also helped me prove the existence of bugs in third party code, since I was able to generate examples showing that a specific function was not respecting certain properties. Without that I would have spent a lot of time trying to manually find an example, let alone the simplest possible example.
IanCal 1y ago
Huge second. I’ve never use pbt and failed to find a new bug. I recommended it in a job interview, they used it and discovered a pretty clear bug on their first test. It’s really powerful.

rmunn 1y ago

I love property-based testing, especially the way it can uncover edge cases you wouldn't have thought about. Haven't used Hypothesis yet, but I once had FsCheck (property-based testing for F#) find a case where the data structure I was writing failed when there were exactly 24 items in the list and you tried to append a 25th. That was a test case I wouldn't have thought to write on my own, but the particular number (it was always the 25th item that failed) quickly led me to find the bug. Once my property tests were running overnight and not finding any failures after thousands and thousands of random cases, I started to feel a lot more confident that I'd nailed down the bugs.

tombert 1y ago
I had a similar thing, with F# as well actually. We had some code that used a square root, and in some very rare circumstances, we could get a negative number, which would throw an exception. I don't think i would have even considered that possibility if FsCheck hadn't generated it.
teiferer 1y ago
That example caught my attention. What was it in your code that made length 24 special?
- rmunn 1y ago
  I was trying out an implementation of Relaxed Radix-Balanced Trees, a data structure for immutable lists that allows splitting and joining them in constant* time. In a normal RRB Tree implementation, each leaf in the tree has up to 32 items in it, and a tree node that's not a leaf has up to 32 pointers to leaves or to lower tree nodes. There's also a tail, a single array of up to 32 which, when full, will be promoted to a leaf and a new empty tail array created. (Or the full tail will be promoted the next time an item is appended and a new tail of length 1 will be created). For my implementation, I was using nodes of size 4 rather than size 32, because that let me test much easier; then after testing, I would switch the branching constant to 32 and run final tests. With a branching factor of 4, that means that a tree of size 24 contained one root node with two children, a left child that was a full tree of 16 (a tree node pointing to four leaf nodes, all of them full of 4 items), and a right child that was a leaf node of size 4. That makes 20 items in the tree, plus a full tail array of 4 making 24 items. The failure came when the next item was appended. Now, if you've never implemented an RRB tree you won't have noticed the mistake. But in the previous paragraph, when I said that the right child of the root was a leaf? That was the mistake; in an RRB tree all leaves are supposed to be at the same level, and the right child of the root should have been a one-child node, whose child was a leaf of size 4. My code that promoted the root up one level when it was full (when the tree was size 20, a full 16-item tree plus a tail of 4) was faulty, and had created a new root with the previous root as the left child and the previous tail as the right child, instead of creating a one-child node as the right child of the new root with the 4-item leaf as its child. The tree still functioned for all read-only purposes (traversal, lookups, etc) and even for appends, as long as the appends did no more than fill the tail. Once the tail was full (making a total of 24 items total stored in the structure) and I needed to append a 25th item, the tail-promotion code found an illegal tree state and did the wrong thing. I no longer remember which wrong thing it did, just that once that 25th item was appended, the resulting tree no longer passed all the property checks. * Constant for all practical purposes. In fact it's O(log32 N), but since log32 N is no more than 7 when N is 2^32, the operation is bounded by a constant. So it's O(1) for all practical purposes: most lists of any reasonable size will translate into trees whose height is no more than 2 or 3, which means only 2 or 3 nodes will need to be split when splitting the entire list (and the RRB tree structure that models the list) into two parts. And likewise for joining two RRB structures together (a list append operation): there might be up to O(H) merged nodes where H is the height of the tree, which again no more than log32 N for a tree of branching factor 32.

pyuser583 1y ago

Make sure to read the docs and understand this well. It has its own vocabulary that can be very counterintuitive.

klntsky 1y ago

It seems to only implement a half of QuickCheck idea, because there is no counterexample shrinking. Good effort though! I wonder how hard would it be to derive generators for any custom types in python - probably not too hard, because types are just values

pfdietz 1y ago
The way it does counterexample shrinking is the most clever part of Hypothesis.
- ctenb 1y ago
  Do you have a reference where it is explained? It's not part of the docs as far as I can tell
  - pfdietz 1y ago
    You might try this blog entry (or other blog entries that talk about shrinking): https://hypothesis.works/articles/how-hypothesis-works/ https://hypothesis.works/articles/how-hypothesis-works/
Jtsummers 1y ago
> because there is no counterexample shrinking Hypothesis does shrink the examples, though.
- eru 1y ago
  And Hypothesis is miles ahead of QuickCheck in how it handles shrinking! Not only does it shrink automatically, it has no problem preserving invariants from generation in your shrinking; like only prime numbers or only strings that begin with a vowel etc.
  - lgas 1y ago
    QuickCheck also shrinks automatically and preserves invariants though?

teiferer 1y ago

This approach has two fundamental problems. 1. It requires you to essentially re-implement the business logic of the SUT (subject-under-test) so that you can assert it. Is your function doing a+b? Then instead of asserting that f(1, 2) == 3 you need to do f(a, b) == a+b since the framework provides a and b. You can do a simpler version that's less efficient, but in the end of the day, you somehow need to derive the expected outputs from input arguments, just like your SUT does. Any logical error that might be slipped into your SUT implementation has a high risk of also slipping into your test and will therefore be hidden by the complexity, even though it would be obvious from just looking at a few well thought through examples. 2. Despite some anecdata in the comments here, the chances are slim that this approach will find edge cases that you couldn't think of. You basically just give up and leave edge case finding to chance. Testing for 0 or -1 or 1-more-than-list-length are obvious cases which both you the human test writer and some test framework can easily generate, and they are often actual edge cases. But what really constitutes an edge case depends on your implementation. You as the developer know the implementation and have a chance of coming up with the edge cases. You know the dark corners of your code. Random tests are just playing the lottery, replacing thinking hard.

robertfw 1y ago
I feel like this talk by John Hughes showed that there is real value in this approach with production systems of varying levels of complexity, with two different examples of using the approach to find very low level bugs that you'd never think to test for in traditional approaches. https://www.youtube.com/watch?v=zi0rHwfiX1Q https://www.youtube.com/watch?v=zi0rHwfiX1Q
vlovich123 1y ago
I have not met anyone that says you should only fuzz/property test, but claiming it can’t possibly find bugs or is unlikely to is silly. I’ve caught numerous non-obvious problems, including a non-fatal but undesirable off-by-1 error in math heavy code due to property testing. It works well when it’s an “np”-hard style problem where the code is harder than the verification. It does not work well for a+b but most problems it’s generally easier to write assertions that have to hold when executing your function. But if it’s not don’t use it - like all testing, it’s an art to determine when it’s useful and how to write it well. Hypothesis in particular does something neat where it tries to generate random inputs that are more likely to execute novel paths within the code under test. That’s not replicated in Rust but is super helpful about reaching more paths of your code and that’s simply not able to be done manually if you have a lot of non obvious boundary conditions.

cat-whisperer 1y ago

A little off topic but theirs’ this https://youtu.be/64t-gPC33cc https://youtu.be/64t-gPC33cc this is a great video by jon about property testing in Rust

__mharrison__ 1y ago

I've taught this in my testing courses. I find that (pytest) fixtures are often good enough for coming up with multiple tests but are simple enough to implement.

thdhhghgbhy 1y ago

Is there something this nice for JS, with the decorators like that?

ngruhn 1y ago
No decorators, but fast-check has add-ons to various test frameworks. E.g. if you use Vitest you can write: import { test, fc } from '@fast-check/vitest' test.prop([fc.array(fc.double())])('sort is correct', (lst) => { expect(lst).toEqual(lst.toSorted()) }) https://www.npmjs.com/package/@fast-check/vitest?activeTab=readme https://www.npmjs.com/package/@fast-check/vitest?activeTab=r...
- chamomeal 1y ago
  Fast check is fantastic!! I found it to be pretty verbose but I think that’s just a typescript limitation. It’s VERY well typed, which was a nice surprise. Such a great library. Shrinking, model based testing, it’s really comprehensive
aidos 1y ago
Not decorators (or at least not last time I looked) but we use fast-check. Was already familiar with and using Hypothesis in Python so went in search of something with similar nice ergonomics. Am happy with fast-check in that regard. https://fast-check.dev/ https://fast-check.dev/
eru 1y ago
The decorators are a nice approach in Python, but they aren't really core to what Hypothesis does, nor what makes it better than eg Haskell's QuickCheck.
greener_grass 1y ago
Decorators are just function application with more syntax.

spooky_deep 1y ago

Property based testing is fantastic. Why is it not more popular? My theory is that only code written in functional languages has complex properties you can actually test. In imperative programs, you might have a few utils that are appropriate for property testing - things like to_title_case(str) - but the bulk of program logic can only be tested imperatively with extensive mocking.

eru 1y ago
But wouldn't that apply just as much to example based testing?
ibizaman 1y ago
I actually used property testing very successfully to test a DB driver and a migration to another DB driver in Go. I wrote up about it here https://blog.tiserbox.com/posts/2024-02-27-stateful-property-testing-in-go.html https://blog.tiserbox.com/posts/2024-02-27-stateful-property...
- imiric 1y ago
  Thanks for sharing! Your article illustrates well the benefits of this approach. One drawback I see is that property-based tests inevitably need to be much more complex than example-based ones. This means that bugs are much more likely, they're more difficult to maintain, etc. You do mention that it's a lot of code, but I wonder if the complexity is worth it in the long run. I suppose that since testing these scenarios any other way would be even more tedious and error-prone, the answer is "yes". But it's something to keep in mind.
  - dragonwriter 1y ago
    > One drawback I see is that property-based tests inevitably need to be much more complex than example-based ones. I don’t think that’s true, I just think the complexity is more explicit (in code) rather than implicit (in the process of coming up with examples). Example-based testing usually involves defining conditions and properties to be tested, then involves constructing sets of examples to test them and which attempt to anticipated edge cases from the description of the requirements (black box) or from knowledge of how the code is implemented (white box). Property based testing involves defining the conditiosn and properties, writing code that generates the conditions and for each property writing a bit of code that can refute it by passing if and only if it is true of the subject under test for a particular set of inputs. With a library like Hypothesis which both has good generators for basic types and good abstractions for combining and adapting generators, the latter seems to be less complex overall, as well as moving the complexity into a form where it is explicit and easy to maintain/adapt, whereas adapting example-based tests to requirements changes involves either throwing out examples and starting over or revalidating and updating examples individually.

imiric 1y ago

Coincidentally, I recently stumbled upon a similar library for Go[1]. I haven't used it, or property-based testing, but I can see how it could be useful. [1]: https://github.com/flyingmutant/rapid https://github.com/flyingmutant/rapid

benrutter 1y ago

I love the idea of hypothesis! Haven't found a lot of use cases for it yet, I think the quick start example helps explain why. Essentially, you're testing that "my_sort" returns the same as python's standard "sort". Of course, this means you need a second function that acts the same as the function you wrote. In real life, if you had that you probably wouldn't have written the function "my_sort" at all. Obviously it's just a tiny example, but in general I often find writing a test that will cover every hypothetical case isn't realistic. Anyone here use this testing in the wild? Where's it most useful? Do you have the issue I described? Is there an easy way to overcome it?

rowanG077 1y ago
On more complex logic you can often reason about many invariants. I don't think using sorting and then comparing to the existing sort is a good example. Precisely for the reason you mention. A sort itself is a good example though. You can do much better then just compare to an existing sort implementation. Because you can easily check if an array is sorted with a linear scan. If your sort is stable you can do an additional check that for all pairs of adjacent elements that compare as equal the order of them is the same as in the input array.
thekoma 1y ago
Indeed. For a sorting algorithm it would be more insightful to show it test the actual property of something that is sorted: every consecutive element is larger than the previous one (or the other way around). You don’t need a sorting function to test the “sorted” property.
aflukasz 1y ago
> Anyone here use this testing in the wild? Where's it most useful? Do you have the issue I described? Is there an easy way to overcome it? One example would be when you have a naive implementation of some algorithm and you want to introduce a second one, optimized but with much more complex implementation. Then this naive one will act as a model for comparisons. Another case that comes to mind is when you have rather simple properties to test (like: does it finish without a crash, within a given time?, does not cross some boundaries on the output?), and want to easily run over a sensible set of varying inputs.

ed_blackburn 1y ago

I'm using sqlglot to parse hundreds of old mysql back up files to find diffs of schemas. The joys of legacy code. I've found hypothesis to be super helpful for tightening up my parser. I've identified properties (invariants) and built some strategies. I can now generate many more permutations of DDL than I'd thought of before. And I have absolutely confidence in what I'm running. I started off TDD covered the basics. Learned what I needed to learn about the files I'm dealing with, edge cases, sqlglot and then I moved onto Hypothesis for extra confidence. I'm curious to see if it'll help with commands for APIs. I nothing else it'll help me appreciate how liberal my API is when perhaps I don't want it to be?

groundhogstate 1y ago
You might find schemathesis useful for API testing (which IIRC is built on Hypothesis). Certainly helped me find a stack of unhandled edge cases. YMMV, but once I first set up Schemathesis on one of my MVPs, it took several hours to iron out the kinks. But thereafter, if built into CI, I've found it guards against regression quite effectively.
- IanCal 1y ago
  https://schemathesis.readthedocs.io/en/stable/ https://schemathesis.readthedocs.io/en/stable/ This looks really good. I like using hypothesis for APIs so this is a really nice layer on top, thanks for pointing this one out.
- agumonkey 1y ago
  pretty nice tool, helped us poke into our app in ways we didn't foresee also pretty happy to see the reports of thousands or api calls that matched the expected responses, that increases the confidence you have in your system
  - dmitry_dygalo 1y ago
    Hi! Author of Schemathesis here, really glad to hear it helped you uncover those edge cases. If any of those bugs were in public or open-source APIs, I’d love to feature them in our new “trophy case”: https://github.com/schemathesis/schemathesis/issues/new?template=trophy-submission.yml https://github.com/schemathesis/schemathesis/issues/new?temp...

vlade11115 1y ago

One cool application of this is schemathesis. I really enjoyed it, and I found more input validation bugs in my code than I can count. Very useful for public APIs.

rtu8lu 1y ago

Great project. I used it a lot, but now I mostly prefer ad hoc generators. Hypothesis combinators quickly become unmaintainable mess for non-trivial objects. Also, shrinking is not such a big deal when you can generate your data in a complexity-sorted order.

ctenb 1y ago
What is complexity sorted order?
- rtu8lu 1y ago
  When you generate randomized data in an order from the simplest to the most complex.
  - ctenb 1y ago
    Isn't that very wasteful or difficult to do in practice? If you consider that shrinkers generally take lower numbers to be 'simpler' than higher numbers, complexity-ordering requires you to generate all the numbers from low to high
    - rtu8lu 1y ago
      Not really. There are many ways depending on your needs. For example, you can partition your space first, then generate randomly inside each of the subspaces. Let's say I need 200 numbers from -1000 to 999. The first range will be 0 to +99, the second -1 to -100, then +100 to +199, and so on. So, to generate a random number I just need an index and the bounds.

widikidiw 1y ago

[dead]

languagehacker 1y ago

My first impression on property-based testing is that it's likely to introduce flakes that distract engineers from their task at hand, forcing them to investigate a new failed test in an area that they weren't even touching.

toast_x 1y ago

this is pretty sick - i imagine this speeds up hardening a lot. will try it out - thank you!

jamesponddotco 1y ago

Never heard of “property-based testing” before. Coming from Go, I mostly use table and fuzzy tests. Is this approach something that makes sense in Go as well? I see at least one package[1][2] for Go, but I wonder if the approach itself makes sense for the language. [1]: https://github.com/flyingmutant/rapid https://github.com/flyingmutant/rapid [2]: Oh, I just found out about testing/quick.

akshayshah 1y ago
Rapid is excellent. It also integrates with the standard library's fuzz testing, which is handy to persist a high-priority corpus of inputs that have caused bugs in the past. Testing/quick is adequate for small things and doesn't introduce new dependencies, but it's also frozen. Many years ago, the Go team decided that PBT is complex enough that it shouldn't be in stdlib. Here's a small example of a classic PBT technique used in a very practical Go project: https://github.com/connectrpc/connect-go/blob/cb2e11fb88c9a61804043355a619c12d4a30a1a5/header_test.go#L26 https://github.com/connectrpc/connect-go/blob/cb2e11fb88c9a6...

anordin95 1y ago

I am a huge fan of the "Explanations" page of their docs: "These explanation pages are oriented towards deepening your understanding of Hypothesis, including its design philosophy." I feel much more software documentation could greatly benefit from this approach. Describing the "why" and design tradeoffs helps me grok a system far better than the typical quickstart or tutorials which show snippets but offer little understanding. That is, they rarely help me answer: is this going to solve my problem and how?

lwhsiao 1y ago
You would like the Diátaxis framework: https://diataxis.fr/ https://diataxis.fr/ That is the structure they (any many others) are following :).
- anordin95 1y ago
  I know it and do like it! Even for those that follow Diataxis, in my experience the "Explanation" sections are often lacking, especially compared to the "How-To" or "Tutorial" ones. Python is one example that comes to mind. They do have explanations here: https://docs.python.org/3/howto/index.html https://docs.python.org/3/howto/index.html. And, to be fair, they are generally excellent in my opinion! But they're far from front and center and there's much less overall content compared to the other Diataxis types I think (granted, I haven't rigorously checked).