5 ms·
Why Property Testing Finds Bugs Unit Testing Does Not (2021)
- arnsholt 1y agoI haven't used PBT much, but I did once get a lot of mileage out of it, which was when I was implementing a fairly gnarly edit-distance algorithm. In that case, I used PBT to check that the required properties of metric functions (d(x,x)=0, d(x,y)>0, d(x,y)=d(y,x), d(x,z) <= d(x,y)+d(y,z)) held for my implementation, which helped shake out a fair few stupid implementation mistakes.
- frogulis 1y agoAw man, I was nodding along with the "most examples suck" section and then... it ended :(
- pfdietz 1y agoI'll give you the good example I've been doing for the last two decades: testing a compiler. The complexity here is the complete opposite of the simple toy examples. What are the edge cases of an optimizing compiler? How do you even approach them, if they're buried deep in a chain of transformations? The properties are simple things like "the compiler shouldn't crash, the compiled code shouldn't crash, and code compiled with different optimization levels should do the same thing." This assumes the randomly generated code doesn't touch undefined behavior in the language spec. Here's a recent example of a bug found by this approach. The Common Lisp code stimulating the bug has been automatically minimized: https://bugs.launchpad.net/sbcl/+bug/2109837 https://bugs.launchpad.net/sbcl/+bug/2109837 with the bug fix https://sourceforge.net/p/sbcl/sbcl/ci/1abebf7addda1a43d6d24fa8d1d7eefd36a7b3af/ https://sourceforge.net/p/sbcl/sbcl/ci/1abebf7addda1a43d6d24...
- chipsrafferty 1y agoUnfortunately I have no idea what I'm looking at here.
- pfdietz 1y agoThe function constructs two lambda expressions (source code for anonymous functions) that should be equivalent. One has some extra declarations. It then compiles the two lambda expressions and calls the compiled code on the same arguments, and gets different values (which is the bug).
- hikarudo 1y agoHere's a great talk by John Hughes, one of the authors of QuickCheck, with real-life examples: https://www.youtube.com/watch?v=zi0rHwfiX1Q https://www.youtube.com/watch?v=zi0rHwfiX1Q
- frogulis 1y agoThanks, that's gotta be one of the best talks I've ever watched. A passionate speaker, talking about fascinating, useful tech, giving specific real world examples of utility, AND showing how to actually apply it for interesting "dirty" situations.
- nickpsecurity 1y agoThey should put some good ones in the article.
- djoldman 1y agoIsn't unit testing a subset of property testing? Seems like a unit test tests specific input and property testing tests more than one input.
- skybrian 1y agoYes, but converting a test to be data-driven adds some complexity (how do you debug just the failures?) and generating the inputs randomly makes it harder to know what properties to assert. Also, if you never look at the test data, it might give you false confidence about how thoroughly your code is being tested.
- stonemetal12 1y agoYes, Property testing is a generalization of unit testing.
- andreareina 1y agoThey’re different, complementary approaches. Example based testing has a known input and output. Unless you have an oracle (not usually the case) you won’t know the exact output you should get from a generated input. So instead you depend on something that should be true no matter the input. e.g. if you have a serialize/deserialize pair then `deserialize(serialize(x))` better equal `x`.
- Jtsummers 1y agoSort of. But you can also use PBT for integration and end-to-end testing, too. A user tries to schedule something and it's overlapping with an existing event, what happens? A user adds 10 events with their various requirements (none overlapping) do they successfully schedule? Set the expectations and model the expected behavior, verify that the system matches that expectation. The approach works at all testing scales.
- chriswarbo 1y agoYes, if you've got a mixture of unit tests and property tests then you can write them all using a single PBT framework, rather than needing two test frameworks.
- esafak 1y agoI independently discovered PBT when I was a junior, and suggested we use it. My coworkers rejected it because tests should be predictable, and it's the programmer's job to pick the edge cases.
- skybrian 1y agoYou do need to be able to reproduce a test failure, which can be done by printing the inputs or the random seed used in a way that makes it trivial to rerun it.
- alkonaut 1y agoI think it goes without saying. PBT shouldn't be random-random (e.g. use a timestamp or cryptographic seed), it should be deterministically pseudorandom if it uses random values. You shouldn't be able to run it 10 times and get 9 pass and one failure. It's either 10 passes or 10 failures.
- Akronymus 1y ago> You shouldn't be able to run it 10 times and get 9 pass and one failure. It's either 10 passes or 10 failures. With property based testing, it actually CAN be 9 passes and 1 failure, because that one single fail can be hitting an edge case the others just aren't. In fact, only a few failures are more likely than it being all failures
- bluGill 1y agoThat is the one thing about PBT that worries me. I can write code and all tests pass, then next week the edge case I missed randomly is hit by a coworker who now has to figure out why their change broke my code (it didn't). I can tell you from experience that random failures cause loss of trust. People learn to ignore failures and just keep hitting rebuild until the tests pass. People will not investigate test failures in code that they don't understand.
- pfdietz 1y agoOne can use property testing to automatically generate unit test inputs. The property becomes "does this do something new that I wanted to test for but wasn't?" (or, rather, the negation; inputs "pass" and can be ignored if they do nothing new.) This could be code coverage, or it could be killing code mutants.
- bluGill 1y agoUnfortunately it ends before it gets to the good stuff. It has me interested that maybe PBT can find some bugs that unit testing wouldn't - however I'm not sure how to write a PBT that would catch those bugs. The obvious tests that drive PBT advocates to drink are not interesting - unit tests will catch all the errors and because there is no randomness they will catch the errors faster in general. However how do I write a property for non-trivial functions it not clear at all, and the article stops.
- drowsspa 1y agoYeah, for simple nearly mathematical functions it's quite clear how to do it. I find it hard to extend this to more business-focused inputs
- zelos 1y agoI remember using something similar a long time ago - basically fuzz testing, I suppose you could call it property based testing where the property is "whatever edits the user does, we should be able to save and reopen the document without crashing". It found so many bugs: file corruption, crashes, memory leaks, pathological performance issues. The kind of issues that standard unit testing doesn't find.
- bpshaver 1y agoDoes the article he links to towards the end of the article address your concerns? > Without complex input spaces, there's no explosion of edge cases, which minimizes the actual benefit of PBT. The real benefits come when you have complex input spaces. Unfortunately, you need to be good at PBT to write complex input strategies. I wrote a bit about it here... Here's the link: https://www.hillelwayne.com/post/property-testing-complex-inputs/ https://www.hillelwayne.com/post/property-testing-complex-in...
- bluGill 1y agoIt is a start, but I still feel like I'm not sure how I'd apply that to my own domain.
- moi2388 1y agoWhy are we even testing to begin with, and not using theorem provers like lean to prove without any doubt that our commutative function is indeed commutative?
- klysm 1y agoI agree with the sentiment, but formal methods are tricky
- pfdietz 1y agoIt's an example of Alfred Whitehead's quote: “Civilization advances by extending the number of important operations which we can perform without thinking about them.” We can do property based testing with less thinking than if we try to prove correctness. We are exploiting the ability of the computer to run millions or even billions of tests. It's the enormous power of today's computers that enables this to work. The theorem prover approach would work only if it could be automatic: press a button, get the proof (after some acceptable delay). Otherwise, look at all the expensive manual effort you just signed up for. You might think that as computers get faster, theorem proving becomes easier. But testing becomes easier also. It's not clear testing will ever lose this race.
- recroad 1y agoCan you please explain more and maybe give some examples that resonate with people who don't have the understanding that you do?
- moi2388 1y agoAn example would be critical systems such as defence and Aerospace, where they use for example Ada Spark to formally proof certain bugs cannot occur
- bluGill 1y agoMultiple problems. Proving code is only as good as the requirements which are often garbage - the customer often doesn't know what they even want. Even if you put in effort, requirements as the proof needs are often very abstract from the customer requirements and so your program can be proved but still be wrong because it doesn't do what the customer really wanted. In any complex program is a reasonable to state that several requirements are wrong and thus even if your prove your code correct it will be wrong. Often the problem itself cannot even be formally defined - a spell checker cannot be proved correct because human languages are not formally defined, not that you can't prove one, just that whatever you prove will be wrong. Many systems are very complex. You can (should!) prove simple algorithms, but put everything together and a proof is not something we can do at all. There are too many halting problems like things in large programs. Tests solve some of the above problems: They can (do not confuse with what they do!) be a simple example of "yes when inputs are exactly x,y,z then I expect that result". A bunch of simple examples that make sense can often be close enough. We do a lot more theorem proving than most people realize. Types which many languages have are a form of formal proof. They don't cover everything, but even in C++ they cover a lot of issues. I think the best answer is a combination: prove the things we know how to prove, and test the rest and hope that between the two we have covered enough to prevent bugs.
- tikhonj 1y agoI wrote some property-based tests for a parsing library. Eventually one of the tests for case-insensitive parsing failed... because it hit the letter "DŽ" which has upper, lower and title-case. The funny thing is that the parsing library was correct and it was the test property that was wrong—but I still learned about an edge case I had never considered! This has been a common pattern for "simpler" property-based tests I've written: I write a test, it fails right away, and it turns out that my code is fine, but my property was wrong. And this is almost always useful; if I write an incorrect property, it means that some assumption I had about my code is incorrect, so the exercise directly improves my conceptual model of whatever I'm doing.
- nottorp 1y agoThat's Unicode, which has a whole section of hell dedicated to it.
- riwsky 1y agoUpdate: It’s actually been divided into multiple sections; if your code assumes a single section, it will break as of the addition of multibyte characters in Unicode v7.1
- atmavatar 1y agoWhen the robots rise up and exterminate the human race, Unicode will be one of the reasons we deserve it.
- rented_mule 1y agoIs the root of the problem Unicode? Or is the root of the problem the complexity of the union of written human languages? To the extent that it's the latter, Unicode is just the messenger.
- nottorp 1y agoI would shed a tear but then I remember that they have not one, not two but four canonical forms... And what do they do about it? They add more emoji. Besides, even if it's justified, it's still sections 7.1-A to 7.3-D of hell.
- metalrain 1y agoI would love to used PBT more, but many tests I write have only one answer per input. Think sum like aggregations. For then it's not clear how would one derive the answer from the generated inputs, that is what code is for. But PBT can be great for pruning out crashes you don't expect while parsing.
- chriswarbo 1y ago> I would love to used PBT more, but many tests I write have only one answer per input. Think sum like aggregations. Not quite sure what you mean by "only one answer per input" (that it's a function, i.e. a 1:1 mapping?), but there are of lots of properties that aggregations might typically need to satisfy, e.g. off the top of my head: # Identity element forAll(pre, post) { assertEqual( agg(pre ++ [agg([])] ++ post), agg(pre ++ post) ) } # Invariant to order forAll(elems, seed) { assertEqual( agg(elems), agg(permute(elems, seed)) ) } # Left-associative forAll(xs, ys, zs) { assertEqual( agg([agg(xs ++ ys)] ++ zs), agg(xs ++ ys ++ zs) ) } # Right-associative forAll(xs, ys, zs) { assertEqual( agg(xs ++ [agg(yz ++ zs)]), agg(xs ++ ys ++ zs) ) } (FYI these typical properties of a (commutative) monoid, which is an algebraic structure that describes many "aggregation-like" operations)
- jononor 1y agoNumerical aggregates often have the property that their output is in the range min and max of the input. An aggregate on discrete values my have the property that the output is one of the elements in the input. It may also have a no-NaN property, or maybe no-NaN unless NaN in input.
- Sukera 1y agoFor aggregation-like things, the interesting properties are often about the properties of the accumulation function, and not the entire aggregation, which should then be correct by extension. So for your `sum` example, you'd use PBT to test that your `+` works first, and only then coming up with things that should hold on top of that when repeatedly applying your operation. For example, once you have a list of numbers and know it's sum, adding one additional number to the list and taking the sum again should lead to the same result as if you had added the number to the sum directly (barring non-associative shenanigans like with floating point - but that should have already been found in the addition step ;) ). There's a bunch of these kinds of patterns (the above was inspired by [0]) that are useful in practice, but unfortunately rarely talked about. I suppose that's because most people end up replicating their TDD workflows and just throwing more randomness at it, instead of throwing more properties at their code. [0] https://fsharpforfunandprofit.com/posts/property-based-testing-2/ https://fsharpforfunandprofit.com/posts/property-based-testi...
- recroad 1y ago> The majority of errors you find with testing are either issues with an entire "partition" of inputs or "boundary" inputs, like INT_MIN. I don't find this to be true at all. Most bugs I find are business scenarios that I didn't consider or mismatches in API expectations etc. Rarely is a bug for me coming from not considering edge cases of min and maximum values for integers, floats etc.