8 ms·

I would love to used PBT more, but many tests I write have only one answer per input. Think sum like aggregations. For then it's not clear how would one derive

I would love to used PBT more, but many tests I write have only one answer per input. Think sum like aggregations. For then it's not clear how would one derive the answer from the generated inputs, that is what code is for. But PBT can be great for pruning out crashes you don't expect while parsing.

chriswarbo 1y ago
> I would love to used PBT more, but many tests I write have only one answer per input. Think sum like aggregations. Not quite sure what you mean by "only one answer per input" (that it's a function, i.e. a 1:1 mapping?), but there are of lots of properties that aggregations might typically need to satisfy, e.g. off the top of my head: # Identity element forAll(pre, post) { assertEqual( agg(pre ++ [agg([])] ++ post), agg(pre ++ post) ) } # Invariant to order forAll(elems, seed) { assertEqual( agg(elems), agg(permute(elems, seed)) ) } # Left-associative forAll(xs, ys, zs) { assertEqual( agg([agg(xs ++ ys)] ++ zs), agg(xs ++ ys ++ zs) ) } # Right-associative forAll(xs, ys, zs) { assertEqual( agg(xs ++ [agg(yz ++ zs)]), agg(xs ++ ys ++ zs) ) } (FYI these typical properties of a (commutative) monoid, which is an algebraic structure that describes many "aggregation-like" operations)
- jononor 1y ago
  Numerical aggregates often have the property that their output is in the range min and max of the input. An aggregate on discrete values my have the property that the output is one of the elements in the input. It may also have a no-NaN property, or maybe no-NaN unless NaN in input.
Sukera 1y ago
For aggregation-like things, the interesting properties are often about the properties of the accumulation function, and not the entire aggregation, which should then be correct by extension. So for your `sum` example, you'd use PBT to test that your `+` works first, and only then coming up with things that should hold on top of that when repeatedly applying your operation. For example, once you have a list of numbers and know it's sum, adding one additional number to the list and taking the sum again should lead to the same result as if you had added the number to the sum directly (barring non-associative shenanigans like with floating point - but that should have already been found in the addition step ;) ). There's a bunch of these kinds of patterns (the above was inspired by [0]) that are useful in practice, but unfortunately rarely talked about. I suppose that's because most people end up replicating their TDD workflows and just throwing more randomness at it, instead of throwing more properties at their code. [0] https://fsharpforfunandprofit.com/posts/property-based-testing-2/ https://fsharpforfunandprofit.com/posts/property-based-testi...
tikhonj 1y ago
Good example for sums: you can write a property that checks whether the sum of a list is the same before and after you randomly shuffle the elements. In practice sum is a sufficiently well-understood function that this property will only catch the edge-cases people know about up-front (integer overflow, floating point issues...). But for more complex cases, this kind of property will catch problems you didn't think about. And even if you decide that the bug is not important—sometimes we have no real choice but to live with these edgecases—at least you'll know about them explicitly and be able to document them.

chriswarbo 1y ago

> I would love to used PBT more, but many tests I write have only one answer per input. Think sum like aggregations. Not quite sure what you mean by "only one answer per input" (that it's a function, i.e. a 1:1 mapping?), but there are of lots of properties that aggregations might typically need to satisfy, e.g. off the top of my head: # Identity element forAll(pre, post) { assertEqual( agg(pre ++ [agg([])] ++ post), agg(pre ++ post) ) } # Invariant to order forAll(elems, seed) { assertEqual( agg(elems), agg(permute(elems, seed)) ) } # Left-associative forAll(xs, ys, zs) { assertEqual( agg([agg(xs ++ ys)] ++ zs), agg(xs ++ ys ++ zs) ) } # Right-associative forAll(xs, ys, zs) { assertEqual( agg(xs ++ [agg(yz ++ zs)]), agg(xs ++ ys ++ zs) ) } (FYI these typical properties of a (commutative) monoid, which is an algebraic structure that describes many "aggregation-like" operations)

jononor 1y ago
Numerical aggregates often have the property that their output is in the range min and max of the input. An aggregate on discrete values my have the property that the output is one of the elements in the input. It may also have a no-NaN property, or maybe no-NaN unless NaN in input.

Sukera 1y ago

For aggregation-like things, the interesting properties are often about the properties of the accumulation function, and not the entire aggregation, which should then be correct by extension. So for your `sum` example, you'd use PBT to test that your `+` works first, and only then coming up with things that should hold on top of that when repeatedly applying your operation. For example, once you have a list of numbers and know it's sum, adding one additional number to the list and taking the sum again should lead to the same result as if you had added the number to the sum directly (barring non-associative shenanigans like with floating point - but that should have already been found in the addition step ;) ). There's a bunch of these kinds of patterns (the above was inspired by [0]) that are useful in practice, but unfortunately rarely talked about. I suppose that's because most people end up replicating their TDD workflows and just throwing more randomness at it, instead of throwing more properties at their code. [0] https://fsharpforfunandprofit.com/posts/property-based-testing-2/ https://fsharpforfunandprofit.com/posts/property-based-testi...

tikhonj 1y ago

Good example for sums: you can write a property that checks whether the sum of a list is the same before and after you randomly shuffle the elements. In practice sum is a sufficiently well-understood function that this property will only catch the edge-cases people know about up-front (integer overflow, floating point issues...). But for more complex cases, this kind of property will catch problems you didn't think about. And even if you decide that the bug is not important—sometimes we have no real choice but to live with these edgecases—at least you'll know about them explicitly and be able to document them.