4 ms·

Would be very valuable if someone could write a summary of novel ideas for practitioners (if there are any).

chriswarbo 2y ago
Not read the whole thing yet, but the idea to represent generators as free monads which can be interpreted in multiple ways is very promising. In particular, this may be a way to unify various generator implementations, like the random generators from QuickCheck, hedgehog, falsify, SmartCheck, etc. with the enumerations provided by Smallcheck, FEAT, LeanCheck, etc. and maybe even the weirder ones like LazySmallcheck, extrapolate, fitspec, speculate, lazy-search, spectacular, the various logic-programming implementations of property-checking, etc. In principle, the same framework would extend to parsers too (atto/mega/parsec, etc.), though parsing is often a bottleneck in real applications, so many would avoid such abstraction/indirection. Still, it would be nice if libraries could define parsers for their datatypes, in such a way that it's easily reusable as a generator in tests (without having the library package depending on any test-generator packages). They also demonstrate finding the "derivative" of a generator (i.e. the result of fixing its next choice to a particular outcome), which is useful for avoiding choices that cause preconditions to fail (something which many PBT users seem to struggle with: assuming that the framework will somehow avoid problematic values, rather than just rejection-sampling them as most do).
chriswarbo 2y ago
One of the later chapters describes the Tyche tool, which seems to be available as this VisualStudio(Code?) extension https://marketplace.visualstudio.com/items?itemName=HarrisonGoldstein.tyche https://marketplace.visualstudio.com/items?itemName=Harrison... and has a standalone version at https://tyche-pbt.github.io/tyche-extension/ https://tyche-pbt.github.io/tyche-extension/ It reads JSON data (described in the paper), which can be generated during property checking, e.g. using https://github.com/tyche-pbt/tyche-quickcheck https://github.com/tyche-pbt/tyche-quickcheck or a Hypothesis equivalent.

Not read the whole thing yet, but the idea to represent generators as free monads which can be interpreted in multiple ways is very promising. In particular, this may be a way to unify various generator implementations, like the random generators from QuickCheck, hedgehog, falsify, SmartCheck, etc. with the enumerations provided by Smallcheck, FEAT, LeanCheck, etc. and maybe even the weirder ones like LazySmallcheck, extrapolate, fitspec, speculate, lazy-search, spectacular, the various logic-programming implementations of property-checking, etc. In principle, the same framework would extend to parsers too (atto/mega/parsec, etc.), though parsing is often a bottleneck in real applications, so many would avoid such abstraction/indirection. Still, it would be nice if libraries could define parsers for their datatypes, in such a way that it's easily reusable as a generator in tests (without having the library package depending on any test-generator packages). They also demonstrate finding the "derivative" of a generator (i.e. the result of fixing its next choice to a particular outcome), which is useful for avoiding choices that cause preconditions to fail (something which many PBT users seem to struggle with: assuming that the framework will somehow avoid problematic values, rather than just rejection-sampling them as most do).

chriswarbo 2y ago

One of the later chapters describes the Tyche tool, which seems to be available as this VisualStudio(Code?) extension https://marketplace.visualstudio.com/items?itemName=HarrisonGoldstein.tyche https://marketplace.visualstudio.com/items?itemName=Harrison... and has a standalone version at https://tyche-pbt.github.io/tyche-extension/ https://tyche-pbt.github.io/tyche-extension/ It reads JSON data (described in the paper), which can be generated during property checking, e.g. using https://github.com/tyche-pbt/tyche-quickcheck https://github.com/tyche-pbt/tyche-quickcheck or a Hypothesis equivalent.