3 ms·

The distinction between property based testing and fuzzing is basically just a rough cluster of vibes. It describes a real difference, but the borders are prett

The distinction between property based testing and fuzzing is basically just a rough cluster of vibes. It describes a real difference, but the borders are pretty vague and precisely deciding which things are fuzzing and which are PBT isn’t really that critical. - Quick running tests, detailed assertions —> PBT - Longer tests, just looking for a crash —> fuzzing. - In between, who knows? https://hypothesis.works/articles/what-is-property-based-testing/ https://hypothesis.works/articles/what-is-property-based-tes...

ncruces 2y ago
Interesting read. I guess I'm mostly interested in on number 2 here: Under this point of view, a property-based testing library is really two parts: 1. A fuzzer. 2. A library of tools for making it easy to construct property-based tests using that fuzzer. What should I (we?) be building on top of Go's fuzzer to make it easier to construct property-based tests? My current strategy is to interpret a byte stream as a sequence of "commands", let those be transitions that drive my "state machine", then test all invariants I can think of at every step.
- deleted 2y ago
  [deleted]
- randomdata 2y ago
  > What should I (we?) be building on top of Go's fuzzer to make it easier to construct property-based tests? What's the intent behind building on top of Go's fuzzer as opposed to an approach like testing/quick takes?
  - ncruces 2y ago
    The testing/quick package is frozen and is not accepting new features. Also: https://news.ycombinator.com/item?id=30385195 https://news.ycombinator.com/item?id=30385195 But, it occurs to me I could try and build on the same API, and compare them for effectiveness.
    - randomdata 2y ago
      > The testing/quick package is frozen and is not accepting new features. Indeed. And respect to software that knows when it is complete, but is there some relevance to this here? > Also: https://news.ycombinator.com/item?id=30385195 https://news.ycombinator.com/item?id=30385195 While the distinction between fuzzing and PBT is, er, fuzzy, what is unquestionable is that PBT is not meant to be coverage driven. It is a means to document behaviour for other developers in a way that is, in some cases, more descriptive than providing a fixed set of examples or other "traditional" ways of providing such documentation. Seeking coverage and "interesting" cases, rather than providing documentation, is decidedly the role of fuzzing.
- PartiallyTyped 2y ago
  I don’t know much about that specific fuzzer; but all you need is a/ consistent and controlled randomness, b/ combinators, c/ reducers. What you describe (commands) is commonly used with model based testing and distributed systems.
  - westurner 2y ago
    TLA+, Formal Methods in Python: FizzBee, Nagini, Deal-solver, Dafny: https://news.ycombinator.com/item?id=39938759 https://news.ycombinator.com/item?id=39938759 : > Python is Turing complete, but does [TLA,] need to be? Is there an in-Python syntax that can be expanded in place by tooling for pretty diffs; How much overlap between existing runtime check DbC decorators and these modeling primitives and feature extraction transforms should there be? (In order to: minimize cognitive overload for human review; sufficiently describe the domains, ranges, complexity costs, inconstant timings, and the necessary and also the possible outcomes given concurrency,) From "S2n-TLS – A C99 implementation of the TLS/SSL protocol" https://news.ycombinator.com/item?id=38510025 https://news.ycombinator.com/item?id=38510025 : > But formal methods (and TLA+ for distributed computation) don't eliminate side channels. [in CPUs e.g. with branch prediction, GPUs, TPUs/NPUs, Hypervisors, OS schedulers, IPC,] Still though, coverage-based fuzzing; From https://news.ycombinator.com/item?id=30786239 https://news.ycombinator.com/item?id=30786239 : > OSS-Fuzz runs CloudFuzz[Lite?] for many open source repos and feeds OSV OpenSSF Vulnerability Format: https://github.com/google/osv#current-data-sources https://github.com/google/osv#current-data-sources From "Automated Unit Test Improvement using Large Language Models at Meta" https://news.ycombinator.com/item?id=39416628 https://news.ycombinator.com/item?id=39416628 : > "Fuzz target generation using LLMs" (2023), OSSF//fuzz-introspector* > gh topic: https://github.com/topics/coverage-guided-fuzzing https://github.com/topics/coverage-guided-fuzzing The Fuzzing computational task is similar to the Genetic Algorithm computational task, in that both explore combinatorial Hilbert spaces of potentially infinite degree and thus there is need for parallelism and thus there is need for partitioning for distributed computation. (But there is no computational oracle to predict that any particular sequence of combinations of inputs under test will deterministically halt on any of the distributed workers, so second-order methods like gradient descent help to skip over apparently desolate territory when the error hasn't changed in awhile) The Fuzzing computational task: partition the set of all combinations of inputs for distributed execution with execute-once or consensus to resolve redundant results. DbC Design-By-Contract patterns include Preconditions and Postconditions (which include tests of Invariance) We test Preconditions to exclude Inputs that do not meet the specified Ranges, and we verify the Ranges of Outputs in Postconditions. We test Invariance to verify that there haven't been side-effects in other scopes; that variables and their attributes haven't changed after the function - the Command - returns. DbC: https://en.wikipedia.org/wiki/Design_by_contract https://en.wikipedia.org/wiki/Design_by_contract : > Design by contract has its roots in work on formal verification, formal specification and Hoare logic. TLA+ > Language: https://en.wikipedia.org/wiki/TLA%2B#Language https://en.wikipedia.org/wiki/TLA%2B#Language Formal verification: https://en.wikipedia.org/wiki/Formal_verification https://en.wikipedia.org/wiki/Formal_verification From https://news.ycombinator.com/item?id=38138319 https://news.ycombinator.com/item?id=38138319 : > Property testing: https://en.wikipedia.org/wiki/Property_testing https://en.wikipedia.org/wiki/Property_testing > awesome-python-testing#property-based-testing: https://github.com/cleder/awesome-python-testing#property-based-testing https://github.com/cleder/awesome-python-testing#property-ba... > Fuzzing: https://en.wikipedia.org/wiki/Fuzzing https://en.wikipedia.org/wiki/Fuzzing Software testing > Categorization > [..., Property testing, Metamorphic testing] https://en.wikipedia.org/wiki/Software_testing#Categorization https://en.wikipedia.org/wiki/Software_testing#Categorizatio... -- From https://news.ycombinator.com/item?id=28494885#28513982 https://news.ycombinator.com/item?id=28494885#28513982 : > https://github.com/dafny-lang/dafny https://github.com/dafny-lang/dafny #read-more > Dafny Cheat Sheet: https://docs.google.com/document/d/1kz5_yqzhrEyXII96eCF1YoHZhnb_6dzv-K3u79bMMis/ https://docs.google.com/document/d/1kz5_yqzhrEyXII96eCF1YoHZ... > Looks like there's a Haskell-to-Dafny converter. haskell2dafny: https://gitlab.doc.ic.ac.uk/dcw/haskell-subset-to-dafny-translator/-/tree/master/05.haskell2dafny https://gitlab.doc.ic.ac.uk/dcw/haskell-subset-to-dafny-tran... -- Controlled randomness: tests of randomness, random uniform not random norm, rngd, tests of randomness: From https://news.ycombinator.com/item?id=40630177 https://news.ycombinator.com/item?id=40630177 : > google/paranoid_crypto.lib.randomness_tests: https://github.com/google/paranoid_crypto/tree/main/paranoid_crypto/lib/randomness_tests https://github.com/google/paranoid_crypto/tree/main/paranoid... docs: https://github.com/google/paranoid_crypto/blob/main/docs/randomness_tests.md#tests https://github.com/google/paranoid_crypto/blob/main/docs/ran...
yen223 2y ago
Unit tests are fuzzers if your system is chaotic enough

ncruces 2y ago

Interesting read. I guess I'm mostly interested in on number 2 here: Under this point of view, a property-based testing library is really two parts: 1. A fuzzer. 2. A library of tools for making it easy to construct property-based tests using that fuzzer. What should I (we?) be building on top of Go's fuzzer to make it easier to construct property-based tests? My current strategy is to interpret a byte stream as a sequence of "commands", let those be transitions that drive my "state machine", then test all invariants I can think of at every step.

deleted 2y ago
[deleted]
randomdata 2y ago
> What should I (we?) be building on top of Go's fuzzer to make it easier to construct property-based tests? What's the intent behind building on top of Go's fuzzer as opposed to an approach like testing/quick takes?
- ncruces 2y ago
  The testing/quick package is frozen and is not accepting new features. Also: https://news.ycombinator.com/item?id=30385195 https://news.ycombinator.com/item?id=30385195 But, it occurs to me I could try and build on the same API, and compare them for effectiveness.
  - randomdata 2y ago
    > The testing/quick package is frozen and is not accepting new features. Indeed. And respect to software that knows when it is complete, but is there some relevance to this here? > Also: https://news.ycombinator.com/item?id=30385195 https://news.ycombinator.com/item?id=30385195 While the distinction between fuzzing and PBT is, er, fuzzy, what is unquestionable is that PBT is not meant to be coverage driven. It is a means to document behaviour for other developers in a way that is, in some cases, more descriptive than providing a fixed set of examples or other "traditional" ways of providing such documentation. Seeking coverage and "interesting" cases, rather than providing documentation, is decidedly the role of fuzzing.
PartiallyTyped 2y ago
I don’t know much about that specific fuzzer; but all you need is a/ consistent and controlled randomness, b/ combinators, c/ reducers. What you describe (commands) is commonly used with model based testing and distributed systems.
- westurner 2y ago
  TLA+, Formal Methods in Python: FizzBee, Nagini, Deal-solver, Dafny: https://news.ycombinator.com/item?id=39938759 https://news.ycombinator.com/item?id=39938759 : > Python is Turing complete, but does [TLA,] need to be? Is there an in-Python syntax that can be expanded in place by tooling for pretty diffs; How much overlap between existing runtime check DbC decorators and these modeling primitives and feature extraction transforms should there be? (In order to: minimize cognitive overload for human review; sufficiently describe the domains, ranges, complexity costs, inconstant timings, and the necessary and also the possible outcomes given concurrency,) From "S2n-TLS – A C99 implementation of the TLS/SSL protocol" https://news.ycombinator.com/item?id=38510025 https://news.ycombinator.com/item?id=38510025 : > But formal methods (and TLA+ for distributed computation) don't eliminate side channels. [in CPUs e.g. with branch prediction, GPUs, TPUs/NPUs, Hypervisors, OS schedulers, IPC,] Still though, coverage-based fuzzing; From https://news.ycombinator.com/item?id=30786239 https://news.ycombinator.com/item?id=30786239 : > OSS-Fuzz runs CloudFuzz[Lite?] for many open source repos and feeds OSV OpenSSF Vulnerability Format: https://github.com/google/osv#current-data-sources https://github.com/google/osv#current-data-sources From "Automated Unit Test Improvement using Large Language Models at Meta" https://news.ycombinator.com/item?id=39416628 https://news.ycombinator.com/item?id=39416628 : > "Fuzz target generation using LLMs" (2023), OSSF//fuzz-introspector* > gh topic: https://github.com/topics/coverage-guided-fuzzing https://github.com/topics/coverage-guided-fuzzing The Fuzzing computational task is similar to the Genetic Algorithm computational task, in that both explore combinatorial Hilbert spaces of potentially infinite degree and thus there is need for parallelism and thus there is need for partitioning for distributed computation. (But there is no computational oracle to predict that any particular sequence of combinations of inputs under test will deterministically halt on any of the distributed workers, so second-order methods like gradient descent help to skip over apparently desolate territory when the error hasn't changed in awhile) The Fuzzing computational task: partition the set of all combinations of inputs for distributed execution with execute-once or consensus to resolve redundant results. DbC Design-By-Contract patterns include Preconditions and Postconditions (which include tests of Invariance) We test Preconditions to exclude Inputs that do not meet the specified Ranges, and we verify the Ranges of Outputs in Postconditions. We test Invariance to verify that there haven't been side-effects in other scopes; that variables and their attributes haven't changed after the function - the Command - returns. DbC: https://en.wikipedia.org/wiki/Design_by_contract https://en.wikipedia.org/wiki/Design_by_contract : > Design by contract has its roots in work on formal verification, formal specification and Hoare logic. TLA+ > Language: https://en.wikipedia.org/wiki/TLA%2B#Language https://en.wikipedia.org/wiki/TLA%2B#Language Formal verification: https://en.wikipedia.org/wiki/Formal_verification https://en.wikipedia.org/wiki/Formal_verification From https://news.ycombinator.com/item?id=38138319 https://news.ycombinator.com/item?id=38138319 : > Property testing: https://en.wikipedia.org/wiki/Property_testing https://en.wikipedia.org/wiki/Property_testing > awesome-python-testing#property-based-testing: https://github.com/cleder/awesome-python-testing#property-based-testing https://github.com/cleder/awesome-python-testing#property-ba... > Fuzzing: https://en.wikipedia.org/wiki/Fuzzing https://en.wikipedia.org/wiki/Fuzzing Software testing > Categorization > [..., Property testing, Metamorphic testing] https://en.wikipedia.org/wiki/Software_testing#Categorization https://en.wikipedia.org/wiki/Software_testing#Categorizatio... -- From https://news.ycombinator.com/item?id=28494885#28513982 https://news.ycombinator.com/item?id=28494885#28513982 : > https://github.com/dafny-lang/dafny https://github.com/dafny-lang/dafny #read-more > Dafny Cheat Sheet: https://docs.google.com/document/d/1kz5_yqzhrEyXII96eCF1YoHZhnb_6dzv-K3u79bMMis/ https://docs.google.com/document/d/1kz5_yqzhrEyXII96eCF1YoHZ... > Looks like there's a Haskell-to-Dafny converter. haskell2dafny: https://gitlab.doc.ic.ac.uk/dcw/haskell-subset-to-dafny-translator/-/tree/master/05.haskell2dafny https://gitlab.doc.ic.ac.uk/dcw/haskell-subset-to-dafny-tran... -- Controlled randomness: tests of randomness, random uniform not random norm, rngd, tests of randomness: From https://news.ycombinator.com/item?id=40630177 https://news.ycombinator.com/item?id=40630177 : > google/paranoid_crypto.lib.randomness_tests: https://github.com/google/paranoid_crypto/tree/main/paranoid_crypto/lib/randomness_tests https://github.com/google/paranoid_crypto/tree/main/paranoid... docs: https://github.com/google/paranoid_crypto/blob/main/docs/randomness_tests.md#tests https://github.com/google/paranoid_crypto/blob/main/docs/ran...

yen223 2y ago

Unit tests are fuzzers if your system is chaotic enough