4 ms·

My understanding was that property tests and fuzzing are orthogonal in roughly the same way unit and integration tests are: you do property test to ensure well-

My understanding was that property tests and fuzzing are orthogonal in roughly the same way unit and integration tests are: you do property test to ensure well-delimited pieces of your code satisfy well-defined constraints, and you fuzz to ensure the whole thing just doesn’t go completely nuts when faced with something unusual. But I could be wrong because I never could figure out how to set up fuzzing for some sort of moderately-involved realistic thing (a CSV parser? something like that).

rdtsc 5y ago
I think property based testing could be used for integration tests as well. There are essentially two major steps to use it: 1) Define generators to produce input. These could start as random with some constraints. Say, "positive integers between 10 and 100" and then those can be combined in various ways with other generators like "positive integers between 10 and 100 in a tuple with a utf8 encoded binary between lengths 0 and 256" and so on. 2) Define properties which model your system. Given the inputs from generators, calling your application APIs should keep certain properties true. It should crash in only certain ways, or return only HTTP 200 responses, etc. These could be applied at any level from the simplest pure functions up to an HTTP API. These steps don't have to be taking in the same order. I end up starting with 2 going back to 1, then 2 again etc. The difference from basic fuzzing is that once you find a failure, it has a consistent and automatic way to backtrack and try to simplify the input down to a minimum which still causes the failure. So, if a giant 1Gb worth of JSON fails the fuzz test, it can be hard to see why it failed. With property testing, however, it might be simplified down a simple case with one utf8 encoded string of length 10 which has some specific emoji or something like that.

rdtsc 5y ago

I think property based testing could be used for integration tests as well. There are essentially two major steps to use it: 1) Define generators to produce input. These could start as random with some constraints. Say, "positive integers between 10 and 100" and then those can be combined in various ways with other generators like "positive integers between 10 and 100 in a tuple with a utf8 encoded binary between lengths 0 and 256" and so on. 2) Define properties which model your system. Given the inputs from generators, calling your application APIs should keep certain properties true. It should crash in only certain ways, or return only HTTP 200 responses, etc. These could be applied at any level from the simplest pure functions up to an HTTP API. These steps don't have to be taking in the same order. I end up starting with 2 going back to 1, then 2 again etc. The difference from basic fuzzing is that once you find a failure, it has a consistent and automatic way to backtrack and try to simplify the input down to a minimum which still causes the failure. So, if a giant 1Gb worth of JSON fails the fuzz test, it can be hard to see why it failed. With property testing, however, it might be simplified down a simple case with one utf8 encoded string of length 10 which has some specific emoji or something like that.