4 ms·

What are the benefits of building out a whole separate part of the test framework that handles this? Is there a way to fuzz non-string inputs?

by kubb 5y ago

masklinn 5y ago
> What are the benefits of building out a whole separate part of the test framework that handles this? Instead of making it part of the base runner? Maybe so the beta bits can work and it’ll be folded in more directly afterwards? Also if it includes coverage guiding being able to know that a run is fuzzing or non-fuzzing would avoid having to include the fuzzing instrumentation for non-fuzzing run, mayhaps? > Is there a way to fuzz non-string inputs? From the design doc it looks like you can have any number of parameters and > Fuzzing of built-in types (e.g. simple types, maps, arrays) and types which implement the BinaryMarshaler and TextMarshaler interfaces are supported.
- kubb 5y ago
  I was thinking a library to generate fuzzed inputs that you could use in normal tests. It's good to hear that it's not limited to strings.
  - masklinn 5y ago
    > I was thinking a library to generate fuzzed inputs that you could use in normal tests. The fuzzer input is random data. You don't need a special library to generate random data. In fact that's what the post tells you with respect to type compatibility: > types which implement the BinaryMarshaler and TextMarshaler interfaces are supported these are just tools to convert from/to binary (unstructured or utf8).
    - kubb 5y ago
      Are you sure it's random bytes? Some fuzzers start with a given input and then mutate it to increase coverage of the code under test.
      - masklinn 5y ago
        > Some fuzzers start with a given input and then mutate it to increase coverage of the code under test. Yes, and this one does, but you requested a library to generate inputs didn't you? A library can't get coverage feedback from the SUT, which as I wrote in my previous comment would be why you'd be building a dedicated test framework.
alephnan 5y ago
> Is there a way to fuzz non-string inputs? Fuzzing usually revolves around strings because of escape characters, escape sequences. There is a much larger set of string characters than there are for the 10 or so numeric digits. Numbers don’t have the same problems that strings do, because numbers are usually interpreted only as data, whereas strings can be interpreted as data or computation.
- pdpi 5y ago
  > Fuzzing usually revolves around strings because of escape characters, escape sequences. Not always. AFL has been used to detect issues around processing plain old binary data (eg https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-8637 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-8637) I would argue that anything involving a parser of some description (either binary or text-based) is a good candidate for fuzzing.
tjpnz 5y ago
>What are the benefits of building out a whole separate part of the test framework that handles this? I'm guessing that you won't always want to execute these alongside other tests. Go has also taken the same approach with benchmarks.
jlouis 5y ago
Fuzz testing is expensive. And you gotta start somewhere. String inputs is a good start, and you can use those to test other inputs by factoring through conversion functions.

What are the benefits of building out a whole separate part of the test framework that handles this? Is there a way to fuzz non-string inputs?

by kubb 5y ago

What are the benefits of building out a whole separate part of the test framework that handles this? Is there a way to fuzz non-string inputs?

masklinn 5y ago

> What are the benefits of building out a whole separate part of the test framework that handles this? Instead of making it part of the base runner? Maybe so the beta bits can work and it’ll be folded in more directly afterwards? Also if it includes coverage guiding being able to know that a run is fuzzing or non-fuzzing would avoid having to include the fuzzing instrumentation for non-fuzzing run, mayhaps? > Is there a way to fuzz non-string inputs? From the design doc it looks like you can have any number of parameters and > Fuzzing of built-in types (e.g. simple types, maps, arrays) and types which implement the BinaryMarshaler and TextMarshaler interfaces are supported.

kubb 5y ago
I was thinking a library to generate fuzzed inputs that you could use in normal tests. It's good to hear that it's not limited to strings.
- masklinn 5y ago
  > I was thinking a library to generate fuzzed inputs that you could use in normal tests. The fuzzer input is random data. You don't need a special library to generate random data. In fact that's what the post tells you with respect to type compatibility: > types which implement the BinaryMarshaler and TextMarshaler interfaces are supported these are just tools to convert from/to binary (unstructured or utf8).
  - kubb 5y ago
    Are you sure it's random bytes? Some fuzzers start with a given input and then mutate it to increase coverage of the code under test.
    - masklinn 5y ago
      > Some fuzzers start with a given input and then mutate it to increase coverage of the code under test. Yes, and this one does, but you requested a library to generate inputs didn't you? A library can't get coverage feedback from the SUT, which as I wrote in my previous comment would be why you'd be building a dedicated test framework.

alephnan 5y ago

> Is there a way to fuzz non-string inputs? Fuzzing usually revolves around strings because of escape characters, escape sequences. There is a much larger set of string characters than there are for the 10 or so numeric digits. Numbers don’t have the same problems that strings do, because numbers are usually interpreted only as data, whereas strings can be interpreted as data or computation.

pdpi 5y ago
> Fuzzing usually revolves around strings because of escape characters, escape sequences. Not always. AFL has been used to detect issues around processing plain old binary data (eg https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-8637 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-8637) I would argue that anything involving a parser of some description (either binary or text-based) is a good candidate for fuzzing.

tjpnz 5y ago

>What are the benefits of building out a whole separate part of the test framework that handles this? I'm guessing that you won't always want to execute these alongside other tests. Go has also taken the same approach with benchmarks.

jlouis 5y ago

Fuzz testing is expensive. And you gotta start somewhere. String inputs is a good start, and you can use those to test other inputs by factoring through conversion functions.