21 ms·

I, probably like many others, want struct-based params. I am thinking of a lib that can fuzz a func w/ struct param(s). It takes the struct params, extracts fi

I, probably like many others, want struct-based params. I am thinking of a lib that can fuzz a func w/ struct param(s). It takes the struct params, extracts fields into a deterministic list, builds a dynamic function via reflect.MakeFunc where each field is an arg and the body of the puts args on structs and invokes the original function. That dynamically created function is then passed to f.Fuzz. Does a lib like this exist?

morelisp 5y ago
testing/quick contains an interface and basic implementation for randomly generating arbitrary structs. https://pkg.go.dev/testing/quick#Value https://pkg.go.dev/testing/quick#Value For traditional fuzzing you usually want a "blacker" barrier between the input and processing. E.g. in an algorithm with exponential growth it's not helpful to know that a raw 1000000 element input slice runs out of memory; it is helpful to know if a mere 100 bytes over the wire can somehow trigger that. If you do need arbitrary input, it's easy enough to treat the fuzz input as JSON or whatever. But most code that you're interested in testing starting from a truly arbitrary state takes only a small number of scalars or a byte stream, not complex structures. If you have complex structures it's probably because you have complex invariants arbitrary data won't satisfy. (And you're probably as much or more interested in verifying your code maintains those invariants during whatever preparatory steps you have which outputs those structures, not just their use as inputs.)
- kodablah 5y ago
  > For traditional fuzzing you usually want a "blacker" barrier between the input and processing. They clearly didn't want it that opaque and bytes only since they accept primitives and accept arbitrary param count. Why not just accept byte slice only and make people binary encode even primitives? The reason is it takes a long time to build a corpus when unnecessarily making the fuzzer figure out UTF-8, JSON, little-endian ints, etc. Using structured typing is a reasonable approach for an in-language fuzzer. They currently have a user-defined length of user-defined types, so it's only natural to translate from more complex types and save the generator time.
  - morelisp 5y ago
    > Why not just accept byte slice only and make people binary encode even primitives? It's a good question since this is what go-fuzz did and it worked pretty well; I don't see the reason in the design documentation. But I don't think it's "clearly" because > it takes a long time to build a corpus when unnecessarily making the fuzzer figure out UTF-8, JSON, little-endian ints, etc. Not because it doesn't in some cases (JSON it can, UTF-8 and le ints not really), but because having the fuzzer be aware of that doesn't seem to be the best way to solve the problem; this has been discussed in lots of fuzzing literature from AFL onwards. Structurally-aware mutators might as well also be []byte mutators so you can also more quickly generate JSONL, JSON-in-HTTP-request, JSON-in-CSV-column, and not only bare JSON. For the specific issue of corpus bootstrapping and measuring coverage, it would seem to be more greatly helped by the approach mentioned in "Instrument specific packages only" in the draft proposal. And this is putting aside my other point: For any complex structure, you have non-well-formed values with other semantic invariants which randomly-generated structures won't satisfy. (And on a more minor implementation note, you may also have some private fields.) To solve both you'll need to go all the way to custom mutators, which is interesting but a much larger issue - specifically, issue 48815. What kind of use case are you considering where you want to start with an completely-unconstrained-yet-complex struct and not some external representation of that struct?

morelisp 5y ago

testing/quick contains an interface and basic implementation for randomly generating arbitrary structs. https://pkg.go.dev/testing/quick#Value https://pkg.go.dev/testing/quick#Value For traditional fuzzing you usually want a "blacker" barrier between the input and processing. E.g. in an algorithm with exponential growth it's not helpful to know that a raw 1000000 element input slice runs out of memory; it is helpful to know if a mere 100 bytes over the wire can somehow trigger that. If you do need arbitrary input, it's easy enough to treat the fuzz input as JSON or whatever. But most code that you're interested in testing starting from a truly arbitrary state takes only a small number of scalars or a byte stream, not complex structures. If you have complex structures it's probably because you have complex invariants arbitrary data won't satisfy. (And you're probably as much or more interested in verifying your code maintains those invariants during whatever preparatory steps you have which outputs those structures, not just their use as inputs.)

kodablah 5y ago
> For traditional fuzzing you usually want a "blacker" barrier between the input and processing. They clearly didn't want it that opaque and bytes only since they accept primitives and accept arbitrary param count. Why not just accept byte slice only and make people binary encode even primitives? The reason is it takes a long time to build a corpus when unnecessarily making the fuzzer figure out UTF-8, JSON, little-endian ints, etc. Using structured typing is a reasonable approach for an in-language fuzzer. They currently have a user-defined length of user-defined types, so it's only natural to translate from more complex types and save the generator time.
- morelisp 5y ago
  > Why not just accept byte slice only and make people binary encode even primitives? It's a good question since this is what go-fuzz did and it worked pretty well; I don't see the reason in the design documentation. But I don't think it's "clearly" because > it takes a long time to build a corpus when unnecessarily making the fuzzer figure out UTF-8, JSON, little-endian ints, etc. Not because it doesn't in some cases (JSON it can, UTF-8 and le ints not really), but because having the fuzzer be aware of that doesn't seem to be the best way to solve the problem; this has been discussed in lots of fuzzing literature from AFL onwards. Structurally-aware mutators might as well also be []byte mutators so you can also more quickly generate JSONL, JSON-in-HTTP-request, JSON-in-CSV-column, and not only bare JSON. For the specific issue of corpus bootstrapping and measuring coverage, it would seem to be more greatly helped by the approach mentioned in "Instrument specific packages only" in the draft proposal. And this is putting aside my other point: For any complex structure, you have non-well-formed values with other semantic invariants which randomly-generated structures won't satisfy. (And on a more minor implementation note, you may also have some private fields.) To solve both you'll need to go all the way to custom mutators, which is interesting but a much larger issue - specifically, issue 48815. What kind of use case are you considering where you want to start with an completely-unconstrained-yet-complex struct and not some external representation of that struct?