4 ms·

I don't get the part about custom mutators: > If the data can’t be parsed into a valid TLV, instead of throwing it away, return a syntactically correct dummy T

I don't get the part about custom mutators: > If the data can’t be parsed into a valid TLV, instead of throwing it away, return a syntactically correct dummy TLV. This can be anything, as long as it can be successfully unpacked. If you're creating a dummy value, how is that better than failing? How does that give your fuzzer better coverage?

pstrateman 3y ago
The file format they choose is difficult for a fuzzer to produce valid examples by random chance. The file format isn't what's being fuzzed, so trying to accept as many things as possible as valid is useful. It's a trick to make the fuzzer faster.
gavinhoward 3y ago
Not an expert, but I am a power user of fuzzers. The problem is that the space of invalid inputs is far larger than the space of valid inputs. Sometimes orders of magnitude larger, say billions or more invalid inputs to one valid input. Naive fuzzing will hit so many error cases that it will hardly produce a valid input. For the ratio that I mentioned, you might run a fuzzer for a billion runs and only get one valid input in the bunch. Using a custom mutator and returning a dummy value will give the fuzzer a starting point from a valid input and makes generating other valid inputs more likely. For my part, I prefer to use custom mutators to generate valid test cases most of the time, but I want some invalid inputs because error handling is where most bugs are.
- vlovich123 3y ago
  I personally would separate that into two separate fuzz cases - one that generates only valid inputs and one that generates only invalid inputs and spend more resources on verifying the latter because validating invalid inputs is more important. I didn’t read the article but I like property testing for this where your mutator takes random values and uses that to generate a valid input somehow rather than stubbing in a static value. Where I can use a static value is where I wouldn’t be fuzzing the validation of that value. Of course certs are complicated beasts so I’m sure the cURL people did what made sense to them.
- PoignardAzur 3y ago
  Your post doesn't address my question (and neither do any of the others replies). The article says: > If the data can’t be parsed into a valid TLV, instead of throwing it away, return a syntactically correct dummy TLV. This can be anything, as long as it can be successfully unpacked. Is the syntactically correct dummy value the same each time? If so, how does that lead to new coverage? If it's a different valid TLV value each time, then it's not really a dummy value, is it? In any case, why bother with this "if invalid replace with dummy" step? Why not generate/mutate a valid TLV value from the start?
  - aaron695 3y ago
    [dead]
  - MajesticHobo2 3y ago
    > Is the syntactically correct dummy value the same each time? If so, how does that lead to new coverage? Per my understanding, the dummy value is constant but is only used once (to create the first valid TLV). Everything after that is a mutation of the original value that, due to the custom mutator logic, is a valid TLV. The mutations are where new coverage comes from. > In any case, why bother with this "if invalid replace with dummy" step? Why not generate/mutate a valid TLV value from the start? I'm guessing to handle both when there are seed files that are already valid (which you'll want to use instead of a dummy value), and when there aren't any valid seed files.
- ackbar03 3y ago
  You think generative ai/deep learning tools could help with this?
stefan_ 3y ago
Remember TLV is for "Type Length Value". It's better to take your fuzzer output for value (or possibly type and value) but generate the length and final TLV yourself than having tons of fuzzer generated sequences already fail at the very basic type/length check that is unlikely to be vulnerable in software like cURL (but can be in many others..).
mkj 3y ago
You want the failures to occur deeper in the programs so you cover more code paths.

pstrateman 3y ago

The file format they choose is difficult for a fuzzer to produce valid examples by random chance. The file format isn't what's being fuzzed, so trying to accept as many things as possible as valid is useful. It's a trick to make the fuzzer faster.

gavinhoward 3y ago

Not an expert, but I am a power user of fuzzers. The problem is that the space of invalid inputs is far larger than the space of valid inputs. Sometimes orders of magnitude larger, say billions or more invalid inputs to one valid input. Naive fuzzing will hit so many error cases that it will hardly produce a valid input. For the ratio that I mentioned, you might run a fuzzer for a billion runs and only get one valid input in the bunch. Using a custom mutator and returning a dummy value will give the fuzzer a starting point from a valid input and makes generating other valid inputs more likely. For my part, I prefer to use custom mutators to generate valid test cases most of the time, but I want some invalid inputs because error handling is where most bugs are.

vlovich123 3y ago
I personally would separate that into two separate fuzz cases - one that generates only valid inputs and one that generates only invalid inputs and spend more resources on verifying the latter because validating invalid inputs is more important. I didn’t read the article but I like property testing for this where your mutator takes random values and uses that to generate a valid input somehow rather than stubbing in a static value. Where I can use a static value is where I wouldn’t be fuzzing the validation of that value. Of course certs are complicated beasts so I’m sure the cURL people did what made sense to them.
PoignardAzur 3y ago
Your post doesn't address my question (and neither do any of the others replies). The article says: > If the data can’t be parsed into a valid TLV, instead of throwing it away, return a syntactically correct dummy TLV. This can be anything, as long as it can be successfully unpacked. Is the syntactically correct dummy value the same each time? If so, how does that lead to new coverage? If it's a different valid TLV value each time, then it's not really a dummy value, is it? In any case, why bother with this "if invalid replace with dummy" step? Why not generate/mutate a valid TLV value from the start?
- aaron695 3y ago
  [dead]
- MajesticHobo2 3y ago
  > Is the syntactically correct dummy value the same each time? If so, how does that lead to new coverage? Per my understanding, the dummy value is constant but is only used once (to create the first valid TLV). Everything after that is a mutation of the original value that, due to the custom mutator logic, is a valid TLV. The mutations are where new coverage comes from. > In any case, why bother with this "if invalid replace with dummy" step? Why not generate/mutate a valid TLV value from the start? I'm guessing to handle both when there are seed files that are already valid (which you'll want to use instead of a dummy value), and when there aren't any valid seed files.
ackbar03 3y ago
You think generative ai/deep learning tools could help with this?

stefan_ 3y ago

Remember TLV is for "Type Length Value". It's better to take your fuzzer output for value (or possibly type and value) but generate the length and final TLV yourself than having tons of fuzzer generated sequences already fail at the very basic type/length check that is unlikely to be vulnerable in software like cURL (but can be in many others..).

mkj 3y ago

You want the failures to occur deeper in the programs so you cover more code paths.