5 ms·

This automated approach reminds me of fuzzing.

stevekemp 6y ago
Fuzz-testing can be amazingly useful at finding bugs, crashes, and security issues. I'm looking forward to seeing more people use it, once it has been added to the golang testing tools: https://go.googlesource.com/proposal/+/master/design/draft-fuzzing.md https://go.googlesource.com/proposal/+/master/design/draft-f... In several of my projects I thought I had good testing, but still found issues within 20 minutes of fuzzing. Wonderful to be able to find them so easily.
isbvhodnvemrwvn 6y ago
That's pretty much it, but it fuzzes the code rather than data.
masklinn 6y ago
"property testing" is closer to fuzzing: it runs a parametric test with randomly generated inputs (following an input specification). Usefully, good property testing systems will try to reduce the input when they find a failure (simplify possibly extremely complex input to try and minimise them to the smallest set of operations triggering them). Mutation testing alters the system under test to identify under-specified (under-tested) portions, rather than explore the input space.
- feanaro 6y ago
  The system under test could be considered as an input to the specification / test suite. Mutation testing could then be viewed as fuzzing of the test suite, with every pass result after a mutation being considered a failure of the test suite.
  - pfdietz 6y ago
    And with the property-based approach, test inputs can be generated that automatically increase coverage or that kill mutants. The bane of mutation testing is equivalent mutants: mutations that aren't actually bugs, because they leave the program correct.
    - nullc 6y ago
      > The bane of mutation testing is equivalent mutants: mutations that aren't actually bugs, because they leave the program correct. That's one reason I check the hash of the optimized binaries-- often the compiler manages to convert different but equivalent code into the same binary. It's still not enough, but it's an easy filter to apply with no false exclusions.
      - pfdietz 6y ago
        Good idea.
- dllthomas 6y ago
  Arguably fuzzing is property testing with a constrained set of properties, although in practice the communities differ.
  - dnautics 6y ago
    I don't think fuzzing usually has a concept of automatic shrinking.
    - dllthomas 6y ago
      I think shrinking is very important, but I guess I don't see it as part of the definition. Both sides have some particulars of their approaches that aren't well reflected on the other side, but I still say it's reasonable to think of them as doing fundamentally the same thing.

Fuzz-testing can be amazingly useful at finding bugs, crashes, and security issues. I'm looking forward to seeing more people use it, once it has been added to the golang testing tools: https://go.googlesource.com/proposal/+/master/design/draft-fuzzing.md https://go.googlesource.com/proposal/+/master/design/draft-f... In several of my projects I thought I had good testing, but still found issues within 20 minutes of fuzzing. Wonderful to be able to find them so easily.

isbvhodnvemrwvn 6y ago

That's pretty much it, but it fuzzes the code rather than data.

masklinn 6y ago

"property testing" is closer to fuzzing: it runs a parametric test with randomly generated inputs (following an input specification). Usefully, good property testing systems will try to reduce the input when they find a failure (simplify possibly extremely complex input to try and minimise them to the smallest set of operations triggering them). Mutation testing alters the system under test to identify under-specified (under-tested) portions, rather than explore the input space.

feanaro 6y ago
The system under test could be considered as an input to the specification / test suite. Mutation testing could then be viewed as fuzzing of the test suite, with every pass result after a mutation being considered a failure of the test suite.
- pfdietz 6y ago
  And with the property-based approach, test inputs can be generated that automatically increase coverage or that kill mutants. The bane of mutation testing is equivalent mutants: mutations that aren't actually bugs, because they leave the program correct.
  - nullc 6y ago
    > The bane of mutation testing is equivalent mutants: mutations that aren't actually bugs, because they leave the program correct. That's one reason I check the hash of the optimized binaries-- often the compiler manages to convert different but equivalent code into the same binary. It's still not enough, but it's an easy filter to apply with no false exclusions.
    - pfdietz 6y ago
      Good idea.
dllthomas 6y ago
Arguably fuzzing is property testing with a constrained set of properties, although in practice the communities differ.
- dnautics 6y ago
  I don't think fuzzing usually has a concept of automatic shrinking.
  - dllthomas 6y ago
    I think shrinking is very important, but I guess I don't see it as part of the definition. Both sides have some particulars of their approaches that aren't well reflected on the other side, but I still say it's reasonable to think of them as doing fundamentally the same thing.