7 ms·

Add experimental fuzz test support for Go 1.17

121 points by typical182 6y ago

shakezula 6y ago
I like this idea a lot and I think I support adding it directly to the testing library. Fuzz testing has been pretty valuable to me in my time using it with Go and I would love to see it have more widespread support.
- omginternets 6y ago
  Agreed. GoFuzz has been a huge asset for me, but it occasionally suffers bugs due to the fact that it's hacking the runtime to some extent. IMHO that makes fuzzing an excellent candidate for inclusion in the standard toolchain.
sitkack 6y ago
Does anyone have experience with Gopter, a Golang Property Based testing library? https://github.com/leanovate/gopter https://github.com/leanovate/gopter
- motiejus 6y ago
  I do. I implemented a specialized json-patch and tested it with gopter. My colleague implemented an equivalent of `openssl passwd` in go and used gopter to test it (comparing outputs between his implementation and what openssl returns). Needless to say, gopter found subtle bugs in both cases. I came to property-testing from PropEr (erlang), and was pleasantly surprised how well gopter worked for us.
  - jacques_chester 6y ago
    > My colleague implemented an equivalent of `openssl passwd` in go and used gopter to test it (comparing outputs between his implementation and what openssl returns). Could you talk more about this? I'd love to see the code or any other docs you might have.
    - motiejus 6y ago
      It's a re-implementation of https://akkadia.org/drepper/SHA-crypt.txt https://akkadia.org/drepper/SHA-crypt.txt in Go. Because Linux does not support shadow in bcrypt or scrypt. Also see https://access.redhat.com/articles/1519843 https://access.redhat.com/articles/1519843. Unfortunately, none of that is open source.
  - nerpderp82 6y ago
    I would like to +1 jacque's request. I would also be really interested in a rundown of gopter.
slaymaker1907 6y ago
I really hope this is a trend and randomized testing just becomes part of standard testing tools. I used jqwik extensively at my last job and it was very useful for finding issues with null on objects with lots of fields. It's much easier to just describe how to construct data rather than coming up with test data and edge cases yourself. Sure you might have test cases for every nullable field, but do you have tests for every pair of nullable fields? I also found some interesting behavior this way. StringUtils.trimToNull(str) == null is actually not the same as checking if a string is blank due to weird unicode peculiarities.
benhoyt 6y ago
Go tests and benchmarks are so easy to write and run: just add TestFoo and BenchmarkFoo functions to a bar_test.go file, and "go test" does the rest. It's currently doable, but it requires a 3rd party library (go-fuzz) and a bit of fluffing around. This will make fuzz testing an equally first-class citizen with standard Go tooling (just add FuzzFoo), and as such we'll probably see a lot more people testing with fuzzing. I used go-fuzz in GoAWK and it found several bugs (see https://benhoyt.com/writings/goawk/#fuzz-testing https://benhoyt.com/writings/goawk/#fuzz-testing), and almost everyone who's done fuzz testing has similar reports. Certainly go-fuzz has found many, many bugs in Go itself: https://github.com/dvyukov/go-fuzz#trophies https://github.com/dvyukov/go-fuzz#trophies For what it's worth, I wrote an article for LWN about the upcoming support for built-in fuzzing in Go: https://lwn.net/Articles/829242/ https://lwn.net/Articles/829242/ (of course, if you want full details, read the full proposal).
- zimpenfish 6y ago
  > and almost everyone who's done fuzz testing has similar reports. Added fuzzing at a recent place and yep, everything went to hell. About to add fuzzing at current place where everything will almost certainly go to hell but at least there's institutional will to fix the issues and no legacy crippling what fixes can be made.
- latch 6y ago
  I never understood their reason for the verbose and unfriendly ergonomics: if a != b { t.Errorf("expected %s to be equal to %s", a, b) } Instead of something like: t.Expect(a).To.Equal(b) And, of course, it gets uglier when you're trying to test a function that returns a error,value tuple, which, again, is something that a more friendly testing library can help with.
  - efwfwef 6y ago
    I'm not sure what you find ugly, actually I find your example ugly because it seems to be full of magic
  - lhorie 6y ago
lhorie 6y ago
At the risk of turning this into a Rust thread, is fuzzing less relevant in languages with more strict handling of nullables? I can see it being very useful in go-land, since it's relatively easy to make mistakes w/ nils that the compiler won't complain about, but I'm curious how far having a stronger type system can go vs fuzzing.
- steveklabnik 6y ago
  Fuzzing is still very relevant in Rust. It tends to find panics rather than segfaults, but that's still bugs. https://github.com/rust-fuzz/trophy-case https://github.com/rust-fuzz/trophy-case
  - baby 6y ago
    Just to add, we're a heavy user of fuzzing in Diem[1] and we found a good number of bugs thanks to it : ) https://github.com/diem/diem/ https://github.com/diem/diem/
agbell 6y ago
Is there a clear dividing line between fuzzing tests and property based testing? They seem to use different words, but do similar things, like generate interesting input and making sure things still work when that interesting input is applied. Is the difference instrumentation vs hand written generators and hand written invariants?
- teeray 6y ago
  Philosophically, I’ve found that property-based tests are more like “no matter what this sorting algorithm does, it absolutely should have the same number of elements before and after.” You’re so confident in that assertion that you’re willing to transcend specific test cases and subject the logic under test to limited randomness. Fuzz testing is more like a blunt instrument for finding tests that you should have written, by devising more and more creative input. There’s less intention behind it other than “this shouldn’t explode, and if it does I want to know.”
  - whateveracct 6y ago
    The stretch goal of structured fizzing beyond []byte sounds like property testing to me
    - agbell 6y ago
      Agree, it seems like the limit of one looks like the other and we should talk about them together. And Property testing has some of the same practical implications that Rob Pike mentions of fuzzing, can it take a long time and its not clear how to do it in CI without slowing things down (or at least it seems that way to me).
malandrew 6y ago
While completely unrelated to the new fuzz feature, one library I'd love to see added to the standard library is one for more standard units besides time, such as a distance library. It would be awesome to be able to do something like `foo := 100 * distance.Meters` which returns a distance.Distance type. At the company I work at we do a lot of stuff with distances and time and with the time library it's a lot easy to have strong time typing throughout the codebase, but without a standard distance library, I've seen things in meters, kilometers, miles, etc. With every new version they can introduce new unit libraries until there are ones for velocity, acceleration, area, volume, mass, temperature, etc. This would make golang a great option for many business uses and scientific uses. The other thing golang should standardize in it's stdlib that is testing related is a clock libary like facebookgo/clock library for time control. It's already standard practice to have the rand library be injected with a seed so you can have determinism for tests. The same is need for controlling time for many use cases if you want determinism in tests involving time. Making this part of the stdlib and making it best practice to always dependency inject a clock instead of using a global like time.Now() would go a long way to improving more complex unit/integration testing where time is involved. https://pkg.go.dev/github.com/facebookgo/clock https://pkg.go.dev/github.com/facebookgo/clock

121 points by typical182 6y ago

I like this idea a lot and I think I support adding it directly to the testing library. Fuzz testing has been pretty valuable to me in my time using it with Go and I would love to see it have more widespread support.

omginternets 6y ago
Agreed. GoFuzz has been a huge asset for me, but it occasionally suffers bugs due to the fact that it's hacking the runtime to some extent. IMHO that makes fuzzing an excellent candidate for inclusion in the standard toolchain.

sitkack 6y ago

Does anyone have experience with Gopter, a Golang Property Based testing library? https://github.com/leanovate/gopter https://github.com/leanovate/gopter

motiejus 6y ago
I do. I implemented a specialized json-patch and tested it with gopter. My colleague implemented an equivalent of `openssl passwd` in go and used gopter to test it (comparing outputs between his implementation and what openssl returns). Needless to say, gopter found subtle bugs in both cases. I came to property-testing from PropEr (erlang), and was pleasantly surprised how well gopter worked for us.
- jacques_chester 6y ago
  > My colleague implemented an equivalent of `openssl passwd` in go and used gopter to test it (comparing outputs between his implementation and what openssl returns). Could you talk more about this? I'd love to see the code or any other docs you might have.
  - motiejus 6y ago
    It's a re-implementation of https://akkadia.org/drepper/SHA-crypt.txt https://akkadia.org/drepper/SHA-crypt.txt in Go. Because Linux does not support shadow in bcrypt or scrypt. Also see https://access.redhat.com/articles/1519843 https://access.redhat.com/articles/1519843. Unfortunately, none of that is open source.
- nerpderp82 6y ago
  I would like to +1 jacque's request. I would also be really interested in a rundown of gopter.

slaymaker1907 6y ago

I really hope this is a trend and randomized testing just becomes part of standard testing tools. I used jqwik extensively at my last job and it was very useful for finding issues with null on objects with lots of fields. It's much easier to just describe how to construct data rather than coming up with test data and edge cases yourself. Sure you might have test cases for every nullable field, but do you have tests for every pair of nullable fields? I also found some interesting behavior this way. StringUtils.trimToNull(str) == null is actually not the same as checking if a string is blank due to weird unicode peculiarities.

benhoyt 6y ago

Go tests and benchmarks are so easy to write and run: just add TestFoo and BenchmarkFoo functions to a bar_test.go file, and "go test" does the rest. It's currently doable, but it requires a 3rd party library (go-fuzz) and a bit of fluffing around. This will make fuzz testing an equally first-class citizen with standard Go tooling (just add FuzzFoo), and as such we'll probably see a lot more people testing with fuzzing. I used go-fuzz in GoAWK and it found several bugs (see https://benhoyt.com/writings/goawk/#fuzz-testing https://benhoyt.com/writings/goawk/#fuzz-testing), and almost everyone who's done fuzz testing has similar reports. Certainly go-fuzz has found many, many bugs in Go itself: https://github.com/dvyukov/go-fuzz#trophies https://github.com/dvyukov/go-fuzz#trophies For what it's worth, I wrote an article for LWN about the upcoming support for built-in fuzzing in Go: https://lwn.net/Articles/829242/ https://lwn.net/Articles/829242/ (of course, if you want full details, read the full proposal).

zimpenfish 6y ago
> and almost everyone who's done fuzz testing has similar reports. Added fuzzing at a recent place and yep, everything went to hell. About to add fuzzing at current place where everything will almost certainly go to hell but at least there's institutional will to fix the issues and no legacy crippling what fixes can be made.
latch 6y ago
I never understood their reason for the verbose and unfriendly ergonomics: if a != b { t.Errorf("expected %s to be equal to %s", a, b) } Instead of something like: t.Expect(a).To.Equal(b) And, of course, it gets uglier when you're trying to test a function that returns a error,value tuple, which, again, is something that a more friendly testing library can help with.
- efwfwef 6y ago
  I'm not sure what you find ugly, actually I find your example ugly because it seems to be full of magic
- lhorie 6y ago

lhorie 6y ago

At the risk of turning this into a Rust thread, is fuzzing less relevant in languages with more strict handling of nullables? I can see it being very useful in go-land, since it's relatively easy to make mistakes w/ nils that the compiler won't complain about, but I'm curious how far having a stronger type system can go vs fuzzing.

steveklabnik 6y ago
Fuzzing is still very relevant in Rust. It tends to find panics rather than segfaults, but that's still bugs. https://github.com/rust-fuzz/trophy-case https://github.com/rust-fuzz/trophy-case
- baby 6y ago
  Just to add, we're a heavy user of fuzzing in Diem[1] and we found a good number of bugs thanks to it : ) https://github.com/diem/diem/ https://github.com/diem/diem/

agbell 6y ago

Is there a clear dividing line between fuzzing tests and property based testing? They seem to use different words, but do similar things, like generate interesting input and making sure things still work when that interesting input is applied. Is the difference instrumentation vs hand written generators and hand written invariants?

teeray 6y ago
Philosophically, I’ve found that property-based tests are more like “no matter what this sorting algorithm does, it absolutely should have the same number of elements before and after.” You’re so confident in that assertion that you’re willing to transcend specific test cases and subject the logic under test to limited randomness. Fuzz testing is more like a blunt instrument for finding tests that you should have written, by devising more and more creative input. There’s less intention behind it other than “this shouldn’t explode, and if it does I want to know.”
- whateveracct 6y ago
  The stretch goal of structured fizzing beyond []byte sounds like property testing to me
  - agbell 6y ago
    Agree, it seems like the limit of one looks like the other and we should talk about them together. And Property testing has some of the same practical implications that Rob Pike mentions of fuzzing, can it take a long time and its not clear how to do it in CI without slowing things down (or at least it seems that way to me).

malandrew 6y ago

While completely unrelated to the new fuzz feature, one library I'd love to see added to the standard library is one for more standard units besides time, such as a distance library. It would be awesome to be able to do something like `foo := 100 * distance.Meters` which returns a distance.Distance type. At the company I work at we do a lot of stuff with distances and time and with the time library it's a lot easy to have strong time typing throughout the codebase, but without a standard distance library, I've seen things in meters, kilometers, miles, etc. With every new version they can introduce new unit libraries until there are ones for velocity, acceleration, area, volume, mass, temperature, etc. This would make golang a great option for many business uses and scientific uses. The other thing golang should standardize in it's stdlib that is testing related is a clock libary like facebookgo/clock library for time control. It's already standard practice to have the rand library be injected with a seed so you can have determinism for tests. The same is need for controlling time for many use cases if you want determinism in tests involving time. Making this part of the stdlib and making it best practice to always dependency inject a clock instead of using a global like time.Now() would go a long way to improving more complex unit/integration testing where time is involved. https://pkg.go.dev/github.com/facebookgo/clock https://pkg.go.dev/github.com/facebookgo/clock