5 ms·

What's a good tutorial for fuzzing? I work with many code bases in my consulting, none of which have come close to being "fuzzed". I've googled around and the l

What's a good tutorial for fuzzing? I work with many code bases in my consulting, none of which have come close to being "fuzzed". I've googled around and the learning curve is steeper than expected. I am approaching it like learning a new unit-test framework. Perhaps that's wrong? Is it like integrating a code coverage tool or memory analysis like valgrind? This article seems like a bunch of high-level entreaties to go fuzz etc. maybe I missed something.

cjbprime 7y ago
Here's an example in Golang: https://blog.kowalczyk.info/article/n/fuzzing-markdown-parser-written-in-go.html https://blog.kowalczyk.info/article/n/fuzzing-markdown-parse... The short version is that you strip down the interesting main loop of your program to accept an item of input, process it, and exit, and then the fuzzer takes over.
hannob 7y ago
It depends really a lot on what your target looks like, both in terms of what interfaces it has and what programming language you're using. A very simple case that's a good start is if you have C or C++ code and already have some form of command line tool that parses an input file and does something with it noninteractively. Then you can easily use afl, ideally coupled with address sanitizer (a compiler feature to catch memory safety bugs). If you want to up your game, don't target applications, target functoins. The go-to tool is libfuzzer. If you leave C/C++ then the answer depends really on what programming language and what you even want to test for. (Looking for memory corruption bugs is just the most common thing to do with fuzzing, so if you don't have those, you need something else to look for.)
server_bot 7y ago
For C/C++: This should help you get started with AFL and more traditional security research workflows: https://www.ixiacom.com/company/blog/how-use-fuzzing-security-research https://www.ixiacom.com/company/blog/how-use-fuzzing-securit... Consider DeepState for a more development-focused workflow, akin to writing unit tests: https://github.com/trailofbits/deepstate https://github.com/trailofbits/deepstate
wyldfire 7y ago
> I am approaching it like learning a new unit-test framework. That's reasonable. If you just take a step back from afl or libfuzzer and consider how you would test your product with arbitrary inputs, the critical steps that you have to take are pretty similar regardless of the framework. It has to be noninteractive and it should ideally be something whose duration doesn't vary by many orders of magnitude (so you can set sane timeouts that you consider failures). It shouldn't have any nondeterminism or behaviors that consider other inputs outside of the ones generated by the fuzzing framework. It's great if you can avoid filesystem interaction.
matt_d 7y ago
"Generating Software Tests" (https://www.fuzzingbook.org/ https://www.fuzzingbook.org/) is pretty great (independent of your programming language) - arguably a must read for anyone interested in software testing. John Regehr (the author of the blog post) has written more great posts: - How to Fuzz an ADT Implementation - https://blog.regehr.org/archives/896 https://blog.regehr.org/archives/896 - Better Random Testing by Leaving Features Out - https://blog.regehr.org/archives/591 https://blog.regehr.org/archives/591 - Tricking a Whitebox Testcase Generator - https://blog.regehr.org/archives/672 https://blog.regehr.org/archives/672 - Fuzzers Need Taming - https://blog.regehr.org/archives/925 https://blog.regehr.org/archives/925 - Levels of Fuzzing - https://blog.regehr.org/archives/1039 https://blog.regehr.org/archives/1039 - API Fuzzing vs. File Fuzzing: A Cautionary Tale - https://blog.regehr.org/archives/1269 https://blog.regehr.org/archives/1269 - Reducers are Fuzzers - https://blog.regehr.org/archives/1284 https://blog.regehr.org/archives/1284 In terms of software, DeepState (https://github.com/trailofbits/deepstate https://github.com/trailofbits/deepstate) may be a good place to start for C and C++. Relevant links: - Fuzzing an API with DeepState: https://blog.trailofbits.com/2019/01/22/fuzzing-an-api-with-deepstate-part-1/ https://blog.trailofbits.com/2019/01/22/fuzzing-an-api-with-..., https://blog.trailofbits.com/2019/01/23/fuzzing-an-api-with-deepstate-part-2/ https://blog.trailofbits.com/2019/01/23/fuzzing-an-api-with-... - NDSS 18 paper, "DeepState: Symbolic Unit Testing for C and C++": https://www.cefns.nau.edu/~adg326/bar18.pdf https://www.cefns.nau.edu/~adg326/bar18.pdf In terms of choosing among fuzzing solutions, https://blog.trailofbits.com/2018/10/05/how-to-spot-good-fuzzing-research/ https://blog.trailofbits.com/2018/10/05/how-to-spot-good-fuz... is also worth a read -- as well as the article it refers to, http://www.pl-enthusiast.net/2018/08/23/evaluating-empirical-evaluations-for-fuzz-testing/ http://www.pl-enthusiast.net/2018/08/23/evaluating-empirical.... For a broad survey, see "The Art, Science, and Engineering of Fuzzing": https://arxiv.org/abs/1812.00140 https://arxiv.org/abs/1812.00140, https://jiliac.com/pdf/fuzzing_survey19.pdf https://jiliac.com/pdf/fuzzing_survey19.pdf More resources: - Effective File Format Fuzzing – Thoughts, Techniques and Results (Black Hat Europe 2016): https://j00ru.vexillium.org/talks/blackhat-eu-effective-file-format-fuzzing-thoughts-techniques-and-results/ https://j00ru.vexillium.org/talks/blackhat-eu-effective-file... - libFuzzer – a library for coverage-guided fuzz testing: http://tutorial.libFuzzer.info http://tutorial.libFuzzer.info, http://llvm.org/docs/LibFuzzer.html http://llvm.org/docs/LibFuzzer.html, https://github.com/ouspg/libfuzzerfication https://github.com/ouspg/libfuzzerfication - Materials of "Modern fuzzing of C/C++ Projects" workshop: https://github.com/Dor1s/libfuzzer-workshop https://github.com/Dor1s/libfuzzer-workshop - Introduction to using libFuzzer with llvm-toolset: https://developers.redhat.com/blog/2019/03/05/introduction-to-using-libfuzzer-with-llvm-toolset/ https://developers.redhat.com/blog/2019/03/05/introduction-t... - Fuzzing workflows - a fuzz job from start to finish: https://foxglovesecurity.com/2016/03/15/fuzzing-workflows-a-fuzz-job-from-start-to-finish/ https://foxglovesecurity.com/2016/03/15/fuzzing-workflows-a-... - Materials from "Fuzzing with AFL" workshop (SteelCon 2017, BSides London and Bristol 2019): https://github.com/ThalesIgnite/afl-training https://github.com/ThalesIgnite/afl-training - Making Your Library More Reliable with Fuzzing (C++Now 2018; Marshall Clow): https://www.youtube.com/watch?v=LlLJRHToyUk https://www.youtube.com/watch?v=LlLJRHToyUk, https://github.com/boostcon/cppnow_presentations_2018/blob/master/05-10-2018_thursday/making_your_library_more_reliable_with_fuzzing__marshall_clow__cppnow_05182018.pdf https://github.com/boostcon/cppnow_presentations_2018/blob/m... - C++ Weekly - Ep 85 - Fuzz Testing - https://www.youtube.com/watch?v=gO0KBoqkOoU https://www.youtube.com/watch?v=gO0KBoqkOoU - The Art of Fuzzing – Slides and Demos: https://sec-consult.com/en/blog/2017/11/the-art-of-fuzzing-slides-and-demos/ https://sec-consult.com/en/blog/2017/11/the-art-of-fuzzing-s...
stusmall 7y ago
I was in the same place for a long time. Recently I found the bar to entry in fuzzing rust code amazingly low. The docs aren't great yet but the tools are super easy to use and well put together. I started with the rust fuzz book: https://rust-fuzz.github.io/book/ https://rust-fuzz.github.io/book/
gbmor 7y ago
One thing I do occasionally is to push /dev/urandom into functions in various tests. Read n bytes, use it as input. Not quite as extensive as fuzzing libraries, but it works.

cjbprime 7y ago

Here's an example in Golang: https://blog.kowalczyk.info/article/n/fuzzing-markdown-parser-written-in-go.html https://blog.kowalczyk.info/article/n/fuzzing-markdown-parse... The short version is that you strip down the interesting main loop of your program to accept an item of input, process it, and exit, and then the fuzzer takes over.

hannob 7y ago

It depends really a lot on what your target looks like, both in terms of what interfaces it has and what programming language you're using. A very simple case that's a good start is if you have C or C++ code and already have some form of command line tool that parses an input file and does something with it noninteractively. Then you can easily use afl, ideally coupled with address sanitizer (a compiler feature to catch memory safety bugs). If you want to up your game, don't target applications, target functoins. The go-to tool is libfuzzer. If you leave C/C++ then the answer depends really on what programming language and what you even want to test for. (Looking for memory corruption bugs is just the most common thing to do with fuzzing, so if you don't have those, you need something else to look for.)

server_bot 7y ago

For C/C++: This should help you get started with AFL and more traditional security research workflows: https://www.ixiacom.com/company/blog/how-use-fuzzing-security-research https://www.ixiacom.com/company/blog/how-use-fuzzing-securit... Consider DeepState for a more development-focused workflow, akin to writing unit tests: https://github.com/trailofbits/deepstate https://github.com/trailofbits/deepstate

wyldfire 7y ago

> I am approaching it like learning a new unit-test framework. That's reasonable. If you just take a step back from afl or libfuzzer and consider how you would test your product with arbitrary inputs, the critical steps that you have to take are pretty similar regardless of the framework. It has to be noninteractive and it should ideally be something whose duration doesn't vary by many orders of magnitude (so you can set sane timeouts that you consider failures). It shouldn't have any nondeterminism or behaviors that consider other inputs outside of the ones generated by the fuzzing framework. It's great if you can avoid filesystem interaction.

matt_d 7y ago

"Generating Software Tests" (https://www.fuzzingbook.org/ https://www.fuzzingbook.org/) is pretty great (independent of your programming language) - arguably a must read for anyone interested in software testing. John Regehr (the author of the blog post) has written more great posts: - How to Fuzz an ADT Implementation - https://blog.regehr.org/archives/896 https://blog.regehr.org/archives/896 - Better Random Testing by Leaving Features Out - https://blog.regehr.org/archives/591 https://blog.regehr.org/archives/591 - Tricking a Whitebox Testcase Generator - https://blog.regehr.org/archives/672 https://blog.regehr.org/archives/672 - Fuzzers Need Taming - https://blog.regehr.org/archives/925 https://blog.regehr.org/archives/925 - Levels of Fuzzing - https://blog.regehr.org/archives/1039 https://blog.regehr.org/archives/1039 - API Fuzzing vs. File Fuzzing: A Cautionary Tale - https://blog.regehr.org/archives/1269 https://blog.regehr.org/archives/1269 - Reducers are Fuzzers - https://blog.regehr.org/archives/1284 https://blog.regehr.org/archives/1284 In terms of software, DeepState (https://github.com/trailofbits/deepstate https://github.com/trailofbits/deepstate) may be a good place to start for C and C++. Relevant links: - Fuzzing an API with DeepState: https://blog.trailofbits.com/2019/01/22/fuzzing-an-api-with-deepstate-part-1/ https://blog.trailofbits.com/2019/01/22/fuzzing-an-api-with-..., https://blog.trailofbits.com/2019/01/23/fuzzing-an-api-with-deepstate-part-2/ https://blog.trailofbits.com/2019/01/23/fuzzing-an-api-with-... - NDSS 18 paper, "DeepState: Symbolic Unit Testing for C and C++": https://www.cefns.nau.edu/~adg326/bar18.pdf https://www.cefns.nau.edu/~adg326/bar18.pdf In terms of choosing among fuzzing solutions, https://blog.trailofbits.com/2018/10/05/how-to-spot-good-fuzzing-research/ https://blog.trailofbits.com/2018/10/05/how-to-spot-good-fuz... is also worth a read -- as well as the article it refers to, http://www.pl-enthusiast.net/2018/08/23/evaluating-empirical-evaluations-for-fuzz-testing/ http://www.pl-enthusiast.net/2018/08/23/evaluating-empirical.... For a broad survey, see "The Art, Science, and Engineering of Fuzzing": https://arxiv.org/abs/1812.00140 https://arxiv.org/abs/1812.00140, https://jiliac.com/pdf/fuzzing_survey19.pdf https://jiliac.com/pdf/fuzzing_survey19.pdf More resources: - Effective File Format Fuzzing – Thoughts, Techniques and Results (Black Hat Europe 2016): https://j00ru.vexillium.org/talks/blackhat-eu-effective-file-format-fuzzing-thoughts-techniques-and-results/ https://j00ru.vexillium.org/talks/blackhat-eu-effective-file... - libFuzzer – a library for coverage-guided fuzz testing: http://tutorial.libFuzzer.info http://tutorial.libFuzzer.info, http://llvm.org/docs/LibFuzzer.html http://llvm.org/docs/LibFuzzer.html, https://github.com/ouspg/libfuzzerfication https://github.com/ouspg/libfuzzerfication - Materials of "Modern fuzzing of C/C++ Projects" workshop: https://github.com/Dor1s/libfuzzer-workshop https://github.com/Dor1s/libfuzzer-workshop - Introduction to using libFuzzer with llvm-toolset: https://developers.redhat.com/blog/2019/03/05/introduction-to-using-libfuzzer-with-llvm-toolset/ https://developers.redhat.com/blog/2019/03/05/introduction-t... - Fuzzing workflows - a fuzz job from start to finish: https://foxglovesecurity.com/2016/03/15/fuzzing-workflows-a-fuzz-job-from-start-to-finish/ https://foxglovesecurity.com/2016/03/15/fuzzing-workflows-a-... - Materials from "Fuzzing with AFL" workshop (SteelCon 2017, BSides London and Bristol 2019): https://github.com/ThalesIgnite/afl-training https://github.com/ThalesIgnite/afl-training - Making Your Library More Reliable with Fuzzing (C++Now 2018; Marshall Clow): https://www.youtube.com/watch?v=LlLJRHToyUk https://www.youtube.com/watch?v=LlLJRHToyUk, https://github.com/boostcon/cppnow_presentations_2018/blob/master/05-10-2018_thursday/making_your_library_more_reliable_with_fuzzing__marshall_clow__cppnow_05182018.pdf https://github.com/boostcon/cppnow_presentations_2018/blob/m... - C++ Weekly - Ep 85 - Fuzz Testing - https://www.youtube.com/watch?v=gO0KBoqkOoU https://www.youtube.com/watch?v=gO0KBoqkOoU - The Art of Fuzzing – Slides and Demos: https://sec-consult.com/en/blog/2017/11/the-art-of-fuzzing-slides-and-demos/ https://sec-consult.com/en/blog/2017/11/the-art-of-fuzzing-s...

stusmall 7y ago

I was in the same place for a long time. Recently I found the bar to entry in fuzzing rust code amazingly low. The docs aren't great yet but the tools are super easy to use and well put together. I started with the rust fuzz book: https://rust-fuzz.github.io/book/ https://rust-fuzz.github.io/book/

gbmor 7y ago

One thing I do occasionally is to push /dev/urandom into functions in various tests. Read n bytes, use it as input. Not quite as extensive as fuzzing libraries, but it works.