4 ms·

The Smart Fuzzer Revolution [video]

73 points by nuclx 10y ago

TempleOS 9y ago
Frasier is a chimpanzee https://glot.io/snippets/ee0wub5ywy https://glot.io/snippets/ee0wub5ywy God damn you are one stupid fuck.
bullsandabears 9y ago
Does anyone know of a list of categorized and recommended fuzzers for different purposes, or just more specifically for smart fuzzing web API's and how to get started with it? Search results for this kind of stuff are hard to parse cause they're either dated or the use cases are very specific one-offs. I realize answering my questions with the given broadly-defined tools may be the required manual expertise they refer to in the presentation. But I'm just looking for a foothold somewhere at the least.
- dguido 9y ago
  Very short answer: For web testing, you should run Burp Suite in headless mode. Lots of people use Carbonator to do this: https://portswigger.net/bappstore/showbappdetails.aspx?uuid=e3a26fff8e1d401dade52f3a8d42d06b https://portswigger.net/bappstore/showbappdetails.aspx?uuid=... Burp Suite is planning on adding native support for continuous integration... integration in the second half of 2017. If you're reading between the lines: there are _very few_ security testing tools that are built well. So you're asking the wrong question. You don't need a huge list. There are only a small handful of fuzzers or analysis tools I would recommend at all, and Burp is it for web testing. Most projects out there are hobby projects from people trying to learn something new and ignoring what has already been done. They don't serve a very useful purpose other than as a learning or teaching tool. We used tried and true basics for our CRS: Radamsa, KLEE, our own open-source binary lifter, and a Python symbolic execution framework built around Z3. Nothing new, or hip, or magic.
  - bullsandabears 9y ago
    Big thanks for this reply. I was looking to cut through all the noise you mentioned. I'm finding all I was looking for in the Radamsa readme for starting and next steps. Use cases expressed in unix pipes > clear_as_day.txt. Fuzzing newb at this point but I'll be using your original and other linked presentations for context while working through basics.
  - _pdp_ 9y ago
    Burp suite does not do a good job at fuzzing APIs - not biased but true. APIs require more structured fuzzers that expose application level problems - not like burp's fuzzers which are working on raw HTTP requests which was useful sometime ago when you had to find bugs in the actual server implementation. This is not relevant in the web application security space anymore apart from the fields of research which is exactly what most web shops not interested to do. You can still use Burp for that but the user needs to do all the heavy lifting by hand. How does Burp do recursive XML or JSON fuzzing? It doesn't. You can write a plugin for that but that defeats the purpose of using an off the shelf tool.
    - dguido 9y ago
      Yep, it's maybe a little more complicated than I let on. We went through the same process you described on a recent engagement and here was the outcome: https://github.com/trailofbits/protofuzz https://github.com/trailofbits/protofuzz
- _pdp_ 9y ago
  There are some commercial efforts but not much in the open source space which in my opinion is a bug problem because APIs should be fuzzed properly - especially when written with dynamic scripting languages.
  - bullsandabears 9y ago
    Thanks for clarifying this and your info on Burp's capabilities above. What are some of the commercial options you just mentioned? I'm dealing with a large app not written with C but does have API endpoints over http. For a start on some black-box fuzzing, I'm thinking I could use Radamsa's client/server capabilities.
dguido 9y ago
I'm the speaker in this video. AMA! If you're interested in more of the technical details of how a CRS (automatic bug finding system) works, I recommend watching this presentation from my colleague Artem Dinaburg. "Making A Scalable Automated Hacking System" * https://www.youtube.com/watch?v=pOuO5m1ljRI https://www.youtube.com/watch?v=pOuO5m1ljRI * https://github.com/trailofbits/presentations/blob/master/Cyber%20Grand%20Challenge/Dinaburg_INFILTRATE_2016.pdf https://github.com/trailofbits/presentations/blob/master/Cyb... You should also keep your eye on https://github.com/trailofbits https://github.com/trailofbits -- we are releasing the final component of our CRS as open-source in a very short time. Manticore, our symbolic execution framework, will be up there soon! I'm happy to give you early access if you get in touch with me on Twitter.
- nuclx 9y ago
  Thanks for the additional information. When experimenting with libFuzzer to test an audio processing library, I was impressed by the results and also the ease of setup. In-process fuzzing is really the best option for that use-case, which is why I chose libFuzzer over AFL. An open-source alternative of Microsoft's SAGE/Springfield would be cool. I'm sure there are things to come with the efforts in CRS you mentioned. Looking forward to where this goes and hope that your 2- and 5-year outlooks hold true.
  - dguido 9y ago
    > An open-source alternative of Microsoft's SAGE/Springfield would be cool We're working on one (!) and I hope we can offer it for free to non-commercial projects. For now, there is Microsoft Springfield [1] for Windows software and Google OSS-Fuzz [2] for open-source software. It is extraordinarily hard to not only get the tech for something like that working but bring it to market. As noted in the video, nearly all the individual pieces of our CRS are open-source but you actually do not want a "CRS." The competition DARPA designed for them involved more than what is necessary to provide value to a development team, e.g., you don't want something that writes IDS signatures, considers "gameplay" or resource contention, or attempts to write automatic patches. You want something that accurately finds and reproduces bugs. We open-sourced the tools we wrote to do that or used tools that were already open-source, like Grr, Manticore, Radamsa, KLEE, and Z3. [1] https://www.microsoft.com/en-us/springfield/ https://www.microsoft.com/en-us/springfield/ [2] https://github.com/google/oss-fuzz https://github.com/google/oss-fuzz

73 points by nuclx 10y ago

Frasier is a chimpanzee https://glot.io/snippets/ee0wub5ywy https://glot.io/snippets/ee0wub5ywy God damn you are one stupid fuck.

bullsandabears 9y ago

Does anyone know of a list of categorized and recommended fuzzers for different purposes, or just more specifically for smart fuzzing web API's and how to get started with it? Search results for this kind of stuff are hard to parse cause they're either dated or the use cases are very specific one-offs. I realize answering my questions with the given broadly-defined tools may be the required manual expertise they refer to in the presentation. But I'm just looking for a foothold somewhere at the least.

dguido 9y ago
Very short answer: For web testing, you should run Burp Suite in headless mode. Lots of people use Carbonator to do this: https://portswigger.net/bappstore/showbappdetails.aspx?uuid=e3a26fff8e1d401dade52f3a8d42d06b https://portswigger.net/bappstore/showbappdetails.aspx?uuid=... Burp Suite is planning on adding native support for continuous integration... integration in the second half of 2017. If you're reading between the lines: there are _very few_ security testing tools that are built well. So you're asking the wrong question. You don't need a huge list. There are only a small handful of fuzzers or analysis tools I would recommend at all, and Burp is it for web testing. Most projects out there are hobby projects from people trying to learn something new and ignoring what has already been done. They don't serve a very useful purpose other than as a learning or teaching tool. We used tried and true basics for our CRS: Radamsa, KLEE, our own open-source binary lifter, and a Python symbolic execution framework built around Z3. Nothing new, or hip, or magic.
- bullsandabears 9y ago
  Big thanks for this reply. I was looking to cut through all the noise you mentioned. I'm finding all I was looking for in the Radamsa readme for starting and next steps. Use cases expressed in unix pipes > clear_as_day.txt. Fuzzing newb at this point but I'll be using your original and other linked presentations for context while working through basics.
- _pdp_ 9y ago
  Burp suite does not do a good job at fuzzing APIs - not biased but true. APIs require more structured fuzzers that expose application level problems - not like burp's fuzzers which are working on raw HTTP requests which was useful sometime ago when you had to find bugs in the actual server implementation. This is not relevant in the web application security space anymore apart from the fields of research which is exactly what most web shops not interested to do. You can still use Burp for that but the user needs to do all the heavy lifting by hand. How does Burp do recursive XML or JSON fuzzing? It doesn't. You can write a plugin for that but that defeats the purpose of using an off the shelf tool.
  - dguido 9y ago
    Yep, it's maybe a little more complicated than I let on. We went through the same process you described on a recent engagement and here was the outcome: https://github.com/trailofbits/protofuzz https://github.com/trailofbits/protofuzz
_pdp_ 9y ago
There are some commercial efforts but not much in the open source space which in my opinion is a bug problem because APIs should be fuzzed properly - especially when written with dynamic scripting languages.
- bullsandabears 9y ago
  Thanks for clarifying this and your info on Burp's capabilities above. What are some of the commercial options you just mentioned? I'm dealing with a large app not written with C but does have API endpoints over http. For a start on some black-box fuzzing, I'm thinking I could use Radamsa's client/server capabilities.

dguido 9y ago

I'm the speaker in this video. AMA! If you're interested in more of the technical details of how a CRS (automatic bug finding system) works, I recommend watching this presentation from my colleague Artem Dinaburg. "Making A Scalable Automated Hacking System" * https://www.youtube.com/watch?v=pOuO5m1ljRI https://www.youtube.com/watch?v=pOuO5m1ljRI * https://github.com/trailofbits/presentations/blob/master/Cyber%20Grand%20Challenge/Dinaburg_INFILTRATE_2016.pdf https://github.com/trailofbits/presentations/blob/master/Cyb... You should also keep your eye on https://github.com/trailofbits https://github.com/trailofbits -- we are releasing the final component of our CRS as open-source in a very short time. Manticore, our symbolic execution framework, will be up there soon! I'm happy to give you early access if you get in touch with me on Twitter.

nuclx 9y ago
Thanks for the additional information. When experimenting with libFuzzer to test an audio processing library, I was impressed by the results and also the ease of setup. In-process fuzzing is really the best option for that use-case, which is why I chose libFuzzer over AFL. An open-source alternative of Microsoft's SAGE/Springfield would be cool. I'm sure there are things to come with the efforts in CRS you mentioned. Looking forward to where this goes and hope that your 2- and 5-year outlooks hold true.
- dguido 9y ago
  > An open-source alternative of Microsoft's SAGE/Springfield would be cool We're working on one (!) and I hope we can offer it for free to non-commercial projects. For now, there is Microsoft Springfield [1] for Windows software and Google OSS-Fuzz [2] for open-source software. It is extraordinarily hard to not only get the tech for something like that working but bring it to market. As noted in the video, nearly all the individual pieces of our CRS are open-source but you actually do not want a "CRS." The competition DARPA designed for them involved more than what is necessary to provide value to a development team, e.g., you don't want something that writes IDS signatures, considers "gameplay" or resource contention, or attempts to write automatic patches. You want something that accurately finds and reproduces bugs. We open-sourced the tools we wrote to do that or used tools that were already open-source, like Grr, Manticore, Radamsa, KLEE, and Z3. [1] https://www.microsoft.com/en-us/springfield/ https://www.microsoft.com/en-us/springfield/ [2] https://github.com/google/oss-fuzz https://github.com/google/oss-fuzz