4 ms·
I've done some of work in the area. Most of the companies that use, or even require fuzzing in their software development procedures, have a certain problem: th
by tshtf 17y ago
I've done some of work in the area. Most of the companies that use, or even require fuzzing in their software development procedures, have a certain problem: the same people who write the parsers write the fuzzers. When they write these fuzzers, they make some too-strong assumptions on the formats of the data involved.
Writing smarter fuzzers may be needed to fully exercise certain file formats, say where certain blocks of data need to match a CRC elsewhere. But smart fuzzers fail where the authors fail to account for edge conditions. Having both in your toolset is invaluable.
Regardless, it's sad for companies like Apple that they're not doing any of this.
- tptacek 17y agoDumb fuzzers find things in Microsoft code all the time, and Microsoft spends huge on external teams to write fuzzers.
- davi 17y agoSo what's your analysis of why the external teams are ineffective?
- tptacek 17y agoThey're not.
- davi 17y agoOK, so Microsoft finds lots of bugs by this technique (according to you), but then this guy says that he finds still more bugs, with (according to him) pretty primitive methods. And then the guy says, '"I found bugs, lots of bugs. That was both surprising and disappointing." And it also made him ask why vendors like Microsoft, Apple and Adobe, which have teams of security engineers and scores of machines running fuzzers looking for flaws, hadn't found these bugs long ago.' So given your assertion that these teams are very effective at what they do, why do you think this guy is very easily (according to him) able to find additional bugs? There's a mismatch between what you're saying and the gist of this article. I'm not saying you're wrong -- far from it. I'm looking to you for a resolution to the discrepancy.
- jballanc 17y agoThis guy is in the security business. If you think your software is secure, he's out of work. If he creates the impression that he didn't even have to try that hard to find these vulnerabilities, then you get paranoid and want to hire him. To be clear, I'm not saying the vulnerabilities weren't real, but to buy his story about them being "easy" to find, well...
- tptacek 17y agoI tend towards the "Charlie is just better than most of us" view of the world, too.
- tshtf 17y agoI tried to bring out 2 points in my comment: 1) The people who write the parsers should not write the fuzzers 2) Using fuzzers based upon more rigorous standards of the relevant file formats may not be optimal. There is an advantage to dumb fuzzers; with defense in depth one would use both.
- tptacek 17y agoI think that when you contract a senior consultant from iSec Partners or Leviathan to test, I don't know LiveMeeting?, you're getting a high-quality "deep" fuzzer and you're getting the dumb fuzzer too. They are getting both. They're spending many tens of thousands of dollars per release cycle to get people to write and run these tools. They evaluate projects based on whether they find things. They find things. They keep the tools and the run them on future releases. The point I'm making is, even if you do an amazing job fuzzing your software, you will clearly still miss things.