4 ms·

Pwn2Own winner tells Apple, Microsoft to find their own bugs

30 points by Sandman 17y ago

nfnaaron 17y ago
"He went into the project figuring that he wouldn't find any vulnerabilities with the dumb fuzzer. "But I found bugs, lots of bugs. That was both surprising and disappointing." And it also made him ask why vendors like Microsoft, Apple and Adobe, which have teams of security engineers and scores of machines running fuzzers looking for flaws, hadn't found these bugs long ago." Or, maybe the companies have already found those bugs and more, but focus their efforts on bugs that surface in the wild and don't spend resources on the others. If they fix all the bugs, then they spend money on some bugs that would never have surfaced. Just a guess.
- viraptor 17y ago
  That seems to make sense from the business point of view (well - not every business works like that, but I'm sure there are people out there who would agree). Unfortunately. Maybe internet needs more hackers these days? Leaving bugs unresolved should never equal saving money.
- tptacek 17y ago
  Waiting to fix bugs until they become an issue is smart, if a bit ruthless. Waiting to fix security flaws until they become public is negligent. From firsthand experience with at least one of the companies implicated in the recent stories: they aren't sitting on things. What Charlie Miller is finding is news to them.
- aidenn0 17y ago
  At one company I worked for, we introduced a fuzzer into the testing of one of our products. Due to a longer than normal release cycle, we were able to collect data on bugs customers found vs bugs the fuzzer found. A lot of bugs the fuzzer found were not found by customers, but a lot of the bugs found by the customers were found by the fuzzer. Furthermore the bugs encountered by the fuzzer identified modules that tended to be buggy and allowed even more bugs to be found by inspection in them. I guess it comes down to how much you value you place on fixing bugs before customers run into them. I certainly feel like in the case of security issues, the value is very high, especially since this is a class of bugs that attackers will specifically search for!
tshtf 17y ago
I've done some of work in the area. Most of the companies that use, or even require fuzzing in their software development procedures, have a certain problem: the same people who write the parsers write the fuzzers. When they write these fuzzers, they make some too-strong assumptions on the formats of the data involved. Writing smarter fuzzers may be needed to fully exercise certain file formats, say where certain blocks of data need to match a CRC elsewhere. But smart fuzzers fail where the authors fail to account for edge conditions. Having both in your toolset is invaluable. Regardless, it's sad for companies like Apple that they're not doing any of this.
- tptacek 17y ago
  Dumb fuzzers find things in Microsoft code all the time, and Microsoft spends huge on external teams to write fuzzers.
  - davi 17y ago
    So what's your analysis of why the external teams are ineffective?
    - tptacek 17y ago
      They're not.
      - davi 17y ago
        OK, so Microsoft finds lots of bugs by this technique (according to you), but then this guy says that he finds still more bugs, with (according to him) pretty primitive methods. And then the guy says, '"I found bugs, lots of bugs. That was both surprising and disappointing." And it also made him ask why vendors like Microsoft, Apple and Adobe, which have teams of security engineers and scores of machines running fuzzers looking for flaws, hadn't found these bugs long ago.' So given your assertion that these teams are very effective at what they do, why do you think this guy is very easily (according to him) able to find additional bugs? There's a mismatch between what you're saying and the gist of this article. I'm not saying you're wrong -- far from it. I'm looking to you for a resolution to the discrepancy.
tkiley 17y ago
This is somewhat equivalent to WikiLeaks' release strategy: tell the organization that you have evidence of XYZ problem, but don't describe the exact nature and scope of the evidence, then pressure them to come clean and fix the problem themselves; hopefully this leads to more comprehensive "cleanup" efforts that have a larger positive long-term effect. How relevant is the WikiLeaks strategy in the field of security?
- tptacek 17y ago
  It depends on the vendor. There are vendors for which the "announce and hold back" strategy will probably work: they're the ones who market based on security. It is a Big Deal if someone has an IIS remote, or a reliable Flash, or Apache/WebSphere. We'll see how big a deal it is for Google and Apple. Marketing based on security means more than just talking about security on your web page. It means making a business out of people who buy based on product security. I like my Mac, and I like Apple, but I have no illusions about the resilience of OS X.
pxlpshr 17y ago
He sounds naive and his logic is quite the cliche. Surprised he didn't close it with a line from the hacker's manifesto. I'm not sure what it is with some people acting like the smartest guy in the room while refusing to share their knowledge for the betterment of society. Entrepreneurs benefit tremendously from mentors and advice from those who've made it. I'd be surprised if he didn't learn a vast portion of his trade from the openness of information. I guess he feels empowered w/ his :15 minutes and this is how he leverages it? Weak^10. +++the pxlpshr+++
- potatolicious 17y ago
  I didn't get that sense from the article - he's not refusing to release the info for his own benefit, or to be a dick; he's ostensibly doing it because he feels like this just encourages the "we'll fix stuff when other people find it in the wild" mentality, which doesn't help make software more secure in the long run. He wants software producers to actively try to break their software so that his ostensibly simplistic tools can't find wide open holes like this. I'm sure some egotism rolls into it somewhere, but I don't think it's altogether a bad stance... you shouldn't be waiting for your users to tell you where your security holes are.
  - pxlpshr 17y ago
    you shouldn't be waiting for your users to tell you where your security holes are. while I agree with that ideology, in reality you just can't assume that to be the case when you have multiple people touching the code, and features added regularly that may inadvertently introduce holes. what software has ever been 100% bullet proof? Also, I think that statement largely contradicts the notion of 'release early, iterate often' considering it's grounded on the notion of integrating feedback and bugs reported by users. Are major corporations held at a different standard simply because they have more resources? Even with thousands of employees, it's still a glass box compared to the millions that will use it.
djb_hackernews 17y ago
Anyone able to give more detail? Maybe not on Millers specifics but how does inserting random data into a program become an exploitable vulnerability?
- evgen 17y ago
  http://en.wikipedia.org/wiki/Fuzz_testing http://en.wikipedia.org/wiki/Fuzz_testing

30 points by Sandman 17y ago

"He went into the project figuring that he wouldn't find any vulnerabilities with the dumb fuzzer. "But I found bugs, lots of bugs. That was both surprising and disappointing." And it also made him ask why vendors like Microsoft, Apple and Adobe, which have teams of security engineers and scores of machines running fuzzers looking for flaws, hadn't found these bugs long ago." Or, maybe the companies have already found those bugs and more, but focus their efforts on bugs that surface in the wild and don't spend resources on the others. If they fix all the bugs, then they spend money on some bugs that would never have surfaced. Just a guess.

viraptor 17y ago
That seems to make sense from the business point of view (well - not every business works like that, but I'm sure there are people out there who would agree). Unfortunately. Maybe internet needs more hackers these days? Leaving bugs unresolved should never equal saving money.
tptacek 17y ago
Waiting to fix bugs until they become an issue is smart, if a bit ruthless. Waiting to fix security flaws until they become public is negligent. From firsthand experience with at least one of the companies implicated in the recent stories: they aren't sitting on things. What Charlie Miller is finding is news to them.
aidenn0 17y ago
At one company I worked for, we introduced a fuzzer into the testing of one of our products. Due to a longer than normal release cycle, we were able to collect data on bugs customers found vs bugs the fuzzer found. A lot of bugs the fuzzer found were not found by customers, but a lot of the bugs found by the customers were found by the fuzzer. Furthermore the bugs encountered by the fuzzer identified modules that tended to be buggy and allowed even more bugs to be found by inspection in them. I guess it comes down to how much you value you place on fixing bugs before customers run into them. I certainly feel like in the case of security issues, the value is very high, especially since this is a class of bugs that attackers will specifically search for!

tshtf 17y ago

I've done some of work in the area. Most of the companies that use, or even require fuzzing in their software development procedures, have a certain problem: the same people who write the parsers write the fuzzers. When they write these fuzzers, they make some too-strong assumptions on the formats of the data involved. Writing smarter fuzzers may be needed to fully exercise certain file formats, say where certain blocks of data need to match a CRC elsewhere. But smart fuzzers fail where the authors fail to account for edge conditions. Having both in your toolset is invaluable. Regardless, it's sad for companies like Apple that they're not doing any of this.

tptacek 17y ago
Dumb fuzzers find things in Microsoft code all the time, and Microsoft spends huge on external teams to write fuzzers.
- davi 17y ago
  So what's your analysis of why the external teams are ineffective?
  - tptacek 17y ago
    They're not.
    - davi 17y ago
      OK, so Microsoft finds lots of bugs by this technique (according to you), but then this guy says that he finds still more bugs, with (according to him) pretty primitive methods. And then the guy says, '"I found bugs, lots of bugs. That was both surprising and disappointing." And it also made him ask why vendors like Microsoft, Apple and Adobe, which have teams of security engineers and scores of machines running fuzzers looking for flaws, hadn't found these bugs long ago.' So given your assertion that these teams are very effective at what they do, why do you think this guy is very easily (according to him) able to find additional bugs? There's a mismatch between what you're saying and the gist of this article. I'm not saying you're wrong -- far from it. I'm looking to you for a resolution to the discrepancy.

tkiley 17y ago

This is somewhat equivalent to WikiLeaks' release strategy: tell the organization that you have evidence of XYZ problem, but don't describe the exact nature and scope of the evidence, then pressure them to come clean and fix the problem themselves; hopefully this leads to more comprehensive "cleanup" efforts that have a larger positive long-term effect. How relevant is the WikiLeaks strategy in the field of security?

tptacek 17y ago
It depends on the vendor. There are vendors for which the "announce and hold back" strategy will probably work: they're the ones who market based on security. It is a Big Deal if someone has an IIS remote, or a reliable Flash, or Apache/WebSphere. We'll see how big a deal it is for Google and Apple. Marketing based on security means more than just talking about security on your web page. It means making a business out of people who buy based on product security. I like my Mac, and I like Apple, but I have no illusions about the resilience of OS X.

pxlpshr 17y ago

He sounds naive and his logic is quite the cliche. Surprised he didn't close it with a line from the hacker's manifesto. I'm not sure what it is with some people acting like the smartest guy in the room while refusing to share their knowledge for the betterment of society. Entrepreneurs benefit tremendously from mentors and advice from those who've made it. I'd be surprised if he didn't learn a vast portion of his trade from the openness of information. I guess he feels empowered w/ his :15 minutes and this is how he leverages it? Weak^10. +++the pxlpshr+++

potatolicious 17y ago
I didn't get that sense from the article - he's not refusing to release the info for his own benefit, or to be a dick; he's ostensibly doing it because he feels like this just encourages the "we'll fix stuff when other people find it in the wild" mentality, which doesn't help make software more secure in the long run. He wants software producers to actively try to break their software so that his ostensibly simplistic tools can't find wide open holes like this. I'm sure some egotism rolls into it somewhere, but I don't think it's altogether a bad stance... you shouldn't be waiting for your users to tell you where your security holes are.
- pxlpshr 17y ago
  you shouldn't be waiting for your users to tell you where your security holes are. while I agree with that ideology, in reality you just can't assume that to be the case when you have multiple people touching the code, and features added regularly that may inadvertently introduce holes. what software has ever been 100% bullet proof? Also, I think that statement largely contradicts the notion of 'release early, iterate often' considering it's grounded on the notion of integrating feedback and bugs reported by users. Are major corporations held at a different standard simply because they have more resources? Even with thousands of employees, it's still a glass box compared to the millions that will use it.

djb_hackernews 17y ago

Anyone able to give more detail? Maybe not on Millers specifics but how does inserting random data into a program become an exploitable vulnerability?

evgen 17y ago
http://en.wikipedia.org/wiki/Fuzz_testing http://en.wikipedia.org/wiki/Fuzz_testing