4 ms·

Hey, HN, CISO of Yahoo here, typing on a phone at a kid's birthday, so excuse the formatting. We run a very progressive bug bounty program that allows bugs lik

by secalex 13y ago

Hey, HN, CISO of Yahoo here, typing on a phone at a kid's birthday, so excuse the formatting. We run a very progressive bug bounty program that allows bugs like this to be posted publicly. Every once in a while we might miss something out of the thousands of invalid reports we receive every month, and we made a mistake in the triage of this bug. The bug is fixed and we won't make the same mistake again. We definitely consider info disclosure to be a class of issue that needs to be addressed and to infer otherwise from one mistake is incorrect. There are a handful of companies experimenting with this kind of open bounty model, and if we want it to survive (I certainly do) then we are going to all have to be willing to iterate to fix the problem, and move on.

tomcorrigan 13y ago
That's great, but you were aware of this issue for a month. If the whole point of having a bug bounty program is that you benefit from the distributed intelligence of the community you should perhaps place a bit more faith in your unpaid and unrewarded labor. Do you really receive thousands of invalid security reports every month or is user Schofield way out of their depth. Why should another dev ever bother submitting a security issue to Yahoo if they have to deal with such obstinate silliness?
- donohoe 13y ago
  Did you not read the part where they stated they missed this one and didn't remedy it as they should have?
- epriest 13y ago
  > Do you really receive thousands of invalid security reports every month or is user Schofield way out of their depth. Although our project is much smaller, we also run a bounty program through HackerOne, and publish aggregate results every month. You can see them under the "Security" headers of the changelog for the last few months to get a quick sense of the overall composition of reports that come through a channel like this, at least for our project: http://phabricator.org/changelog/ http://phabricator.org/changelog/ For example, last month we received 49 reports, of which we believe 5 were legitimate security issues which we fixed and awarded. Although the signal on this channel is extremely valuable, it's embedded in a lot of noise, and separating the two is often difficult and time consuming. It wouldn't surprise me if we made mistakes with a few reports even at this relatively small scale, and we have a much easier task than larger projects do. I'm extremely supportive of HackerOne, but I'm always a little worried we'll make a mistake and end up tried in the court of public opinion when we triaged >99% of the reports correctly and the overall impact of the program is hugely beneficial for researchers, for us, and for our users. Of course, we should be aiming for 100%, and getting it right almost all the time isn't a free pass for the cases when things go wrong, but seeing just the cases where an issue wasn't handled correctly discards a lot of context.
  - collingreene 13y ago
    To echo this sentiment: In 2013 facebook received 14,763 submissions which lead to 687 paid issues, 1 : 21 signal to noise. Facebook errs on the side of paying out as often as possible even for lame bugs (apache shows its version number in some talent acquisitions blog), code we didn't write, defense in depth type stuff, instances where the reporter was wrong and there wasn't actually a bug but in the process of investigating the non-bug we happened to find a bug on our own etc. Given all that, I would (personal opinion) put the number of useful, impactful security issues we received in 2013 at about 70. If we use this guide its 1 : 211 signal to noise. In this sea of noise the reports submitted are often in other languages or submitted by less clueful people. This yahoo example the reporter explained the issue pretty well but in my experience this is a rarity. A legit issue could come from anyone though, even the guy who writes a sentence of Polish and sends you a 30sec youtube video in 320x480. Basically doing a bug bounty right is very hard. Stuff like this will happen. By running a bug bounty at all you are opening your company up to situations like this but the bigger picture is that you care about security enough to still do it for the valid security issues bug bounties find. It is a strong signal to me that a company actually cares about security and we shouldn't lose focus of that in the midst of pitchfork-waving "but yahoo was WRONG". We recently released some stats that support all this here: https://www.facebook.com/notes/facebook-bug-bounty/bug-bounty-highlights-and-updates/818902394790655 https://www.facebook.com/notes/facebook-bug-bounty/bug-bount...
    - secalex 13y ago
      Thank you for your data. I'm hoping to do a talk this fall with detailed stats after we have a whole year on this platform, but to a first order approximation your ratios do not look far off from our experience.

by secalex 13y ago

tomcorrigan 13y ago

That's great, but you were aware of this issue for a month. If the whole point of having a bug bounty program is that you benefit from the distributed intelligence of the community you should perhaps place a bit more faith in your unpaid and unrewarded labor. Do you really receive thousands of invalid security reports every month or is user Schofield way out of their depth. Why should another dev ever bother submitting a security issue to Yahoo if they have to deal with such obstinate silliness?

donohoe 13y ago
Did you not read the part where they stated they missed this one and didn't remedy it as they should have?
epriest 13y ago
> Do you really receive thousands of invalid security reports every month or is user Schofield way out of their depth. Although our project is much smaller, we also run a bounty program through HackerOne, and publish aggregate results every month. You can see them under the "Security" headers of the changelog for the last few months to get a quick sense of the overall composition of reports that come through a channel like this, at least for our project: http://phabricator.org/changelog/ http://phabricator.org/changelog/ For example, last month we received 49 reports, of which we believe 5 were legitimate security issues which we fixed and awarded. Although the signal on this channel is extremely valuable, it's embedded in a lot of noise, and separating the two is often difficult and time consuming. It wouldn't surprise me if we made mistakes with a few reports even at this relatively small scale, and we have a much easier task than larger projects do. I'm extremely supportive of HackerOne, but I'm always a little worried we'll make a mistake and end up tried in the court of public opinion when we triaged >99% of the reports correctly and the overall impact of the program is hugely beneficial for researchers, for us, and for our users. Of course, we should be aiming for 100%, and getting it right almost all the time isn't a free pass for the cases when things go wrong, but seeing just the cases where an issue wasn't handled correctly discards a lot of context.
- collingreene 13y ago
  To echo this sentiment: In 2013 facebook received 14,763 submissions which lead to 687 paid issues, 1 : 21 signal to noise. Facebook errs on the side of paying out as often as possible even for lame bugs (apache shows its version number in some talent acquisitions blog), code we didn't write, defense in depth type stuff, instances where the reporter was wrong and there wasn't actually a bug but in the process of investigating the non-bug we happened to find a bug on our own etc. Given all that, I would (personal opinion) put the number of useful, impactful security issues we received in 2013 at about 70. If we use this guide its 1 : 211 signal to noise. In this sea of noise the reports submitted are often in other languages or submitted by less clueful people. This yahoo example the reporter explained the issue pretty well but in my experience this is a rarity. A legit issue could come from anyone though, even the guy who writes a sentence of Polish and sends you a 30sec youtube video in 320x480. Basically doing a bug bounty right is very hard. Stuff like this will happen. By running a bug bounty at all you are opening your company up to situations like this but the bigger picture is that you care about security enough to still do it for the valid security issues bug bounties find. It is a strong signal to me that a company actually cares about security and we shouldn't lose focus of that in the midst of pitchfork-waving "but yahoo was WRONG". We recently released some stats that support all this here: https://www.facebook.com/notes/facebook-bug-bounty/bug-bounty-highlights-and-updates/818902394790655 https://www.facebook.com/notes/facebook-bug-bounty/bug-bount...
  - secalex 13y ago
    Thank you for your data. I'm hoping to do a talk this fall with detailed stats after we have a whole year on this platform, but to a first order approximation your ratios do not look far off from our experience.