4 ms·

What's the general signal to noise ratio for bug reports?

by daddyo 9y ago

dsacco 9y ago
About 10:1 noise:signal. This comes from a variety of experiences: I used to manage a bug bounty for a mid-size company on Bugcrowd; in 2014 I surveyed people managing a bunch of programs across different sizes; I've participated in bug bounty programs for companies of different sizes. The more you offer for rewards and the more recognizable your company name, the more you will be spammed by people submitting reports like (I kid you not): "You have the OPTIONS method allowed on your site this is really serious." The last time I looked at the numbers, Google had over 80,000 bug bounty reports per year, with about 10% of them being valid and maybe another order of magnitude being high severity (I'm fuzzy on the last bit). It's probably over 100,000 per year at this point. It's not uncommon for recognizable but smaller companies to receive one or more per day. I'm aware of full-time security engineers at Facebook and Google who do almost nothing but respond to bug bounty reports. It's a lot like resumes - people who have essentially no qualifications, experience or (most importantly) a real vulnerability finding will nevertheless spam boilerplate bug reports to as many companies as they can. Take a look at the list of exclusions on a given program - you'll see that many of them explicitly call out common invalid findings that are so ridiculous it's kafkaesque. HackerOne and Bugcrowd provide a lot of technical sophistication to prime companies for success, but there is an organizational component that is very difficult. If your program is very active, it requires dedication to tune it so you're not flushing engineer-hours away responding to nonsense. This is not to say they're bad - quite the opposite, I think they're fantastic. But I generally recommend smaller companies set up a vulnerability disclosure program through a solid third party, and do so without a monetary reward until they can commit to dealing with a reasonable deluge of reports.
- thaumasiotes 9y ago
  My favorite bug bounty report so far read, in its entirety, "try it ASAP".
  - arkadiyt 9y ago
    I've received reports for things like "source code disclosure" where they link to our jQuery.
    - illumin8 9y ago
      LOL - I'd like to report that I was able to download the entire source code of your website by right-clicking and selecting "View Page Source..."
      - thaumasiotes 9y ago
        If only that were true... modern web pages frequently have basically nothing of any value in the page source; it's all dynamically loaded.
- sundvor 9y ago
  ~10% valid submissions still sounds like a fantastic number to me. Sure you have to sort out the bad ones, but it's still a solid stream of valid reports.
  - jcims 9y ago
    It's a pain when you're in the thick of it, but it really is a great way to round out your security program. There's an astonishing number of incredibly skilled and motivated folks out there, and a well-run bounty program can create a nice symbiotic relationship that benefits both. One other thing that never really gets any press is the fact that a good chunk of the folks sending in reports are young people in impoverished nations. Some of them can be pretty tricky to deal with, but if you hold a hard line on professional expectations you can see them flourish in pretty short order to be some of the best reporters out there. I only spent a short amount of time on the program I was with, but it was very rewarding. A+++, highly recommended.
    - sundvor 9y ago
      That's great about the young people! Thanks for relating this.
jcims 9y ago
Whoever runs this definitely works in a bug bounty program: https://twitter.com/cluelesssec https://twitter.com/cluelesssec
robbiemitchell 9y ago
Is there a ticketing system of some kind in play there? I imagine there would be steps like "Respond to user" before resolving/closing.
martenmickos 9y ago
Generally in the world of bug bounty programs, the signal-to-noise ratio (SNR) is around 10-20%. Even at this low rate, it is not too bad. Let's say you receive 10 reports. You can relatively quickly identify the 8-9 noisy reports to find the 1-2 valid ones. Of course, a higher SNR is always better. It saves you time and effort. On HackerOne, the average SNR across all programs is over 30%. The platform can automatically filter out certain reports that are duplicates or out of scope. The platform maintains an average signal rating for each hacker (aka security researcher). Companies can limit access to their programs to hackers with a certain signal or higher. This will significantly increase SNR for the program. Companies can also opt for a HackerOne program with triage included, in which case the SNR rises close to 100%.
- txutxu 9y ago
  > The platform maintains an average signal rating for each hacker (aka security researcher). Companies can limit access to their programs to hackers with a certain signal or higher. This will significantly increase SNR for the program. So if a new user of the platform, finds a valid or high impact bug, will be unable to report... less noise but a high value bug unreported in that case...

What's the general signal to noise ratio for bug reports?

by daddyo 9y ago

What's the general signal to noise ratio for bug reports?

About 10:1 noise:signal. This comes from a variety of experiences: I used to manage a bug bounty for a mid-size company on Bugcrowd; in 2014 I surveyed people managing a bunch of programs across different sizes; I've participated in bug bounty programs for companies of different sizes. The more you offer for rewards and the more recognizable your company name, the more you will be spammed by people submitting reports like (I kid you not): "You have the OPTIONS method allowed on your site this is really serious." The last time I looked at the numbers, Google had over 80,000 bug bounty reports per year, with about 10% of them being valid and maybe another order of magnitude being high severity (I'm fuzzy on the last bit). It's probably over 100,000 per year at this point. It's not uncommon for recognizable but smaller companies to receive one or more per day. I'm aware of full-time security engineers at Facebook and Google who do almost nothing but respond to bug bounty reports. It's a lot like resumes - people who have essentially no qualifications, experience or (most importantly) a real vulnerability finding will nevertheless spam boilerplate bug reports to as many companies as they can. Take a look at the list of exclusions on a given program - you'll see that many of them explicitly call out common invalid findings that are so ridiculous it's kafkaesque. HackerOne and Bugcrowd provide a lot of technical sophistication to prime companies for success, but there is an organizational component that is very difficult. If your program is very active, it requires dedication to tune it so you're not flushing engineer-hours away responding to nonsense. This is not to say they're bad - quite the opposite, I think they're fantastic. But I generally recommend smaller companies set up a vulnerability disclosure program through a solid third party, and do so without a monetary reward until they can commit to dealing with a reasonable deluge of reports.

thaumasiotes 9y ago
My favorite bug bounty report so far read, in its entirety, "try it ASAP".
- arkadiyt 9y ago
  I've received reports for things like "source code disclosure" where they link to our jQuery.
  - illumin8 9y ago
    LOL - I'd like to report that I was able to download the entire source code of your website by right-clicking and selecting "View Page Source..."
    - thaumasiotes 9y ago
      If only that were true... modern web pages frequently have basically nothing of any value in the page source; it's all dynamically loaded.
sundvor 9y ago
~10% valid submissions still sounds like a fantastic number to me. Sure you have to sort out the bad ones, but it's still a solid stream of valid reports.
- jcims 9y ago
  It's a pain when you're in the thick of it, but it really is a great way to round out your security program. There's an astonishing number of incredibly skilled and motivated folks out there, and a well-run bounty program can create a nice symbiotic relationship that benefits both. One other thing that never really gets any press is the fact that a good chunk of the folks sending in reports are young people in impoverished nations. Some of them can be pretty tricky to deal with, but if you hold a hard line on professional expectations you can see them flourish in pretty short order to be some of the best reporters out there. I only spent a short amount of time on the program I was with, but it was very rewarding. A+++, highly recommended.
  - sundvor 9y ago
    That's great about the young people! Thanks for relating this.

jcims 9y ago

Whoever runs this definitely works in a bug bounty program: https://twitter.com/cluelesssec https://twitter.com/cluelesssec

robbiemitchell 9y ago

Is there a ticketing system of some kind in play there? I imagine there would be steps like "Respond to user" before resolving/closing.

martenmickos 9y ago

Generally in the world of bug bounty programs, the signal-to-noise ratio (SNR) is around 10-20%. Even at this low rate, it is not too bad. Let's say you receive 10 reports. You can relatively quickly identify the 8-9 noisy reports to find the 1-2 valid ones. Of course, a higher SNR is always better. It saves you time and effort. On HackerOne, the average SNR across all programs is over 30%. The platform can automatically filter out certain reports that are duplicates or out of scope. The platform maintains an average signal rating for each hacker (aka security researcher). Companies can limit access to their programs to hackers with a certain signal or higher. This will significantly increase SNR for the program. Companies can also opt for a HackerOne program with triage included, in which case the SNR rises close to 100%.

txutxu 9y ago
> The platform maintains an average signal rating for each hacker (aka security researcher). Companies can limit access to their programs to hackers with a certain signal or higher. This will significantly increase SNR for the program. So if a new user of the platform, finds a valid or high impact bug, will be unable to report... less noise but a high value bug unreported in that case...