3 ms·

I'm pretty surprised at this: >For every valid report they get, they get 3 that aren't valid. Because taking the time to write and to submit an invalid report

I'm pretty surprised at this: >For every valid report they get, they get 3 that aren't valid. Because taking the time to write and to submit an invalid report is a total waste of the reporter's time. Reports aren't the type of thing that someone will accidentally say "oh this is a severe vulnerability! here's some cash" even though the researcher has submitted bullshit. So can you talk about "3 that aren't valid" for every valid report? Who makes these? Weird, obscure cranks, of the type who in other industries would be churning out perpetual motion devices? I would expect 80%+ of vulnerability reports to be serious and real - quite different from what you just wrote.

rommelsLegacy 10y ago
I guess it's various consulting firms that want to expand there market like e. g. KPMG cyber security division and the likes.
venantius 10y ago
As someone who has been on the receiving end of a bug bounty's mailbox, 3-to-1 sounds about right. We got a ton of invalid "security vulnerabilities" that were essentially either people reporting OAuth as a vulnerability or not understanding how XSS actually worked. Most of these came from teenagers in southeast Asia.
- logicallee 10y ago
  super-interesting - thanks.
- charleslmunger 10y ago
  I've also seen lots of reports of some serious vulnerability - "major product X doesn't validate TLS certificates" - where it looks like the reporter forgot they had added their own cert to the OS's trust store.
mistercow 10y ago
A lot of vulnerabilities are basically the same thing. From what I've seen, even legitimate researchers will have form reports. It's just an obvious optimization. From there it's pretty easy to see that "vulnerability spam" would be a thing.
tptacek 10y ago
Whether you run a bug bounty or not, there are now hundreds of people in Asia and Eastern Europe who hope to make $500 every time they find a page without X-Frame-Options set. A lot of them have pirated Burp Suite and are hoping to simply cash in on the scanner output.
- dsacco 10y ago
  I used to manage a bug bounty for a bank. It's nightmarish how accurate this is.
dsacco 10y ago
Here is some context for what I'm about to write: I managed a bug bounty for a sizable arm of BBVA. I have temporarily managed bug bounties for many smaller tech companies. In 2014, I surveyed the the industry as BugCrowd and Hackerone were coming into prominence. Bug bounties, on average, have a signal:noise ratio that is horrible. I advocate for the programs completely, but they require a lot of planning in order to prevent them from becoming overwhelming. I personally know security engineers at Google and Facebook whose full time job is sorting through nonsense bug bounty reports. What Tom said about people in Eurasia wanting to cash in on bug bounties is spot on. If you start a bug bounty program, expect to get a deluge of nonsense reports for bullshit like having the Trace HTTP method allowed on an API endpoint. For every one valid report, three - five will be bullshit. Of those that do not reproduce, half will be incomprehensible, and the other half will blatantly ignore the program guidelines or be low effort spam ("content spoofing"). If you offer a cash reward, expect the ratio of valid to invalid to be closer to 1 in 10. It's a numbers game for these people. The security research industry is bifurcated between rare, sophisticated and highly paid freelancers who do it mostly for passion outside of their day jobs, and opportunistic amateurs who couldn't write a curl command.
- willstrafach 10y ago
  BBVA? I have been trying to get in touch with them regarding an app of theirs (Access Key Extranet by BBVA Bancomer, S.A. https://appsto.re/us/4QEKfb.i https://appsto.re/us/4QEKfb.i) which does not properly validate TLS certificates.
- logicallee 10y ago
  Thanks for this detailed write-up! Super informative.

rommelsLegacy 10y ago

I guess it's various consulting firms that want to expand there market like e. g. KPMG cyber security division and the likes.

venantius 10y ago

As someone who has been on the receiving end of a bug bounty's mailbox, 3-to-1 sounds about right. We got a ton of invalid "security vulnerabilities" that were essentially either people reporting OAuth as a vulnerability or not understanding how XSS actually worked. Most of these came from teenagers in southeast Asia.

logicallee 10y ago
super-interesting - thanks.
charleslmunger 10y ago
I've also seen lots of reports of some serious vulnerability - "major product X doesn't validate TLS certificates" - where it looks like the reporter forgot they had added their own cert to the OS's trust store.

mistercow 10y ago

A lot of vulnerabilities are basically the same thing. From what I've seen, even legitimate researchers will have form reports. It's just an obvious optimization. From there it's pretty easy to see that "vulnerability spam" would be a thing.

tptacek 10y ago

Whether you run a bug bounty or not, there are now hundreds of people in Asia and Eastern Europe who hope to make $500 every time they find a page without X-Frame-Options set. A lot of them have pirated Burp Suite and are hoping to simply cash in on the scanner output.

dsacco 10y ago
I used to manage a bug bounty for a bank. It's nightmarish how accurate this is.

dsacco 10y ago

Here is some context for what I'm about to write: I managed a bug bounty for a sizable arm of BBVA. I have temporarily managed bug bounties for many smaller tech companies. In 2014, I surveyed the the industry as BugCrowd and Hackerone were coming into prominence. Bug bounties, on average, have a signal:noise ratio that is horrible. I advocate for the programs completely, but they require a lot of planning in order to prevent them from becoming overwhelming. I personally know security engineers at Google and Facebook whose full time job is sorting through nonsense bug bounty reports. What Tom said about people in Eurasia wanting to cash in on bug bounties is spot on. If you start a bug bounty program, expect to get a deluge of nonsense reports for bullshit like having the Trace HTTP method allowed on an API endpoint. For every one valid report, three - five will be bullshit. Of those that do not reproduce, half will be incomprehensible, and the other half will blatantly ignore the program guidelines or be low effort spam ("content spoofing"). If you offer a cash reward, expect the ratio of valid to invalid to be closer to 1 in 10. It's a numbers game for these people. The security research industry is bifurcated between rare, sophisticated and highly paid freelancers who do it mostly for passion outside of their day jobs, and opportunistic amateurs who couldn't write a curl command.

willstrafach 10y ago
BBVA? I have been trying to get in touch with them regarding an app of theirs (Access Key Extranet by BBVA Bancomer, S.A. https://appsto.re/us/4QEKfb.i https://appsto.re/us/4QEKfb.i) which does not properly validate TLS certificates.
logicallee 10y ago
Thanks for this detailed write-up! Super informative.