3 ms·

I see bug bounty programs as just being another example of InfoSec charlatanism. People do bug bounties because InfoSec people say you should do bug bounties, a

I see bug bounty programs as just being another example of InfoSec charlatanism. People do bug bounties because InfoSec people say you should do bug bounties, and not doing what InfoSec people say you should do implies some sort of malfeasance. But the value of them is remarkably questionable. For starters they are absolutely not a replacement for pen testing, having a bug bounty is not a sufficient substitute for any of the pen testing that you should be doing. The value of the bugs you have reported are also typically incredibly low value. Most of the reports you get through bug bounty programs are just the output of open source scanning and static analysis tools. You get no-effort reports for things like frame-able content and “your mobile app has a dependency, which has a dependency, which was compiled with marginally sub-optimal flags”. Actually valuable reports do make it through, but I seriously doubt having a bug bounty program is more effective than publishing a security email address on your website. They’re mostly just spam generating services that invite people to try pressure you into coughing up some money for largely trivial nonsense.

tptacek 6y ago
Bounty submissions are mostly scanner spam; maybe almost entirely. But every once in awhile you get a serious and important report, and you wouldn't have gotten it without the bounty. Retain 3 different pentesting firms to hit an app and their reports will overlap maybe 70-80%; similarly, bounty people find different stuff as well.
- AmericanChopper 6y ago
  > But every once in awhile you get a serious and important report Yes this is true. > and you wouldn't have gotten it without the bounty Of this I am highly skeptical. Most companies that I’ve worked at who don’t have bug bounty programs get the occasional serious report. These reports were submitted to companies before bug bounty programs were even thought up. Bug bounty services just seek to capture value from something that was happening long before they were invented, and I’m highly skeptical of them having a significant impact on generating serious reports that otherwise wouldn’t exist. The other issue is that everybody has a finite security budget, and dealing with the spam is going to (perhaps significantly) eat into that. In those cases, it’s taking money that could be spent on something with a decent ROI, and redirecting it to essentially just creating busy work for your security team (or spending their budget on paying somebody else to do that busy work). I think they really only exist because a lot of security professionals don’t really know what they’re doing, and a lot of their employers don’t really know what they’re supposed to be doing.
  - tptacek 6y ago
    I watched it happen, several times, on projects where I not only had 6 weeks of undivided time to run an app assessment before I took over their bounty program, but the client had contracted assessments from other firms prior to me being involved. It's also a refrain I got from almost everyone I asked about running bounty programs. A fundamental truth I don't think anybody who does appsec assessments professionally can escape: multiple audits of the same target will turn up different bugs. It happens even if the same people do both assessments! I've never talked to a software security professional who pushed back on this; maybe you'll be the first.
    - AmericanChopper 6y ago
      > A fundamental truth I don't think anybody who does appsec assessments professionally can escape: multiple audits of the same target will turn up different bugs. I completely agree with this. > I watched it happen, several times I don’t doubt you have, but this is anecdata. I wouldn’t want to draw any conclusions from this, especially because it conflicts with a lot of my own anecdata. Auditing/security testing is something you could infinitely devote incrementally more budget to, and infinitely receive incremental returns from. While I question it provides as much benefit as you’re describing, it can certainly provide a benefit. My issue with it is that, in my experience, it has always had a bad RoI. The resources that you have to devote to dealing with the spam make extracting any value from bounty programs a very expensive exercise. I’ve personally seen people devoting more than one day a week to managing bounty programs that they were lucky to get one good report per year out of. Imagine how much more value they could have gotten from having a security engineer spending a full day per week threat modeling with the product engineers. Perhaps the ROI is different if your department is over-funded, or if you have a big brand that people want to write self-promotion blogs about. But any time I hear a security professional parroting the importance of bug bounty programs, the cynic in my wonders whether they’re just looking for some low-skill busy work for themselves.
      - tptacek 6y ago
        I think we'd have to struggle to find a clear debate between the two of us. I just wanted to make the point that people run bounty programs because they do in fact turn up black swan bugs, even after the repeated pentests that most of these firms have already engaged.

tptacek 6y ago

Bounty submissions are mostly scanner spam; maybe almost entirely. But every once in awhile you get a serious and important report, and you wouldn't have gotten it without the bounty. Retain 3 different pentesting firms to hit an app and their reports will overlap maybe 70-80%; similarly, bounty people find different stuff as well.

AmericanChopper 6y ago
> But every once in awhile you get a serious and important report Yes this is true. > and you wouldn't have gotten it without the bounty Of this I am highly skeptical. Most companies that I’ve worked at who don’t have bug bounty programs get the occasional serious report. These reports were submitted to companies before bug bounty programs were even thought up. Bug bounty services just seek to capture value from something that was happening long before they were invented, and I’m highly skeptical of them having a significant impact on generating serious reports that otherwise wouldn’t exist. The other issue is that everybody has a finite security budget, and dealing with the spam is going to (perhaps significantly) eat into that. In those cases, it’s taking money that could be spent on something with a decent ROI, and redirecting it to essentially just creating busy work for your security team (or spending their budget on paying somebody else to do that busy work). I think they really only exist because a lot of security professionals don’t really know what they’re doing, and a lot of their employers don’t really know what they’re supposed to be doing.
- tptacek 6y ago
  I watched it happen, several times, on projects where I not only had 6 weeks of undivided time to run an app assessment before I took over their bounty program, but the client had contracted assessments from other firms prior to me being involved. It's also a refrain I got from almost everyone I asked about running bounty programs. A fundamental truth I don't think anybody who does appsec assessments professionally can escape: multiple audits of the same target will turn up different bugs. It happens even if the same people do both assessments! I've never talked to a software security professional who pushed back on this; maybe you'll be the first.
  - AmericanChopper 6y ago
    > A fundamental truth I don't think anybody who does appsec assessments professionally can escape: multiple audits of the same target will turn up different bugs. I completely agree with this. > I watched it happen, several times I don’t doubt you have, but this is anecdata. I wouldn’t want to draw any conclusions from this, especially because it conflicts with a lot of my own anecdata. Auditing/security testing is something you could infinitely devote incrementally more budget to, and infinitely receive incremental returns from. While I question it provides as much benefit as you’re describing, it can certainly provide a benefit. My issue with it is that, in my experience, it has always had a bad RoI. The resources that you have to devote to dealing with the spam make extracting any value from bounty programs a very expensive exercise. I’ve personally seen people devoting more than one day a week to managing bounty programs that they were lucky to get one good report per year out of. Imagine how much more value they could have gotten from having a security engineer spending a full day per week threat modeling with the product engineers. Perhaps the ROI is different if your department is over-funded, or if you have a big brand that people want to write self-promotion blogs about. But any time I hear a security professional parroting the importance of bug bounty programs, the cynic in my wonders whether they’re just looking for some low-skill busy work for themselves.
    - tptacek 6y ago
      I think we'd have to struggle to find a clear debate between the two of us. I just wanted to make the point that people run bounty programs because they do in fact turn up black swan bugs, even after the repeated pentests that most of these firms have already engaged.