5 ms·

I feel that bug bounties are still undervalued, and the market is still inefficient because the prices are unilaterally set by companies. The only other public

I feel that bug bounties are still undervalued, and the market is still inefficient because the prices are unilaterally set by companies. The only other publicly disclosed signals for market price come from third party companies and state actors. The other signals are not public and hard to quantify, they come from trying to weaponize and monetize exploits yourself. This results in potentially incurring various forms of liability, or reducing that by selling information to a different broker, who will eventually find someone to weaponize or monetize a piece of the exploit. This part is a much more efficient market, but it is not vertically integrated. The prime bug bounties seem to be trending upwards in value, with the bottom being crowded and with non-serious companies testing the waters. Does anyone have any ideas to make the value of bug bounties be more dynamic and elastic, trend upwards towards their true value inline with the growth of the sector?

contravariant 6y ago
I don't think it's entirely fair to say there is no competition in this market. In the end companies are still competing for a limited supply of ethical hackers. Except that it is a bit of an unusual market, since both sides are operating with very limited information. Perhaps it will become more stable when people realise you can sell negative results as well (either implicitly by cooperating, or through an actual market). Anything that can bridge the gap between hackers wanting to get paid for their efforts and companies wanting to pay for results would help. More volume would also help, in the end it's hard to sustain a whole industry on just tens or hundreds of millions per year.
- vmception 6y ago
  > I don't think it's entirely fair to say there is no competition in this market Good thing I never said that. The market is inefficient. I like that idea of monetizing negative results. More transaction volume generally helps, I think external market signals and their actual value are whats really missing. Like, how much would someone pay for an exploit should dictate how much a company should pay for letting a crowd try to fix it where only one person/group gets the payment. If there was a way for additional participants to get paid for bothering to look at all that would be helpful too, and different people can focus on how people try to game that instead of worrying about the idea of people gaming it.
  - albntomat0 6y ago
    Aren't negative results already monetized, through standard pen-testing servcies? The usefulness of a negative result is dependent on the thoroughness and skill of attacker, while a positive result is useful no matter who submits it. An unknown person finding a vulnerability has clear value, but a negative result from an unknown person doesn't mean much.
    - contravariant 6y ago
      A negative result has value as long as it's clear what vulnerabilities have been tested. Of course it's pointless if you don't trust the person testing it, but that's always going to be a problem. I guess you might need to have reproducibility to make it truly useful.
      - albntomat0 6y ago
        That seems to me to be mostly limited to automated tools, or checklist level evaluation? The good/dangerous bugs require creativity and deep system knowledge to find, which is hard to trust that an unknown person has.
        ozim 6y ago
        Then comes risk management and I would tell you "good bugs that require creativity and deep system knowledge to find" are not dangerous. It is sexy and all to sell "we hacked you using emacs through sendmail by the moon phase exploit that happens every 2 months". But really how many people will be able to exploit that? My company is not target of a state sponsored actors. Automated tools, checklist evaluation and decent administrators will be good enough for most of companies. Ordering pentests from time to time is good practice to check if administrators are doing their job as in "trust but verify".
        bashwizard 6y ago
        It's not. Running scanners and report scan results won't get you paid so it's a complete waste of time. Trust is easy. You either have a proof of concept and can show impact in your report or gtfo basically.
Proven 6y ago
Open market for exploits.
tptacek 6y ago
There are several other important price inputs to this market, including scanning services (thousands quarterly) and consulting penetration testing (tens of thousands annually). Most companies that offer bounties have also had pentests done; nobody believes that pentesters are underpaid (though maybe they should). The reality is probably that 80% of the vulnerabilities disclosed on bounty platforms just aren't worth that much. Certainly: companies that lack security expertise but manage bounties tend to radically overpay bounties; I'd of course be curious to see a breakdown of that $100MM by bug class.
- albntomat0 6y ago
  > Companies that lack security expertise but manage bounties tend to radically overpay bounties Could you share your reasoning on this? Are you saying they're overpaying for a low threat type of vulnerability, have a large number of vulns that should have been caught earlier, or something different? Genuinely curious! Thanks
  - tptacek 6y ago
    I'm saying that people are submitting sev:info's and getting paid $500 for them.
    - albntomat0 6y ago
      Makes sense, thanks for the clarification.
Taniwha 6y ago
Obviously there's a way to make the market more efficient, let black hats compete to purchase exploits on the same platforms .... of course that's likely illegal, for good reason. Instead we have two markets with only a few people playing in both so price signals are likely weak
cosmodisk 6y ago
"reducing that by selling information to a different broker, who will eventually find someone to weaponize or monetize a piece of the exploit" This has become the standard way of monetisation for the more capable ones who don't want or don't see the point to do the crime itself. ATM skimming kits, carjacking kits, etc. The problem is that security is not rewarded at all.Even a top level expert with very deep knowledge gets peanuts in comparison to the cost businesses incur when they get hacked.
AmericanChopper 6y ago
I see bug bounty programs as just being another example of InfoSec charlatanism. People do bug bounties because InfoSec people say you should do bug bounties, and not doing what InfoSec people say you should do implies some sort of malfeasance. But the value of them is remarkably questionable. For starters they are absolutely not a replacement for pen testing, having a bug bounty is not a sufficient substitute for any of the pen testing that you should be doing. The value of the bugs you have reported are also typically incredibly low value. Most of the reports you get through bug bounty programs are just the output of open source scanning and static analysis tools. You get no-effort reports for things like frame-able content and “your mobile app has a dependency, which has a dependency, which was compiled with marginally sub-optimal flags”. Actually valuable reports do make it through, but I seriously doubt having a bug bounty program is more effective than publishing a security email address on your website. They’re mostly just spam generating services that invite people to try pressure you into coughing up some money for largely trivial nonsense.
- tptacek 6y ago
  Bounty submissions are mostly scanner spam; maybe almost entirely. But every once in awhile you get a serious and important report, and you wouldn't have gotten it without the bounty. Retain 3 different pentesting firms to hit an app and their reports will overlap maybe 70-80%; similarly, bounty people find different stuff as well.
  - AmericanChopper 6y ago
    > But every once in awhile you get a serious and important report Yes this is true. > and you wouldn't have gotten it without the bounty Of this I am highly skeptical. Most companies that I’ve worked at who don’t have bug bounty programs get the occasional serious report. These reports were submitted to companies before bug bounty programs were even thought up. Bug bounty services just seek to capture value from something that was happening long before they were invented, and I’m highly skeptical of them having a significant impact on generating serious reports that otherwise wouldn’t exist. The other issue is that everybody has a finite security budget, and dealing with the spam is going to (perhaps significantly) eat into that. In those cases, it’s taking money that could be spent on something with a decent ROI, and redirecting it to essentially just creating busy work for your security team (or spending their budget on paying somebody else to do that busy work). I think they really only exist because a lot of security professionals don’t really know what they’re doing, and a lot of their employers don’t really know what they’re supposed to be doing.

contravariant 6y ago

I don't think it's entirely fair to say there is no competition in this market. In the end companies are still competing for a limited supply of ethical hackers. Except that it is a bit of an unusual market, since both sides are operating with very limited information. Perhaps it will become more stable when people realise you can sell negative results as well (either implicitly by cooperating, or through an actual market). Anything that can bridge the gap between hackers wanting to get paid for their efforts and companies wanting to pay for results would help. More volume would also help, in the end it's hard to sustain a whole industry on just tens or hundreds of millions per year.

vmception 6y ago
> I don't think it's entirely fair to say there is no competition in this market Good thing I never said that. The market is inefficient. I like that idea of monetizing negative results. More transaction volume generally helps, I think external market signals and their actual value are whats really missing. Like, how much would someone pay for an exploit should dictate how much a company should pay for letting a crowd try to fix it where only one person/group gets the payment. If there was a way for additional participants to get paid for bothering to look at all that would be helpful too, and different people can focus on how people try to game that instead of worrying about the idea of people gaming it.
- albntomat0 6y ago
  Aren't negative results already monetized, through standard pen-testing servcies? The usefulness of a negative result is dependent on the thoroughness and skill of attacker, while a positive result is useful no matter who submits it. An unknown person finding a vulnerability has clear value, but a negative result from an unknown person doesn't mean much.
  - contravariant 6y ago
    A negative result has value as long as it's clear what vulnerabilities have been tested. Of course it's pointless if you don't trust the person testing it, but that's always going to be a problem. I guess you might need to have reproducibility to make it truly useful.
    - albntomat0 6y ago
      That seems to me to be mostly limited to automated tools, or checklist level evaluation? The good/dangerous bugs require creativity and deep system knowledge to find, which is hard to trust that an unknown person has.
      - ozim 6y ago
        Then comes risk management and I would tell you "good bugs that require creativity and deep system knowledge to find" are not dangerous. It is sexy and all to sell "we hacked you using emacs through sendmail by the moon phase exploit that happens every 2 months". But really how many people will be able to exploit that? My company is not target of a state sponsored actors. Automated tools, checklist evaluation and decent administrators will be good enough for most of companies. Ordering pentests from time to time is good practice to check if administrators are doing their job as in "trust but verify".
      - bashwizard 6y ago
        It's not. Running scanners and report scan results won't get you paid so it's a complete waste of time. Trust is easy. You either have a proof of concept and can show impact in your report or gtfo basically.

Proven 6y ago

Open market for exploits.

tptacek 6y ago

There are several other important price inputs to this market, including scanning services (thousands quarterly) and consulting penetration testing (tens of thousands annually). Most companies that offer bounties have also had pentests done; nobody believes that pentesters are underpaid (though maybe they should). The reality is probably that 80% of the vulnerabilities disclosed on bounty platforms just aren't worth that much. Certainly: companies that lack security expertise but manage bounties tend to radically overpay bounties; I'd of course be curious to see a breakdown of that $100MM by bug class.

albntomat0 6y ago
> Companies that lack security expertise but manage bounties tend to radically overpay bounties Could you share your reasoning on this? Are you saying they're overpaying for a low threat type of vulnerability, have a large number of vulns that should have been caught earlier, or something different? Genuinely curious! Thanks
- tptacek 6y ago
  I'm saying that people are submitting sev:info's and getting paid $500 for them.
  - albntomat0 6y ago
    Makes sense, thanks for the clarification.

Taniwha 6y ago

Obviously there's a way to make the market more efficient, let black hats compete to purchase exploits on the same platforms .... of course that's likely illegal, for good reason. Instead we have two markets with only a few people playing in both so price signals are likely weak

cosmodisk 6y ago

"reducing that by selling information to a different broker, who will eventually find someone to weaponize or monetize a piece of the exploit" This has become the standard way of monetisation for the more capable ones who don't want or don't see the point to do the crime itself. ATM skimming kits, carjacking kits, etc. The problem is that security is not rewarded at all.Even a top level expert with very deep knowledge gets peanuts in comparison to the cost businesses incur when they get hacked.

AmericanChopper 6y ago

I see bug bounty programs as just being another example of InfoSec charlatanism. People do bug bounties because InfoSec people say you should do bug bounties, and not doing what InfoSec people say you should do implies some sort of malfeasance. But the value of them is remarkably questionable. For starters they are absolutely not a replacement for pen testing, having a bug bounty is not a sufficient substitute for any of the pen testing that you should be doing. The value of the bugs you have reported are also typically incredibly low value. Most of the reports you get through bug bounty programs are just the output of open source scanning and static analysis tools. You get no-effort reports for things like frame-able content and “your mobile app has a dependency, which has a dependency, which was compiled with marginally sub-optimal flags”. Actually valuable reports do make it through, but I seriously doubt having a bug bounty program is more effective than publishing a security email address on your website. They’re mostly just spam generating services that invite people to try pressure you into coughing up some money for largely trivial nonsense.

tptacek 6y ago
Bounty submissions are mostly scanner spam; maybe almost entirely. But every once in awhile you get a serious and important report, and you wouldn't have gotten it without the bounty. Retain 3 different pentesting firms to hit an app and their reports will overlap maybe 70-80%; similarly, bounty people find different stuff as well.
- AmericanChopper 6y ago
  > But every once in awhile you get a serious and important report Yes this is true. > and you wouldn't have gotten it without the bounty Of this I am highly skeptical. Most companies that I’ve worked at who don’t have bug bounty programs get the occasional serious report. These reports were submitted to companies before bug bounty programs were even thought up. Bug bounty services just seek to capture value from something that was happening long before they were invented, and I’m highly skeptical of them having a significant impact on generating serious reports that otherwise wouldn’t exist. The other issue is that everybody has a finite security budget, and dealing with the spam is going to (perhaps significantly) eat into that. In those cases, it’s taking money that could be spent on something with a decent ROI, and redirecting it to essentially just creating busy work for your security team (or spending their budget on paying somebody else to do that busy work). I think they really only exist because a lot of security professionals don’t really know what they’re doing, and a lot of their employers don’t really know what they’re supposed to be doing.