5 ms·

So I'd be interested to know your thoughts on bug bounties as against "traditional" security reviews. Have these areas of your application been through externa

So I'd be interested to know your thoughts on bug bounties as against "traditional" security reviews. Have these areas of your application been through external reviews before being opened up to bug bounty or did you decide to start there? I was thinking that for the amount you've paid out in bounties you could've engaged a reasonable team for several man-months, so was interested in what led you more down the bug bounty line for this.

user5994461 10y ago
> you could've engaged a reasonable team for several man-months Doubt that. 1) The daily rate for this kind of work is high. 2) It's hard to find people who can execute. (read: security as in pen testing, not security as in filling a PCI compliance checklist). 3) Multiply 1 and 2 by the number of people you want in the team.
- raesene6 10y ago
  Sorry, but got to disagree with you there. I'm a security tester and have been in the industry for 15+ years either as a buyer or provider of services, for small and large companies, I've been involved in procurement of multi hundred $k tests and involved in the delivery similar sized engagements. You absolutely can get good security testing consultants for $2k/day for example, and probably less depending on the region and exact speciality. for $368k at a $2k/day rate that would be 184 person-days or several person months (as stated). It's well known there are teams who specialize in specific technologies, and with a sophisticated customer, not too tricky to interview and make sure you get the right people.
  - user5994461 10y ago
    Sorry. Should have clarified. They could get pen testers and it would cost as much as they paid so far. Managing a pen test or a bounty program are very different things. I don't think that they planned to spend $300k in exploits when they first rolled out the bounties. A company serious about security should have both anyway: Security audits + bounty programs. They fundamentally cover different things, with some overlap.
    - raesene6 10y ago
      Ahh indeed, that was kind of my initial question for the top-commentor. I'm interested in whether Shopify explored getting security consultants in to review this area before going for a bug bounty on it, or went straight to the bug bounty. My personal feeling is that the order of play should be Internal Security Review --> External Security Review --> Bug Bounty as you can use the first two stages to catch all the basic stuff and some of the advanced stuff then leave the bug bounty to pay out for things the first two elements missed, but that you still want to know about. So I'm interested in data that suggests that companies are either going for that route, or have decided to cut out the external consultant review and go straight to bug bounty. In this case I'd guess part of that would be whether Shopify did indeed expect a $300k+ bug bounty programme or whether that was a surprise to them.
      - user5994461 10y ago
        I think companies should get standard compliance stuff done first (they're in payments so they certainly have lots of these to have) + a set of standard vulnerability scans (Nesus / FireEye). These things are "cheap" and easy to get, it's standard package 1-week-audit-for-XYZ. Then get custom pen testing and bug bounty programs later. They're a lot of work to get done and get right. pen testing is a lot of investment and preparation upfront[1], bug bounty is on a longer term. [1] Don't bring people at $2k a day if you didn't think through what they're gonna do.
      - tptacek 10y ago
        I think what 'xal is trying to say is that this bounty had more to do with security marketing than with accomplishing a particular tactical security goal. Their comment even concludes with a note that most or all the findings were accounted for with a sandboxing design they'd already planned.
        raesene6 10y ago
        Interesting approach to marketing, I'll give them that. Will be interesting to see how it works out for them in the medium-long term.
  - tptacek 10y ago
    $2k/day is a reasonable rate, just a shade towards the high side for vanilla web security work.
xal 10y ago
Of course we use both. One provides depth, the other breadth.
EiNSTeiN_ 10y ago
You seem to assume we set out to pay this amount to begin with. Indeed for this amount we could have went other ways, but hindsight is 20/20. No one expected to get so many valid sumbmissions in such a short time. We set the payout amounts this high as a way to attract talent at the beginning of the program, which worked quite well to bootstrap it.
- raesene6 10y ago
  Ah no I didn't really assume that, it was part of my question, as to whether it was a deliberate strategy to jump straight to bug bounty with an expectation of a potentially large number of reported issues or an unexpected event, where the number of reported bugs and their severity, was more than you'd expected. So (and I'm guessing from your comment you work for Shopify) I guess I can take from that, that it was the latter.
- codinghorror 10y ago
  I literally just got off the phone with Hacker One on Friday and they recommend the exact OPPOSITE of what you did. Start with low or no bounty to get the easy stuff off the plate and figure out what class of reports you want -- then ramp up the bounty over time. Which is what we'll be doing!

user5994461 10y ago

> you could've engaged a reasonable team for several man-months Doubt that. 1) The daily rate for this kind of work is high. 2) It's hard to find people who can execute. (read: security as in pen testing, not security as in filling a PCI compliance checklist). 3) Multiply 1 and 2 by the number of people you want in the team.

raesene6 10y ago
Sorry, but got to disagree with you there. I'm a security tester and have been in the industry for 15+ years either as a buyer or provider of services, for small and large companies, I've been involved in procurement of multi hundred $k tests and involved in the delivery similar sized engagements. You absolutely can get good security testing consultants for $2k/day for example, and probably less depending on the region and exact speciality. for $368k at a $2k/day rate that would be 184 person-days or several person months (as stated). It's well known there are teams who specialize in specific technologies, and with a sophisticated customer, not too tricky to interview and make sure you get the right people.
- user5994461 10y ago
  Sorry. Should have clarified. They could get pen testers and it would cost as much as they paid so far. Managing a pen test or a bounty program are very different things. I don't think that they planned to spend $300k in exploits when they first rolled out the bounties. A company serious about security should have both anyway: Security audits + bounty programs. They fundamentally cover different things, with some overlap.
  - raesene6 10y ago
    Ahh indeed, that was kind of my initial question for the top-commentor. I'm interested in whether Shopify explored getting security consultants in to review this area before going for a bug bounty on it, or went straight to the bug bounty. My personal feeling is that the order of play should be Internal Security Review --> External Security Review --> Bug Bounty as you can use the first two stages to catch all the basic stuff and some of the advanced stuff then leave the bug bounty to pay out for things the first two elements missed, but that you still want to know about. So I'm interested in data that suggests that companies are either going for that route, or have decided to cut out the external consultant review and go straight to bug bounty. In this case I'd guess part of that would be whether Shopify did indeed expect a $300k+ bug bounty programme or whether that was a surprise to them.
    - user5994461 10y ago
      I think companies should get standard compliance stuff done first (they're in payments so they certainly have lots of these to have) + a set of standard vulnerability scans (Nesus / FireEye). These things are "cheap" and easy to get, it's standard package 1-week-audit-for-XYZ. Then get custom pen testing and bug bounty programs later. They're a lot of work to get done and get right. pen testing is a lot of investment and preparation upfront[1], bug bounty is on a longer term. [1] Don't bring people at $2k a day if you didn't think through what they're gonna do.
    - tptacek 10y ago
      I think what 'xal is trying to say is that this bounty had more to do with security marketing than with accomplishing a particular tactical security goal. Their comment even concludes with a note that most or all the findings were accounted for with a sandboxing design they'd already planned.
      - raesene6 10y ago
        Interesting approach to marketing, I'll give them that. Will be interesting to see how it works out for them in the medium-long term.
- tptacek 10y ago
  $2k/day is a reasonable rate, just a shade towards the high side for vanilla web security work.

xal 10y ago

Of course we use both. One provides depth, the other breadth.

EiNSTeiN_ 10y ago

You seem to assume we set out to pay this amount to begin with. Indeed for this amount we could have went other ways, but hindsight is 20/20. No one expected to get so many valid sumbmissions in such a short time. We set the payout amounts this high as a way to attract talent at the beginning of the program, which worked quite well to bootstrap it.

raesene6 10y ago
Ah no I didn't really assume that, it was part of my question, as to whether it was a deliberate strategy to jump straight to bug bounty with an expectation of a potentially large number of reported issues or an unexpected event, where the number of reported bugs and their severity, was more than you'd expected. So (and I'm guessing from your comment you work for Shopify) I guess I can take from that, that it was the latter.
codinghorror 10y ago
I literally just got off the phone with Hacker One on Friday and they recommend the exact OPPOSITE of what you did. Start with low or no bounty to get the easy stuff off the plate and figure out what class of reports you want -- then ramp up the bounty over time. Which is what we'll be doing!