4 ms·

Not a Google employee but have worked for a bug bounty I agree everything you've stated would be desirable, and if there was a strong culture and policy of s

by jcims 5y ago

*Not a Google employee but have worked for a bug bounty* I agree everything you've stated would be desirable, and if there was a strong culture and policy of supporting bounty programs from the CEO on down, this could potentially be achievable. However: dupes - On the bounty side dupes are extremely common and buddies telling buddies about their finds is going to drive fraud up quite a bit. In my triage work I saw very clear attempts at this regularly. wontfix - This one is largely due to the fact that many bug bounties don't have authority over or even shared reporting structure with the product teams. There's probably room for a consolation prize as long as the bug is in scope but that's about it. The fact that the bug goes away later could be a fix or could just be part of a new release. This should be extremely rare though and is worth following up with the program (again, as long as its in scope). sharing issues - This is going to struggle mightily with legal without good contracts and NDAs for each researcher. status updates - Agree its frustrating but is challenged by the product team/bug bounty alignment noted above. Most bounty programs don't get info from devs either and if product teams don't listen about fixing bugs the likelihood that they are going to regularly report on fixes is almost nil. shared comp - Unless I'm missing your point you can self-organize outside of the bounty program (and many do) for this.

stingraycharles 5y ago
Sounds like the conclusion is that the bounty programs needs to work closer together with the product teams if it wants to be more effective. Phrased differently, internal organizational challenges should never be a valid reason why a bug is disqualified. It’s completely irrelevant from an outsider’s perspective.
- jcims 5y ago
  I don't know if 'extremely valid' is good English but it's how I would characterize your assessment. I totally agree. The challenge, however, is constructing a sustainable model to incentivize product teams to reciprocate this closer working relationship. I would say all but the smallest of companies running bug bounties also have an internal security function that is already doing reporting on vulnerabilities, time to fix, etc. etc. So whatever internal 'reputation' there might be across product teams is well established (and in my experience the culture around bug fixing across product orgs is consistent from both internally discovered and externally reported bugs) Another thing that happens is that there is typically a backlog on bugfixes, so if a researcher reports a new bug that's Medium priority, it's not going to get prioritized against a backlog of Criticals or Highs. Most of the lack of feedback from devs is simply the fact that there's been no action. Once the bug is front and center, the fixes are extremely simple and done within a few days and rolled out.
  - stingraycharles 5y ago
    I think there are two separate issues, though: * properly qualifying a bug bounty submission * actually fixing the thing From what I understood from your original comment, was that sometimes things get qualified as “wontfix” because of internal struggles. I understand that these struggles exist, as with any large org with conflicting priorities, especially when the issues are not 0day severity. I think, however, from an outsider’s perspective, acknowledging it’s a valid bug, but also saying “this will not get fixed any time soon” is infinitely better than just marking it as wontfix. As such, I’d argue that the minimum effort the product teams would have to put into the bug bounty program is properly qualifying the submissions. If that, for some reason, is too much to ask, I submit it’s a higher level, leadership problem that they want to have a bug bounty program without the necessary resources to qualify the bugs.
    - jcims 5y ago
      Your points are totally valid and I agree with them in principle, I just see them as existing on a spectrum. One pretty interesting example is GitLab's bounty program because you can see both sides, both the reported issues and how they are tracked on the engineering side: https://hackerone.com/gitlab/hacktivity?type=team https://hackerone.com/gitlab/hacktivity?type=team https://gitlab.com/gitlab-org/gitlab/-/issues?label_name=HackerOne https://gitlab.com/gitlab-org/gitlab/-/issues?label_name=Hac... There's very good organizational alignment here from what I can see, and overall the program seems to be quite adaptive and well supported (e.g. the comment about eliminating the 50% bounty here https://hackerone.com/reports/1154542#activity-11394543 https://hackerone.com/reports/1154542#activity-11394543, 'unduping' a report here https://hackerone.com/reports/402658 https://hackerone.com/reports/402658 ). It's about as best-case as you can get for a bug bounty...relatively small, single-product company that exposed to the sunshine effect of a public issue tracker. That said, if you peruse every one of those issues you're going to find instances where they drop the ball, take too long, make mistakes, etc. etc. So I would just say that if you ask anyone that has run a bug bounty, they will likely say they were surprised how difficult it was. You'll also generally find that they are advocates for the researcher community and want to run the best program they can. Everything else is largely a function of the dark art of business prioritization and the quality of the experience will be a function of how well they and their team is able to navigate it.
dylan604 5y ago
>dupes - On the bounty side dupes are extremely common and buddies telling buddies about their finds is going to drive fraud up quite a bit. In my triage work I saw very clear attempts at this regularly. wouldn't looking at the logs of when the reports were taken pretty much clearly show the first person to make the report? how is this a thing that gets confused to be an issue?
- jcims 5y ago
  Yes. There was never a real problem figuring out who reported it first. Duplicate reports are likely the most frustrating thing most security researchers will encounter. They put in a ton of work into finding the bug, developing a proof of concept and writing up a detailed report with the hope and expectation of being awarded for their effort. So when the triage team comes back and say it’s a duplicate and there will be no award, it's incredibly maddening. If someone needs to vent in my direction because of that, I can totally understand. The *problem* was trying to remain diplomatic with people who would sit there and repeatedly claim they were actually first, or that it wasn't a dupe, or that all of this was far too sophisticated for me to understand. Then, finding no sympathy from me, go to Twitter to wail and moan and bash the program with impunity because they know that the org won't respond in kind.
  - dylan604 5y ago
    >they know that the org won't respond in kind. Maybe they should be allowed to. If it is someone that is seriously being that petulant about something, then the Org could post dates of correspondence, and even quote petulant tempertantrum once it escalates beyond civility. Once the user name gets out there, other bounty programs could just put a blanket ignor and drive the petulant person into obscurity. But of course it is the today&now, and nobody actually believes facts anymore.
    - jcims 5y ago
      I worked with some true geniuses back then, and the idea of watching them systematically dismantle idiots in a public forum would give me chills. Alas, it wasn't to be. :)

*Not a Google employee but have worked for a bug bounty* I agree everything you've stated would be desirable, and if there was a strong culture and policy of s

by jcims 5y ago

stingraycharles 5y ago

Sounds like the conclusion is that the bounty programs needs to work closer together with the product teams if it wants to be more effective. Phrased differently, internal organizational challenges should never be a valid reason why a bug is disqualified. It’s completely irrelevant from an outsider’s perspective.

jcims 5y ago
I don't know if 'extremely valid' is good English but it's how I would characterize your assessment. I totally agree. The challenge, however, is constructing a sustainable model to incentivize product teams to reciprocate this closer working relationship. I would say all but the smallest of companies running bug bounties also have an internal security function that is already doing reporting on vulnerabilities, time to fix, etc. etc. So whatever internal 'reputation' there might be across product teams is well established (and in my experience the culture around bug fixing across product orgs is consistent from both internally discovered and externally reported bugs) Another thing that happens is that there is typically a backlog on bugfixes, so if a researcher reports a new bug that's Medium priority, it's not going to get prioritized against a backlog of Criticals or Highs. Most of the lack of feedback from devs is simply the fact that there's been no action. Once the bug is front and center, the fixes are extremely simple and done within a few days and rolled out.
- stingraycharles 5y ago
  I think there are two separate issues, though: * properly qualifying a bug bounty submission * actually fixing the thing From what I understood from your original comment, was that sometimes things get qualified as “wontfix” because of internal struggles. I understand that these struggles exist, as with any large org with conflicting priorities, especially when the issues are not 0day severity. I think, however, from an outsider’s perspective, acknowledging it’s a valid bug, but also saying “this will not get fixed any time soon” is infinitely better than just marking it as wontfix. As such, I’d argue that the minimum effort the product teams would have to put into the bug bounty program is properly qualifying the submissions. If that, for some reason, is too much to ask, I submit it’s a higher level, leadership problem that they want to have a bug bounty program without the necessary resources to qualify the bugs.
  - jcims 5y ago
    Your points are totally valid and I agree with them in principle, I just see them as existing on a spectrum. One pretty interesting example is GitLab's bounty program because you can see both sides, both the reported issues and how they are tracked on the engineering side: https://hackerone.com/gitlab/hacktivity?type=team https://hackerone.com/gitlab/hacktivity?type=team https://gitlab.com/gitlab-org/gitlab/-/issues?label_name=HackerOne https://gitlab.com/gitlab-org/gitlab/-/issues?label_name=Hac... There's very good organizational alignment here from what I can see, and overall the program seems to be quite adaptive and well supported (e.g. the comment about eliminating the 50% bounty here https://hackerone.com/reports/1154542#activity-11394543 https://hackerone.com/reports/1154542#activity-11394543, 'unduping' a report here https://hackerone.com/reports/402658 https://hackerone.com/reports/402658 ). It's about as best-case as you can get for a bug bounty...relatively small, single-product company that exposed to the sunshine effect of a public issue tracker. That said, if you peruse every one of those issues you're going to find instances where they drop the ball, take too long, make mistakes, etc. etc. So I would just say that if you ask anyone that has run a bug bounty, they will likely say they were surprised how difficult it was. You'll also generally find that they are advocates for the researcher community and want to run the best program they can. Everything else is largely a function of the dark art of business prioritization and the quality of the experience will be a function of how well they and their team is able to navigate it.

dylan604 5y ago

>dupes - On the bounty side dupes are extremely common and buddies telling buddies about their finds is going to drive fraud up quite a bit. In my triage work I saw very clear attempts at this regularly. wouldn't looking at the logs of when the reports were taken pretty much clearly show the first person to make the report? how is this a thing that gets confused to be an issue?

jcims 5y ago
Yes. There was never a real problem figuring out who reported it first. Duplicate reports are likely the most frustrating thing most security researchers will encounter. They put in a ton of work into finding the bug, developing a proof of concept and writing up a detailed report with the hope and expectation of being awarded for their effort. So when the triage team comes back and say it’s a duplicate and there will be no award, it's incredibly maddening. If someone needs to vent in my direction because of that, I can totally understand. The *problem* was trying to remain diplomatic with people who would sit there and repeatedly claim they were actually first, or that it wasn't a dupe, or that all of this was far too sophisticated for me to understand. Then, finding no sympathy from me, go to Twitter to wail and moan and bash the program with impunity because they know that the org won't respond in kind.
- dylan604 5y ago
  >they know that the org won't respond in kind. Maybe they should be allowed to. If it is someone that is seriously being that petulant about something, then the Org could post dates of correspondence, and even quote petulant tempertantrum once it escalates beyond civility. Once the user name gets out there, other bounty programs could just put a blanket ignor and drive the petulant person into obscurity. But of course it is the today&now, and nobody actually believes facts anymore.
  - jcims 5y ago
    I worked with some true geniuses back then, and the idea of watching them systematically dismantle idiots in a public forum would give me chills. Alas, it wasn't to be. :)