7 ms·

I would rather see more transparency once you are a reporter than shinier leaderboards. It is extremely frustrating to spend a week reverse engineering a vulner

I would rather see more transparency once you are a reporter than shinier leaderboards. It is extremely frustrating to spend a week reverse engineering a vulnerability in an opaque cloud service only to be told it was a known issue (but private), won’t be fixed (but is within 48 hours), and that you don’t qualify for any compensation. I would like to see private issues shared with reporters when they are independently discovered. I would like to see status updates from developers. I would like to see some kind of shared compensation system that acknowledges it can take more than one person to investigate a problem before it is fixable and that even time spent replicating a vulnerability has value.

anonymouswacker 5y ago
Sounds like a full time job vs. a gig.
- notatoad 5y ago
  yeah, these are all features of working on the development team of a project. involving every developer who wants to work on a bounty at that level would be an insane amount of management overhead. It makes sense if you're vetting people beforehand to make sure that giving them this level of access and communication is worth the effort, but that's what a job interview is.
  - deckar01 5y ago
    Not work on the bug, just be kept in the loop like the initial reporter. Putting in the same effort as the first reporter should earn you the same trust that is afforded to the first reporter.
    - ehsankia 5y ago
      If you're not going to win a prize, why would you need to be kept in the loop about an internal vulnerability that is being worked on? As for getting a prize, it goes back to the dupe issue which is the source of a lot of abuse. There's no way to prove you also worked on it or if you just got the info from your friend and want to double your winnings.
jcims 5y ago
*Not a Google employee but have worked for a bug bounty* I agree everything you've stated would be desirable, and if there was a strong culture and policy of supporting bounty programs from the CEO on down, this could potentially be achievable. However: dupes - On the bounty side dupes are extremely common and buddies telling buddies about their finds is going to drive fraud up quite a bit. In my triage work I saw very clear attempts at this regularly. wontfix - This one is largely due to the fact that many bug bounties don't have authority over or even shared reporting structure with the product teams. There's probably room for a consolation prize as long as the bug is in scope but that's about it. The fact that the bug goes away later could be a fix or could just be part of a new release. This should be extremely rare though and is worth following up with the program (again, as long as its in scope). sharing issues - This is going to struggle mightily with legal without good contracts and NDAs for each researcher. status updates - Agree its frustrating but is challenged by the product team/bug bounty alignment noted above. Most bounty programs don't get info from devs either and if product teams don't listen about fixing bugs the likelihood that they are going to regularly report on fixes is almost nil. shared comp - Unless I'm missing your point you can self-organize outside of the bounty program (and many do) for this.
- stingraycharles 5y ago
  Sounds like the conclusion is that the bounty programs needs to work closer together with the product teams if it wants to be more effective. Phrased differently, internal organizational challenges should never be a valid reason why a bug is disqualified. It’s completely irrelevant from an outsider’s perspective.
  - jcims 5y ago
    I don't know if 'extremely valid' is good English but it's how I would characterize your assessment. I totally agree. The challenge, however, is constructing a sustainable model to incentivize product teams to reciprocate this closer working relationship. I would say all but the smallest of companies running bug bounties also have an internal security function that is already doing reporting on vulnerabilities, time to fix, etc. etc. So whatever internal 'reputation' there might be across product teams is well established (and in my experience the culture around bug fixing across product orgs is consistent from both internally discovered and externally reported bugs) Another thing that happens is that there is typically a backlog on bugfixes, so if a researcher reports a new bug that's Medium priority, it's not going to get prioritized against a backlog of Criticals or Highs. Most of the lack of feedback from devs is simply the fact that there's been no action. Once the bug is front and center, the fixes are extremely simple and done within a few days and rolled out.
sirdarckcat 5y ago
about duplicates - google has this thing called grants https://bughunters.google.com/about/rules/5479188746993664 https://bughunters.google.com/about/rules/5479188746993664 that pay people for doing security research, even if they don't find any bugs. we agree that doing security research is valuable even if no bugs are fixed. about having access to private bugs, we don't want to share vulnerabilities with others without the researcher's permission, but the original researcher can make bugs public on the new website, you can see some of them here https://bughunters.google.com/report/reports https://bughunters.google.com/report/reports
cwkoss 5y ago
> only to be told it was a known issue (but private), won’t be fixed (but is within 48 hours), and that you don’t qualify for any compensation This kind of issue is rampant and opaque among bug bounty programs. IMO if a company says a bug is a wontfix, that's an immediate moral justification for public disclosure. If they say it's a known issue, they don't give a timeline of when it will be fixed, and it's still not fixed in a week, that's also moral justification for public disclosure. If multiple people have reported the same vulnerability, there is a high likelihood that the bug is known by even more people. Bugs that are not hastily addressed are wasting participants in your bug bounty's time. Companies with bug bounty programs need to treat security researchers with more respect, and when they don't the moral imperative shifts towards public disclosure so that others are warned that vulnerabilities exist and are not being addressed. Too many companies set up a bug bounty program as a box checking exercise and then have a lackadaisical attitude about addressing reports. I found a pretty obvious XSS on Tesla's website. Submitted through bugcrowd and got no information besides "marked as duplicate". Publicly disclosed, bugcrowd temporarily suspended me for disclosing, but it was fixed within a week. Nothing lights a fire under people's asses like airing their dirty laundry. If they had told me "we are working on a fix and expect it to be live in 3 weeks" I would have respected that and held off on disclosure.
- tptacek 5y ago
  For what it's worth: you don't need an artificial moral justification to post immediately. Post immediately if that's your thing.
  - cwkoss 5y ago
    I specified "moral" because most bug bounty programs' terms have a blanket prohibition against public disclosure (at least until vulnerability is resolved), and in some cases public disclosure could be legally ambiguous as well (because CFAA is so vague and broad).
tptacek 5y ago
It's really clear to me why people want more transparency on this stuff. I'd want it too if I was submitting to bounties. But the transparency you're asking for is difficult to actually provide. Meanwhile, for a vendor at Google's scale, there is essentially zero upside to screwing over bounty hunters. At any realistic valuation for a vulnerability, these are rounding error sums to the business. In fact, the exact opposite incentive exists: these bounty programs are deemed to be performing well when they pay out more money, not less. The people managing these bounties aren't paying with their own money. They'd rather make you happy and encourage you to submit more stuff. The two phenomena you describe here are real and common. But it's just simply the case that vendors are generally working through backlogs of issues, triaged by severity. If you're told your finding is a dupe of a private issue, it is overwhelmingly likely that it is. If you pay for every independent discovery of an issue, people game that; worse than the dead weight loss of the bogus bounties, you set up crazy incentives on your dev team to fix marginal issues because they're being gamed, rather than triaging according to real severity. And, bounty hunters do turn up real bugs that aren't real security issues, but are still bugs. If those bugs are easy to fix, they're going to get fixed! You have the same weird gaming and precedent issues if you start paying out non-exploitable bugfix findings; you encourage people to find and report non-exploitable bugs, and you screw up the team incentives on what to fix. I don't expect people to like any of this logic, because it boils down to "you should just trust the Google VRP people". But: you should. This isn't worth the cortisol. If it's driving you up a wall, maybe don't participate? There are other ways to market security bug finding skills. :) If you've never worked triage on a bounty before, my guess is that you can't really imagine how terrible the median interaction is. Maybe the next evolution of these programs will be long-term contract relationships with trusted, successful vuln hunters that address some of these concerns by separating out the people who are good at this stuff from the median submitter (the median submitter invests 2 weeks trying to litigate whether copying a cookie out of the Chrome inspector and pasting it into curl constitutes an account takeover vulnerability).
- jcims 5y ago
  >If you've never worked triage on a bounty before, my guess is that you can't really imagine how terrible the median interaction is. I deleted three sentences about this very topic in my earlier comment because it turned into an ugly rant, lol. >Maybe the next evolution of these programs will be long-term contract relationships with trusted, successful vuln hunters I'm actually somewhat surprised that bounty programs missed the whole 'gig workers are employees' issue.
thaumasiotes 5y ago
> I would like to see private issues shared with reporters when they are independently discovered. Other people have covered most of the rest of your comment, but there are things to say about this one specifically. The reporting platforms support it. It's a very frequent request from researchers who file duplicate reports. It's rare for a company to do this, because (1) it doesn't do anything to help with the issue being reported; (2) it doesn't change how the report will be handled; and (3) researchers hate it. They don't hate it when they file a duplicate report and want to see what scooped them. But they hate it when their report gets shared with someone who filed a duplicate finding. Sharing duplicate reports causes a lot more fights than it solves. And it tends to piss off the higher-productivity researchers in an attempt to soothe lower-productivity ones, who generally aren't soothed anyway. The ticket I saw the most outrage on over this issue was one that I duped to another ticket with a higher number. (HackerOne assigns ticket numbers serially. Fun!) How could I justify calling this report a duplicate of one that, as anyone could see, was filed later? Well, the second report began > Hi, I reported this through email and was told that in order to claim a reward I should open a ticket in the HackerOne program... (This situation is odd enough, and easy enough to explain without sharing private information, that I could explain it to the guy on the lower ticket number without causing problems (or sharing the other ticket directly. That's a big no-no.). I present it here as an example illustrating that, even if you think you have ironclad evidence that the program is out to screw you over, it probably isn't.) I definitely saw companies doing things that were unfair considering the program as a whole. That generally happened as part of an effort to preserve a relationship with a researcher who frequently filed useful reports. I never saw a ticket duped inappropriately. If you want my advice on how to get the most money out of your bug reports, it's this: - Don't antagonize the team handling you. - As much as you can get away with, demonstrate what you can do with the issue you report. The program will say "we investigate the impact of every issue and pay out according to the highest-severity potential impact." They are sincere. But they don't have the motivation you do to find the highest-severity potential impact. Any impact you actually demonstrate will automatically be considered, because nobody had to notice it was possible. - Sometimes an issue might ambiguously fall into a low-paying category, or maybe a high-paying category. Try to characterize it as belonging to the high-paying category. - Decide whether you're a "deep" guy who finds complex issues on a handful of platforms or a "shallow" guy who finds the same set of related issues everywhere he can with minimal effort. Both approaches work. If you're a "shallow" guy, make sure that when you report an issue, it's really there, and don't argue too much over what you think it should be worth. If you're a "deep" guy, you have more scope to develop a personal relationship with the team who handles you. - Watch for existing programs to add new platforms. This often happens when a company with an existing program makes a new acquisition. The new platform probably has a lot of low-hanging fruit. - Clean up after yourself. The most outrage I've ever seen on the company side was someone who demonstrated full-on RCE on a company server. He installed a webshell. And his webshell was still up when he filed his report. Don't be that guy. You can be disqualified from a bug that would have paid tens of thousands of dollars.
eyeareque 5y ago
So much this. It’s insanely annoying when they say something isn’t a risk and then they fix it shortly after. Google should improve this problem over new webpages.

anonymouswacker 5y ago

Sounds like a full time job vs. a gig.

notatoad 5y ago
yeah, these are all features of working on the development team of a project. involving every developer who wants to work on a bounty at that level would be an insane amount of management overhead. It makes sense if you're vetting people beforehand to make sure that giving them this level of access and communication is worth the effort, but that's what a job interview is.
- deckar01 5y ago
  Not work on the bug, just be kept in the loop like the initial reporter. Putting in the same effort as the first reporter should earn you the same trust that is afforded to the first reporter.
  - ehsankia 5y ago
    If you're not going to win a prize, why would you need to be kept in the loop about an internal vulnerability that is being worked on? As for getting a prize, it goes back to the dupe issue which is the source of a lot of abuse. There's no way to prove you also worked on it or if you just got the info from your friend and want to double your winnings.

jcims 5y ago

*Not a Google employee but have worked for a bug bounty* I agree everything you've stated would be desirable, and if there was a strong culture and policy of supporting bounty programs from the CEO on down, this could potentially be achievable. However: dupes - On the bounty side dupes are extremely common and buddies telling buddies about their finds is going to drive fraud up quite a bit. In my triage work I saw very clear attempts at this regularly. wontfix - This one is largely due to the fact that many bug bounties don't have authority over or even shared reporting structure with the product teams. There's probably room for a consolation prize as long as the bug is in scope but that's about it. The fact that the bug goes away later could be a fix or could just be part of a new release. This should be extremely rare though and is worth following up with the program (again, as long as its in scope). sharing issues - This is going to struggle mightily with legal without good contracts and NDAs for each researcher. status updates - Agree its frustrating but is challenged by the product team/bug bounty alignment noted above. Most bounty programs don't get info from devs either and if product teams don't listen about fixing bugs the likelihood that they are going to regularly report on fixes is almost nil. shared comp - Unless I'm missing your point you can self-organize outside of the bounty program (and many do) for this.

stingraycharles 5y ago
Sounds like the conclusion is that the bounty programs needs to work closer together with the product teams if it wants to be more effective. Phrased differently, internal organizational challenges should never be a valid reason why a bug is disqualified. It’s completely irrelevant from an outsider’s perspective.
- jcims 5y ago
  I don't know if 'extremely valid' is good English but it's how I would characterize your assessment. I totally agree. The challenge, however, is constructing a sustainable model to incentivize product teams to reciprocate this closer working relationship. I would say all but the smallest of companies running bug bounties also have an internal security function that is already doing reporting on vulnerabilities, time to fix, etc. etc. So whatever internal 'reputation' there might be across product teams is well established (and in my experience the culture around bug fixing across product orgs is consistent from both internally discovered and externally reported bugs) Another thing that happens is that there is typically a backlog on bugfixes, so if a researcher reports a new bug that's Medium priority, it's not going to get prioritized against a backlog of Criticals or Highs. Most of the lack of feedback from devs is simply the fact that there's been no action. Once the bug is front and center, the fixes are extremely simple and done within a few days and rolled out.

sirdarckcat 5y ago

about duplicates - google has this thing called grants https://bughunters.google.com/about/rules/5479188746993664 https://bughunters.google.com/about/rules/5479188746993664 that pay people for doing security research, even if they don't find any bugs. we agree that doing security research is valuable even if no bugs are fixed. about having access to private bugs, we don't want to share vulnerabilities with others without the researcher's permission, but the original researcher can make bugs public on the new website, you can see some of them here https://bughunters.google.com/report/reports https://bughunters.google.com/report/reports

cwkoss 5y ago

> only to be told it was a known issue (but private), won’t be fixed (but is within 48 hours), and that you don’t qualify for any compensation This kind of issue is rampant and opaque among bug bounty programs. IMO if a company says a bug is a wontfix, that's an immediate moral justification for public disclosure. If they say it's a known issue, they don't give a timeline of when it will be fixed, and it's still not fixed in a week, that's also moral justification for public disclosure. If multiple people have reported the same vulnerability, there is a high likelihood that the bug is known by even more people. Bugs that are not hastily addressed are wasting participants in your bug bounty's time. Companies with bug bounty programs need to treat security researchers with more respect, and when they don't the moral imperative shifts towards public disclosure so that others are warned that vulnerabilities exist and are not being addressed. Too many companies set up a bug bounty program as a box checking exercise and then have a lackadaisical attitude about addressing reports. I found a pretty obvious XSS on Tesla's website. Submitted through bugcrowd and got no information besides "marked as duplicate". Publicly disclosed, bugcrowd temporarily suspended me for disclosing, but it was fixed within a week. Nothing lights a fire under people's asses like airing their dirty laundry. If they had told me "we are working on a fix and expect it to be live in 3 weeks" I would have respected that and held off on disclosure.

tptacek 5y ago
For what it's worth: you don't need an artificial moral justification to post immediately. Post immediately if that's your thing.
- cwkoss 5y ago
  I specified "moral" because most bug bounty programs' terms have a blanket prohibition against public disclosure (at least until vulnerability is resolved), and in some cases public disclosure could be legally ambiguous as well (because CFAA is so vague and broad).

tptacek 5y ago

It's really clear to me why people want more transparency on this stuff. I'd want it too if I was submitting to bounties. But the transparency you're asking for is difficult to actually provide. Meanwhile, for a vendor at Google's scale, there is essentially zero upside to screwing over bounty hunters. At any realistic valuation for a vulnerability, these are rounding error sums to the business. In fact, the exact opposite incentive exists: these bounty programs are deemed to be performing well when they pay out more money, not less. The people managing these bounties aren't paying with their own money. They'd rather make you happy and encourage you to submit more stuff. The two phenomena you describe here are real and common. But it's just simply the case that vendors are generally working through backlogs of issues, triaged by severity. If you're told your finding is a dupe of a private issue, it is overwhelmingly likely that it is. If you pay for every independent discovery of an issue, people game that; worse than the dead weight loss of the bogus bounties, you set up crazy incentives on your dev team to fix marginal issues because they're being gamed, rather than triaging according to real severity. And, bounty hunters do turn up real bugs that aren't real security issues, but are still bugs. If those bugs are easy to fix, they're going to get fixed! You have the same weird gaming and precedent issues if you start paying out non-exploitable bugfix findings; you encourage people to find and report non-exploitable bugs, and you screw up the team incentives on what to fix. I don't expect people to like any of this logic, because it boils down to "you should just trust the Google VRP people". But: you should. This isn't worth the cortisol. If it's driving you up a wall, maybe don't participate? There are other ways to market security bug finding skills. :) If you've never worked triage on a bounty before, my guess is that you can't really imagine how terrible the median interaction is. Maybe the next evolution of these programs will be long-term contract relationships with trusted, successful vuln hunters that address some of these concerns by separating out the people who are good at this stuff from the median submitter (the median submitter invests 2 weeks trying to litigate whether copying a cookie out of the Chrome inspector and pasting it into curl constitutes an account takeover vulnerability).

jcims 5y ago
>If you've never worked triage on a bounty before, my guess is that you can't really imagine how terrible the median interaction is. I deleted three sentences about this very topic in my earlier comment because it turned into an ugly rant, lol. >Maybe the next evolution of these programs will be long-term contract relationships with trusted, successful vuln hunters I'm actually somewhat surprised that bounty programs missed the whole 'gig workers are employees' issue.

thaumasiotes 5y ago

> I would like to see private issues shared with reporters when they are independently discovered. Other people have covered most of the rest of your comment, but there are things to say about this one specifically. The reporting platforms support it. It's a very frequent request from researchers who file duplicate reports. It's rare for a company to do this, because (1) it doesn't do anything to help with the issue being reported; (2) it doesn't change how the report will be handled; and (3) researchers hate it. They don't hate it when they file a duplicate report and want to see what scooped them. But they hate it when their report gets shared with someone who filed a duplicate finding. Sharing duplicate reports causes a lot more fights than it solves. And it tends to piss off the higher-productivity researchers in an attempt to soothe lower-productivity ones, who generally aren't soothed anyway. The ticket I saw the most outrage on over this issue was one that I duped to another ticket with a higher number. (HackerOne assigns ticket numbers serially. Fun!) How could I justify calling this report a duplicate of one that, as anyone could see, was filed later? Well, the second report began > Hi, I reported this through email and was told that in order to claim a reward I should open a ticket in the HackerOne program... (This situation is odd enough, and easy enough to explain without sharing private information, that I could explain it to the guy on the lower ticket number without causing problems (or sharing the other ticket directly. That's a big no-no.). I present it here as an example illustrating that, even if you think you have ironclad evidence that the program is out to screw you over, it probably isn't.) I definitely saw companies doing things that were unfair considering the program as a whole. That generally happened as part of an effort to preserve a relationship with a researcher who frequently filed useful reports. I never saw a ticket duped inappropriately. If you want my advice on how to get the most money out of your bug reports, it's this: - Don't antagonize the team handling you. - As much as you can get away with, demonstrate what you can do with the issue you report. The program will say "we investigate the impact of every issue and pay out according to the highest-severity potential impact." They are sincere. But they don't have the motivation you do to find the highest-severity potential impact. Any impact you actually demonstrate will automatically be considered, because nobody had to notice it was possible. - Sometimes an issue might ambiguously fall into a low-paying category, or maybe a high-paying category. Try to characterize it as belonging to the high-paying category. - Decide whether you're a "deep" guy who finds complex issues on a handful of platforms or a "shallow" guy who finds the same set of related issues everywhere he can with minimal effort. Both approaches work. If you're a "shallow" guy, make sure that when you report an issue, it's really there, and don't argue too much over what you think it should be worth. If you're a "deep" guy, you have more scope to develop a personal relationship with the team who handles you. - Watch for existing programs to add new platforms. This often happens when a company with an existing program makes a new acquisition. The new platform probably has a lot of low-hanging fruit. - Clean up after yourself. The most outrage I've ever seen on the company side was someone who demonstrated full-on RCE on a company server. He installed a webshell. And his webshell was still up when he filed his report. Don't be that guy. You can be disqualified from a bug that would have paid tens of thousands of dollars.

eyeareque 5y ago

So much this. It’s insanely annoying when they say something isn’t a risk and then they fix it shortly after. Google should improve this problem over new webpages.