5 ms·

I worked on Chrome for most of its life. The internal bug tracker was closed when the product went public (we manually migrated some bugs we were actively work

by evmar 14y ago

I worked on Chrome for most of its life. The internal bug tracker was closed when the product went public (we manually migrated some bugs we were actively working on, and the rest got comments of the form "if you really want this fixed, we're sorry to say that you need to refile it on the public tracker"). The real truth is sadder: many bugs are lost, much like yours. In the old days I (and a few friends) personally reviewed every bug marked "Linux" and we attempted to triage them, but at one point I realized bugs were filed faster than we could classify or fix them. That isn't saying new bugs were being introduced rapidly, but rather that there's a lot of random spam ("my site doesn't work" / "does it work in other browsers?" / "no") and not-yet-implemented feature requests that appear to users as bugs. The latest bug on the public tracker is bug number 174163. Chrome was released in late 2008, so let's round up and say the tracker's been up for five years. That's averaging nearly 35k bugs a year, or 95 a day, and in reality the rate was slower in the early years so it's much faster than that now. Even just reading each bug and understanding whether it's reporting an actual issue takes hours. I have come to believe that a publicly writable bug tracker is a bad idea. Something like what SQLite does (where once users must first establish via some other mechanism they know how to file useful bugs -- yours look good at a glance, but I'd recommend attaching the demo html to the tracker so the link doesn't eventually 404) likely makes a lot of people mad but at least they're not shouting into a vacuum. Recently (like in the last few months) I heard they've allocated more resources to looking at the tracker, so maybe things will improve. It's honestly just a hard problem.

brownbat 14y ago
> It's honestly just a hard problem. We really need a way to reset those bug report counters to zero in one fell swoop. The solution, of course, is planned obsolescence. Bonus: http://xkcd.com/844/ http://xkcd.com/844/
abstractbill 14y ago
This makes me think there's a market for a better bug tracker - maybe one where people submitting bugs have karma, and you can sort bugs by that.
- bo1024 14y ago
  It's a tough problem. You really want to be able to take advantage of the tail of contributors -- the people who use one particular product, use it a lot, but have never filed bug reports for other products. I do think a stackoverflow-type solution would be worth a shot -- if you post a good bug report, it will be hopefully easy for people to understand or replicate and will get voted up, without requiring previous posts on your part. That's if you have a "crowd" actually willing to look at these things (might work for Chrome, not sure about small projects).
  - hayksaakian 14y ago
    Maybe the starring system can be this?
  - RachelF 14y ago
    You often feel that bug reporting just goes to dev/null
- tasuki 14y ago
  I like how Ubuntu handles this - each bug has a "Bug heat" number, which is determined by how many people say the bug affects them and by other criteria.
blaabjerg 14y ago
> The latest bug on the public tracker is bug number 174163. Chrome was released in late 2008, so let's round up and say the tracker's been up for five years. That's averaging nearly 35k bugs a year, or 95 a day, and in reality the rate was slower in the early years so it's much faster than that now. That honestly doesn't sound too scary. I did QA for Opera for a few years, and triaging (for specific platforms, at least) is remarkably quick after a while. A glance and a quick test on two platforms should let you toss a bug in either of four bins: a) WFM/invalid/bullshit b) Duplicate (it's scary how many bug id's you remember after a while) c) Likely a core/cross-platform problem (somebody else's problem) d) Worthy of further investigation Provided that you have some sort of bucket in which to toss the cross-platform/core bugs, only the bugs in category d should take longer than a couple of minutes each to triage. Some would argue that bugs in category d are already triaged at that point and should be put in some ANALYZE bucket.
- guelo 14y ago
  I think the real problem is that any talented ambitious employee that spends enough 40-hour weeks to get good at bug triaging is going to want to get the hell out of there as soon as possible. The work is close to data entry in its rewardishness.
  - blaabjerg 14y ago
    That's definitely true. It's important to spread the workload over an optimal number of employees so they do enough triaging to stay in the loop, but not enough to get sick and tired of it. A delicate balance. It's an important chore, though.
  - i386 14y ago
    I love doing triage. It's one of the best times to learn more about how your customers actually use the product.
    - greggman 14y ago
      Apply at Google for the Chrome team :-) We'd be happy to have you do triage
mmilo 14y ago
The reason it takes hours to verify whether an issue is even an issue is because people that consume tech describe problems like this "My shit just broke, help!" while people that build tech need something like "I was logged in as an admin user on the dashboard interface using Firefox v.18 and I clicked on the submit button and validation didn’t work". Most bug trackers worry about how best to show you a list of issues, the thing is if most of those issues aren’t reported correctly they’re useless. We’re currently trying to resolve this problem for web devs specifically by creating an embedded tracker, where you integrate the bug reporting directly into a site or web app. Meaning people can report issues in the way that’s natural for them e.g. "this is broken" and the tracker handles capturing all the other data a dev needs to verify and fix the problem. Our app can be found at http://www.bugherd.com http://www.bugherd.com if it’s of any interest to anyone.
- Zak 14y ago
  A big part of the problem is social, not technical. A bug report is essentially a set of instructions for a person who knows how to operate the software, but not necessarily anything else about the user's situation to follow. Most people are not used to writing instructions without conversational feedback and have a hard time figuring out what information in relevant and what to leave out. For a high-volume end-user project like Chrome or Firefox, a wizard/flowchart type UI probably ought to be the default, with a more streamlined UI available to people who have been approved by admins or met some criteria programmed in to the system. An example of the process a user might go through is: What's the problem? -> A web page doesn't look right in Chrome -> Have you tried it in another browser? -> No -> What other browsers do you have? -> None -> Ok. Here's a list of browsers that run on your OS and where to get them -> Firefox -> Ok, does it look right in Firefox? -> Yes -> Do you know how to take a screenshot? And so on until the user has likely provided enough information for a good bug report. There could be short-circuits in the process - for example, when asking if the user has tried another browser, a small link a bit outside the main UI could say "I've tried another browser and I have screenshots from both".
  - bruceboughton 14y ago
    I'm not sure that encouraging users to download another browser to try the page in would (a) encourage many bug reports (b) be a great idea for user retention.
ihaveajob 14y ago
I imagine by now someone at Google would have looked at the problem and written a basic bug clustering mechanism to group reports on the same issue. There's usually a very clear Pareto pattern when it comes to bugs, and fixing issues for 80% of users would be huge.
DeepDuh 14y ago
Sometimes adding humans is the best option? How about just hire some student workers to sweep through bug reports remotely? Even some Mechanical Turk approach might work, if you can create some checklist of what a useful bug report must have (and if you got this, it could also be integrated into the bug tracker workflow).
- aidenn0 14y ago
  That's hard to do because you end up with situations where the bug submitter figures out s/he knows more about the product than the person on the other end of the line, and stops responding out of frustration.
  - DeepDuh 14y ago
    So? Just close those reports then. I thought the issue here is to filter out anything spammy while not loosing any potentially valuable reports. A human should be able to filter out at least 90% of the spammy stuff while not loosing any good ones, thus significantly reducing the workload of engineers in higher pay grades. Google sometimes give me the impression they want to absolutely only hire the best engineers, but I'd argue that many non automatable tasks can be done more efficiently by having some reasonably trained lower rank staff. See also all the complaints about missing service when trying to use Google products commercially - if they do want to seriously enter that market, this stuff needs to change significantly. Other things that come to mind: Automated Youtube takedowns and Gmail account blocks. Do you really want to save those few million bucks a year for not hiring support staff, then have lots of disgruntled users blogging about how evil you are and then pouring those millions into PR campaigns?
    - aidenn0 14y ago
      " I thought the issue here is to filter out anything spammy while not loosing any potentially valuable reports. " Yes! But, in my comment I pointed out a specific example of where you lose a potentially valuable report.
blablabla123 14y ago
I think for such projects it would make sense to seek help from volunteers. People who don't necessarily contribute to the code base, but who could help organize the many bugs. After all it's just a question of man power to remove duplicates, prioritize by commonness and make sure long-term bugs really get fixed. That latter has the advantage that the code base really gets improved. In fact the Ubuntu project seems to do something similar eventhough it is IMHO not well enough organized: there is ubuntuforums.org, when I recently asked a question there, within minutes I got a response. The response was not helpful at all but still. The moderators were really doing a lot to keep the forum organized, and for instanced pulled me into a separate thread to not spam another bug. (Which was in their opinion a different issue.) Taking this a step further and bridging it with the actual Ubuntu bug tracker on launchpad would be awesome. Maybe with launchpad having only limited write permissions for the public.

by evmar 14y ago

brownbat 14y ago

> It's honestly just a hard problem. We really need a way to reset those bug report counters to zero in one fell swoop. The solution, of course, is planned obsolescence. Bonus: http://xkcd.com/844/ http://xkcd.com/844/

abstractbill 14y ago

This makes me think there's a market for a better bug tracker - maybe one where people submitting bugs have karma, and you can sort bugs by that.

bo1024 14y ago
It's a tough problem. You really want to be able to take advantage of the tail of contributors -- the people who use one particular product, use it a lot, but have never filed bug reports for other products. I do think a stackoverflow-type solution would be worth a shot -- if you post a good bug report, it will be hopefully easy for people to understand or replicate and will get voted up, without requiring previous posts on your part. That's if you have a "crowd" actually willing to look at these things (might work for Chrome, not sure about small projects).
- hayksaakian 14y ago
  Maybe the starring system can be this?
- RachelF 14y ago
  You often feel that bug reporting just goes to dev/null
tasuki 14y ago
I like how Ubuntu handles this - each bug has a "Bug heat" number, which is determined by how many people say the bug affects them and by other criteria.

blaabjerg 14y ago

> The latest bug on the public tracker is bug number 174163. Chrome was released in late 2008, so let's round up and say the tracker's been up for five years. That's averaging nearly 35k bugs a year, or 95 a day, and in reality the rate was slower in the early years so it's much faster than that now. That honestly doesn't sound too scary. I did QA for Opera for a few years, and triaging (for specific platforms, at least) is remarkably quick after a while. A glance and a quick test on two platforms should let you toss a bug in either of four bins: a) WFM/invalid/bullshit b) Duplicate (it's scary how many bug id's you remember after a while) c) Likely a core/cross-platform problem (somebody else's problem) d) Worthy of further investigation Provided that you have some sort of bucket in which to toss the cross-platform/core bugs, only the bugs in category d should take longer than a couple of minutes each to triage. Some would argue that bugs in category d are already triaged at that point and should be put in some ANALYZE bucket.

guelo 14y ago
I think the real problem is that any talented ambitious employee that spends enough 40-hour weeks to get good at bug triaging is going to want to get the hell out of there as soon as possible. The work is close to data entry in its rewardishness.
- blaabjerg 14y ago
  That's definitely true. It's important to spread the workload over an optimal number of employees so they do enough triaging to stay in the loop, but not enough to get sick and tired of it. A delicate balance. It's an important chore, though.
- i386 14y ago
  I love doing triage. It's one of the best times to learn more about how your customers actually use the product.
  - greggman 14y ago
    Apply at Google for the Chrome team :-) We'd be happy to have you do triage

mmilo 14y ago

The reason it takes hours to verify whether an issue is even an issue is because people that consume tech describe problems like this "My shit just broke, help!" while people that build tech need something like "I was logged in as an admin user on the dashboard interface using Firefox v.18 and I clicked on the submit button and validation didn’t work". Most bug trackers worry about how best to show you a list of issues, the thing is if most of those issues aren’t reported correctly they’re useless. We’re currently trying to resolve this problem for web devs specifically by creating an embedded tracker, where you integrate the bug reporting directly into a site or web app. Meaning people can report issues in the way that’s natural for them e.g. "this is broken" and the tracker handles capturing all the other data a dev needs to verify and fix the problem. Our app can be found at http://www.bugherd.com http://www.bugherd.com if it’s of any interest to anyone.

Zak 14y ago
A big part of the problem is social, not technical. A bug report is essentially a set of instructions for a person who knows how to operate the software, but not necessarily anything else about the user's situation to follow. Most people are not used to writing instructions without conversational feedback and have a hard time figuring out what information in relevant and what to leave out. For a high-volume end-user project like Chrome or Firefox, a wizard/flowchart type UI probably ought to be the default, with a more streamlined UI available to people who have been approved by admins or met some criteria programmed in to the system. An example of the process a user might go through is: What's the problem? -> A web page doesn't look right in Chrome -> Have you tried it in another browser? -> No -> What other browsers do you have? -> None -> Ok. Here's a list of browsers that run on your OS and where to get them -> Firefox -> Ok, does it look right in Firefox? -> Yes -> Do you know how to take a screenshot? And so on until the user has likely provided enough information for a good bug report. There could be short-circuits in the process - for example, when asking if the user has tried another browser, a small link a bit outside the main UI could say "I've tried another browser and I have screenshots from both".
- bruceboughton 14y ago
  I'm not sure that encouraging users to download another browser to try the page in would (a) encourage many bug reports (b) be a great idea for user retention.

ihaveajob 14y ago

I imagine by now someone at Google would have looked at the problem and written a basic bug clustering mechanism to group reports on the same issue. There's usually a very clear Pareto pattern when it comes to bugs, and fixing issues for 80% of users would be huge.

DeepDuh 14y ago

Sometimes adding humans is the best option? How about just hire some student workers to sweep through bug reports remotely? Even some Mechanical Turk approach might work, if you can create some checklist of what a useful bug report must have (and if you got this, it could also be integrated into the bug tracker workflow).

aidenn0 14y ago
That's hard to do because you end up with situations where the bug submitter figures out s/he knows more about the product than the person on the other end of the line, and stops responding out of frustration.
- DeepDuh 14y ago
  So? Just close those reports then. I thought the issue here is to filter out anything spammy while not loosing any potentially valuable reports. A human should be able to filter out at least 90% of the spammy stuff while not loosing any good ones, thus significantly reducing the workload of engineers in higher pay grades. Google sometimes give me the impression they want to absolutely only hire the best engineers, but I'd argue that many non automatable tasks can be done more efficiently by having some reasonably trained lower rank staff. See also all the complaints about missing service when trying to use Google products commercially - if they do want to seriously enter that market, this stuff needs to change significantly. Other things that come to mind: Automated Youtube takedowns and Gmail account blocks. Do you really want to save those few million bucks a year for not hiring support staff, then have lots of disgruntled users blogging about how evil you are and then pouring those millions into PR campaigns?
  - aidenn0 14y ago
    " I thought the issue here is to filter out anything spammy while not loosing any potentially valuable reports. " Yes! But, in my comment I pointed out a specific example of where you lose a potentially valuable report.

blablabla123 14y ago

I think for such projects it would make sense to seek help from volunteers. People who don't necessarily contribute to the code base, but who could help organize the many bugs. After all it's just a question of man power to remove duplicates, prioritize by commonness and make sure long-term bugs really get fixed. That latter has the advantage that the code base really gets improved. In fact the Ubuntu project seems to do something similar eventhough it is IMHO not well enough organized: there is ubuntuforums.org, when I recently asked a question there, within minutes I got a response. The response was not helpful at all but still. The moderators were really doing a lot to keep the forum organized, and for instanced pulled me into a separate thread to not spam another bug. (Which was in their opinion a different issue.) Taking this a step further and bridging it with the actual Ubuntu bug tracker on launchpad would be awesome. Maybe with launchpad having only limited write permissions for the public.