5 ms·
Ask HN: How do you measure bug report quality?
As software developers, some of us spend a lot of our time fixing defects reported by internal testers and QA teams. The quality of defect reports seems to significantly impact the time it takes for developers to diagnose and resolve issues. Providing defect report templates and examples have helped thus far, and good old conversation is useful too, but I'd like to take it a step further and use metrics and data-driven experimentation.
Does your organization measure bug report quality? If so, what metrics do you use and what recommendations do you have?
- jlg23 11y agoUhm, you gave the answer yourself? The only sane metric is "the time it takes for developers to diagnose and resolve issues".
- takmusashi 11y agoYeah, time to diagnose & resolve is definitely a key metric. It gets challenging when trying to tease out the benefit from a high quality bug report versus other factors such as how well the developer happens to know that part of the code base or how simple the required fix happens to be. Perhaps some statistics would be in order to somehow account for these other factors...
- DrScump 11y agoThat strikes me as an unfair metric. Really complex problems or intermittent behaviors exist; a reporter can do a highly diligent job of isolating and reproducing a bug whose cause is not easily diagnosable. That difficulty is hardly the fault of the reporter. If a bug reporter thoroughly documents a problem (and its attempted workarounds and behaviors) diligently and clearly, just be damn thankful that s/he took the time and made the effort.
- jlg23 11y ago> If a bug reporter thoroughly documents a problem (and its attempted workarounds and behaviors) diligently and clearly That is exactly the metric I am referring. Of course, some bugs take longer to solve than others, but I can only start when I am able to reproduce the bug. The less steps to reproduce them, the better. The more concise the bug report, the better. Once you can reproduce it, I consider it essentially fixed, its only up to the developer to wrap their head around it and actually implement the fix it. And on, I am not "damn thankful" for the effort alone; I had to fire testers for showing much effort but not being helpful at all, even after long periods of training. The last time I was awarded for effort alone was in pre-school. When you buy a car that breaks down every 100 miles you probably would return it and not watch the manufacturer try to fix it 20 times, lauding their "effort"? Good testers are worth their weight in gold, they are very rare, they save us developers insane amounts of time and thus the company we work for a lot of money.
- makecheck 11y agoA few guidelines: - Is the bug report focused (one issue), and not riddled with a list of unrelated problems? - Does it include full environment detail, relevant to the domain (e.g. hardware/OS, versions of the programs involved, environment variable settings, any recent changes, etc.)? - When reporting a crash, does it include all logs? - Is it professional and to-the-point, free of distracting and irrelevant statements like angry rants? - Does the submitter respond regularly, instead of treating bugs like black holes when follow-up questions are asked or test patches are released?
- takmusashi 11y agoThanks for the ideas and suggestions!