3 ms·
A significant issue with this is simply designing a study or a way of measuring. First of all, define "bug". Is that a ticket filed in a system? What about the
by gregmac 5y ago
A significant issue with this is simply designing a study or a way of measuring.
First of all, define "bug". Is that a ticket filed in a system? What about the "bug" that was found and fixed by the developer testing it a few minutes after finishing the new feature (and thus never had a ticket filed)? What about QA finding a problem with a new feature while testing it (and sending the feature ticket back to the developer)?
Consider: Filing a bug ticket itself takes a bunch of time. If you find and fix a bug early enough then even the overhead time of filing/verifying/closing the ticket isn't spent. How do you measure the time compared to a bug that effectively didn't exist?
You can't do any like-for-like tests: two developers could take significantly different amounts of time to fix the same bug. If essentially the same bug occurs again in another area of code, the fix still can't be compared: if the developer applies essentially the same fix, it'll be much faster than the first time; if they apply a more systematic fix like refactoring the code to avoid that type of bug, that's not comparable.
There's also some bias that will make production bugs take longer. A team that has good quality controls (unit and automated testing, QA team, staging environments) will likely catch a lot of bugs before they get to production. As a result, the bugs that do end up in production are more likely to be complicated ones -- such as requiring several obscure conditions to be met, or only happening at large scale -- and those naturally take more time to address, even if in some cases the time is just spent figuring out the reproduction steps or replicating the environment.
This all makes comparing overall trends pretty difficult -- maybe to the point of being entirely useless.