5 ms·

The top answer has it exactly right. This should be Root Cause, since ultimately even bugs caused by human errors are really caused by systemic flaws. Some exam

The top answer has it exactly right. This should be Root Cause, since ultimately even bugs caused by human errors are really caused by systemic flaws. Some examples: * SQLite has 1177 times as many tests as code (not a typo) * Live television is broadcast with a 5-10 second delay * IMVU automatically reverts commits pushed to the site if regressions are detected * Netflix implemented a system called the Chaos Monkey, which randomly shuts down processes. Ensures the system can survive any failure * VLC, Unity3D, Windows, 3D Studio Max and many more applications phone home crashes, which allows developers to quickly patch frequent issues * Code reviews and pair programming ensure no one person's mistake can break critical code sections * Similarly, multiple people should sign off on copy written. For newsletters and press releases the whole team should, since they can't be withdrawn * Well designed systems automatically backup, and those backups are automatically tested, so nothing is deleted forever Change "who" to "why" and a horrible idea turns into a brilliant one. Well designed systems can reduce the risk of almost any mistake, at the cost of speed and flexibility. Ultimately it's up to the company to decide where the balance lies, and to live with the consequences. Startups will accept a drastically different risk profile to banks and Fortune 500 companies.

SoftwareMaven 14y ago
I'm usually arguing against stereotyping management around here, but I find it a stretch to believe a boss who thinks "Person to Blame" is going to be willing to pay the cost for true root cause analysis. I have a feeling it's more like: <git annotate>; "Oh, it's Bob's fault."
- gte910h 14y ago
  If you tell me to blame someone every time there is an issue, and you don't let me find out why, I'm going to blame you. Because it's your fault as the boss of the department till you let us prove specifically otherwise. That's what responsibility is.
- reitzensteinm 14y ago
  I completely agree with that - the boss in the story sounds like a lost cause. My post was more of a speaking in general kind of thing. I was going to go off on a tangent about the importance of good management, but I haven't ever had a boss (self employed then startups), so by missing half the picture I didn't really feel qualified.
- kls 14y ago
  We actually had a tradition at one of the places I worked at, where someone along the way brought in a trophy and the title was changed to something along the lines of best bug write. Anytime a bug broke the system the trophy was passed to the person that broke the system. It was actually light hearted and effective. We would make a big deal of passing the trophy, but no one wanted that trophy. It was not punitive, but more of a joke, kind of a simple reminder that I don't want to be the guy with the trophy on his desk.
  - deleted 14y ago
    [deleted]
ww520 14y ago
Changing who to why is brilliant. That encourages investigation and understanding of the problem.
- krishnakv 14y ago
  +1 on "root cause" being the field rather than "person to blame". "person to blame" pre-supposes that the root cause is a person, which is not true in most cases. While programmers do introduce bugs due to oversight, I have seen that systemic issues (build environment doesn't allow for tests to be written, missing patches in legacy architecture, bad staffing decisions etc...) introduce a large percentage of defects, especially in larger projects. If your manager is really interested in reducing defect injection, you will want to champion root cause analysis (read up on it - Karl Wiegers and others have some some great work on this).
sseveran 14y ago
While the why is important the who is important as well. In any organization of sufficient size there are going to be B grade developers, or even just young developers. It is important to understand if certain people are writing most of the major bugs. Now there might be a good reason for it such as the difficulty of certain pieces of the application, so bug density is not the only thing to look at. However knowing who is writing bugs that cause outages is very important for proper accountability.
- gerts 14y ago
  yes, who wrote the bug ...and who reviewed the bad design ...and who reviewed the bad code ...and who wrote the bad tests ...and who built the spaghetti system that can't isolate failures ...and who pushed new code to production without smoke-testing ...and who demanded the product ship before it was ready
  - zithtar 14y ago
    I'll agree on everything except that last part. When is the last time in any business you've seen or even heard of non-technical management (including clients) accepting blame for a technical malfunction?
mleonhard 14y ago
* Well designed systems automatically backup, and those backups are automatically tested, so nothing is deleted forever How much software automatically backs itself up? I'm solving this problem with my startup, http://www.restbackup.com/ http://www.restbackup.com/ . I provide online backups that software makers can bundle and sell to users.

SoftwareMaven 14y ago

I'm usually arguing against stereotyping management around here, but I find it a stretch to believe a boss who thinks "Person to Blame" is going to be willing to pay the cost for true root cause analysis. I have a feeling it's more like: <git annotate>; "Oh, it's Bob's fault."

gte910h 14y ago
If you tell me to blame someone every time there is an issue, and you don't let me find out why, I'm going to blame you. Because it's your fault as the boss of the department till you let us prove specifically otherwise. That's what responsibility is.
reitzensteinm 14y ago
I completely agree with that - the boss in the story sounds like a lost cause. My post was more of a speaking in general kind of thing. I was going to go off on a tangent about the importance of good management, but I haven't ever had a boss (self employed then startups), so by missing half the picture I didn't really feel qualified.
kls 14y ago
We actually had a tradition at one of the places I worked at, where someone along the way brought in a trophy and the title was changed to something along the lines of best bug write. Anytime a bug broke the system the trophy was passed to the person that broke the system. It was actually light hearted and effective. We would make a big deal of passing the trophy, but no one wanted that trophy. It was not punitive, but more of a joke, kind of a simple reminder that I don't want to be the guy with the trophy on his desk.
- deleted 14y ago
  [deleted]

ww520 14y ago

Changing who to why is brilliant. That encourages investigation and understanding of the problem.

krishnakv 14y ago
+1 on "root cause" being the field rather than "person to blame". "person to blame" pre-supposes that the root cause is a person, which is not true in most cases. While programmers do introduce bugs due to oversight, I have seen that systemic issues (build environment doesn't allow for tests to be written, missing patches in legacy architecture, bad staffing decisions etc...) introduce a large percentage of defects, especially in larger projects. If your manager is really interested in reducing defect injection, you will want to champion root cause analysis (read up on it - Karl Wiegers and others have some some great work on this).

sseveran 14y ago

While the why is important the who is important as well. In any organization of sufficient size there are going to be B grade developers, or even just young developers. It is important to understand if certain people are writing most of the major bugs. Now there might be a good reason for it such as the difficulty of certain pieces of the application, so bug density is not the only thing to look at. However knowing who is writing bugs that cause outages is very important for proper accountability.

gerts 14y ago
yes, who wrote the bug ...and who reviewed the bad design ...and who reviewed the bad code ...and who wrote the bad tests ...and who built the spaghetti system that can't isolate failures ...and who pushed new code to production without smoke-testing ...and who demanded the product ship before it was ready
- zithtar 14y ago
  I'll agree on everything except that last part. When is the last time in any business you've seen or even heard of non-technical management (including clients) accepting blame for a technical malfunction?

mleonhard 14y ago

* Well designed systems automatically backup, and those backups are automatically tested, so nothing is deleted forever How much software automatically backs itself up? I'm solving this problem with my startup, http://www.restbackup.com/ http://www.restbackup.com/ . I provide online backups that software makers can bundle and sell to users.