7 ms·

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident inves

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a huge improvement in safety across the whole industry. In software there is still a certain arrogance of quickly calling the user (or other software professional) stupid, thinking it can't happen to you. But in reality given enough time, everyone makes at least one stupid mistake, it's how humans work.

suzakus 6y ago
It's a piece of software for scoreboards. Not the Therac-25, nor an airplane.
- t0mas88 6y ago
  It isn't about the importance of this one database, it's about the cultural issue in most of the sector that the parent comment was pointing out: we far too often blame the user/operator calling them stupid, while every human makes mistake, it's inevitable.
  - dmitriid 6y ago
    The Russian joke about investigating is "Punish the innocent, award the uninvolved"
- qz2 6y ago
  This time. Some days it’s just an on line community that gets burned to the ground. Other days it’s just a service tied into hundred of small businesses that gets burned to the ground. Other says it’s massive financial platform getting burned to the ground. I’m responsible for the latter but the former two have had a much larger impact for many people when they occur. Trivialising the lax administrative discipline because a product isn’t deemed important is a slippery slope. We need to start building competence in to what we do regardless of what it is rather than run on apologies because it’s cheaper.
  - wizzwizz4 6y ago
    Prismo is an example of the first: https://fediverse.blog/~/Prismo/on-prismo-data-loss https://fediverse.blog/~/Prismo/on-prismo-data-loss The project never recovered.
- ozim 6y ago
  Parent is not advocating about going as strict with procedures as operating an airplane. Post is saying about "a bit of aviation safety culture" then it highlights a specific part that would be useful. Safety culture element highlighted is: not blaming a single person but finding out how to prevent accident that happened from happening again. Which is reasonable because you don't want to impose some strict rules that are expensive up front. This way you just introduce measure to prevent same thing in the future, in the context of your project.
  - suzakus 6y ago
    Ah, I misread. That's what I get for commenting late at night :( Thanks for clarifying!
watwut 6y ago
Yeah, but "pilot was drinking alcohol" would be considerate issue, would lead to fired pilot and would lead to more alcohol testing. I understand what you are taking about, but aviation has also strong expectations on pilots.
- janoc 6y ago
  Of course it would. But then there should be a process that identifies such pilot before they even get on the plane, there are two crew in the cockpit, so if one crewman does something unsafe or inappropriate, the other person is there to notice it, call it out and, in the extreme case, to take control of the plane. Also, if the guy or gal has alcohol problems, it would likely be visible on their flying performance over time, it should be noticed during the periodic medicals, etc. So while a drunk pilot could be the immediate cause of a crash, it is not the only one. If any of those other things I have mentioned functioned as designed (or were in place to start with - not all flying is airline flying!), the accident wouldn't have happened. If you focus only on the "drunk pilot, case closed", you will never identify deficiencies you may have elsewhere and which have contributed to the problem.
  - watwut 6y ago
    Note how none of those is in article. Article is like "Could we blame alcohol? Oh no, surely not.".
- t0mas88 6y ago
  Believe it or not, even "pilot is an alcoholic" is still part of the no blame culture in aviation. As long as the pilot reports himself he'll not be fired for that. Look up the HIMS program to read more details.
  - deleted 6y ago
    [deleted]
  - user5994461 6y ago
    You can google and find cases of US pilots getting fired and sentenced to a year in prison for flying intoxicated. Maybe they don't get fired if they report themselves unable to fly beforehand but I wouldn't quite call that a no blame culture.
neillyons 6y ago
This sounds quite interesting. Any books you could recommend on the "pilot error" topic.
- janoc 6y ago
  Not sure about books but look up the Swiss cheese model. It is widely used approach and not only in aviation. Most industrial accidents and incidents are investigated with this in mind.
- permarad 6y ago
  The Design of Everyday Things by Donald A. Norman. He covers pilot error a lot in this book in how it falls back on design and usability. Very interesting read.
- miketery 6y ago
  When I was getting my pilots license I used to read accident reports from Canada's Transportation Safety Board [1]. I'm sure the NTSB (America's version) has similar calibre reports [2]. There is also Cockpit Resource Management [3] which addresses the human factor in great detail (how people work with each other, and how prepared are people). In general what you learn from reading these things is that its rarely one big error or issue - but many small things leading to the failure event. 1 - https://www.tsb.gc.ca/eng/rapports-reports/aviation/index.html https://www.tsb.gc.ca/eng/rapports-reports/aviation/index.ht... 2 - https://www.ntsb.gov/investigations/AccidentReports/Pages/AccidentReports.aspx https://www.ntsb.gov/investigations/AccidentReports/Pages/Ac... 3 - https://en.wikipedia.org/wiki/Crew_resource_management https://en.wikipedia.org/wiki/Crew_resource_management
  - masklinn 6y ago
    The old "they write the right stuff" essay on the On-Board Shuttle Group also talked about this mindset of errors getting through the process as being first and foremost a problem with the process to be examined in detail and fixed.
- benkelaar 6y ago
  This is also one of the core tenets of SRE. The chapter on blameless postmortems is quite nice: https://landing.google.com/sre/sre-book/chapters/postmortem-culture/ https://landing.google.com/sre/sre-book/chapters/postmortem-...
janoc 6y ago
It is not only that but also realizing that there is never a single cause to an accident or incident. Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc. So even when the accident is ultimately caused by a pilot's actions, there is always a chain of events where if any of the segments were broken the accident wouldn't have happened. While we can't prevent a bonkers pilot from crashing a plane, we could perhaps prevent a bonkers crew member from flying the plane in the first place. Aka the Swiss cheese model. You don't want to let the holes to align. This approach is widely used in accident investigations and not only in aviation. Most industrial accidents are investigated like this, trying to understand the entire chain of events in order that processes could be improved and the problem prevented in the future. Oh and there is one more key part in aviation that isn't elsewhere. The goal of an accident or incident investigation IS NOT TO APPORTION BLAME. It is to learn from it. That's why pilots in airlines with a healthy safety culture are encouraged to report problems, unsafe practices, etc. and this is used to fix the process instead of firing people. Once you start to play the blame game, people won't report problems - and you are flying blind into a disaster sooner or later.
- globular-toast 6y ago
  There is sometimes a single cause, but as the parent comment pointed out, that should never be the case and is a flaw in the system. We are gradually working towards single errors being correctable, but we're not there yet. On the railways in Britain the failures were extensively documented. Years ago it was possible for a single failure to cause a loss. But over the years the systems have been patched and if you look at more recent incidents it is always a multitude of factors aligning that cause the loss. Sometimes it's amazing how precisely these individual elements have to align, but it's just probability. As demonstrated by the article here, we are still in the stage where single failures can cause a loss. But it's a bit different because there is no single universal body regulating every computer system.
  - janoc 6y ago
oblio 6y ago
The user? Start a discussion about using better programming language and you'll see people, even here, blaming the developer. The common example is C: "C is a sharp tool, but with a sufficiently smart, careful and experienced developer it does what you want (you're holding it wrong"). Developers still do this to each other.
m463 6y ago
That reminds me of the time during the rise of the PC when windows would do something wrong, from a confusing interface all the way up to a blue screen of death. What happened is that users started blaming themselves for what was going wrong, or start thinking they needed a new PC because problems would become more frequent. From the perspective of a software guy, it was obvious that windows was the culprit but people would assign blame elsewhere and frequently point the finger at themselves. so yes - an FAA investigation would end up unraveling the nonsense and point to windows. That said, aviation level of safety is reliable and dependable and few single points of failure and... there are no private kit jets darnit! There is a continuum from nothing changes & everything works to everything changes & nothing works. You have to choose the appropriate place on the dial for the task. Sounds like this is a one-man band.
ENOTTY 6y ago
The idea that multiple failures must occur for catastrophic failure is found in certain parts of the computing community. https://en.wikipedia.org/wiki/Defense_in_depth_(computing) https://en.wikipedia.org/wiki/Defense_in_depth_(computing)

suzakus 6y ago

It's a piece of software for scoreboards. Not the Therac-25, nor an airplane.

t0mas88 6y ago
It isn't about the importance of this one database, it's about the cultural issue in most of the sector that the parent comment was pointing out: we far too often blame the user/operator calling them stupid, while every human makes mistake, it's inevitable.
- dmitriid 6y ago
  The Russian joke about investigating is "Punish the innocent, award the uninvolved"
qz2 6y ago
This time. Some days it’s just an on line community that gets burned to the ground. Other days it’s just a service tied into hundred of small businesses that gets burned to the ground. Other says it’s massive financial platform getting burned to the ground. I’m responsible for the latter but the former two have had a much larger impact for many people when they occur. Trivialising the lax administrative discipline because a product isn’t deemed important is a slippery slope. We need to start building competence in to what we do regardless of what it is rather than run on apologies because it’s cheaper.
- wizzwizz4 6y ago
  Prismo is an example of the first: https://fediverse.blog/~/Prismo/on-prismo-data-loss https://fediverse.blog/~/Prismo/on-prismo-data-loss The project never recovered.
ozim 6y ago
Parent is not advocating about going as strict with procedures as operating an airplane. Post is saying about "a bit of aviation safety culture" then it highlights a specific part that would be useful. Safety culture element highlighted is: not blaming a single person but finding out how to prevent accident that happened from happening again. Which is reasonable because you don't want to impose some strict rules that are expensive up front. This way you just introduce measure to prevent same thing in the future, in the context of your project.
- suzakus 6y ago
  Ah, I misread. That's what I get for commenting late at night :( Thanks for clarifying!

watwut 6y ago

Yeah, but "pilot was drinking alcohol" would be considerate issue, would lead to fired pilot and would lead to more alcohol testing. I understand what you are taking about, but aviation has also strong expectations on pilots.

janoc 6y ago
Of course it would. But then there should be a process that identifies such pilot before they even get on the plane, there are two crew in the cockpit, so if one crewman does something unsafe or inappropriate, the other person is there to notice it, call it out and, in the extreme case, to take control of the plane. Also, if the guy or gal has alcohol problems, it would likely be visible on their flying performance over time, it should be noticed during the periodic medicals, etc. So while a drunk pilot could be the immediate cause of a crash, it is not the only one. If any of those other things I have mentioned functioned as designed (or were in place to start with - not all flying is airline flying!), the accident wouldn't have happened. If you focus only on the "drunk pilot, case closed", you will never identify deficiencies you may have elsewhere and which have contributed to the problem.
- watwut 6y ago
  Note how none of those is in article. Article is like "Could we blame alcohol? Oh no, surely not.".
t0mas88 6y ago
Believe it or not, even "pilot is an alcoholic" is still part of the no blame culture in aviation. As long as the pilot reports himself he'll not be fired for that. Look up the HIMS program to read more details.
- deleted 6y ago
  [deleted]
- user5994461 6y ago
  You can google and find cases of US pilots getting fired and sentenced to a year in prison for flying intoxicated. Maybe they don't get fired if they report themselves unable to fly beforehand but I wouldn't quite call that a no blame culture.

neillyons 6y ago

This sounds quite interesting. Any books you could recommend on the "pilot error" topic.

janoc 6y ago
Not sure about books but look up the Swiss cheese model. It is widely used approach and not only in aviation. Most industrial accidents and incidents are investigated with this in mind.
permarad 6y ago
The Design of Everyday Things by Donald A. Norman. He covers pilot error a lot in this book in how it falls back on design and usability. Very interesting read.
miketery 6y ago
When I was getting my pilots license I used to read accident reports from Canada's Transportation Safety Board [1]. I'm sure the NTSB (America's version) has similar calibre reports [2]. There is also Cockpit Resource Management [3] which addresses the human factor in great detail (how people work with each other, and how prepared are people). In general what you learn from reading these things is that its rarely one big error or issue - but many small things leading to the failure event. 1 - https://www.tsb.gc.ca/eng/rapports-reports/aviation/index.html https://www.tsb.gc.ca/eng/rapports-reports/aviation/index.ht... 2 - https://www.ntsb.gov/investigations/AccidentReports/Pages/AccidentReports.aspx https://www.ntsb.gov/investigations/AccidentReports/Pages/Ac... 3 - https://en.wikipedia.org/wiki/Crew_resource_management https://en.wikipedia.org/wiki/Crew_resource_management
- masklinn 6y ago
  The old "they write the right stuff" essay on the On-Board Shuttle Group also talked about this mindset of errors getting through the process as being first and foremost a problem with the process to be examined in detail and fixed.
benkelaar 6y ago
This is also one of the core tenets of SRE. The chapter on blameless postmortems is quite nice: https://landing.google.com/sre/sre-book/chapters/postmortem-culture/ https://landing.google.com/sre/sre-book/chapters/postmortem-...

janoc 6y ago

It is not only that but also realizing that there is never a single cause to an accident or incident. Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc. So even when the accident is ultimately caused by a pilot's actions, there is always a chain of events where if any of the segments were broken the accident wouldn't have happened. While we can't prevent a bonkers pilot from crashing a plane, we could perhaps prevent a bonkers crew member from flying the plane in the first place. Aka the Swiss cheese model. You don't want to let the holes to align. This approach is widely used in accident investigations and not only in aviation. Most industrial accidents are investigated like this, trying to understand the entire chain of events in order that processes could be improved and the problem prevented in the future. Oh and there is one more key part in aviation that isn't elsewhere. The goal of an accident or incident investigation IS NOT TO APPORTION BLAME. It is to learn from it. That's why pilots in airlines with a healthy safety culture are encouraged to report problems, unsafe practices, etc. and this is used to fix the process instead of firing people. Once you start to play the blame game, people won't report problems - and you are flying blind into a disaster sooner or later.

globular-toast 6y ago
There is sometimes a single cause, but as the parent comment pointed out, that should never be the case and is a flaw in the system. We are gradually working towards single errors being correctable, but we're not there yet. On the railways in Britain the failures were extensively documented. Years ago it was possible for a single failure to cause a loss. But over the years the systems have been patched and if you look at more recent incidents it is always a multitude of factors aligning that cause the loss. Sometimes it's amazing how precisely these individual elements have to align, but it's just probability. As demonstrated by the article here, we are still in the stage where single failures can cause a loss. But it's a bit different because there is no single universal body regulating every computer system.
- janoc 6y ago

oblio 6y ago

The user? Start a discussion about using better programming language and you'll see people, even here, blaming the developer. The common example is C: "C is a sharp tool, but with a sufficiently smart, careful and experienced developer it does what you want (you're holding it wrong"). Developers still do this to each other.

m463 6y ago

That reminds me of the time during the rise of the PC when windows would do something wrong, from a confusing interface all the way up to a blue screen of death. What happened is that users started blaming themselves for what was going wrong, or start thinking they needed a new PC because problems would become more frequent. From the perspective of a software guy, it was obvious that windows was the culprit but people would assign blame elsewhere and frequently point the finger at themselves. so yes - an FAA investigation would end up unraveling the nonsense and point to windows. That said, aviation level of safety is reliable and dependable and few single points of failure and... there are no private kit jets darnit! There is a continuum from nothing changes & everything works to everything changes & nothing works. You have to choose the appropriate place on the dial for the task. Sounds like this is a one-man band.

ENOTTY 6y ago

The idea that multiple failures must occur for catastrophic failure is found in certain parts of the computing community. https://en.wikipedia.org/wiki/Defense_in_depth_(computing) https://en.wikipedia.org/wiki/Defense_in_depth_(computing)