2 ms·

> Aviation is designed around having redundancies, tolerances, and contingencies for these small fuckups. Are there lessons for software engineering to be lear

> Aviation is designed around having redundancies, tolerances, and contingencies for these small fuckups. Are there lessons for software engineering to be learned from aviation?

sokoloff 5y ago
I think so. I’ve enshrined an adaptation of FAR 91.3 into our technical operations handbook (which is incorporated into our SOX compliance policies). The intention was to make sure that a specific group is clearly, unambiguously, and completely in charge. They are the final authority as to the production operation. (We call this group “Problem Management”, though in the ITIL framework this is more akin to critical incident management) “Problem Management is exercising emergency authority here” clears away any ambiguity and the confidence this gives them means they are free to act to resolve an incident rather than wonder who they need to call to give the blessing. https://www.law.cornell.edu/cfr/text/14/91.3 https://www.law.cornell.edu/cfr/text/14/91.3 When onboarding new members to that group, we go over this, including the origin story as part of helping them understand how (and that) we expect them to use this. In other areas? Probably. Aviation tries not to bet lives on a single piece of equipment working perfectly. Where that’s unavoidable (like the Jesus nut and pin on a helicopter), inspection and maintenance procedures are more stringent.
klodolph 5y ago
SREs I know talk about aviation incidents all the time, spinning it into a “what lessons do we learn?” - Aviation incidents are investigated and the investigation is typically bent on discovering root causes, not assigning blame. Do the same thing with bugs & security breaches, you’ll learn more. - Pilots spend time in simulators practicing unusual situations like water landings or engine failure—do the same thing in software, simulate failure scenarios and intrusion events and see how your engineers and technicians handle those events. You develop new training materials and redesign systems based on what you learned in simulations. - Pilots rely heavily on checklists rather than memory to ensure that all steps have been completed. This translates well to QA, deployment, various tasks people you need to do in production like migrations or whatnot. - Pilots require good UX to do their job and cannot be overloaded with too many tasks. Sometime an incident will be traced to dangerous things that don’t seem dangerous when you look at the UI (like thrust reversers), or traced to systems that assigned too many duties to pilots (who get overloaded). For example, if you have a button that deletes a database, it should require a serious confirmation step, like a molly guard on a plane. - Tasks must be explicitly delegated to individual people (which is what happens on an airplane). For example, in the Gimli Glider incident, the flight engineer role was eliminated and the engineer's responsibilities were not explicitly delegated to the pilot or copilot, but assigned to “both”. Same should be true for tasks in software engineering and incident response. We also talk about how medicine seems uninterested in learning these lessons, for various reasons.
marcosdumay 5y ago
For software reliability, many of them. A large number are mainstream. If you take a distributed systems book, you will see many, and test suites are an evolution of the aviation checklists. For security, I don't think there are any. Aviation isn't great on security anyway.

> Aviation is designed around having redundancies, tolerances, and contingencies for these small fuckups. Are there lessons for software engineering to be learned from aviation?

sokoloff 5y ago

I think so. I’ve enshrined an adaptation of FAR 91.3 into our technical operations handbook (which is incorporated into our SOX compliance policies). The intention was to make sure that a specific group is clearly, unambiguously, and completely in charge. They are the final authority as to the production operation. (We call this group “Problem Management”, though in the ITIL framework this is more akin to critical incident management) “Problem Management is exercising emergency authority here” clears away any ambiguity and the confidence this gives them means they are free to act to resolve an incident rather than wonder who they need to call to give the blessing. https://www.law.cornell.edu/cfr/text/14/91.3 https://www.law.cornell.edu/cfr/text/14/91.3 When onboarding new members to that group, we go over this, including the origin story as part of helping them understand how (and that) we expect them to use this. In other areas? Probably. Aviation tries not to bet lives on a single piece of equipment working perfectly. Where that’s unavoidable (like the Jesus nut and pin on a helicopter), inspection and maintenance procedures are more stringent.

klodolph 5y ago

SREs I know talk about aviation incidents all the time, spinning it into a “what lessons do we learn?” - Aviation incidents are investigated and the investigation is typically bent on discovering root causes, not assigning blame. Do the same thing with bugs & security breaches, you’ll learn more. - Pilots spend time in simulators practicing unusual situations like water landings or engine failure—do the same thing in software, simulate failure scenarios and intrusion events and see how your engineers and technicians handle those events. You develop new training materials and redesign systems based on what you learned in simulations. - Pilots rely heavily on checklists rather than memory to ensure that all steps have been completed. This translates well to QA, deployment, various tasks people you need to do in production like migrations or whatnot. - Pilots require good UX to do their job and cannot be overloaded with too many tasks. Sometime an incident will be traced to dangerous things that don’t seem dangerous when you look at the UI (like thrust reversers), or traced to systems that assigned too many duties to pilots (who get overloaded). For example, if you have a button that deletes a database, it should require a serious confirmation step, like a molly guard on a plane. - Tasks must be explicitly delegated to individual people (which is what happens on an airplane). For example, in the Gimli Glider incident, the flight engineer role was eliminated and the engineer's responsibilities were not explicitly delegated to the pilot or copilot, but assigned to “both”. Same should be true for tasks in software engineering and incident response. We also talk about how medicine seems uninterested in learning these lessons, for various reasons.

marcosdumay 5y ago

For software reliability, many of them. A large number are mainstream. If you take a distributed systems book, you will see many, and test suites are an evolution of the aviation checklists. For security, I don't think there are any. Aviation isn't great on security anyway.