7 ms·

Software Engineering Lessons from Aviation

98 points by riceo100 7y ago

cjbprime 7y ago
This was great! > 1. Don’t kill yourself > 2. Don’t kill anyone else Could we reorder these, though? Every once in a while a plane will hit a house and kill its occupants (and the pilot, usually) and it's so awful. I think not killing others as a pilot is so much more important than not killing yourself.
- jacobush 7y ago
  Well, if you kill yourself the plane may then go on killing others. This is double plus ungood. https://en.wikipedia.org/wiki/Helios_Airways_Flight_522 https://en.wikipedia.org/wiki/Helios_Airways_Flight_522
- maxxxxx 7y ago
  I think the idea is that if the pilot dies then most likely passengers or others will die too. Someone needs to control the plane.
- X6S1x6Okd1st 7y ago
  That ordering reminds me of the first rule of search and rescue: don't create another victim. If your job is to save a life and that life depends on you, you don't do anyone any favors if you die
jefffoster 7y ago
I think there's a lot to learn from the aviation industry. I did a talk at my companies internal conference on this (turned into words at https://medium.com/ingeniouslysimple/why-dont-planes-crash-14a0579a5e2d https://medium.com/ingeniouslysimple/why-dont-planes-crash-1...). For me it's the mindset that differs. Too often as software engineers we find a bug and just fix it. Aviation goes a step deeper and finds the environment that created the bug and stops that. Unfortunately, the recent 737 MAX incidents seem to have changed this. From what I understand the reaction to the problems sounds more like what I'd expect a software business to do, rather than the airline industry!
- 0x445442 7y ago
  The blog post was good but it was just another variation highlighting the age old conundrum... fast, good and cheap, pick two. Those that value quality are going to be swimming up stream in most organization that develop software because the bean counters always go straight to fast and cheap.
  - maxxxxx 7y ago
    I work in regulated industry and you really don’t want the level of scrutiny regulated processes have in other industries. Innovation would slow down to a crawl or pretty much stop. In a lot of industries you can make trade offs quality vs speed or innovation and be better off by not having perfect quality.
    - retiredcoder 7y ago
      I strongly agree. However to my experience companies decide on quality vs speed based on other factors rather than business needs. For example, in my previous job, CTO called “ overengineering” whenever something went against his will, while “innovation/we are not a Corp” was his tail wind. So much office politics for such a small company :/
    - mcguire 7y ago
      ...until you start injuring people or costing significant numbers of zeros.
- ellius 7y ago
  After fixing a recent bug, I asked my client company what if any postmortem process they had. I informally noted about 8 factors that had driven the resolution time to ~8 hours from what probably could have been 1 or 2. Some of them were things we had no control over, but a good 4-5 were things in the application team's immediate control or within its orbit. These are issues that will definitely recur in troubleshooting future bugs, and doing a proper postmortem could easily save 250+ man hours over the course of a year. What's more, fixing some of these issues would also aid in application development. So you're looking at immediate cost savings and improved development speed just by doing a napkin postmortem on a simple bug. I can't imagine how much more efficient an organization with an ingrained and professional postmortem culture would be.
billfruit 7y ago
Though article isn't about software development in the aviation industry, a few thoughts on that: The industry is really slow to change its practices and tools. Like the use of C for most software, I do feel a more safer language out to be preferred. Use of 1553 bus for inter device communication, the bus and protocol aren't general, it is very opinionated/rigid about the manner in which communication should happen. And the hardware parts for it are horrendously expensive compared to most ethernet, IP equipment. There is an aviation ethernet standard, but adoption of it has been slow.
- HeyLaughingBoy 7y ago
  it is very opinionated/rigid about the manner in which communication should happen This could be a strong factor in its popularity. If things must happen in a certain order, then the behavior of the system becomes easier to verify. Ease of verification should never be understated in safety-critical systems.
  - billfruit 7y ago
    Yet the industry uses largely the C language, which isn't a model for safety or ease of verification.
    - HeyLaughingBoy 7y ago
      Well, nobody's perfect ;-)
    - magduf 7y ago
      It is, compared to other languages, because it's simple and deterministic. The #1 most important thing with avionics systems and software is determinism. That's why they even disable CPU caches on avionics systems.
- magduf 7y ago
  >The industry is really slow to change its practices and tools. Like the use of C for most software, I do feel a more safer language out to be preferred. And what language would that be, where it has absolute determinism (which rules out anything with GC)? They tried using Ada years ago for avionics. The problem here is that no one knows Ada any more, and no one really wants to make a career out of it since it isn't used anywhere else. So, in practice, C and (a narrow subset of) C++ get used. Maybe Rust would be a good choice in the future.
myl 7y ago
"...plenty of episodes of Mayday/Air Crash Investigation available on Youtube too. (Be warned though, all doomed flights take off from one of the busiest airports in the world .)" Great show. Comment is spot on, and don't forget "investigators were under extreme pressure".
sn 7y ago
Checklists and written procedures are very important. One of the earlier things I did when coming into my company was create a written procedure for software upgrades until we had time to automate it with ansible. One thing I have not had very good discipline about is I want to use checklists both for code submitted for review and when I'm doing reviews. Lint checkers etc. can only go so far. If anyone has published checklists for code reviews I'd be curious to see them. This one seems reasonable: https://www.liberty.edu/media/1414/%5B6401%5Dcode_review_checklist.pdf https://www.liberty.edu/media/1414/%5B6401%5Dcode_review_che... though I'd add concurrency to the list.
starpilot 7y ago
Not killing yourself and a checklist (like we learned in driver's ed but apply informally at best) also apply to driving a car.
- bdamm 7y ago
  Uh, no. In a car, if things go badly you pull off the road and work on a solution. If things to really badly, you have seatbelts, airbags, crumple zones, and a thick frame to help you out. In an airplane, if things to badly, you keep flying until you land. If things go really badly, remember that everything is built to be light weight, and unless the crash is well controlled, everything will be destroyed and everyone will die. If your engine quits, your cabin ruptures, your instrumentation fails, you keep flying. And you need instruments; in poor visibility, your own sensory inputs are in fact faulty, and won't help you figure out which way is down. Unlike in a car, where it's pretty obvious where the ground is, for example.
shamino 7y ago
Nathan Marz talks about this previously, with unique insights: http://nathanmarz.com/blog/how-becoming-a-pilot-made-me-a-better-programmer.html http://nathanmarz.com/blog/how-becoming-a-pilot-made-me-a-be...
skookumchuck 7y ago
The article talks about pilot procedures, not engineering procedures specific to aviation.
marcosdumay 7y ago
I still hold my opinion that checklists are for hardware issues. One should not be filling them on software tasks. Instead, software is automated, automatically tested and automatically verified - routine checks are an anti-feature and inversely correlated to quality.
horacio_colbert 7y ago
Thinking of aviation makes me remember the impact of doing things right.

98 points by riceo100 7y ago

This was great! > 1. Don’t kill yourself > 2. Don’t kill anyone else Could we reorder these, though? Every once in a while a plane will hit a house and kill its occupants (and the pilot, usually) and it's so awful. I think not killing others as a pilot is so much more important than not killing yourself.

jacobush 7y ago
Well, if you kill yourself the plane may then go on killing others. This is double plus ungood. https://en.wikipedia.org/wiki/Helios_Airways_Flight_522 https://en.wikipedia.org/wiki/Helios_Airways_Flight_522
maxxxxx 7y ago
I think the idea is that if the pilot dies then most likely passengers or others will die too. Someone needs to control the plane.
X6S1x6Okd1st 7y ago
That ordering reminds me of the first rule of search and rescue: don't create another victim. If your job is to save a life and that life depends on you, you don't do anyone any favors if you die

jefffoster 7y ago

I think there's a lot to learn from the aviation industry. I did a talk at my companies internal conference on this (turned into words at https://medium.com/ingeniouslysimple/why-dont-planes-crash-14a0579a5e2d https://medium.com/ingeniouslysimple/why-dont-planes-crash-1...). For me it's the mindset that differs. Too often as software engineers we find a bug and just fix it. Aviation goes a step deeper and finds the environment that created the bug and stops that. Unfortunately, the recent 737 MAX incidents seem to have changed this. From what I understand the reaction to the problems sounds more like what I'd expect a software business to do, rather than the airline industry!

0x445442 7y ago
The blog post was good but it was just another variation highlighting the age old conundrum... fast, good and cheap, pick two. Those that value quality are going to be swimming up stream in most organization that develop software because the bean counters always go straight to fast and cheap.
- maxxxxx 7y ago
  I work in regulated industry and you really don’t want the level of scrutiny regulated processes have in other industries. Innovation would slow down to a crawl or pretty much stop. In a lot of industries you can make trade offs quality vs speed or innovation and be better off by not having perfect quality.
  - retiredcoder 7y ago
    I strongly agree. However to my experience companies decide on quality vs speed based on other factors rather than business needs. For example, in my previous job, CTO called “ overengineering” whenever something went against his will, while “innovation/we are not a Corp” was his tail wind. So much office politics for such a small company :/
  - mcguire 7y ago
    ...until you start injuring people or costing significant numbers of zeros.
ellius 7y ago
After fixing a recent bug, I asked my client company what if any postmortem process they had. I informally noted about 8 factors that had driven the resolution time to ~8 hours from what probably could have been 1 or 2. Some of them were things we had no control over, but a good 4-5 were things in the application team's immediate control or within its orbit. These are issues that will definitely recur in troubleshooting future bugs, and doing a proper postmortem could easily save 250+ man hours over the course of a year. What's more, fixing some of these issues would also aid in application development. So you're looking at immediate cost savings and improved development speed just by doing a napkin postmortem on a simple bug. I can't imagine how much more efficient an organization with an ingrained and professional postmortem culture would be.

billfruit 7y ago

Though article isn't about software development in the aviation industry, a few thoughts on that: The industry is really slow to change its practices and tools. Like the use of C for most software, I do feel a more safer language out to be preferred. Use of 1553 bus for inter device communication, the bus and protocol aren't general, it is very opinionated/rigid about the manner in which communication should happen. And the hardware parts for it are horrendously expensive compared to most ethernet, IP equipment. There is an aviation ethernet standard, but adoption of it has been slow.

HeyLaughingBoy 7y ago
it is very opinionated/rigid about the manner in which communication should happen This could be a strong factor in its popularity. If things must happen in a certain order, then the behavior of the system becomes easier to verify. Ease of verification should never be understated in safety-critical systems.
- billfruit 7y ago
  Yet the industry uses largely the C language, which isn't a model for safety or ease of verification.
  - HeyLaughingBoy 7y ago
    Well, nobody's perfect ;-)
  - magduf 7y ago
    It is, compared to other languages, because it's simple and deterministic. The #1 most important thing with avionics systems and software is determinism. That's why they even disable CPU caches on avionics systems.
magduf 7y ago
>The industry is really slow to change its practices and tools. Like the use of C for most software, I do feel a more safer language out to be preferred. And what language would that be, where it has absolute determinism (which rules out anything with GC)? They tried using Ada years ago for avionics. The problem here is that no one knows Ada any more, and no one really wants to make a career out of it since it isn't used anywhere else. So, in practice, C and (a narrow subset of) C++ get used. Maybe Rust would be a good choice in the future.

myl 7y ago

"...plenty of episodes of Mayday/Air Crash Investigation available on Youtube too. (Be warned though, all doomed flights take off from one of the busiest airports in the world .)" Great show. Comment is spot on, and don't forget "investigators were under extreme pressure".

sn 7y ago

Checklists and written procedures are very important. One of the earlier things I did when coming into my company was create a written procedure for software upgrades until we had time to automate it with ansible. One thing I have not had very good discipline about is I want to use checklists both for code submitted for review and when I'm doing reviews. Lint checkers etc. can only go so far. If anyone has published checklists for code reviews I'd be curious to see them. This one seems reasonable: https://www.liberty.edu/media/1414/%5B6401%5Dcode_review_checklist.pdf https://www.liberty.edu/media/1414/%5B6401%5Dcode_review_che... though I'd add concurrency to the list.

starpilot 7y ago

Not killing yourself and a checklist (like we learned in driver's ed but apply informally at best) also apply to driving a car.

bdamm 7y ago
Uh, no. In a car, if things go badly you pull off the road and work on a solution. If things to really badly, you have seatbelts, airbags, crumple zones, and a thick frame to help you out. In an airplane, if things to badly, you keep flying until you land. If things go really badly, remember that everything is built to be light weight, and unless the crash is well controlled, everything will be destroyed and everyone will die. If your engine quits, your cabin ruptures, your instrumentation fails, you keep flying. And you need instruments; in poor visibility, your own sensory inputs are in fact faulty, and won't help you figure out which way is down. Unlike in a car, where it's pretty obvious where the ground is, for example.

shamino 7y ago

Nathan Marz talks about this previously, with unique insights: http://nathanmarz.com/blog/how-becoming-a-pilot-made-me-a-better-programmer.html http://nathanmarz.com/blog/how-becoming-a-pilot-made-me-a-be...

skookumchuck 7y ago

The article talks about pilot procedures, not engineering procedures specific to aviation.

marcosdumay 7y ago

I still hold my opinion that checklists are for hardware issues. One should not be filling them on software tasks. Instead, software is automated, automatically tested and automatically verified - routine checks are an anti-feature and inversely correlated to quality.

horacio_colbert 7y ago

Thinking of aviation makes me remember the impact of doing things right.