4 ms·
> EVERYTHING MUST FAIL NORMAL. I'm often surprised how little discussion I see of failure analysis and especially failure design is in software engineering. It
by godelski 3y ago
> EVERYTHING MUST FAIL NORMAL.
I'm often surprised how little discussion I see of failure analysis and especially failure design is in software engineering. It is one of the most important concepts in engineering, but I guess we still operate like software can't kill you.
For those unfamiliar with this type of analysis I'll give you 3 examples. 1) Imagine you are building skyscrapers in city with a bunch of other skyscrapers. You don't expect them to fall but you question what happens if they do. You will explicitly design them to fall in on themselves (say after... idk... a plane hits them?) so that they do not fall on their side and take out other buildings around them. An important aspect of security design is explicitly around this whole idea. When your systems fail do you want them to "fail open" or "fail closed"? 2) Fail open would be like TSA, where if you start firing off rounds in the queue the system fails and you get "access" to the rest of the airport. 3) Failing closed is typical of safes, where you accept the fact that you will need to take a blowtorch and drill into the safe. (the TSA thing did happen, just not in America)
While these examples are a bit extreme the ideas are incredibly important for many products. I'll give a good example of this by looking at Android Auto. If a user gets frustrated with the system and is not able to accomplish their task in the way __they want to__, they will disconnect their phone, look down at it, do the thing, and reconnect. No matter how much you want to push voice operations, if the person doesn't like that, they fail over to a more dangerous set of actions than if you had just let them use the console a little longer or let them do a specific task. At least their eyes would be closer towards the road. I'm often amazed that there are lack of consideration for these type of things in home and auto platforms where there are often high stakes. How things fail is one of the most important things when designing systems (I'd argue it even matters for things like law and medicine).
- syndicatedjelly 3y ago> I'm often surprised how little discussion I see of failure analysis and especially failure design is in software engineering. It is one of the most important concepts in engineering, but I guess we still operate like software can't kill you. This is a major reason why many "traditional" engineers do not consider programmers to be true engineers.
- godelski 3y agoYeah the jokes tend to be about how little concern there is for quality control or things breaking. I'm a fan of "move fast and break things" for getting things started and new projects but I'm not sure why that mentality continues into mature products. I think a lot of software dev teams would benefit from referencing the TRL system[0]. I did work as an engineer in a aerospace startup and you definitely move fast and hack a lot at low TRL but you treat mid TRL very differently. Helps to keep scope and pull away from these 26 "2 week sprint" nonesense. Doesn't make sense for mature products, you're just going to break things that already work. Now I will add though that it isn't uncommon for "traditional" engineers to also move too slowly. I think there's a balance in there. Don't be Boeing but don't be a startup either if you're Google. [0] https://en.wikipedia.org/wiki/Technology_readiness_level https://en.wikipedia.org/wiki/Technology_readiness_level
- mewpmewp2 3y agoDoes it not matter what the consequences of failure are to determine how much effort should be put into handling failure?
- syndicatedjelly 3y agoOf course it matters. That’s the “end effect” portion of any proper risk management process. The severity and likelihood of various end effects drives improvement in the design, creation, and usage of the item.
- mewpmewp2 3y agoBut why are people surprised then that in most software the process is not as thorough as in aerospace or elsewhere with physical things. Most of the time physical devices failing will have much more severe consequences, than a website going down for an hour. In some cases there are APIs which many other critical systems use and frequently those APIs do have much more rigorous process in making sure that they are reliable. All of that seems regulated by governments and markets to me. They have promises of reliability and big financial penalties if they were to fail on that reliability.
- bencelaszlo 3y ago> I guess we still operate like software can't kill you. Software is killing people since decades: https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25
- fl7305 3y ago> Failing closed is typical of safes Unless it's a very large safe where people can walk inside, like a bank vault. The fire department gets very grumpy with you if they fail closed and can trap people inside. So if you want to force open a bank vault, stage a purported terrorist attack, and the FBI will cut power to the building as standard operating procedure. Voila, vault opens.
- godelski 3y agoSure, but this is literally the essence of failure analysis. Because you're demonstrating how nuance is essential to the process and how this means that there are no universal concepts. That simplicity and generalizability are often in contention. I'm not at why people don't understand that nuance is important because if it wasn't we wouldn't need experts. It's also weird to see people thinking nuance is required in their domain (explicitly or implicitly by means of assuming everyone else is an idiot for not understanding what they obtained through years of education. Yes I see the irony) but don't think it's required for other areas. Look at any online discussion here involving climate, nuclear, rockets, aerospace, and many other non programming stuff. Your example might also demonstrate this because the failure analysis is considerate of this by having a combination of fail open and fail closed gates in conjunction with the FBI not going to cut power to the bank until they arrive on scene which entirely defeats any terrorist or robber strategy because they the FBI are now the safe door and they're hard to pick open.
- anotherevan 3y agoI seem to remember a certain Christmas movie starring Bruce Willis where this was a plot point.
- moandcompany 3y agoThe concept of "Safety Critical" is something that I wish more software developers and engineers would have exposure to
- godelski 3y ago100%. But I really hope that they can learn this concept and understand that it is generalizable and need but require a system that is critical nor dangerous. That the same ideas are essential for simply designing good user experiences. It's really frustrating that this isn't understood well in ML where people end up talking past one another and therefore the extreme ends of the spectrum end up dominating the conversation with equally idiotic ideas. I can't remember the last time I saw someone discuss a confusion matrix, especially when discussing products. But how's that not like one of the most important aspects of your product (I mean internally in teams developing products, not communication to the public. Or discussions in papers)
- moandcompany 3y agoBeyond confusion matricies, with ML classification, there should also be deep discussion and consideration of the actual costs (consequences) of false positives, negatives to the end users and operators.
- godelski 3y agoIf you want to know the state of things, I was once told that my ethics section (which is requested by the conference) was unnecessary and overly verbose. This was an image generation paper and the section was only like a 4-5 sentence paragraph. I've shifted to throwing them into my appendix because I find readers look at the appendix but reviewers don't and I care more about communicating to the readers.