6 ms·

I strongly agree with this recommendation. Personal Experience: During and after the economic collapse of 2007-2009 I wondered who was at fault; who to blame.

• deleted 6y ago
  [deleted]
• narag 6y ago
  I wrote another comment, before reading the article that says it better than me: Catastrophe requires multiple failures – single point failures are not enough.
• Joeri 6y ago
  This is why I prefer thinking about root solutions rather than root causes. The answer to the question “how can we make sure something like this cannot happen again?” (for a reasonably wide definition of this). The nice thing is that there are usually many right answers, all of which can be implemented, while when looking for root causes there may not actually be one.
  • travmatt 6y ago
    A good example was the AWS S3 outage that occurred when a single engineer mistyped a command[0]. While the outage wouldn't have occurred had an engineer not mistyped a command, that conclusion still would have missed the issue that the system should have some level of resiliency against simple typos - in their case, checking that actions that wouldn't take subsystems below their minimum required capacity. [0] https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/
    • academi 6y ago
      Systems should still be able to be taken offline, though, even if that means failure. For example, let’s say you have a service that uses another service that raised its cost from free to $100/hour and you call it 1000 times per hour. Even though you may not have a fallback, and your service may fail, you need to be able to disable it. In this case, an admin is unavailable and the only recourse would be to lower the capacity to 0, since you have that control. That doesn’t negate the benefit of validation, but don’t be too heavy-handed with validation, just as a reaction to failure without fully thinking it through.
      • jimktrains2 6y ago
        Ideally a destructive command shouldn't be accidently triggerable. At the very least it should require some positive confirmation. Alternatively, a series of actions could be required, such as changing the capacity (which should be the comqmnd where the double checks and positive confirmations happen in my opinion) followed by changing the services usage.
• triangleman 6y ago
  There is a relatively simple root cause of the financial crisis, it's called moral hazard: https://en.m.wikipedia.org/wiki/Greenspan_put https://en.m.wikipedia.org/wiki/Greenspan_put Now, how exactly to solve the problem is a complex question, so I suppose in that respect it's hard to think productively about it.
  • watwut 6y ago
    Afaik, there were actual massive frauds going on.
    • triangleman 6y ago
      Like Madoff? The subprime lending/NINJA loans? In my opinion both of those things would have been harder if everyone was doing more due diligence. Moral hazard took away some of the incentive to do that.
      • simonh 6y ago
        Right, even if all the wrongdoing had all happened together, but the rest of the economic and financial system had been sound, we’d have been ok.
      • watwut 6y ago
        "everyone was doing due diligence" is not really realistic expectation. It amounts to blaming everyone for small bad thing while excusing massive big bad thing going on.
        • triangleman 6y ago
          I found a decent analogy for what I'm talking about: https://www.history.com/news/beanie-babies-value-criminal-activity https://www.history.com/news/beanie-babies-value-criminal-ac... With a bubble in Beanie Babies there was accompanying fraud and other criminal activity, with most consumers not wittingly participating, but they were affected by it. So with the asset bubble of the 2000's, this kind of behavior affected everyone, as everyone needs a house. In another thread I have said I don't blame the masses who may have made bad decisions in hindsight, as they did not have access to all the relevant information. So I do blame the fraudsters, but in the end I really only blame the Fed for keeping interest rates too low, and of course that can circle back around to society as well if you like. https://youtu.be/d0nERTFo-Sk https://youtu.be/d0nERTFo-Sk
• hansvm 6y ago
  > But what about a case where someone 3 or 4 car lengths ahead makes a quick lane change and an accident occurs behind them? You'd have to be going extremely slowly for 3 or 4 car lengths to be a safe following distance. On a typical 60-70mph freeway you should have a gap of at least 15-20 car lengths, and then accidents like that will happen only when other factors are at play (and if those factors are predictable like water on the road then your distance/speed should be adjusted accordingly). While I think that example was bad, your point about there existing accidents without single points of blame is still valid.
  • csours 6y ago
    I meant to say 3 or 4 cars ahead. Thank you for taking the point.
    • hansvm 6y ago
      That makes sense. To be fair I shouldn't have been so hasty anyway; with your example of a lane change, even if you can control your following distance it's hard to also keep people from other lanes cutting in too closely.
• yholio 6y ago
  > A quick example: in many car accidents, you can easily point to the person who caused the accident This is a bad example: traffic is a complex sistem with a century of ruleset evolution specifically intended to isolate personal responsibility and provide a simple interface for the users, that, when correctly used, guarantees a collision free ride for all participants. The systemic failures of trafic are more related to the fallible nature of its actors. The safety guarantees work only when humans demonstrate almost super-human regard to the safety of others, are never inattentive, tired, in a hurry or influenced by substances or medical conditions etc. We try to align personal incentives to systemic goals with hefty punishments, but there is a diminishing return on that, at some point you have to consider humans unreliable and design your system to be fault-tolerant. Indeed, most modern trafic systems are doing this today with things like impact absorbing railings, speed bumps, wide shoulders and curves etc.
  • mjcohen 6y ago
    "guarantees a collision free ride for all participants"? The only way to win is not to play. I was stationary in a traffic jam when I was rear-ended by the car in back of me. Fortunately, I was not hurt at all. How could I have avoided this? (see above)?
    • yholio 6y ago
      I think you missed the "when correctly used" part. Clearly, the person who hit you was not correctly using the system which places the whole responsibility of this situation onto them. When obeying all rules of the road, situations where rear-ending someone is probable should not arise. The systemic failure here is expecting people not to phase out and pay less attention to the road when driving for hours at high speed on monotonous highways.
    • JadeNB 6y ago
      > I was stationary in a traffic jam when I was rear-ended by the car in back of me. Fortunately, I was not hurt at all. I believe yholio's statement about "when correctly used" was meant to be "when correctly used by all participants"; i.e., no single participant can guarantee their own, or anyone else's, safety, no matter how careful they are. On the other hand, the guarantee is, as yholio notes, of an almost entirely theoretical nature even given universal cooperation: > The safety guarantees work only when humans demonstrate almost super-human regard to the safety of others, are never inattentive, tired, in a hurry or influenced by substances or medical conditions etc.
• roenxi 6y ago
  > During and after the economic collapse of 2007-2009 I wondered who was at fault; who to blame. I kept waiting for a clear answer to "THE root cause". Well, there can be a difference between "who is at fault" and "THE root cause". Quite a large difference, potentially. In the case of '07-'09, maybe nobody was at fault (seems plausible) but there was a very neat root cause. The government handed out a lot of money to people who shouldn't have gotten it, the people responsible were largely protected from bankruptcy and the system forced to reform largely as it was. The people who took excessive risk earned an excessive reward - they should have all gone bankrupt. The financial system should actually have changed, and people who made productive investments and didn't take risk should have become ascendant. Instead we have the same old crowd playing the same old game. Disabling the major feedback mechanism of capitalism is about as root-causal as can be gotten. Nobody in particular chose to disable it though, it was a consensus decision among the powerful.
  • worik 6y ago
    Yes you are correct, root cause != person to blame. But.... That is such a over simplification and ignores a lot, most, of the issues in the GFC. * Regulatory capture * Illegal behaviour by trusted actors (chiefly banks) * Corrupt judges * Carelessness by trusted agencies e.g., credit rating agencies Long long list of the collection of failures in the international financial system in general and American civil society in particular.
    • roenxi 6y ago
      Any issue can be made impossible to solve if enough details are highlighted. Deciding what to eat for breakfast is an impossible collection of nutrient requirements, bodily requirements, long and short term tradeoffs, economic problems and social expectations. Yet somehow we mostly manage. "Oh but it's complicated" is a completely standard line of misdirection that comes up very regularly when people are making repeated bad decisions. Even small children sometimes try it. It is very, very rarely true and particularly in totally synthetic systems like the monetary one. There is always a point of greatest leverage that could be changed and it makes sense to call that the root cause and try changing it. Now I'm totally open to the idea that I don't know what the point of greatest leverage is in the GFC. I haven't read the regulations and I wasn't in the room when the money was being handed out. But there were billions to trillions of dollars in fake wealth that turned out never to have existed. The fact that the banking industry skated through with the same people largely in charge suggests strongly that no serious attempt was made to figure out who exactly was screwing up.
• konjin 6y ago
  The root cause was the deregulation in the 90s and the removal of the Glass-Steagall act. Complex systems are also simple systems when viewed as a black box from the out side. A lion always eats a gazelle given a chance and a bank always explodes if not regulated to within an inch of its life. That people like to pretend internal complexity matches external complexity is a very odd mental quirk. It is false in both directions of implication. Conway's game of life is as simple a game as you can get yet it has the most complex behavior possible.
  • dkarl 6y ago
    Thank you for making this point. The idea that it was an unpredictable failure mode of a complex system, rather than predictable exploitation of a system where safeguards against exploitation were deliberately removed as "inefficient," is exactly what will ensure a repeat in the future.
• apta 6y ago
  > During and after the economic collapse of 2007-2009 I wondered who was at fault; who to blame Sometimes it is possible to at least narrow things down to an underlying inherent instability. In the case of your example, a huge underlying cause is an economic system based on debt (backed by interest and usurious transactions). It's for a reason that usury/interest is banned in Islam, Christianity, and Judaism for example. It's a parasitic practice that makes the economy fundamentally unstable. This includes dangerous practices such as selling debt for debt (again part of the same crisis), and things like stock shorting (which, interestingly enough was also banned during the crisis, at least for some critical company stocks).