4 ms·

Thanks for the summary for those of us who can't watch video right now. There are so many layers of failures that it makes you wonder how many other operations

Thanks for the summary for those of us who can't watch video right now. There are so many layers of failures that it makes you wonder how many other operations on those ships are only working because those fallbacks, automatic switchovers, emergency supplies, and backup systems save the day. We only see the results when all of them fail and the failure happens to result in some external problem that means we all notice.

crote 1y ago
Oh, it gets even worse! The NTSB also had some comments on the ship's equivalent of a black box. Turns out it was impossible to download the data while it was still inside the ship, the manufacturer's software was awful and the various agencies had a group chat to share 3rd party software(!), the software exported thousands of separate files, audio tracks were mixed to the point of being nearly unusable, and the black box stopped recording some metrics after power loss "because it wasn't required to" - despite the data still being available. At least they didn't have anything negative to say about the crew: they reacted timely and adequately - they just didn't stand a chance.
- nothercastle 1y ago
  It’s pretty common for black boxes to be load shed during an emergency. Kind of funny how that was allowed for a long time.
- MengerSponge 1y ago
  "they reacted timely and adequately" and yet: they're indefinitely restricted (detained isn't the right word, but you get it) to Baltimore, while the ship is free to resume service.
arjie 1y ago
It seems to just be standard "normalization of deviance" to use the language of safety engineering. You have 5 layers of fallbacks, so over time skipping any of the middle layers doesn't really have anything fail. So in time you end up with a true safety factor equal only to the last layer. Then that fails and looking back "everything had to go wrong". As Sidney Dekker (of Understanding Human Error fame) says: Murphy's Law is wrong - everything that can go wrong will go right. The problem arises from the operators all assuming that it will keep going right. I remember reading somewhere that part of Qantas's safety record came from the fact that at one time they had the highest number of minor issues. In some sense, you want your error detection curve to be smooth: as you get closer to catastrophe, your warnings should get more severe. On this ship, it appeared everything was A-OK till it bonked a bridge.
- bombcar 1y ago
  This is the most pertinent thing to learn from these NTSB crash investigations - it's not what went wrong at the final disaster, but all the things that went wrong that didn't detect that they were down to one layer of defense. Your car engaging auto brake to prevent a collision shouldn't be a "whew, glad that didn't happen" and more a "oh shit, I need to work on paying attention more."
  - dmurray 1y ago
    Why then does the NTSB point blame so much at the single wiring issue? Shouldn't they have the context to point to the 5 things that went wrong in the Swiss cheese and not pat themselves on the back with having found the almost-irrelevant detail of > Our investigators routinely accomplish the impossible, and this investigation is no different...Finding this single wire was like hunting for a loose rivet on the Eiffel Tower. In the software world, if I had an application that failed when a single DNS query failed, I wouldn't be pointing the blame at DNS and conducting a deep dive into why this particular query timed out. I'd be asking why a single failure was capable of taking down the app for hundreds or thousands of other users.
    - toast0 1y ago
      The faulty wire is the root cause. If it didn't trigger the sequence of events, all of the other things wouldn't have happened. And it's kind of a tricky thing to find, so that's an exciting find. The flushing pump not restarting when power resumed did also cause a blackout in port the day before the incident. But you know, looking into why you always have two blackouts when you have one is something anybody could do; open the main system breaker, let the crew restore it and that flushing pump will likely fail in the same way every time... but figuring out why and how the breaker opened is neat, when it's not something obvious.
      - nothercastle 1y ago
        Operators always like to just clear the fault and move on they have extremely high pressure to make schedule and low incentive to work safely
deleted 1y ago
[deleted]
dboreham 1y ago
Also the zeroth failure mode: someone built a bridge that will collapse if any of the many many large ships that sail beneath it can't steer itself with high precision.
- foobar1962 1y ago
  Ships were a lot smaller when the bridge was designed and built.
  - fl7305 1y ago
    In 1971 there where ships with almost twice the displacement of the Dali. They weren't freight ships destined for Baltimore, but it wasn't hard to imagine future freight ship sizes when designing the bridge in the early 1970s. The London sewer system was designed in the 1850s, when the population was around two million people. It was so overdesigned that it held up to the 1950s, when the population was over 8 million. It didn't start to become a big problem until the 1990s.
- myself248 1y ago
  Right? There's an artificial island in that very harbor, which could be rammed by similar ships all day and give nary a fuck. It's called Fort Carroll and it was built in the *1850s*. Why the bridge piers weren't set into artificial islands, I can't fathom. Sure. Let's build a bridge across a busy port but not make it ship-proof. The bridge was built in the 1970s, had they forgotten how to make artificial islands?
  - potato3732842 1y ago
    If you design a fort and it actually gets used and turns out to suck that WILL be the end of your career in the military even if it only comes out as sucking 20yr later unless you have an airtight case why it's not your fault. That's just how the .mil works. Heads MUST roll. This is completely the opposite from big company bureaucracy and on a literal different planet than civil government bureaucracy. The organizations that made the bridge happen were so much more vast and so much higher turnover and subject to way, way, way looser application of consequences than the one that built the fort it would be literally impossible to get them to produce something so unnecessarily robust for the average use case. This sort of "everything I depend on will just have to not suck because my shit will keel right over if it sucks in the slightest" type engineering is all over the modern world and does work well in a lot of places when you consider lifetime cost. But when it fails bridges fall over and cloudflare (disclaimer, didn't actually read that PM, have no idea what happened) goes down or whatever.
haddonist 1y ago
One of the things Sal Mercogliano stressed is that the crew (and possibly other crews of the same line) modified systems in order to save time. Rather than doing the process of purging high-sulphur fuel that can't be used in USA waters, they had it set so that some of the generators were fed from USA-approved fuel, resulting in redundancy & automatic failover being compromised. It seems probable that the wire failure would not have caused catastrophic overall loss of power if the generators had been in the normal configuration.
myself248 1y ago
Obligatory: https://how.complexsystems.fail/ https://how.complexsystems.fail/

crote 1y ago

Oh, it gets even worse! The NTSB also had some comments on the ship's equivalent of a black box. Turns out it was impossible to download the data while it was still inside the ship, the manufacturer's software was awful and the various agencies had a group chat to share 3rd party software(!), the software exported thousands of separate files, audio tracks were mixed to the point of being nearly unusable, and the black box stopped recording some metrics after power loss "because it wasn't required to" - despite the data still being available. At least they didn't have anything negative to say about the crew: they reacted timely and adequately - they just didn't stand a chance.

nothercastle 1y ago
It’s pretty common for black boxes to be load shed during an emergency. Kind of funny how that was allowed for a long time.
MengerSponge 1y ago
"they reacted timely and adequately" and yet: they're indefinitely restricted (detained isn't the right word, but you get it) to Baltimore, while the ship is free to resume service.

arjie 1y ago

It seems to just be standard "normalization of deviance" to use the language of safety engineering. You have 5 layers of fallbacks, so over time skipping any of the middle layers doesn't really have anything fail. So in time you end up with a true safety factor equal only to the last layer. Then that fails and looking back "everything had to go wrong". As Sidney Dekker (of Understanding Human Error fame) says: Murphy's Law is wrong - everything that can go wrong will go right. The problem arises from the operators all assuming that it will keep going right. I remember reading somewhere that part of Qantas's safety record came from the fact that at one time they had the highest number of minor issues. In some sense, you want your error detection curve to be smooth: as you get closer to catastrophe, your warnings should get more severe. On this ship, it appeared everything was A-OK till it bonked a bridge.

bombcar 1y ago
This is the most pertinent thing to learn from these NTSB crash investigations - it's not what went wrong at the final disaster, but all the things that went wrong that didn't detect that they were down to one layer of defense. Your car engaging auto brake to prevent a collision shouldn't be a "whew, glad that didn't happen" and more a "oh shit, I need to work on paying attention more."
- dmurray 1y ago
  Why then does the NTSB point blame so much at the single wiring issue? Shouldn't they have the context to point to the 5 things that went wrong in the Swiss cheese and not pat themselves on the back with having found the almost-irrelevant detail of > Our investigators routinely accomplish the impossible, and this investigation is no different...Finding this single wire was like hunting for a loose rivet on the Eiffel Tower. In the software world, if I had an application that failed when a single DNS query failed, I wouldn't be pointing the blame at DNS and conducting a deep dive into why this particular query timed out. I'd be asking why a single failure was capable of taking down the app for hundreds or thousands of other users.
  - toast0 1y ago
    The faulty wire is the root cause. If it didn't trigger the sequence of events, all of the other things wouldn't have happened. And it's kind of a tricky thing to find, so that's an exciting find. The flushing pump not restarting when power resumed did also cause a blackout in port the day before the incident. But you know, looking into why you always have two blackouts when you have one is something anybody could do; open the main system breaker, let the crew restore it and that flushing pump will likely fail in the same way every time... but figuring out why and how the breaker opened is neat, when it's not something obvious.
    - nothercastle 1y ago
      Operators always like to just clear the fault and move on they have extremely high pressure to make schedule and low incentive to work safely

deleted 1y ago

[deleted]

dboreham 1y ago

Also the zeroth failure mode: someone built a bridge that will collapse if any of the many many large ships that sail beneath it can't steer itself with high precision.

foobar1962 1y ago
Ships were a lot smaller when the bridge was designed and built.
- fl7305 1y ago
  In 1971 there where ships with almost twice the displacement of the Dali. They weren't freight ships destined for Baltimore, but it wasn't hard to imagine future freight ship sizes when designing the bridge in the early 1970s. The London sewer system was designed in the 1850s, when the population was around two million people. It was so overdesigned that it held up to the 1950s, when the population was over 8 million. It didn't start to become a big problem until the 1990s.
myself248 1y ago
Right? There's an artificial island in that very harbor, which could be rammed by similar ships all day and give nary a fuck. It's called Fort Carroll and it was built in the *1850s*. Why the bridge piers weren't set into artificial islands, I can't fathom. Sure. Let's build a bridge across a busy port but not make it ship-proof. The bridge was built in the 1970s, had they forgotten how to make artificial islands?
- potato3732842 1y ago
  If you design a fort and it actually gets used and turns out to suck that WILL be the end of your career in the military even if it only comes out as sucking 20yr later unless you have an airtight case why it's not your fault. That's just how the .mil works. Heads MUST roll. This is completely the opposite from big company bureaucracy and on a literal different planet than civil government bureaucracy. The organizations that made the bridge happen were so much more vast and so much higher turnover and subject to way, way, way looser application of consequences than the one that built the fort it would be literally impossible to get them to produce something so unnecessarily robust for the average use case. This sort of "everything I depend on will just have to not suck because my shit will keel right over if it sucks in the slightest" type engineering is all over the modern world and does work well in a lot of places when you consider lifetime cost. But when it fails bridges fall over and cloudflare (disclaimer, didn't actually read that PM, have no idea what happened) goes down or whatever.

haddonist 1y ago

One of the things Sal Mercogliano stressed is that the crew (and possibly other crews of the same line) modified systems in order to save time. Rather than doing the process of purging high-sulphur fuel that can't be used in USA waters, they had it set so that some of the generators were fed from USA-approved fuel, resulting in redundancy & automatic failover being compromised. It seems probable that the wire failure would not have caused catastrophic overall loss of power if the generators had been in the normal configuration.

myself248 1y ago

Obligatory: https://how.complexsystems.fail/ https://how.complexsystems.fail/