6 ms·
I strongly recommend watching/reading the entire report, or the summary by Sal Mercogliano of What's Going On In Shipping [0]. Yes, the loose wire was the imme
by crote 1y ago
I strongly recommend watching/reading the entire report, or the summary by Sal Mercogliano of What's Going On In Shipping [0].
Yes, the loose wire was the immediate cause, but there was far more going wrong here. For example:
- The transformer switchover was set to manual rather than automatic, so it didn't automatically fail over to the backup transformer.
- The crew did not routinely train transformer switchover procedures.
- The two generators were both using a single non-redundant fuel pump (which was never intended to supply fuel to the generators!), which did not automatically restart after power was restored.
- The main engine automatically shut down when the primary coolant pump lost power, rather than using an emergency water supply or letting it overheat.
- The backup generator did not come online in time.
It's a classic Swiss Cheese model. A lot of things had to go wrong for this accident to happen. Focusing on that one wire isn't going to solve all the other issues. Wires, just like all other parts, will occasionally fail. One wire failure should never have caused an incident of this magnitude. Sure, there should probably be slightly better procedures for checking the wiring, but next time it'll be a failed sensor, actuator, or controller board.
If we don't focus on providing and ensuring a defense-in-depth, we will sooner or later see another incident like this.
[0]: https://www.youtube.com/watch?v=znWl_TuUPp0 https://www.youtube.com/watch?v=znWl_TuUPp0
- Aurornis 1y agoThanks for the summary for those of us who can't watch video right now. There are so many layers of failures that it makes you wonder how many other operations on those ships are only working because those fallbacks, automatic switchovers, emergency supplies, and backup systems save the day. We only see the results when all of them fail and the failure happens to result in some external problem that means we all notice.
- crote 1y agoOh, it gets even worse! The NTSB also had some comments on the ship's equivalent of a black box. Turns out it was impossible to download the data while it was still inside the ship, the manufacturer's software was awful and the various agencies had a group chat to share 3rd party software(!), the software exported thousands of separate files, audio tracks were mixed to the point of being nearly unusable, and the black box stopped recording some metrics after power loss "because it wasn't required to" - despite the data still being available. At least they didn't have anything negative to say about the crew: they reacted timely and adequately - they just didn't stand a chance.
- nothercastle 1y agoIt’s pretty common for black boxes to be load shed during an emergency. Kind of funny how that was allowed for a long time.
- MengerSponge 1y ago"they reacted timely and adequately" and yet: they're indefinitely restricted (detained isn't the right word, but you get it) to Baltimore, while the ship is free to resume service.
- arjie 1y agoIt seems to just be standard "normalization of deviance" to use the language of safety engineering. You have 5 layers of fallbacks, so over time skipping any of the middle layers doesn't really have anything fail. So in time you end up with a true safety factor equal only to the last layer. Then that fails and looking back "everything had to go wrong". As Sidney Dekker (of Understanding Human Error fame) says: Murphy's Law is wrong - everything that can go wrong will go right. The problem arises from the operators all assuming that it will keep going right. I remember reading somewhere that part of Qantas's safety record came from the fact that at one time they had the highest number of minor issues. In some sense, you want your error detection curve to be smooth: as you get closer to catastrophe, your warnings should get more severe. On this ship, it appeared everything was A-OK till it bonked a bridge.
- bombcar 1y agoThis is the most pertinent thing to learn from these NTSB crash investigations - it's not what went wrong at the final disaster, but all the things that went wrong that didn't detect that they were down to one layer of defense. Your car engaging auto brake to prevent a collision shouldn't be a "whew, glad that didn't happen" and more a "oh shit, I need to work on paying attention more."
- dmurray 1y agoWhy then does the NTSB point blame so much at the single wiring issue? Shouldn't they have the context to point to the 5 things that went wrong in the Swiss cheese and not pat themselves on the back with having found the almost-irrelevant detail of > Our investigators routinely accomplish the impossible, and this investigation is no different...Finding this single wire was like hunting for a loose rivet on the Eiffel Tower. In the software world, if I had an application that failed when a single DNS query failed, I wouldn't be pointing the blame at DNS and conducting a deep dive into why this particular query timed out. I'd be asking why a single failure was capable of taking down the app for hundreds or thousands of other users.
- toast0 1y ago
- deleted 1y ago[deleted]
- dboreham 1y agoAlso the zeroth failure mode: someone built a bridge that will collapse if any of the many many large ships that sail beneath it can't steer itself with high precision.
- foobar1962 1y agoShips were a lot smaller when the bridge was designed and built.
- fl7305 1y agoIn 1971 there where ships with almost twice the displacement of the Dali. They weren't freight ships destined for Baltimore, but it wasn't hard to imagine future freight ship sizes when designing the bridge in the early 1970s. The London sewer system was designed in the 1850s, when the population was around two million people. It was so overdesigned that it held up to the 1950s, when the population was over 8 million. It didn't start to become a big problem until the 1990s.
- myself248 1y agoRight? There's an artificial island in that very harbor, which could be rammed by similar ships all day and give nary a fuck. It's called Fort Carroll and it was built in the *1850s*. Why the bridge piers weren't set into artificial islands, I can't fathom. Sure. Let's build a bridge across a busy port but not make it ship-proof. The bridge was built in the 1970s, had they forgotten how to make artificial islands?
- potato3732842 1y agoIf you design a fort and it actually gets used and turns out to suck that WILL be the end of your career in the military even if it only comes out as sucking 20yr later unless you have an airtight case why it's not your fault. That's just how the .mil works. Heads MUST roll. This is completely the opposite from big company bureaucracy and on a literal different planet than civil government bureaucracy. The organizations that made the bridge happen were so much more vast and so much higher turnover and subject to way, way, way looser application of consequences than the one that built the fort it would be literally impossible to get them to produce something so unnecessarily robust for the average use case. This sort of "everything I depend on will just have to not suck because my shit will keel right over if it sucks in the slightest" type engineering is all over the modern world and does work well in a lot of places when you consider lifetime cost. But when it fails bridges fall over and cloudflare (disclaimer, didn't actually read that PM, have no idea what happened) goes down or whatever.
- haddonist 1y agoOne of the things Sal Mercogliano stressed is that the crew (and possibly other crews of the same line) modified systems in order to save time. Rather than doing the process of purging high-sulphur fuel that can't be used in USA waters, they had it set so that some of the generators were fed from USA-approved fuel, resulting in redundancy & automatic failover being compromised. It seems probable that the wire failure would not have caused catastrophic overall loss of power if the generators had been in the normal configuration.
- myself248 1y agoObligatory: https://how.complexsystems.fail/ https://how.complexsystems.fail/
- pstuart 1y agoHopefully the lesson from this will be received by operators: it's way cheaper to invest in personnel, training, and maintenance than to let the shit hit the fan.
- stackskipton 1y agoWhy? It's cost them 100M (https://www.justice.gov/archives/opa/pr/us-reaches-settlement-over-100m-civil-lawsuit-against-owner-and-operator-vessel-destroyed https://www.justice.gov/archives/opa/pr/us-reaches-settlemen...) but rebuilding the bridge is going to be 5.2Billion so if gundecking all this maintenance for 20+ years has saved more then 100M, they will do it again.
- lazide 1y agoActually, to be even more cynical…. If everyone saved $100M by doing this and it only cost one shipper $100M, then of course everyone else would do it and just hope they aren’t the one who has bad enough luck to hit the bridge. And statistically, almost all of them will be okay!
- sitkack 1y agoThis is the calculus that shows why our current civilization is unlikely to pass the filter.
- lazide 1y agoMaking the calculus apparent is why we might have a chance. Because then anyone who owns a bridge/needs to pay for said bridge damage goes, ‘well clearly the costs of running into a bridge on the runs-into-bridges-due-to-negligence-group isn’t high enough, so we need to either create more rules and inspections, or increase the penalties, or find a way to stop these folks from breaking our bridges, or the like - and actually enforce them’. It’s why airplanes are so safe to fly on, despite all the same financial incentives. If you don’t comply with regulators, you’ll be fined all to hell or flat out forbidden from doing business. And that is enforced. And the regulators take it all very seriously. Ships are mostly given a free pass (except passenger liners, ferries, and hazmat carrying ships) because the typical situation if the owner screws up is ‘loses their asset and the assets of anyone who trusted them’, which is a more socially acceptable self correcting problem than ‘kills hundreds of innocent people who were voters and will have families crying, gnashing their teeth, and pointing fingers on live TV about all this’.
- ChrisMarshallNY 1y agoI have found that 99% of all network problems are bad wires. I remember that the IT guys at my old company, used to immediately throw out every ethernet cable, and replace them with ones right out of the bag; first thing. But these ships tend to be houses of cards. They are not taken care of properly, and run on a shoestring budget. Many of them look like floating wrecks.
- jmonty900 1y agoI recently had a home network outage. The last thing I tested was the in-wall wiring because I just didn't think that would be the cause. It was. Wiring fails!
- nrhrjrjrjtntbt 1y agoOh yeah had outages recently. Turned out to be corroded connector to box in the street. Not a wire per-se but close.
- HeyLaughingBoy 1y agoI remember a customer support call where the hardware they bought from us wasn't working. The last question I asked was "are you sure that the outlet it's plugged into is working?" It wasn't.
- ChrisMarshallNY 1y agoThat's sort of the joke behind this: https://www.youtube.com/watch?v=nn2FB1P_Mn8 https://www.youtube.com/watch?v=nn2FB1P_Mn8
- gerdesj 1y agoIf I see a RJ45 plug with a broken locking thingie, or bare wires (not just bare copper - any internal wire), I chop the plug off. If I come across a CATx (solid core) cable being used as a really long patch lead then I lose my shit or perhaps get a backbox and face plate and modules out along with a POST tool. I don't look after floating fires.
- renhanxue 1y agoThe fuel pump not automatically restarting on power loss may actually have been an intentional safety feature to prevent scenarios like pumping fuel into a fire in or around the generators. Still part of the Swiss cheese model, of course.
- crote 1y agoIt wasn't. They were feeding generators 1 & 2 with the pump intended for flushing the lines while switching between different fuel types. The regular fuel pumps were set up to automatically restart, which is why a set of them came online to feed generator 3 (which automatically spinned up after 1 & 2 failed, and wasn't tied to the fuel-line-flushing pump) after the second blackout.
- p3rls 1y ago[dead]
- kfarr 1y agoAnother case study to add to the maritime chapter of this timeless classic: https://www.amazon.com/Normal-Accidents-Living-High-Risk-Technologies/dp/0691004129 https://www.amazon.com/Normal-Accidents-Living-High-Risk-Tec... Like you said (and illustrated well in the book) it's never just 1 thing, these incidents happen when multiple systems interact and often reflect a the disinvestment in comprehensive safety schemes.
- lostlogin 1y agoShipping, accidents and timeless classics. I was sure you were going to link to Clarke and Dawe, The Front Fell off. https://m.youtube.com/watch?v=3m5qxZm_JqM https://m.youtube.com/watch?v=3m5qxZm_JqM
- FridayoLeary 1y agoJust insane how much criminal negligence went on. Even boeing hardly comes close. What needs to change is obviously a major review of how ships are allowed to operate near bridges and other infrastructure. And far stricter safety standards like aircraft face.
- rolph 1y agoive been in an environment like that. "nuisance" issues like that are deferred bcz they are not really causing a problem, so maintenance spends time on problems with things that make money, rather than what some consider spit n polish on things that have no prior failures.
- hylaride 1y agoTragically, it's the same with modern software development and the growth of technical debt.
- jacquesm 1y agoThe problem is that there are a thousand merchant marine vessels operating right now that are all doing great - until the next loose wire. The problem is that nobody knows about that wire and it worked fine on the last trip. The other systems are all just as marginal as they were on the 'Dali' but that one shitty little wire is masking that. Running a 'tight ship' is great when you have a budget to burn on excellent quality crew. But shipping is so incredibly cut-throat that the crew members make very little money, are effectively modern slaves and tend to carry responsibilities way above their pay grade. They did what they could, and more than that, and for their efforts they were rewarded with what effectively amounted to house arrest while the authorities did their thing. The NTSB of course will focus on the 'hard' causes. But you can see a lot of frustration shine through towards the owners who even in light of the preliminary findings had changed absolutely nothing on the rest of their fleet. The recommendation to inspect the whole ship with an IR camera had me laughing out loud. We're talking about a couple of kilometers of poorly accessible duct work and cabinets. You can do that while in port, but while you're in port most systems are idle or near idle and so you won't ever find an issue like this until you are underway, when vibration goes up and power consumption shoots up compared to being in port. There is no shipping company that is effectively going to do a sea trial after every minor repair, usually there is a technician from some supplier that boards the vessel (often while it is underway), makes some fix and then goes off-board again. Vessels that are not moving are money sinks so the goal is to keep turnaround time in port to an absolute minimum. What should really amaze you is how few of these incidents there are. In spite of this being a regulated industry it is first and foremost an oversight failure, if the regulators would have more budget and more manpower there maybe would be a stronger drive to get things technically in good order (resist temptation: 'shipshape').
- amelius 1y agoYou can also look at the problem from the perspective of the bridge. Why was it possible that a ship took it down? Motors can fail ...
- richardwhiuk 1y agoIt’s not realistically plausible to build bridges that won’t be brought down by that size of ship
- deleted 1y ago[deleted]
- Nifty3929 1y agoAll you said is true - but these investigations are often used for the purpose of determining financial liability and often that comes down to figuring out that one, immediate, proximate thing that caused the accident. A whole bunch of things might have gone wrong, but if only you hadn't done/not-done that one thing, we'd all be fine. So it's all your fault!
- _n_b_ 1y agoRespectfully, have you ever actually read an NTSB report? They're incredibly thorough and consider both causes and contributing factors through a number of lenses with an exclusive focus on preventing accidents from occurring. Also, they're basically inadmissible in court [49 U.S.C.§1154(b)] so are useless for determining financial liability.
- creeble 1y agoI watched Sal's video yesterday, great summary. So much complexity, plenty of redundancy, but not enough adherence to important rules.