12 ms·
Loose wire leads to blackout, contact with Francis Scott Key bridge
- DamnInteresting 11mo agoVideo explanation: https://www.youtube.com/watch?v=bu7PJoxaMZg https://www.youtube.com/watch?v=bu7PJoxaMZg
- bmelton 11mo agoThat was super helpful. I was assuming from skimming the text description that it was a failed crimp A lot of people wildly under-crimp things, but marine vessels not only have nuanced wire requirements, but more stringent crimping requirements that the field at large frustratingly refuses to adhere to despite ABYC and other codes insisting on it
- Aurornis 11mo ago> A lot of people wildly under-crimp things The good tools will crimp to the proper pressure and make it obvious when it has happened. Unfortunately the good tools aren't cheap. Even when they are used, some techs will substitute their own ideas of how a crimp should be made when nobody is watching them.
- DannyBee 11mo agoWhile the US is still very manual at panel building, Europe is not. So outside of waiting time, I can go from eplan to "send me precrimped and labeled wires that were cut, crimped, and labeled by machine and automatically tested to spec" because this now exists as a service accessible even to random folks. It is not even expensive.
- phasetransition 11mo agoCan you give an examples of companies that offer this service?
- DannyBee 10mo agoSure Onepull does it. Precutwire does it Even dirtypcbs does it Rittal and others who make the processing machines can point you towards tons of folks
- potato3732842 11mo agoThis attitude wherein one thinks they can just spend money and offload responsibility is exactly the problem. Abdicating responsibility to those "good tools" are why shit never gets crimped right. People just crimp away without a care in the world. Don't get me wrong, they're great for speed and when all you're doing it working on brand new stuff that fits perfect. But when you're working on something sketchy you really want the feedback of the older styles of tool that have more direct feedback. They have a place, but you have to know what that place is. See also: "the low level alarm would go off if it was empty"
- psunavy03 11mo agoAlthough I was never named to a mishap board, my experience in my prior career in aviation is that the proper way to look at things like this is that while it is valuable to identify and try to fix the ultimate root cause of the mishap, it's also important to keep in mind what we called the "Swiss cheese model." Basically, the line of causation of the mishap has to pass through a metaphorical block of Swiss cheese, and a mishap only occurs if all the holes in the cheese line up. Otherwise, something happens (planned or otherwise) that allows you to dodge the bullet this time. Meaning a) it's important to identify places where firebreaks and redundancies can be put in place to guard against failures further upstream, and b) it's important to recognize times when you had a near-miss, and still fix those root causes as well. Which is why the "retrospectives are useless" crowd spins me up so badly.
- drivers99 11mo ago> it's important to recognize times when you had a near-miss, and still fix those root causes as well. I mentioned this principal to the traffic engineer when someone almost crashed into me because of a large sign that blocked their view. The engineer looked into it and said the sight lines were within spec, but just barely, so they weren't going to do anything about it. Technically the person who almost hit me could have pulled up to where they had a good view, and looked both ways as they were supposed to, but that is relying on one layer of the cheese to fix a hole in another, to use your analogy.
- kennethrc 11mo agoLikewise with decorative hedges and other gardenwork; your post brought to mind this one hotel I stay regularly where a hedge is high enough and close enough to the exit that you have to nearly pull into the street to see if there's oncoming cars. I've mentioned to the FD that it's gonna get someone hurt one day, yet they've done nothing about it for years now.
- avidiax 11mo agoSend certified letters to the owner of the hedge and whatever government agency would enforce rules about road visibility. That puts them "on notice" legally, so that they can be held accountable for not enforcing their rules or taking precautions.
- tialaramex 11mo agoNote that "Don't make mistakes" is no more actionable for maintenance of a huge cargo ship than for your 10MLoC software project. A successful safety strategy must assume there will be mistakes and deliver safe outcomes nevertheless.
- andrewflnr 11mo agoObviously this is the standard line any disaster prevention, and makes sense 99% of the time. But what's the standard line about where this whole protocols-to-catch-mistakes thing bottoms out? Obviously people executing the protocol can make mistakes, or fall victim to normalization of deviance. The same is true for the next level of safety protocol you layer on top of that. At some level, the only answer really is just "don't make mistakes", right? And you're mostly trying to make sure you can do that at a level where it's easier to not make mistakes, like simpler decisions not under time pressure. Am I missing something? I feel like one of us is crazy when people are talking about improving process instead of assigning blame without addressing the base case.
- lmm 11mo agoNormalization of deviance doesn't happen through people "making mistakes", at least not in the conventional sense. It's a deliberate choice, usually a response to bad incentives, or sometimes even a reasonable tradeoff. I mean ultimately establishing a good process requires make good choices and not making bad ones, sure. But the kind of bad decisions that you have to avoid are not really "mistakes" the same way that, like, switching on the wrong generator is a mistake.
- andrewflnr 11mo agoQuite, normalization is another failure mode, besides simple mistakes, that process has to account for.
- potato3732842 11mo agoIt kind of is though. There's a lot less opportunity for failures at the limit and unforeseen scale. Mechanical things also mostly don't keel over or go haywire with no warning.
- gishh 11mo agoThe date for bridge completion was bumped from 2028 to 2030 already. I assume it won't be done until 2038. It is absolutely murdering traffic in the Baltimore area, not having a bridge. I would be super interested in seeing where every single dollar goes for this project, I assume at least 1/3 of it will be skimmed off the top.
- 1970-01-01 11mo agoSo there were two big failures: Electrician not doing work to code; inspector just checking the box during the final inspection.
- nightpool 11mo agoNo, there was a larger failure: whoever designed the control system such that a single loose wire on a single terminal block (!) could take down the entire steering system for a 91,000 ton ship.
- bragr 11mo agoThere's a 3rd failure: the failure to install/upgrade dolphins that could deflect a modern containership, despite the identified need for such. That proposed project seems cheap in retrospect.
- nightpool 11mo agoYes, 100%. Lots of failures across the board here. Especially with large ships and how many different nations they might be registered in, I can't imagine it's easy to have a lot of regulatory oversight into their construction, mechanical inspection or maintenance schedules. I'm curious how modern ports handle this problem, feels like it could cause a ton of issues beyond just catastrophic ones like this one.
- DannyBee 11mo agoThey didn't. If you read the report they were misusing this pump to do fuel supply when it wasn't for that. And it was non redundant when fuel supply pumps are. Its like someone repurposing a husky air compressor to power a pneumatic fire suppression system and then saying the issue is someone tripping over the cord and knocking it out.
- IncreasePosts 11mo agoThe terminal blocks could also have been designed to aid visual inspection.
- buildsjets 11mo agoIn a well engineered control system, any single failure will not result in a loss of control over the system. Was a FMECA (Failure Mode, Effects, and Criticality Analysis) performed on the design prior to implementation in order to find the single points of failure, and identify and mitigate their system level effects? Evidence at hand suggests "No."
- CGMthrowaway 11mo ago"Catastrophe requires multiple failures – single point failures are not enough. The array of defenses works. System operations are generally successful. Overt catastrophic failure occurs when small, apparently innocuous failures join to create opportunity for a systemic accident. Each of these small failures is necessary to cause catastrophe but only the combination is sufficient to permit failure. Put another way, there are many more failure opportunities than overt system accidents. Most initial failure trajectories are blocked by designed system safety components. Trajectories that reach the operational level are mostly blocked, usually by practitioners." https://how.complexsystems.fail/#3 https://how.complexsystems.fail/#3
- jojobas 11mo agoMost cargo ships have a single main engine with plenty of backup-less failure points. They are sort of engineered so these failures can't happen suddenly but you can help yourself to a bunch of videos on how substandard fuel and parts shortages cause week-long poweroffs in a middle of the ocean.
- LeifCarrotson 11mo agoSystem designers and regulators are aware that the main engine is a single point of failure, but they generally consider loss of main engine power to not be an immediate emergency. There are redundant systems to retain electrical and hydraulic power, and losing motive power isn't generally an instant emergency. Power and steering together is an emergency, yes, and steering is degraded without power, but had they still been able to use the rudder they wouldn't have hit the bridge.
- comeonbro 11mo agoA label placed half an inch wrong on misleading affordance -> 200,000 ton bridge collapse, 6 deaths, tens of billions of dollars of economic damage Instant classic destined for the engineering-disasters-drilled-into-1st-year-engineers canon (or are the other swiss cheese holes too confounding) Where do you think it would fit on the list?
- bragr 11mo agoI guess this will still be bellow Therac-25 for CS and CE students, but above for EE, ME, and Civil Engineering.
- ocdtrekkie 11mo agoThe image brings to mind the Cisco ethernet boot infographic: https://www.cisco.com/c/en/us/support/docs/field-notices/636/fn63697.html https://www.cisco.com/c/en/us/support/docs/field-notices/636...
- hexbin010 11mo agoI can't believe I've never seen this. I literally laughed out loud when I got to the image. Thank you! Absolute gold
- protocolture 11mo agoI love this one.
- lostlogin 11mo agoSomeone out there spent ages trying to work this out. Fucking hell.
- deleted 11mo ago[deleted]
- jtokoph 11mo agoIt’s been noted that automatic failover systems did not kick in due to shortcuts being taken by the company: https://youtu.be/znWl_TuUPp0 https://youtu.be/znWl_TuUPp0
- fabian2k 11mo agoThe big problem was that they didn't have the actual fuel pumps running but were using a different pump that was never intended to fulfill this role. And this pump stays off if the power fails for any reason. The bad contact with the wire was just the trigger, that should have been recoverable had the regular fuel pumps been running.
- jojobas 11mo ago"Contact" is a weird choice of words.
- charles_f 11mo agoThought the same, bridge is fallen on its entire length, sounds like a way to undersell it. Such an opportunity to pass on clickbait is interesting in this day and age.
- dhosek 11mo agoI’m not sure that the NTSB is really in the clickbait business. But yes, contact does seem to really be underselling the event.
- crote 11mo agoNot really, because that's where that part of the investigation ends. Pre-contact everything is about the ship and why it hit anything, post-contact everything is about the bridge and why it collapsed. The ship part of the investigation wouldn't look significantly different if the bridge had remained (mostly) intact, or if the ship had run aground inside the harbor instead.
- nocoiner 11mo agoYeah, when the word “allision” was right there!
- ErroneousBosh 11mo agoRight? Like when I read that I thought we're talking a little paint-swapping. No, we are not talking a little paint-swapping.
- analog31 11mo agoReminds me of "fetched up" describing what happened to the Exxon Valdez.
- dopamean 11mo agoI know a little about planes and nothing about ships so maybe this is crazy but it seems to me that if you're moving something that large there should be redundant systems for steering the thing.
- gk1 11mo agoThere are.[1] Unfortunately they take longer to employ than the crew had time. [1] As it happens I open with an anecdote about steering redundancy on ships in this post: https://www.gkogan.co/simple-systems/ https://www.gkogan.co/simple-systems/
- dopamean 11mo agoThanks for this comment!
- cjensen 11mo agoShipping is a low-margin business. That business structure does not incentivize paying for careful analysis of failure modes. Seems to me the only effective and enforceable redundancy that can be easily be imposed by regulation would be mandatory tug boats.
- dboreham 11mo ago> mandatory tug boats Which there are in some places. Where I grew up I'd watch the ships sail into and out of the oil and gas terminals, always accompanied by tugs. More than one in case there's a tug failure.
- protocolture 11mo ago>Seems to me the only effective and enforceable redundancy that can be easily be imposed by regulation would be mandatory tug boats. Way it worked in Sydney harbour 20+ years ago when I briefly worked on the wharves/tugs, was that the big ships had to have both local tugs, and a local pilot who would come aboard and run the ship. Which seemed to me to be quite an expensive operation but I honestly cant recall any big nautical disasters in the habour so I guess it works.
- ocdtrekkie 11mo ago"and WAGO Corporation, the electrical component manufacturer" Sucks to be any of the YouTubers influencers today telling everyone they should use WAGO connectors in all their walls. Seriously though, impressive to trace the issue down this closely. I am at best an amateur DIY electrician, but I am always super careful about the quality of each connection.
- rootusrootus 11mo agoI don't see anything in the report that suggests the connector failed. It sounds like the installer failed. Trust me, they can screw up twist connections too :)
- Polizeiposaune 11mo agoThe WAGO connectors typically used in home wiring have a transparent plastic shell which lets you see whether the wire made it all the way through the spring clip. The ones shown in the NTSB video had an opaque shell around the spring clip.
- ocdtrekkie 11mo agoI think my attempt at humor butthurt a lot of WAGO fans. I used "seriously though" after in my actual... serious comment.
- crote 11mo agoI strongly recommend watching/reading the entire report, or the summary by Sal Mercogliano of What's Going On In Shipping [0]. Yes, the loose wire was the immediate cause, but there was far more going wrong here. For example: - The transformer switchover was set to manual rather than automatic, so it didn't automatically fail over to the backup transformer. - The crew did not routinely train transformer switchover procedures. - The two generators were both using a single non-redundant fuel pump (which was never intended to supply fuel to the generators!), which did not automatically restart after power was restored. - The main engine automatically shut down when the primary coolant pump lost power, rather than using an emergency water supply or letting it overheat. - The backup generator did not come online in time. It's a classic Swiss Cheese model. A lot of things had to go wrong for this accident to happen. Focusing on that one wire isn't going to solve all the other issues. Wires, just like all other parts, will occasionally fail. One wire failure should never have caused an incident of this magnitude. Sure, there should probably be slightly better procedures for checking the wiring, but next time it'll be a failed sensor, actuator, or controller board. If we don't focus on providing and ensuring a defense-in-depth, we will sooner or later see another incident like this. [0]: https://www.youtube.com/watch?v=znWl_TuUPp0 https://www.youtube.com/watch?v=znWl_TuUPp0
- Aurornis 11mo agoThanks for the summary for those of us who can't watch video right now. There are so many layers of failures that it makes you wonder how many other operations on those ships are only working because those fallbacks, automatic switchovers, emergency supplies, and backup systems save the day. We only see the results when all of them fail and the failure happens to result in some external problem that means we all notice.
- crote 11mo agoOh, it gets even worse! The NTSB also had some comments on the ship's equivalent of a black box. Turns out it was impossible to download the data while it was still inside the ship, the manufacturer's software was awful and the various agencies had a group chat to share 3rd party software(!), the software exported thousands of separate files, audio tracks were mixed to the point of being nearly unusable, and the black box stopped recording some metrics after power loss "because it wasn't required to" - despite the data still being available. At least they didn't have anything negative to say about the crew: they reacted timely and adequately - they just didn't stand a chance.
- kylehotchkiss 11mo agoWhen shipowners are willing to cut costs with sketchy moves like registering with a random landlocked African country, why should we believe they'll spend any time or effort reading/implementing NTSB guidelines? It isn't like there's some well respected international body like ITAO calling the shots
- tonymet 11mo agoThe older I get , the more I trust people over rules.
- caminanteblanco 11mo agoHere's the attached report, it has a lot of additional helpful information: https://www.ntsb.gov/investigations/Documents/Board%20Summary%20Contact%20of%20Containership%20Dali%20with%20Francis%20Scott%20Key%20Bridge.pdf https://www.ntsb.gov/investigations/Documents/Board%20Summar...
- airstrike 11mo agoOnly tangentially related but the debate over whether the Francis Scott Key bridge is or was a bridge got so heated on Wikipedia that the page had to be protected, and I finally have a reason for bringing this up Edit wars aside, it's a nice philosophical question. https://en.wikipedia.org/wiki/Francis_Scott_Key_Bridge_(Baltimore) https://en.wikipedia.org/wiki/Francis_Scott_Key_Bridge_(Balt...
- caminanteblanco 11mo ago>The seven highway workers and inspector on the Key Bridge at the time were not notified of the Dali’s emergency situation before the bridge collapsed. We found that, had they been notified about the same time the MDTA Police officers were told to block vehicular traffic, the highway workers may have had sufficient time to drive to a portion of the bridge that did not collapse. Further, we found that effective and immediate communication to evacuate the bridge during an emergency is critical to ensuring the safety of bridge workers.
- deleted 11mo ago[deleted]
- mberning 11mo agoThis is a great example of why “small details” matter. How many times do you think an apprentice has been corrected about this? What percentage of the time does the apprentice say “yeah but it’s just a label”. Lots of things went wrong in this case, but if the person that put the label on that wire did it correctly then this whole catastrophe could have been avoided.
- dboreham 11mo agoMy rule for a couple decades: any failover procedure that only gets run when there's a failure, will not work.
- nacozarina 11mo agoI predicted 10yr & $20B to replace it and stand by that forecast.
- timmmmmmay 11mo agoYou're an optimist!
- ROOFLES 11mo agoNon redundant fuel pump that doesn't even restart on power failure. Main engine shutting of when water pressure drops, backup generator not even starting in time AND shoddy wiring that offlines the whole steering system. Thats what i call GOATED engineering. props to Hyundai HI
- aaronmdjones 11mo ago> Non redundant fuel pump that doesn't even restart on power failure The crew weren't using the redundant fuel pumps. They were using the non-redundant fuel line flushing pump as a generator fuel pump, a task it was never designed for and which was not compliant. That it doesn't restart on restoration of power is by design; you don't want to start flushing your fuel lines when the power returns because this could kill your generators and cause another blackout. > Main engine shutting of (sic) when water pressure drops Yeah, this is quite bad. There ought to be an override one can activate in an emergency in order to run the engines to the point of overheating, under the assumption that even destroying the engine will cause less catastrophic consequences than not having propulsion at the time. > backup generator not even starting in time There were 5 generators on board. Generators 1 through 4 are the main generators on the HV bus side, and the emergency backup generator is on the LV bus side. When the incident occurred, the ship was being powered by generators 3 and 4, which were receiving their fuel via the non-redundant fuel line flushing pump. These generators powered the HV bus, which powered the LV bus via a transformer. The emergency backup generator was not running, so the LV bus was only receiving power from the HV bus via 1 transformer. The incident tripped the circuit breaker for this transformer, disconnecting the HV bus from the LV bus, resulting in the first LV bus blackout. This resulted in main engine shutdown (coolant pump failure) and an automatic emergency backup generator startup. There is an alternate (backup) set of circuit breakers and transformer that could have energised the LV bus, but the transformer switches were left in the manual position, so this failover did not happen automatically and immediately. There were no company procedures or regulations which required them to be left in the automatic position. The LV bus also powered the fuel line flushing pump, so this pump failed. As a result, generators 3 and 4 started to fail (being supplied with fuel by a pump which was no longer operating). The electrical management system automatically commanded the start of generator 2 in response to the failing performance of generators 3 and 4. Generator 1 and generator 2 were fed by the standard fuel pumps, which were available. One main generator is capable of powering the entire ship, so there was no need to start generator 1 as well; this would have just put more load on the HV bus (by having to run the fuel pump for generator 1 as well). Instead of the automatic transformer failover (which was unavailable), the crew manually closed the same circuit breaker that had already tripped, 1 minute after the first LV bus blackout. This restored power to the LV bus via the same transformer that was originally powering it, but did not restart the fuel line flushing pump supplying generators 3 and 4 (which were still running, but spinning down because they were being fed fuel via gravity only). This also restored full steering control, but this in itself was inadequate to control the vessel's course without the engine-driven propeller. The main engine was still offline and takes upwards of half a minute to restart, assuming everyone is in place and ready to do so immediately, which was unlikely. The emergency backup generator finally started 10 seconds later (25 seconds too late by requirements, 70 seconds after the first LV bus blackout). Generator 2 had not yet gotten up to speed and connected to the HV bus before generators 3 and 4 disconnected (having exhausted the gravity-fed fuel in the line ahead of the inoperative fuel line flushing pump), resulting in an HV bus blackout and the second LV bus blackout. With only the emergency backup generator running on the LV side, only one-third of steering control was available, but again, this was inadequate without the engine. 3 seconds later, generator 2 connected to the HV bus. 26 seconds later, a crew member manually activated the alternate transformer, restoring power to the LV bus for the second time. The collision was preventable: - It is no longer a requirement that the engine automatically shuts down due to a loss of coolant pressure. It was at the time the vessel was constructed, but this was never re-evaluated. If it were, the system may have been tweaked to avoid losing the engine. - If the transformer switches were left in the automatic position, the LV bus would have switched over to being powered by the second transformer automatically, and the engine coolant pumps and fuel line flushing pump would not have been lost. - Leaving the emergency backup generator running (instead of in standby configuration) would have kept the LV bus energised after the first transformer tripped, and the engine coolant pumps and fuel line flushing pump would not have been lost. - If the crew had opted to manually activate the second transformer within about half a minute (twice as fast as they reactivated the first one), and restarted the fuel line flushing pump, a second blackout would have been avoided, and the engine could have been restarted in time to steer away. This shows the importance of leaving recovery systems armed and regularly training on power transfer procedures. It also illustrates why you shouldn't be running your main generators from a fuel pump which isn't designed for that task. This same pump setup was found on another ship they operated.
- bell-cot 11mo agoWorth noting: The MV Dali is a 1000-foot-long ship, weighing 50% more than a nuclear aircraft carrier, with a total crew of twenty-two. That's everybody - captain, bridge crew, deck crew, cook, etc. So - how many of those 22 will be your engineering crew? How many of those engineers would be on duty, when this incident happened? And once things start going wrong, and you're sending engineers off to "check why Pump #83, down on Deck H, shows as off-line" or whatever - how many people do you have left in the big, complex engineering control room - trying to figure out what's wrong and fix it, as multiple systems fail, in the maybe 3 1/2 minutes between the first failure and when collision becomes inevitable?
- srmatto 11mo agoIf anyone was curious what is happening with the replacement, I just found this website: https://keybridgerebuild.com/ https://keybridgerebuild.com/
- taco_emoji 11mo agoI was very confused by the word "contact" in the headline, which apparently means "crashed the fuck into and killed six people"
- acyou 11mo agoWe should have federal legislation requiring tugboat assist adequate to recover from complete loss of power and steering, through shipping channels that go under bridges supported by mid span support columns. The mechanism should be that if the Coast Guard catches you without a tug, the ship is permanently banned from the port under threat of seizure and repossession by the US federal government, or your vessel just gets immediately seized and held in port under bond. Insurance providers insuring ships in US waters should also be required to permanently deny insurance coverage to vessels found to be out of compliance, though I doubt the insurance companies would want to play ball.
- fluorinerocket 11mo agoI still hate screw terminal blocks. Spring terminals + ferrules are still the way.
- pardon_me 11mo agoClear plastic viewing windows on the spring terminals are the way to go. It allows for both instant feedback for the installer, and visual inspection or troubleshooting later by a third party. The spring terminals should also be designed to have a secondary latch on this type on (what should be) rugged installation. Finally, critical circuits should be designed to detect open connections, and act accordingly. A single hardware<->software design for this could be a module to apply across all such wiring inputs/outputs. This is simple and cheap enough to do these days. A manual tug-test on the physical would be advisable when installing, to check the spring terminal has gripped the conductor when latched.
- fluorinerocket 11mo agoClear case on the terminal blocks could be nice, though might still get a bit tough to see when you have multi-level TBs all mounted in strip on DIN rail. When I used to be involved in control panels I would always yank on all the wires too :-)