5 ms·

Why the U.S. Navy's new $362M ship broke down

19 points by MarlonPro 11y ago

EvanPlaice 11y ago
Aside from the painfully obvious hyperbole... Having a brand new model of a ship break down during an exercise is the 'best case' scenario for failure. There are important aspects of this that should be taken into account. First, nobody died. Second, the fault didn't occur during combat. Yes, there are some projects that waste insane amounts of funding with underwhelming results like the Army's 5 billion UCP (Universal Camouflage Pattern) and Air Force's F35 programs. Cases like those usually occur due to personality-driven leadership. Imagine you have an officer with a lifetime of domain-specific knowledge but little/no experience with product development. Give him an objective, a line of funding, and the responsibility to draft a contract to make it happen. This is the usual case. Things can go south for a few reasons. The officers in charge don't reach out for outside feedback/advice, and default to relying solely on the advice of the contractors. Poor assumptions are made or requirements are overlooked, delaying production. The procurement process takes such a ridiculously amount of time that COTS equipment goes obsolete and/or can't be sourced by the time it's required for implementation. To make matters worse, contracting guidelines dictate that projects follow a waterfall design model. The specification process all happens up-front. Lots of bureaucracy, lots of documentation, lots of steps where mistakes and bad assumptions can/will be made on a tight time schedule. There are exceptions to the rule but generally field testing doesn't take place until after launch. People wonder why the military spends billions of dollars training and 'playing war' every year. They do it to iron out the kinks in their equipment, organization, and processes. If the services do their job (which most do very well) they will identify, report, and correct problems before they're put to the test in a wartime environment. In software development terms, prototype with a waterfall and iterate over time. The difference being, on software projects iterations happen on the scale of weeks/months. On military projects, iterations happen on a scale of years (ex 3 or 5). The news will jump on any sign of weakness chanting 'muh taxpayer dollars'; oldschool politicians like McCain will sponge up the public attention calling for 'the good old days of big iron' (aka sitting ducks); and, none of that matters because war changes and the tools of war need to adapt to overcome those changes. All I can say is, I respect the fortitude of the service members who do this type of work. It's truly a miserable, thankless job, fraught with unbelievable obstacles and little support. Source: I used to work on a contract under MARCORSYSCOM, which is basically the product development division of the US Marines.
Someone1234 11y ago
Stuff failing right out of the gate should be no surprise if you're familiar with the Bathtub curve concept. In particular when it is marketably different from what came before. It might have redundant systems, but if the same manufacturing fault existed in both then that doesn't really help you much. That's why there is something to be said from taking secondary components from different manufacturing runs or even entirely different manufacturers, so that failures aren't synchronized. As an aside: Are ship engines not tested externally before installation into the hull?
- Zancarius 11y ago
  I know it's only tangentially related to your comment, but you reminded me of something Walter Bright discussed about two years ago on the design of the 757: https://news.ycombinator.com/item?id=6639097 https://news.ycombinator.com/item?id=6639097 Relevant portion applicable to your comment: "The computer control systems were dual, meaning two independent computer boards. The boards were designed independently, had different CPU architectures on board, were programmed in different languages, were developed by different teams, the algorithms used were different, and a third group would check that there was no inadvertent similarity." I realize that the liability fear of failure in commercial aircraft (or perhaps regulatory oversight) is probably a strong motivator, but you'd think that combat systems might be equally well-motivated toward redundancy given the nature of their purpose...
ChuckMcM 11y ago
Always amazed that the folks who buy military gear don't get that when you build entirely new systems they break in ways you didn't know they could break. Often they compare them to the current reliability of 20 - 30 year old technology.
- frozenport 11y ago
  It would be wholly unacceptable for a commercial ship to break a week after entering service.
  - Casseres 11y ago
    Former marine engineer here. Not really. Usually a tech rep sails with the ship to work out any kinks. In this instance, this isn't something you can work out at sea. A ship I was on had its shafts pulled during a dry dock evolution. When we went out to sea, we discovered that one of the shafts was off-center and moving the line shaft bearing around causing cracks in its foundation. We had to lock that shaft in place and continue underway on just the other shaft. Having to lock the shaft in place isn't a normal occurance, but it is a contingency that is prepared for.
scurvy 11y ago
Can they not put a magnetic stub on the oil drain plug like motorcycles? ;) Or is the engine made from aluminum?
- dsfyu404ed 11y ago
  I'm sure they could dump a bushel of hard drive magnets over a pipe somewhere but an modern marine engine already has some sort of magnetic filtration. Also, you don't just turn this kind of thing off while you change the oil so a drain plug isn't a realistic option.
- elbrownos 11y ago
  It's not that the metallic debris has caused the failure, but rather it indicates a serious failure has occurred inside the engine.
  - scurvy 11y ago
    Some debris is completely normal when the engine is first being started and breaking in. This is true for all internal combustion engines. That's why some designs use magnets to capture the debris and shavings. That said, this debris was probably more substantial than shavings.
eloff 11y ago
Like the F35 debacle, trying to make something that's good at everything, you end up with something fit for nothing. There's a lot to be said for simpler, fit for specialized purpose designs. The same applies in computer software. Oracle attempts to be a one size fits all database, but as a row store it can never compete with column stores for analytics. As a disk-based, latching, locking, logging database, it can never compete with in memory stores that avoid those overheads. Really just an expensive pile of bytes that claims to solve all your problems, but is fit for nothing.
Confusion 11y ago
Reporting of a complete loss of propulsion [..] is deeply alarming, particularly given this ship was commissioned just 20 days ago It would be deeply alarming if it happened twice a year for every year to come. This is hopefully only a startup problem.
Casseres 11y ago
This Business Insider article is making incorrect speculations. gCaptain (a website focusing on maritime issues) has a more accurate reporting [0]. If this were to occur during a critical mission, they could continue, but at the cost of further damage and a lengthy dry dock afterwards. This was probably caused by misalignment or a piece of debris (which would cause damage and thus create more debris) left in the system during the installation process, not by something inherently wrong with the design. The Navy statement in the gCaptain article mentions locking the port shaft, so this not a problem with an engine (which can be decoupled from the shaft by a clutch). [0] http://gcaptain.com/u-s-navys-new-lcs-towed-to-port-after-breaking-down-off-virginia/ http://gcaptain.com/u-s-navys-new-lcs-towed-to-port-after-br... (I'm a licensed marine engineer, now working in a different industry.)

19 points by MarlonPro 11y ago

Aside from the painfully obvious hyperbole... Having a brand new model of a ship break down during an exercise is the 'best case' scenario for failure. There are important aspects of this that should be taken into account. First, nobody died. Second, the fault didn't occur during combat. Yes, there are some projects that waste insane amounts of funding with underwhelming results like the Army's 5 billion UCP (Universal Camouflage Pattern) and Air Force's F35 programs. Cases like those usually occur due to personality-driven leadership. Imagine you have an officer with a lifetime of domain-specific knowledge but little/no experience with product development. Give him an objective, a line of funding, and the responsibility to draft a contract to make it happen. This is the usual case. Things can go south for a few reasons. The officers in charge don't reach out for outside feedback/advice, and default to relying solely on the advice of the contractors. Poor assumptions are made or requirements are overlooked, delaying production. The procurement process takes such a ridiculously amount of time that COTS equipment goes obsolete and/or can't be sourced by the time it's required for implementation. To make matters worse, contracting guidelines dictate that projects follow a waterfall design model. The specification process all happens up-front. Lots of bureaucracy, lots of documentation, lots of steps where mistakes and bad assumptions can/will be made on a tight time schedule. There are exceptions to the rule but generally field testing doesn't take place until after launch. People wonder why the military spends billions of dollars training and 'playing war' every year. They do it to iron out the kinks in their equipment, organization, and processes. If the services do their job (which most do very well) they will identify, report, and correct problems before they're put to the test in a wartime environment. In software development terms, prototype with a waterfall and iterate over time. The difference being, on software projects iterations happen on the scale of weeks/months. On military projects, iterations happen on a scale of years (ex 3 or 5). The news will jump on any sign of weakness chanting 'muh taxpayer dollars'; oldschool politicians like McCain will sponge up the public attention calling for 'the good old days of big iron' (aka sitting ducks); and, none of that matters because war changes and the tools of war need to adapt to overcome those changes. All I can say is, I respect the fortitude of the service members who do this type of work. It's truly a miserable, thankless job, fraught with unbelievable obstacles and little support. Source: I used to work on a contract under MARCORSYSCOM, which is basically the product development division of the US Marines.

Someone1234 11y ago

Stuff failing right out of the gate should be no surprise if you're familiar with the Bathtub curve concept. In particular when it is marketably different from what came before. It might have redundant systems, but if the same manufacturing fault existed in both then that doesn't really help you much. That's why there is something to be said from taking secondary components from different manufacturing runs or even entirely different manufacturers, so that failures aren't synchronized. As an aside: Are ship engines not tested externally before installation into the hull?

Zancarius 11y ago
I know it's only tangentially related to your comment, but you reminded me of something Walter Bright discussed about two years ago on the design of the 757: https://news.ycombinator.com/item?id=6639097 https://news.ycombinator.com/item?id=6639097 Relevant portion applicable to your comment: "The computer control systems were dual, meaning two independent computer boards. The boards were designed independently, had different CPU architectures on board, were programmed in different languages, were developed by different teams, the algorithms used were different, and a third group would check that there was no inadvertent similarity." I realize that the liability fear of failure in commercial aircraft (or perhaps regulatory oversight) is probably a strong motivator, but you'd think that combat systems might be equally well-motivated toward redundancy given the nature of their purpose...

ChuckMcM 11y ago

Always amazed that the folks who buy military gear don't get that when you build entirely new systems they break in ways you didn't know they could break. Often they compare them to the current reliability of 20 - 30 year old technology.

frozenport 11y ago
It would be wholly unacceptable for a commercial ship to break a week after entering service.
- Casseres 11y ago
  Former marine engineer here. Not really. Usually a tech rep sails with the ship to work out any kinks. In this instance, this isn't something you can work out at sea. A ship I was on had its shafts pulled during a dry dock evolution. When we went out to sea, we discovered that one of the shafts was off-center and moving the line shaft bearing around causing cracks in its foundation. We had to lock that shaft in place and continue underway on just the other shaft. Having to lock the shaft in place isn't a normal occurance, but it is a contingency that is prepared for.

scurvy 11y ago

Can they not put a magnetic stub on the oil drain plug like motorcycles? ;) Or is the engine made from aluminum?

dsfyu404ed 11y ago
I'm sure they could dump a bushel of hard drive magnets over a pipe somewhere but an modern marine engine already has some sort of magnetic filtration. Also, you don't just turn this kind of thing off while you change the oil so a drain plug isn't a realistic option.
elbrownos 11y ago
It's not that the metallic debris has caused the failure, but rather it indicates a serious failure has occurred inside the engine.
- scurvy 11y ago
  Some debris is completely normal when the engine is first being started and breaking in. This is true for all internal combustion engines. That's why some designs use magnets to capture the debris and shavings. That said, this debris was probably more substantial than shavings.

eloff 11y ago

Like the F35 debacle, trying to make something that's good at everything, you end up with something fit for nothing. There's a lot to be said for simpler, fit for specialized purpose designs. The same applies in computer software. Oracle attempts to be a one size fits all database, but as a row store it can never compete with column stores for analytics. As a disk-based, latching, locking, logging database, it can never compete with in memory stores that avoid those overheads. Really just an expensive pile of bytes that claims to solve all your problems, but is fit for nothing.

Confusion 11y ago

Reporting of a complete loss of propulsion [..] is deeply alarming, particularly given this ship was commissioned just 20 days ago It would be deeply alarming if it happened twice a year for every year to come. This is hopefully only a startup problem.

Casseres 11y ago

This Business Insider article is making incorrect speculations. gCaptain (a website focusing on maritime issues) has a more accurate reporting [0]. If this were to occur during a critical mission, they could continue, but at the cost of further damage and a lengthy dry dock afterwards. This was probably caused by misalignment or a piece of debris (which would cause damage and thus create more debris) left in the system during the installation process, not by something inherently wrong with the design. The Navy statement in the gCaptain article mentions locking the port shaft, so this not a problem with an engine (which can be decoupled from the shaft by a clutch). [0] http://gcaptain.com/u-s-navys-new-lcs-towed-to-port-after-breaking-down-off-virginia/ http://gcaptain.com/u-s-navys-new-lcs-towed-to-port-after-br... (I'm a licensed marine engineer, now working in a different industry.)