8 ms·

A dozen or so [maintenance issues] were considered more serious. They included problems with the ship’s primary navigation system. It was the oldest such system

A dozen or so [maintenance issues] were considered more serious. They included problems with the ship’s primary navigation system. It was the oldest such system among destroyers based in Japan. It was running on Windows 2000, even though other ships had been upgraded. It could not display information from the AIS. The broken email system had a “major impact” on the ship’s day-to-day operations. Microsoft Outlook did not work. Nor could commanders communicate over a classified email system... Technicians were constantly fixing the SPS-73, the other main navigational radar on the Fitzgerald. Sometimes, the radar would show the destroyer heading the wrong way. At other times, it simply locked up and would have to be shut down... A third radar, used for warfare, was slow to acquire targets, but technicians had installed a temporary fix that became permanent. “Problem known since 2012. Declared hopeless,” read notes attached to the repair report... Other equipment had been written off, too. The so-called Bright Bridge console was supposed to help the bridge crew by sharing information from the combat room. The console had been scavenged for spare parts, leaving the station unmanned. I found this section of the article legitimately infuriating. How the hell does a first-line warship of the United States Navy get into a state like this? I can understand the occasional malfunction, but this sounds less like that and more like some sad-sack Russian ship circa 1995, where the crew is struggling to keep the thing afloat with just duct tape and bubble gum. But at least the Russians in the '90s had the excuse that the government that had built the ship had collapsed and the replacement hadn't bothered spending a dime to maintain it. What's our Navy's excuse? Based on the article it sounds like the excuse they've settled on is just blaming everything on the crew, which is shameful. Things don't get this bad in the absence of some kind of systemic rot. If that rot isn't tackled now it'll end up having to be tackled after we lose some future war, in which case, God help us all.

jessaustin 8y ago
Microsoft Outlook did not work. Nor could commanders communicate over a classified email system... A functioning local email system requires very little resources. This is really an indictment of M$, that they couldn't set up a server that would just run for a couple of decades. If sendmail had been installed in the mid-90s when DDG-62 was commissioned, it probably would still be functioning. Yes we know that it would be better to receive regular software updates, but in their absence the system shouldn't just shit the bed.
- brianpgordon 8y ago
  There are many things that could have gone wrong. Maybe they ran out of disk space. Maybe some critical certs expired.
  - jessaustin 8y ago
    Those are issues that would arise within a couple of years, not over 20 years later. If the ship doesn't have anyone qualified to administer computers, then archiving etc. would have to be automated. If the ship used to have such personnel and no longer does, that is certainly a problem at a higher level than the captain.
  - Neil44 8y ago
    I was wondering if the previous power issues mentioned had borked the exchange database. Still, thats a day or two to fix, not weeks.
- da_chicken 8y ago
  > This is really an indictment of M$ No, it really isn't. For two decades all across the world people have kept Outlook running well enough to function in all sorts of scenarios, including those more difficult than a military ship at sea. It's not like Thunderbird and Alpine are magically immune to having corrupted data stores and invalid configurations. This is entirely due to lack of proper maintenance.
  - patrickg_zill 8y ago
    There are phone switches that were made in 1982 that are still running phone service, including 911 calling, for mid-sized towns. And yes, postfix or Exim or qmail, once set up, in a fixed configuration like boat only email, could easily run for decades automatically.
- pjc50 8y ago
  Poor choice perhaps, given how many security holes there were in Sendmail. I have a sneaking suspicion that the failure was something generic like running out of disk space, which nobody had both the authority and time to fix.
crushcrashcrush 8y ago
Corruption my friend. Direct from a deployed friend's mouth: "When laser printers would run out of toner on base in Iraq, we wouldn't order more toner (we couldn't) - we'd order another entire printer and use the toner cartridge in the new one, and shoot at the printer on the range." Your tax dollars at work.
- csours 8y ago
  On the one hand, yes that's terrible. On the other hand, I can totally see several scenarios where this is a reasonable outcome of an unreasonable system: 1. Toner not in order system. 2. Previous orders of toner found to be incompatible with printer. etc.
- genkimind 8y ago
  It has nothing to do with corruption. See my other comment.
  - module0000 8y ago
    Hanlon's razor at work... "Never attribute to malice that which is adequately explained by stupidity"
    - darkpuma 8y ago
      But never forget that malice does exist, and the clever malicious will often conceal their malice behind a veil of plausible stupidity. For instance, "Oops I accidentally forgot my briefcase full of classified documents on the bus" is as old as spycraft itself.
      - jjoonathan 8y ago
        Even calling it "clever" or attributing it to spycraft is a stretch. Kids learn to "play dumb" before they learn to form complete sentences.
        8y ago
deleted 8y ago
[deleted]
dragonwriter 8y ago
> How the hell does a first-line warship of the United States Navy get into a state like this? Command negligence at very high levels, a high optempo without adequate attention to readiness and discipline. > Based on the article it sounds like the excuse they've settled on is just blaming everything on the crew Unless you are including the chain of command above the ship's commander up through and including the 7th Fleet commander as “the crew”, you are wrong. The cluster of 7th Fleet incidents (the Antietam grounding and the Fitzgerald and McCain collisions) resulted in at least the dismissal of the 7th Fleet commander, the Task Force 70 commander, and the Destroyer Squadron 15 commander, and the early retirement of the head of Naval Surface Forces.
- smacktoward 8y ago
  > The cluster of 7th Fleet incidents (the Antietam grounding and the Fitzgerald and McCain collisions) resulted in at least the dismissal of the 7th Fleet commander, the Task Force 70 commander, and the Destroyer Squadron 15 commander, and the early retirement of the head of Naval Surface Forces. This is good to hear, thank you! I'm glad that accountability for these kinds of systemic problems doesn't stop at the captain's chair.
- dv_dt 8y ago
  Yup, from the article: The Navy inquiries determined that there had been widespread problems with leaders regarding shortfalls in training, manning and equipment in the 7th Fleet. The Navy fired admirals, captains and commanders, punished sailors and criminally prosecuted officers for neglecting their duties. Adm. John Richardson, head of the Navy, called the two collisions “avoidable tragedies.” The ships’ commanders and their superiors, he said in a written statement to ProPublica, were responsible for the results. “The tragedies of USS Fitzgerald and USS John S. McCain reminded us that all commanders, from the unit level to the fleet commander, must constantly assess and manage risks and opportunities in a very complex and dynamic environment,” Richardson said. “But at the end of the day, our commanders make decisions and our sailors execute and there is an outcome — a result of that decision. The commander ‘owns’ that outcome.” As an aside, the use of illustrations in this article are really great, from the collision graphics, to the busy sea routes.
genkimind 8y ago
I spent 5+ years on a ship in 7th fleet and am all too familiar with what was described above. I would attribute it to a few things. Mostly the over-extension of resources and people. The operational tempo in 7th fleet is absolutely ridiculous, so everything that isn't "mission accomplishment" gets pushed to the wayside. Training, making sure equipment is properly fixed.. our days were so jam packed full of bullshit that this important stuff suffered. I considered myself lucky if I got 5 hours of unbroken sleep a night while at sea, and most of the people I worked with were the same way. Walking around functionally drunk doesn't lend itself to good problem solving skills. Do you know after these incidences there was massive effort to train people about the importance of sleep? Guess what, this mandatory training forced people to interrupt their sleep to attend. All in all 7th fleet suffers badly from undermanning based on the operational requirements. Ask anybody. AMA I guess.
- lainga 8y ago
  Would you say this is primarily a resource problem? A culture problem with the 7th Fleet? A culture problem in the USN overall?
  - crooked-v 8y ago
    "Every enlisted man in the Navy only gets 4-5 hours of a sleep a night" seems to be a common anecdote service-wide, including from people I know personally who served.
    - kryogen1c 8y ago
      Also, "nights" are not a real thing, not in the biological sense. In the surface ship nuclear power plant world where there is no air conditioning, you can't stand watch for more than 5 hours, which means 4 5s and one 4 hour watch. Think you stand the same watch every day? Nope, not enough people. You normally stand 1 on 2 off (5 and dimes) or one on 3 off (5 and 15s). This puts your sleep window at different spots every night. Subs are air conditioned, so they can stand either 6 and 6s or less commonly 6 and 12s. Their lives are generally worse in every way. Except they all like smelling each other farts and sharing beds, so it works out.
pjc50 8y ago
> Russians in the '90s had the excuse that the government that had built the ship had collapsed and the replacement hadn't bothered spending a dime to maintain it. It sounds like this is uncomfortably close to the truth about the US situation too; the wars in Iraq and Afghanistan had unlimited money to waste, but this must have been at the cost of other parts of the military. And the political system is increasingly dysfunctional above it.
nradov 8y ago
Budget sequestration was a factor. Ships require increasing maintenance as they age, but the military budget was flat or declining.
mysterydip 8y ago
Most of those systems probably had upgrades or fixes, but due to a combination of availability, funding, or politics, did not get implemented.
rangibaby 8y ago
Both sides have an incentive to over-engineer. The arms manufacturer wants to add as many systems as possible because more complexity = more money. The navy as a whole might want to save money, but the parts that make it up want to spend as much money as possible; if they don't, their budget will get cut since they don't "need" the resources allocated to them. What people sometimes forget about crude Russian engineering is that bubblegum and toothpicks are enough to get the thing operational. Simplicity was always the Russian priority because even in the Soviet days they were broke, and even if they had the money to buy parts getting them to a frozen or muddy air(literal)field in the middle of the middle of nowhere was a feat in itself. The US military-industrial complex has functionally unlimited resources, so the things they build require unlimited resources to operate and function.
sonnyblarney 8y ago
And why are fighting ships using MS Exchange? And not some more hardened linux-based, simple UI/system thing that's easier to admin and keep up?
- ForHackernews 8y ago
  If I had to speculate, I'd say most likely because there are zero Linux vendors willing to run the gauntlet of a DoD procurement process.
  - velosol 8y ago
    There's at least Red Hat according to: http://www.esi.mil/AgreementList.aspx?type=2 http://www.esi.mil/AgreementList.aspx?type=2
  - sonnyblarney 8y ago
    I was in the Canadian Army, and buddies in procurement in Ottawa, so I'm roughly familiar. And agree with your point. My question was more rhetorical :) You know what DARPA should be doing? Instead of trying to make 'smart bullets' maybe they should consider that complexity and security of IT is a goddam problem and support products instead of just technology. Someone to make a version of Linux, hardended/simplified it, and then did the same for a few other pieces (i.e. networking gear) and then made 'mil grade' versions of a bunch of things. And then teach every single member of the military how to do basic stuff, i.e. 'know their computer'. Super basic 'MILINUX' skills, so that weapons systems techs could at least fathom what on earth is wrong with their thing and possibly fix it. Same with mail. There is no reason in 2018 that e-mail can't be durable and available for relatively low cost in some system. Also consider now that there should be a lot more IT dudes in the forces.
Tsubasachan 8y ago
Nothing new really. You should read up on the state of the Navy in 1941 just before Pearl Harbour. Or just before Korea in 1950. Armies are in a boom and bust cycle. When the last war is over they lose their funding and priority over other stuff. Until the next war begins.
- davidgould 8y ago
  Except the US military never lost priority after WW2.

jessaustin 8y ago

Microsoft Outlook did not work. Nor could commanders communicate over a classified email system... A functioning local email system requires very little resources. This is really an indictment of M$, that they couldn't set up a server that would just run for a couple of decades. If sendmail had been installed in the mid-90s when DDG-62 was commissioned, it probably would still be functioning. Yes we know that it would be better to receive regular software updates, but in their absence the system shouldn't just shit the bed.

brianpgordon 8y ago
There are many things that could have gone wrong. Maybe they ran out of disk space. Maybe some critical certs expired.
- jessaustin 8y ago
  Those are issues that would arise within a couple of years, not over 20 years later. If the ship doesn't have anyone qualified to administer computers, then archiving etc. would have to be automated. If the ship used to have such personnel and no longer does, that is certainly a problem at a higher level than the captain.
- Neil44 8y ago
  I was wondering if the previous power issues mentioned had borked the exchange database. Still, thats a day or two to fix, not weeks.
da_chicken 8y ago
> This is really an indictment of M$ No, it really isn't. For two decades all across the world people have kept Outlook running well enough to function in all sorts of scenarios, including those more difficult than a military ship at sea. It's not like Thunderbird and Alpine are magically immune to having corrupted data stores and invalid configurations. This is entirely due to lack of proper maintenance.
- patrickg_zill 8y ago
  There are phone switches that were made in 1982 that are still running phone service, including 911 calling, for mid-sized towns. And yes, postfix or Exim or qmail, once set up, in a fixed configuration like boat only email, could easily run for decades automatically.
pjc50 8y ago
Poor choice perhaps, given how many security holes there were in Sendmail. I have a sneaking suspicion that the failure was something generic like running out of disk space, which nobody had both the authority and time to fix.

crushcrashcrush 8y ago

Corruption my friend. Direct from a deployed friend's mouth: "When laser printers would run out of toner on base in Iraq, we wouldn't order more toner (we couldn't) - we'd order another entire printer and use the toner cartridge in the new one, and shoot at the printer on the range." Your tax dollars at work.

csours 8y ago
On the one hand, yes that's terrible. On the other hand, I can totally see several scenarios where this is a reasonable outcome of an unreasonable system: 1. Toner not in order system. 2. Previous orders of toner found to be incompatible with printer. etc.
genkimind 8y ago
It has nothing to do with corruption. See my other comment.
- module0000 8y ago
  Hanlon's razor at work... "Never attribute to malice that which is adequately explained by stupidity"
  - darkpuma 8y ago
    But never forget that malice does exist, and the clever malicious will often conceal their malice behind a veil of plausible stupidity. For instance, "Oops I accidentally forgot my briefcase full of classified documents on the bus" is as old as spycraft itself.
    - jjoonathan 8y ago
      Even calling it "clever" or attributing it to spycraft is a stretch. Kids learn to "play dumb" before they learn to form complete sentences.
      - 8y ago

deleted 8y ago

[deleted]

dragonwriter 8y ago

> How the hell does a first-line warship of the United States Navy get into a state like this? Command negligence at very high levels, a high optempo without adequate attention to readiness and discipline. > Based on the article it sounds like the excuse they've settled on is just blaming everything on the crew Unless you are including the chain of command above the ship's commander up through and including the 7th Fleet commander as “the crew”, you are wrong. The cluster of 7th Fleet incidents (the Antietam grounding and the Fitzgerald and McCain collisions) resulted in at least the dismissal of the 7th Fleet commander, the Task Force 70 commander, and the Destroyer Squadron 15 commander, and the early retirement of the head of Naval Surface Forces.

smacktoward 8y ago
> The cluster of 7th Fleet incidents (the Antietam grounding and the Fitzgerald and McCain collisions) resulted in at least the dismissal of the 7th Fleet commander, the Task Force 70 commander, and the Destroyer Squadron 15 commander, and the early retirement of the head of Naval Surface Forces. This is good to hear, thank you! I'm glad that accountability for these kinds of systemic problems doesn't stop at the captain's chair.
dv_dt 8y ago
Yup, from the article: The Navy inquiries determined that there had been widespread problems with leaders regarding shortfalls in training, manning and equipment in the 7th Fleet. The Navy fired admirals, captains and commanders, punished sailors and criminally prosecuted officers for neglecting their duties. Adm. John Richardson, head of the Navy, called the two collisions “avoidable tragedies.” The ships’ commanders and their superiors, he said in a written statement to ProPublica, were responsible for the results. “The tragedies of USS Fitzgerald and USS John S. McCain reminded us that all commanders, from the unit level to the fleet commander, must constantly assess and manage risks and opportunities in a very complex and dynamic environment,” Richardson said. “But at the end of the day, our commanders make decisions and our sailors execute and there is an outcome — a result of that decision. The commander ‘owns’ that outcome.” As an aside, the use of illustrations in this article are really great, from the collision graphics, to the busy sea routes.

genkimind 8y ago

I spent 5+ years on a ship in 7th fleet and am all too familiar with what was described above. I would attribute it to a few things. Mostly the over-extension of resources and people. The operational tempo in 7th fleet is absolutely ridiculous, so everything that isn't "mission accomplishment" gets pushed to the wayside. Training, making sure equipment is properly fixed.. our days were so jam packed full of bullshit that this important stuff suffered. I considered myself lucky if I got 5 hours of unbroken sleep a night while at sea, and most of the people I worked with were the same way. Walking around functionally drunk doesn't lend itself to good problem solving skills. Do you know after these incidences there was massive effort to train people about the importance of sleep? Guess what, this mandatory training forced people to interrupt their sleep to attend. All in all 7th fleet suffers badly from undermanning based on the operational requirements. Ask anybody. AMA I guess.

lainga 8y ago
Would you say this is primarily a resource problem? A culture problem with the 7th Fleet? A culture problem in the USN overall?
- crooked-v 8y ago
  "Every enlisted man in the Navy only gets 4-5 hours of a sleep a night" seems to be a common anecdote service-wide, including from people I know personally who served.
  - kryogen1c 8y ago
    Also, "nights" are not a real thing, not in the biological sense. In the surface ship nuclear power plant world where there is no air conditioning, you can't stand watch for more than 5 hours, which means 4 5s and one 4 hour watch. Think you stand the same watch every day? Nope, not enough people. You normally stand 1 on 2 off (5 and dimes) or one on 3 off (5 and 15s). This puts your sleep window at different spots every night. Subs are air conditioned, so they can stand either 6 and 6s or less commonly 6 and 12s. Their lives are generally worse in every way. Except they all like smelling each other farts and sharing beds, so it works out.

pjc50 8y ago

> Russians in the '90s had the excuse that the government that had built the ship had collapsed and the replacement hadn't bothered spending a dime to maintain it. It sounds like this is uncomfortably close to the truth about the US situation too; the wars in Iraq and Afghanistan had unlimited money to waste, but this must have been at the cost of other parts of the military. And the political system is increasingly dysfunctional above it.

nradov 8y ago

Budget sequestration was a factor. Ships require increasing maintenance as they age, but the military budget was flat or declining.

mysterydip 8y ago

Most of those systems probably had upgrades or fixes, but due to a combination of availability, funding, or politics, did not get implemented.

rangibaby 8y ago

Both sides have an incentive to over-engineer. The arms manufacturer wants to add as many systems as possible because more complexity = more money. The navy as a whole might want to save money, but the parts that make it up want to spend as much money as possible; if they don't, their budget will get cut since they don't "need" the resources allocated to them. What people sometimes forget about crude Russian engineering is that bubblegum and toothpicks are enough to get the thing operational. Simplicity was always the Russian priority because even in the Soviet days they were broke, and even if they had the money to buy parts getting them to a frozen or muddy air(literal)field in the middle of the middle of nowhere was a feat in itself. The US military-industrial complex has functionally unlimited resources, so the things they build require unlimited resources to operate and function.

sonnyblarney 8y ago

And why are fighting ships using MS Exchange? And not some more hardened linux-based, simple UI/system thing that's easier to admin and keep up?

ForHackernews 8y ago
If I had to speculate, I'd say most likely because there are zero Linux vendors willing to run the gauntlet of a DoD procurement process.
- velosol 8y ago
  There's at least Red Hat according to: http://www.esi.mil/AgreementList.aspx?type=2 http://www.esi.mil/AgreementList.aspx?type=2
- sonnyblarney 8y ago
  I was in the Canadian Army, and buddies in procurement in Ottawa, so I'm roughly familiar. And agree with your point. My question was more rhetorical :) You know what DARPA should be doing? Instead of trying to make 'smart bullets' maybe they should consider that complexity and security of IT is a goddam problem and support products instead of just technology. Someone to make a version of Linux, hardended/simplified it, and then did the same for a few other pieces (i.e. networking gear) and then made 'mil grade' versions of a bunch of things. And then teach every single member of the military how to do basic stuff, i.e. 'know their computer'. Super basic 'MILINUX' skills, so that weapons systems techs could at least fathom what on earth is wrong with their thing and possibly fix it. Same with mail. There is no reason in 2018 that e-mail can't be durable and available for relatively low cost in some system. Also consider now that there should be a lot more IT dudes in the forces.

Tsubasachan 8y ago

Nothing new really. You should read up on the state of the Navy in 1941 just before Pearl Harbour. Or just before Korea in 1950. Armies are in a boom and bust cycle. When the last war is over they lose their funding and priority over other stuff. Until the next war begins.

davidgould 8y ago
Except the US military never lost priority after WW2.