14 ms·

Navy’s flawed technology set the USS John McCain up for disaster

194 points by thomasjudge 7y ago

FlyMoreRockets 7y ago
Very sobering article. Makes me reconsider the proliferation of glass cockpits and fly by wire in aviation. Granted, there is no putting the genie back in the bottle, but failsafe and redundancy need to be a central design tenet from the start. Back in the 80's, I was throttleman aboard a similar sized USN ship. There was no direct link from the bridge to the throttle. Throttle commands were communicated via an indicator and the throttle operator manually turned a big wheel to meet the command. I remember thinking at the time there should be direct throttle control from the bridge. Hindsight is 20:20.
- scrumper 7y ago
  Looks like quite a lot of redundancy here - you can split up controls over various stations really in kind of any way you want. That being the problem. The article led me to think the crash was more a confluence of factors: staff shortages, high operational demands, culture of tolerating cursory training, and a very complex new system that was both substantially harder to understand than traditional fixed stations, and constantly changing. The complexity of the IBNS system seems to be the ultimate culprit, in that the various members of the stressed and tired bridge crew were each operating under a different mental model of the state of the system. Because of IBNS's flexibility, there were too many possible states for a crew to reason about under pressure. Software makes it very easy for states to proliferate, but there hasn't been enough research (nor, more soberly, enough spilled blood) yet around designing flexible systems which are easily comprehensible in urgent situations. Of course this complexity does cause problems in aviation, even with that culture's extreme emphasis on rigorous training, procedure, and human factors. AF 447, for example.
- magduf 7y ago
  Glass cockpits and FBW are great, in theory. In practice, I don't know because I haven't had to use it. However, don't make the mistake of assuming that just because the US Navy can't get a well-engineered system put together means that Boeing and Airbus are guilty of the same. (Their glass-cockpit systems come from other vendors, who also supply to smaller aircraft makers BTW.) Also, how many foreign militaries have had problems like this with basic ship steering? From what I can tell, the US Navy is uniquely broken here. UK, Australia, Japan, France etc. do not have have their ships plowing into slow cargo ships. Commercial aircraft are not flying into the ground left and right. (The 737MAX is pretty bad, but that's not even related to glass cockpits; MCAS was a Boeing-engineered component, not a glass cockpit sourced from some vendor like Thales.)
  - nwallin 7y ago
    > Also, how many foreign militaries have had problems like this with basic ship steering? > From what I can tell, the US Navy is uniquely broken here. UK, Australia, Japan, France etc. do not have have their ships plowing into slow cargo ships. Norway lost a destroyer last year after colliding with a tanker. Keep in mind the US Navy is larger than the next 13 largest navies combined. 2 of 3 of these events happening to the US Navy isn't statistically significant enough to say it's a uniquely US Navy problem. Not to say they don't need to get their shit together.
schwanksta 7y ago
Not to be overlooked: ProPublica actually constructed a 3D version of the navigation system's control board: https://www.propublica.org/article/how-we-reconstructed-the-flawed-navigation-controls-behind-the-navy-worst-maritime-accident-in-40-years https://www.propublica.org/article/how-we-reconstructed-the-...
- mikeyouse 7y ago
  Couldn't find it in your article, but it looks like the 3D models are integrated into the story here: https://features.propublica.org/navy-uss-mccain-crash/navy-installed-touch-screen-steering-ten-sailors-paid-with-their-lives/ https://features.propublica.org/navy-uss-mccain-crash/navy-i...
crmrc114 7y ago
In a panic people get tunnel vision. When you design HMI's for a living you have to consider. If my brother, sister, grandmother was using this system during an emergency could we guide them to make the right decisions. The fact that the prop controls could be split but the UI would not gray out the sliders on the screens that did not have control is a poor user interface design choice. If the controls only appeared "active" on the screen they were transferred too this could have made things much more simple. The fact that the mushroom button for emergency steering yanked control to wherever it was pressed- this should have displayed a red on black box. Emergency Override called from X Station. The issues with the navigation system are a whole other bag of worms- who knows what the mess was there. Disclosure, I have done HMI design and install in production systems for manufacturing, chemical and warehousing operations. I have never built anything for a ship or vehicle. So I may be speaking total stupid to someone in that industry. I would love to hear someone from that market sector drop in and talk about how or why the HMI was setup like this. Sounds like this system was sub-contracted to the lowest bidder and pushed out without enough testing.
- rwmurrayVT 7y ago
  Integrated bridges are being installed all across USN ships, including Mt. Whitney with MSC right now. They're filled with a hodgepodge of HMIs that are all linked to a control system filled with gyros, ECDIS, RADAR, propulsion, etc. etc. System integration is complex and the design work is done by contracted companies. You don't have the first party vendor doing the integration work. It's a recipe for disaster.
  - jandrese 7y ago
    > You don't have the first party vendor doing the integration work. The only reason this should be true is if the Navy is directly providing this service instead. If nobody is coordinating integration, then that's absolute insanity. Integration is the most important job in a distributed system like this, also probably the hardest. Knowing the military they probably thought they could avoid needing an integrator by writing out tens of thousands of pages of specs, never stopping to consider that the specs themselves may be incorrect or unworkable or that they would be so complex that no vendor could ever deliver a product that exactly meets the specs.
remarkEon 7y ago
Absolutely phenomenal reporting. Take the time and read the entire thing in detail. It’s worth it. Can any Navy vets comment or speak to these systems? I really worry about the proliferation of these digital interfaces. I’m curious to know how folks feel about them.
- linuxftw 7y ago
  I can't speak to that specific system, but I can give you plenty of other anecdotes about the systemic unreadiness of the US Navy. I was forced to be the on-duty electrician from time to time, despite not being an ship's electrician. One of the responsibilities for the on-duty electrician was to cut the right circuits in case of a compartment fire. Despite my repeated protests that I didn't know what I was doing and needed additional training, I was forced to be the on-duty electrician. Mind you, there were no other duties I had to perform, just that one, and just in case of a fire, so it's not like my every-day workload was increased. Determined not to get people killed in the event of an emergency, I asked the actual electricians what I should be doing. Turns out, they didn't know either. Their plan was, in case of fire, kill power to the whole ship. I don't think I need to tell you, this is exactly the wrong move to make. There's an ever-growing number of electronic systems onboard ships, and an ever-shrinking number of sailors to run them. Qualifications are a joke in pretty much every department, at least in the surface fleet. Ships are in a constant state of disrepair, it's not uncommon for 30-50% of the communications equipment to be broken at any given time, parts taking months to receive (mostly due to budget restraints). Sailors are routinely sleep deprived. It's not uncommon for a sailor to be up by 6AM, work all day, and then be steering the ship from 00:00 to 04:00, with no sleep, followed by maybe 2 hours of sleep before having to work again the next day, and possibly have another watch from 20:00-12:00. Rinse repeat.
  - jandrese 7y ago
    That last paragraph screams of a massive failure in Crew Resource Management. There are acres of case studies about this subject throughout industry, and when people take them to heart lots of accidents are avoided. Airline pilots are a prime example. How many airliners crash these days? CRM is one of the factors in making them so safe. Truck Drivers now have similar requirements and a much improved safety record as well. This is literally life or death, industries that ignore it are being unnecessarily reckless and unprofessional.
deleted 7y ago
[deleted]
waterside81 7y ago
> In 2014, Navy officials discovered a flaw in the IBNS. One component could not keep track of more than 150 ships at a time without malfunctioning, according to Navy investigators. The Navy’s solution? Sailors were told to delete tracked ships before the total hit the magic number. This blows my mind. Can anyone here guess what's going on under the hood? Is this a magic number that a developer came up during testing to avoid running of out memory/swapping?
- klodolph 7y ago
  Just a guess—probably designed using high-reliability / realtime principles. This might involve e.g. allocating memory only at startup, which would explain things. The limit has to be some number when programming this way.
  - magduf 7y ago
    In a real-time safety-critical OS, you absolutely do allocate memory only at startup, but this kind of OS usually isn't used for any kind of GUI interface, it's used for much simpler systems. Another trait of these systems is that the CPU caches are all disabled, because those prevent determinism. But again, this isn't something you'd do on a GUI system; it's something you'd do on an ABS brake controller or an engine ECU, where the amount of memory ever needed is easy to determine in the design.
    - vonmoltke 7y ago
      This is also done in the signal processors that power these systems. I don't know this system in particular, but the USN has been moving towards commodity computing hardware for these systems. Essentially, a rack of commercial servers (usually IBM) running stripped-down RHEL with a real-time kernel. The software for these systems is firm real-time, and generally uses the same principals as other real-time software (including, but not limited to, up-front allocation).
      - jki275 7y ago
partiallypro 7y ago
This reminds me of "Pentagon Wars."
shitgoose 7y ago
Tried scrolling article up and down. Animated images hijacking the mouse control randomly, showing some motion then freezing. A good illustration.
- stefan_ 7y ago
  This is the latest innovation in "interactive story telling". It is just the ultimate in irony: here is an article crying murder because a touch interface allowed a sailor to un-gang engine power, then slide a big fat green power slider for one engine to half of what the other big green slider for the other engine directly next to it showed. Meanwhile here we are furiously scrolling to hit invisible "scrolled distance" breakpoints to advance a slideshow.
  - ceejayoz 7y ago
    It's even worse if you're used to using spacebar to page-down.
    - aidenn0 7y ago
      Meh, I use spacebar to page-down, but am ready to fall-back at a moments notice because so many sites draw their header over the body, so a spacebar will end up by scrolling by more than a page...
  - thatswrong0 7y ago
    The websites that pull this off "well" do so by giving you feedback that your scrolling is still doing _something_. For example, Apple pulls this off reasonably in their marketing pages by moving things in the background: https://www.apple.com/iphone-11-pro/ https://www.apple.com/iphone-11-pro/ This one just sticks and does nothing, which is exceptionally awkward and feels broken.
- mkl 7y ago
  It's even worse on mobile, locking the page up entirely with Javascript. I assumed it was untested and broken there, and shifted to desktop, but it's broken here too.
unlinked_dll 7y ago
Something unclear from the article, but jumped out at me. It seems that the Achilles heel of the IBNS system was that too much information tended to bork it, but what’s unclear to me (as implied by the article) is if dropping incoming data was actually the sensible thing to do (as is often the case) and if incoming data was prioritized over existing data and that was dropped instead. The other thing I would point out is that the technical failure is not that a system is complex, but that a complex system has a user interface that allows a sequence of actions to leave it in an invalid or unpredictable state. The transfer of throttle control being incomplete while other actions were taken, for example. I think the thing lacking here was consistent and sound changes to the state, lack of fuzzing user input to observe this possibility, and the reliance on training to avoid the issues at all. These are problems that can be solved with design.
- iudqnolq 7y ago
  I haven't worked on any project anywhere approaching significant complexity, but the magic number of 150 supported tracked ships sounds insanely low to me. Is this at all reasonable, or does it indicate the design and implementation were shit? Does anyone know how much processing is needed here? Is it just updating an internal model of locations and warning about impending collisions? Presuming constant speed and heading that sounds easy, not presuming it sounds not feasible even for just a handful of ships. Also, if only 150 ships are supported, wouldn't it makes sense for the computer to automatically untrack them? Maybe allow manual prioritization of what to drop first, but surely within a few days of a competent vendor learning adding too many items borked their system they would put out a patch that does the absolute minimum by automatically preventing adding too many items? > In 2014, Navy officials discovered a flaw in the IBNS. One component could not keep track of more than 150 ships at a time without malfunctioning, according to Navy investigators. The Navy’s solution? Sailors were told to delete tracked ships before the total hit the magic number. > The navigation system could also become overloaded if too much information streamed in from a ship tracking database used worldwide to prevent maritime collisions. The Navy’s second solution was similar to the first: Drop the feed when it became too much. > They were patches on top of patches that left the Navy’s destroyers without a full picture of the seas around them. But none of the problems was serious enough to attract high-level attention. A Navy system designed to track problems in major ship systems did not contain any reports that mentioned the IBNS until last year, according to a Navy official.
userbinator 7y ago
He ordered Bordeaux to take over steering the warship while another sailor controlled its speed. The idea was to avoid distractions by having each man focus on a single task in the heavy maritime traffic. I've not controlled anything that floats bigger than a small fishing boat, and I realise the timescales of reaction here are much larger, but that just sounds wrong to me --- changes in speed can affect direction, and vice-versa. Splitting that to two separate brains that are communicating only via a low-bandwidth voice channel seems like a recipe for disaster.
- exabrial 7y ago
  I'm curious why these things are manually controlled at all. Seems like you should give it a heading and it goes that direction, self-correcting along the way.
  - davengh 7y ago
    Because sometimes he "wrong" action is the right thing to do. No automated system can anticipate everything. Think the NASCAR "drive at the wreck in front of you" - by the time you get there it will have moved. Especially on a ship that might have to spring from "situation normal" to active combat (maneuver/attack/defend all at once) without a pre-knowable trigger condition.
- TeMPOraL 7y ago
  I've not controlled anything that floats bigger than a pedal boat, but to the extent sci-fi shows and navy fictions show can teach me anything, I've always had the impression that the direction and speed of ships are controlled from two different stations on the bridge.
  - dredmorbius 7y ago
    There was a time when the wheelhouse and the bridge were separated, at least in some ships and navies. See this 1941 British Navy training film: https://invidio.us/watch?v=ocUD07fvf48 https://invidio.us/watch?v=ocUD07fvf48
allovernow 7y ago
>Immediate responsibility, the Navy ruled, rested with Sanchez, his officers and senior sailors. They had been lax, even complacent, in their training of the sailors steering the ship. Sanchez had made a critical error in not adding more sailors to stand watch as the McCain navigated the treacherous strait From what I read consistently about the Navy, and hear from veteran friends, any sailors they could have asked to stand watch would have already been overworked and under rested. Everything I read about the U.S. military, and especially the NAVY, seems to indicate a broken culture of bloat and incompetence from the top down. I think we've been too comfortable as the premier unchallenged world superpower and the services have started to rot from within.
- magduf 7y ago
  It's not just the Navy or military; all kinds of things are breaking down in this country, though most people are completely in denial. Just look at how this country compares to others in terms of: obesity, healthcare costs per capita, transportation infrastructure, housing costs, gap between rich and poor, gun-related deaths, I could go on and on.
  - pathseeker 7y ago
    >obesity, healthcare costs per capita Yep, healthcare is garbage. >transportation infrastructure This is subjective. Most Americans care more about roads than public transportation. >housing costs In all but the most expensive metros, US housing is way cheaper per sq ft than most Western European nations. This is partly due to urban sprawl due to the love affair with cars, but it is cheaper. >gap between rich and poor Richer middle class than most other countries. Just because the US is a good place for business and a bad place to be poor says very little about the average experience. >gun-related deaths Yep, that comes with gun ownership. Many people are knowingly making that trade-off. You'd want to compare murder/suicide rates with other countries for any meaningful comparisons.
    - aqme28 7y ago
      > Yep, that comes with gun ownership. Many people are knowingly making that trade-off. You'd want to compare murder/suicide rates with other countries for any meaningful comparisons. And our murder rate is pretty bad, likely thanks to our extremely high rate of gun violence.
LifeLiverTransp 7y ago
Very soon every ship station will come with a miniature tutorial to bring even a layman up to speed on how to steer a destroyer. I cant wait for a the first school class who hijacks a military boat listening to youtube videos while singing pirat shantys.
_Nat_ 7y ago
This story's worrisome in that it makes the Navy sound very vulnerable to electronic warfare. I mean, if overall operations are so hazard-prone that they're not reliable in the absence of hostilities, how could they possibly be resilient against an advanced actor? Is the situation as bad as it sounds from the story?
jschwartzi 7y ago
It strikes me that at least 16 seconds of screwing around could have been saved by changing the caption of the "Emergency Override - To Manual" button to say "Emergency Override - Take Control." The button's caption would then contain enough information for anyone pressing the button to know that they were responsible for all controls at the station after pressing.
dwnvoted2hell 7y ago
Is it weird that John McCain got to be a traitor before and after his death?
kbos87 7y ago
Any entry level UX designer would tell you that you can’t pack more and more control into an opaque and confusing system and still lay all the responsibility on the user when something breaks down. I really wonder, what does UX testing and iteration look like at Northrup-Grumman?
- jdhn 7y ago
  >I really wonder, what does UX testing and iteration look like at Northrup-Grumman? It's probably very limited, if it even exists at all. What I'm betting happens is that somebody says "The Navy needs a new widget for X purpose!". and the only input the designer gets to have is where it goes. I'd love to hear how UX designers in research challenged areas such as the military do their testing. It's not like they can just go on UserTesting.com and get feedback from somebody for $10 a test.
- CapricornNoble 7y ago
  >>I really wonder, what does UX testing and iteration look like at Northrup-Grumman? As someone who worked daily with two different NG-produced command and control systems, it's probably between "non-existent" and "flaming dumpster fire". https://www.northropgrumman.com/Capabilities/C2PC/Pages/default.aspx https://www.northropgrumman.com/Capabilities/C2PC/Pages/defa... https://dzone.com/articles/war-fighter-netbeans-platform https://dzone.com/articles/war-fighter-netbeans-platform Agile Client is the better of the two apps UI/UX-wise, close to a clunky, old version of Google Earth. Newer versions of C2PC are a polished turd, advancing the app's interface from 1995 to 2005.....despite hitting the operating forces around 2017-2018. We had bug submissions on some key mission-critical visualization features in the new version with patch turn-around times looking like 9-12 months at best.
iPhone1 7y ago
DO NOT LET YOUR FRIENDS/FAMILY JOIN THE NAVY. -ACTIVE DUTY SAILOR
- anticensor 7y ago
  Openly defaming your employer might have serious consequences.
AareyBaba 7y ago
Those look like Motif gui widgets from the 1990's on the touchscreen! https://features.propublica.org/navy-uss-mccain-crash/assets/images/ui-overview-9x7.jpg https://features.propublica.org/navy-uss-mccain-crash/assets...
slics 7y ago
These days we are so hyped on the cool shiny objects, but failing at the most basic concept, HSI = Human System Integration: “ Systems are comprised of hardware, software, and human personnel all of which operate within a surrounding environment. Too often, acquisition systems programs fail to consider the human capacity or requirements as part of the system. This leads to poor task allocation between hardware, software, and human users or supporters. To promote ideal task allocation, it is critical that the human element be considered early in system development.”
mannykannot 7y ago
The only thing here that is worse than this shitstorm of a system design is the vindictive attitude of the Navy towards the sailors who have to cope with it. Flogging and hanging from the yardarm may be gone, but the spirit lives on.
- mannykannot 7y ago
  Update: I have this suspicion that the Captain was charged with homicide in order to get him to take a plea deal, so that the problems with this system, and the Navy's failure to properly train crews in its use, would not become an issue in a court-martial.
SQL2219 7y ago
touch screens are a horrible idea for this type of system. I also think they're a bad idea in cars. there is no muscle-memory with a touch screen.
fmakunbound 7y ago
> could not keep track of more than 150 ships at a time without malfunctioning, according to Navy investigators. The Navy’s solution? Sailors were told to delete tracked ships before the total hit the magic number > “Usually when we have a fault with that system,” Sanchez said, “their resolution is to reboot the system.” Of course it is. Software is going to kill us all.
paranoidrobot 7y ago
I find it a little ironic that for an article about unintuitive and broken electronic navigation system, the UX on mobile of this article breaks navigation in weird and unintuitive ways. They value their narrative and slow animated exposition over my ability to scroll and read the article, meaning at multiple times I have to stop and wait for their painfully slow animations before I can continue.
- pbourke 7y ago
  I thought the visuals were well done, but I agree that they should have just scrolled normally on mobile instead of capturing the scroll to advance the animation.
throw7 7y ago
The Navy has a serious, serious leadership problem. Their seals have also been fucking up left and right. This would all be comical if it wasn't so tragic.
galaxyLogic 7y ago
I wonder was the system coded in Ada?

194 points by thomasjudge 7y ago

Very sobering article. Makes me reconsider the proliferation of glass cockpits and fly by wire in aviation. Granted, there is no putting the genie back in the bottle, but failsafe and redundancy need to be a central design tenet from the start. Back in the 80's, I was throttleman aboard a similar sized USN ship. There was no direct link from the bridge to the throttle. Throttle commands were communicated via an indicator and the throttle operator manually turned a big wheel to meet the command. I remember thinking at the time there should be direct throttle control from the bridge. Hindsight is 20:20.

scrumper 7y ago
Looks like quite a lot of redundancy here - you can split up controls over various stations really in kind of any way you want. That being the problem. The article led me to think the crash was more a confluence of factors: staff shortages, high operational demands, culture of tolerating cursory training, and a very complex new system that was both substantially harder to understand than traditional fixed stations, and constantly changing. The complexity of the IBNS system seems to be the ultimate culprit, in that the various members of the stressed and tired bridge crew were each operating under a different mental model of the state of the system. Because of IBNS's flexibility, there were too many possible states for a crew to reason about under pressure. Software makes it very easy for states to proliferate, but there hasn't been enough research (nor, more soberly, enough spilled blood) yet around designing flexible systems which are easily comprehensible in urgent situations. Of course this complexity does cause problems in aviation, even with that culture's extreme emphasis on rigorous training, procedure, and human factors. AF 447, for example.
magduf 7y ago
Glass cockpits and FBW are great, in theory. In practice, I don't know because I haven't had to use it. However, don't make the mistake of assuming that just because the US Navy can't get a well-engineered system put together means that Boeing and Airbus are guilty of the same. (Their glass-cockpit systems come from other vendors, who also supply to smaller aircraft makers BTW.) Also, how many foreign militaries have had problems like this with basic ship steering? From what I can tell, the US Navy is uniquely broken here. UK, Australia, Japan, France etc. do not have have their ships plowing into slow cargo ships. Commercial aircraft are not flying into the ground left and right. (The 737MAX is pretty bad, but that's not even related to glass cockpits; MCAS was a Boeing-engineered component, not a glass cockpit sourced from some vendor like Thales.)
- nwallin 7y ago
  > Also, how many foreign militaries have had problems like this with basic ship steering? > From what I can tell, the US Navy is uniquely broken here. UK, Australia, Japan, France etc. do not have have their ships plowing into slow cargo ships. Norway lost a destroyer last year after colliding with a tanker. Keep in mind the US Navy is larger than the next 13 largest navies combined. 2 of 3 of these events happening to the US Navy isn't statistically significant enough to say it's a uniquely US Navy problem. Not to say they don't need to get their shit together.

schwanksta 7y ago

Not to be overlooked: ProPublica actually constructed a 3D version of the navigation system's control board: https://www.propublica.org/article/how-we-reconstructed-the-flawed-navigation-controls-behind-the-navy-worst-maritime-accident-in-40-years https://www.propublica.org/article/how-we-reconstructed-the-...

mikeyouse 7y ago
Couldn't find it in your article, but it looks like the 3D models are integrated into the story here: https://features.propublica.org/navy-uss-mccain-crash/navy-installed-touch-screen-steering-ten-sailors-paid-with-their-lives/ https://features.propublica.org/navy-uss-mccain-crash/navy-i...

crmrc114 7y ago

In a panic people get tunnel vision. When you design HMI's for a living you have to consider. If my brother, sister, grandmother was using this system during an emergency could we guide them to make the right decisions. The fact that the prop controls could be split but the UI would not gray out the sliders on the screens that did not have control is a poor user interface design choice. If the controls only appeared "active" on the screen they were transferred too this could have made things much more simple. The fact that the mushroom button for emergency steering yanked control to wherever it was pressed- this should have displayed a red on black box. Emergency Override called from X Station. The issues with the navigation system are a whole other bag of worms- who knows what the mess was there. Disclosure, I have done HMI design and install in production systems for manufacturing, chemical and warehousing operations. I have never built anything for a ship or vehicle. So I may be speaking total stupid to someone in that industry. I would love to hear someone from that market sector drop in and talk about how or why the HMI was setup like this. Sounds like this system was sub-contracted to the lowest bidder and pushed out without enough testing.

rwmurrayVT 7y ago
Integrated bridges are being installed all across USN ships, including Mt. Whitney with MSC right now. They're filled with a hodgepodge of HMIs that are all linked to a control system filled with gyros, ECDIS, RADAR, propulsion, etc. etc. System integration is complex and the design work is done by contracted companies. You don't have the first party vendor doing the integration work. It's a recipe for disaster.
- jandrese 7y ago
  > You don't have the first party vendor doing the integration work. The only reason this should be true is if the Navy is directly providing this service instead. If nobody is coordinating integration, then that's absolute insanity. Integration is the most important job in a distributed system like this, also probably the hardest. Knowing the military they probably thought they could avoid needing an integrator by writing out tens of thousands of pages of specs, never stopping to consider that the specs themselves may be incorrect or unworkable or that they would be so complex that no vendor could ever deliver a product that exactly meets the specs.

remarkEon 7y ago

Absolutely phenomenal reporting. Take the time and read the entire thing in detail. It’s worth it. Can any Navy vets comment or speak to these systems? I really worry about the proliferation of these digital interfaces. I’m curious to know how folks feel about them.

linuxftw 7y ago
I can't speak to that specific system, but I can give you plenty of other anecdotes about the systemic unreadiness of the US Navy. I was forced to be the on-duty electrician from time to time, despite not being an ship's electrician. One of the responsibilities for the on-duty electrician was to cut the right circuits in case of a compartment fire. Despite my repeated protests that I didn't know what I was doing and needed additional training, I was forced to be the on-duty electrician. Mind you, there were no other duties I had to perform, just that one, and just in case of a fire, so it's not like my every-day workload was increased. Determined not to get people killed in the event of an emergency, I asked the actual electricians what I should be doing. Turns out, they didn't know either. Their plan was, in case of fire, kill power to the whole ship. I don't think I need to tell you, this is exactly the wrong move to make. There's an ever-growing number of electronic systems onboard ships, and an ever-shrinking number of sailors to run them. Qualifications are a joke in pretty much every department, at least in the surface fleet. Ships are in a constant state of disrepair, it's not uncommon for 30-50% of the communications equipment to be broken at any given time, parts taking months to receive (mostly due to budget restraints). Sailors are routinely sleep deprived. It's not uncommon for a sailor to be up by 6AM, work all day, and then be steering the ship from 00:00 to 04:00, with no sleep, followed by maybe 2 hours of sleep before having to work again the next day, and possibly have another watch from 20:00-12:00. Rinse repeat.
- jandrese 7y ago
  That last paragraph screams of a massive failure in Crew Resource Management. There are acres of case studies about this subject throughout industry, and when people take them to heart lots of accidents are avoided. Airline pilots are a prime example. How many airliners crash these days? CRM is one of the factors in making them so safe. Truck Drivers now have similar requirements and a much improved safety record as well. This is literally life or death, industries that ignore it are being unnecessarily reckless and unprofessional.

deleted 7y ago

[deleted]

waterside81 7y ago

> In 2014, Navy officials discovered a flaw in the IBNS. One component could not keep track of more than 150 ships at a time without malfunctioning, according to Navy investigators. The Navy’s solution? Sailors were told to delete tracked ships before the total hit the magic number. This blows my mind. Can anyone here guess what's going on under the hood? Is this a magic number that a developer came up during testing to avoid running of out memory/swapping?

klodolph 7y ago
Just a guess—probably designed using high-reliability / realtime principles. This might involve e.g. allocating memory only at startup, which would explain things. The limit has to be some number when programming this way.
- magduf 7y ago
  In a real-time safety-critical OS, you absolutely do allocate memory only at startup, but this kind of OS usually isn't used for any kind of GUI interface, it's used for much simpler systems. Another trait of these systems is that the CPU caches are all disabled, because those prevent determinism. But again, this isn't something you'd do on a GUI system; it's something you'd do on an ABS brake controller or an engine ECU, where the amount of memory ever needed is easy to determine in the design.
  - vonmoltke 7y ago
    This is also done in the signal processors that power these systems. I don't know this system in particular, but the USN has been moving towards commodity computing hardware for these systems. Essentially, a rack of commercial servers (usually IBM) running stripped-down RHEL with a real-time kernel. The software for these systems is firm real-time, and generally uses the same principals as other real-time software (including, but not limited to, up-front allocation).
    - jki275 7y ago

partiallypro 7y ago

This reminds me of "Pentagon Wars."

shitgoose 7y ago

Tried scrolling article up and down. Animated images hijacking the mouse control randomly, showing some motion then freezing. A good illustration.

stefan_ 7y ago
This is the latest innovation in "interactive story telling". It is just the ultimate in irony: here is an article crying murder because a touch interface allowed a sailor to un-gang engine power, then slide a big fat green power slider for one engine to half of what the other big green slider for the other engine directly next to it showed. Meanwhile here we are furiously scrolling to hit invisible "scrolled distance" breakpoints to advance a slideshow.
- ceejayoz 7y ago
  It's even worse if you're used to using spacebar to page-down.
  - aidenn0 7y ago
    Meh, I use spacebar to page-down, but am ready to fall-back at a moments notice because so many sites draw their header over the body, so a spacebar will end up by scrolling by more than a page...
- thatswrong0 7y ago
  The websites that pull this off "well" do so by giving you feedback that your scrolling is still doing _something_. For example, Apple pulls this off reasonably in their marketing pages by moving things in the background: https://www.apple.com/iphone-11-pro/ https://www.apple.com/iphone-11-pro/ This one just sticks and does nothing, which is exceptionally awkward and feels broken.
mkl 7y ago
It's even worse on mobile, locking the page up entirely with Javascript. I assumed it was untested and broken there, and shifted to desktop, but it's broken here too.

unlinked_dll 7y ago

Something unclear from the article, but jumped out at me. It seems that the Achilles heel of the IBNS system was that too much information tended to bork it, but what’s unclear to me (as implied by the article) is if dropping incoming data was actually the sensible thing to do (as is often the case) and if incoming data was prioritized over existing data and that was dropped instead. The other thing I would point out is that the technical failure is not that a system is complex, but that a complex system has a user interface that allows a sequence of actions to leave it in an invalid or unpredictable state. The transfer of throttle control being incomplete while other actions were taken, for example. I think the thing lacking here was consistent and sound changes to the state, lack of fuzzing user input to observe this possibility, and the reliance on training to avoid the issues at all. These are problems that can be solved with design.

iudqnolq 7y ago
I haven't worked on any project anywhere approaching significant complexity, but the magic number of 150 supported tracked ships sounds insanely low to me. Is this at all reasonable, or does it indicate the design and implementation were shit? Does anyone know how much processing is needed here? Is it just updating an internal model of locations and warning about impending collisions? Presuming constant speed and heading that sounds easy, not presuming it sounds not feasible even for just a handful of ships. Also, if only 150 ships are supported, wouldn't it makes sense for the computer to automatically untrack them? Maybe allow manual prioritization of what to drop first, but surely within a few days of a competent vendor learning adding too many items borked their system they would put out a patch that does the absolute minimum by automatically preventing adding too many items? > In 2014, Navy officials discovered a flaw in the IBNS. One component could not keep track of more than 150 ships at a time without malfunctioning, according to Navy investigators. The Navy’s solution? Sailors were told to delete tracked ships before the total hit the magic number. > The navigation system could also become overloaded if too much information streamed in from a ship tracking database used worldwide to prevent maritime collisions. The Navy’s second solution was similar to the first: Drop the feed when it became too much. > They were patches on top of patches that left the Navy’s destroyers without a full picture of the seas around them. But none of the problems was serious enough to attract high-level attention. A Navy system designed to track problems in major ship systems did not contain any reports that mentioned the IBNS until last year, according to a Navy official.

userbinator 7y ago

He ordered Bordeaux to take over steering the warship while another sailor controlled its speed. The idea was to avoid distractions by having each man focus on a single task in the heavy maritime traffic. I've not controlled anything that floats bigger than a small fishing boat, and I realise the timescales of reaction here are much larger, but that just sounds wrong to me --- changes in speed can affect direction, and vice-versa. Splitting that to two separate brains that are communicating only via a low-bandwidth voice channel seems like a recipe for disaster.

exabrial 7y ago
I'm curious why these things are manually controlled at all. Seems like you should give it a heading and it goes that direction, self-correcting along the way.
- davengh 7y ago
  Because sometimes he "wrong" action is the right thing to do. No automated system can anticipate everything. Think the NASCAR "drive at the wreck in front of you" - by the time you get there it will have moved. Especially on a ship that might have to spring from "situation normal" to active combat (maneuver/attack/defend all at once) without a pre-knowable trigger condition.
TeMPOraL 7y ago
I've not controlled anything that floats bigger than a pedal boat, but to the extent sci-fi shows and navy fictions show can teach me anything, I've always had the impression that the direction and speed of ships are controlled from two different stations on the bridge.
- dredmorbius 7y ago
  There was a time when the wheelhouse and the bridge were separated, at least in some ships and navies. See this 1941 British Navy training film: https://invidio.us/watch?v=ocUD07fvf48 https://invidio.us/watch?v=ocUD07fvf48

allovernow 7y ago

>Immediate responsibility, the Navy ruled, rested with Sanchez, his officers and senior sailors. They had been lax, even complacent, in their training of the sailors steering the ship. Sanchez had made a critical error in not adding more sailors to stand watch as the McCain navigated the treacherous strait From what I read consistently about the Navy, and hear from veteran friends, any sailors they could have asked to stand watch would have already been overworked and under rested. Everything I read about the U.S. military, and especially the NAVY, seems to indicate a broken culture of bloat and incompetence from the top down. I think we've been too comfortable as the premier unchallenged world superpower and the services have started to rot from within.

magduf 7y ago
It's not just the Navy or military; all kinds of things are breaking down in this country, though most people are completely in denial. Just look at how this country compares to others in terms of: obesity, healthcare costs per capita, transportation infrastructure, housing costs, gap between rich and poor, gun-related deaths, I could go on and on.
- pathseeker 7y ago
  >obesity, healthcare costs per capita Yep, healthcare is garbage. >transportation infrastructure This is subjective. Most Americans care more about roads than public transportation. >housing costs In all but the most expensive metros, US housing is way cheaper per sq ft than most Western European nations. This is partly due to urban sprawl due to the love affair with cars, but it is cheaper. >gap between rich and poor Richer middle class than most other countries. Just because the US is a good place for business and a bad place to be poor says very little about the average experience. >gun-related deaths Yep, that comes with gun ownership. Many people are knowingly making that trade-off. You'd want to compare murder/suicide rates with other countries for any meaningful comparisons.
  - aqme28 7y ago
    > Yep, that comes with gun ownership. Many people are knowingly making that trade-off. You'd want to compare murder/suicide rates with other countries for any meaningful comparisons. And our murder rate is pretty bad, likely thanks to our extremely high rate of gun violence.

LifeLiverTransp 7y ago

Very soon every ship station will come with a miniature tutorial to bring even a layman up to speed on how to steer a destroyer. I cant wait for a the first school class who hijacks a military boat listening to youtube videos while singing pirat shantys.

_Nat_ 7y ago

This story's worrisome in that it makes the Navy sound very vulnerable to electronic warfare. I mean, if overall operations are so hazard-prone that they're not reliable in the absence of hostilities, how could they possibly be resilient against an advanced actor? Is the situation as bad as it sounds from the story?

jschwartzi 7y ago

It strikes me that at least 16 seconds of screwing around could have been saved by changing the caption of the "Emergency Override - To Manual" button to say "Emergency Override - Take Control." The button's caption would then contain enough information for anyone pressing the button to know that they were responsible for all controls at the station after pressing.

dwnvoted2hell 7y ago

Is it weird that John McCain got to be a traitor before and after his death?

kbos87 7y ago

Any entry level UX designer would tell you that you can’t pack more and more control into an opaque and confusing system and still lay all the responsibility on the user when something breaks down. I really wonder, what does UX testing and iteration look like at Northrup-Grumman?

jdhn 7y ago
>I really wonder, what does UX testing and iteration look like at Northrup-Grumman? It's probably very limited, if it even exists at all. What I'm betting happens is that somebody says "The Navy needs a new widget for X purpose!". and the only input the designer gets to have is where it goes. I'd love to hear how UX designers in research challenged areas such as the military do their testing. It's not like they can just go on UserTesting.com and get feedback from somebody for $10 a test.
CapricornNoble 7y ago
>>I really wonder, what does UX testing and iteration look like at Northrup-Grumman? As someone who worked daily with two different NG-produced command and control systems, it's probably between "non-existent" and "flaming dumpster fire". https://www.northropgrumman.com/Capabilities/C2PC/Pages/default.aspx https://www.northropgrumman.com/Capabilities/C2PC/Pages/defa... https://dzone.com/articles/war-fighter-netbeans-platform https://dzone.com/articles/war-fighter-netbeans-platform Agile Client is the better of the two apps UI/UX-wise, close to a clunky, old version of Google Earth. Newer versions of C2PC are a polished turd, advancing the app's interface from 1995 to 2005.....despite hitting the operating forces around 2017-2018. We had bug submissions on some key mission-critical visualization features in the new version with patch turn-around times looking like 9-12 months at best.

iPhone1 7y ago

DO NOT LET YOUR FRIENDS/FAMILY JOIN THE NAVY. -ACTIVE DUTY SAILOR

anticensor 7y ago
Openly defaming your employer might have serious consequences.

AareyBaba 7y ago

Those look like Motif gui widgets from the 1990's on the touchscreen! https://features.propublica.org/navy-uss-mccain-crash/assets/images/ui-overview-9x7.jpg https://features.propublica.org/navy-uss-mccain-crash/assets...

slics 7y ago

These days we are so hyped on the cool shiny objects, but failing at the most basic concept, HSI = Human System Integration: “ Systems are comprised of hardware, software, and human personnel all of which operate within a surrounding environment. Too often, acquisition systems programs fail to consider the human capacity or requirements as part of the system. This leads to poor task allocation between hardware, software, and human users or supporters. To promote ideal task allocation, it is critical that the human element be considered early in system development.”

mannykannot 7y ago

The only thing here that is worse than this shitstorm of a system design is the vindictive attitude of the Navy towards the sailors who have to cope with it. Flogging and hanging from the yardarm may be gone, but the spirit lives on.

mannykannot 7y ago
Update: I have this suspicion that the Captain was charged with homicide in order to get him to take a plea deal, so that the problems with this system, and the Navy's failure to properly train crews in its use, would not become an issue in a court-martial.

SQL2219 7y ago

touch screens are a horrible idea for this type of system. I also think they're a bad idea in cars. there is no muscle-memory with a touch screen.

fmakunbound 7y ago

> could not keep track of more than 150 ships at a time without malfunctioning, according to Navy investigators. The Navy’s solution? Sailors were told to delete tracked ships before the total hit the magic number > “Usually when we have a fault with that system,” Sanchez said, “their resolution is to reboot the system.” Of course it is. Software is going to kill us all.

paranoidrobot 7y ago

I find it a little ironic that for an article about unintuitive and broken electronic navigation system, the UX on mobile of this article breaks navigation in weird and unintuitive ways. They value their narrative and slow animated exposition over my ability to scroll and read the article, meaning at multiple times I have to stop and wait for their painfully slow animations before I can continue.

pbourke 7y ago
I thought the visuals were well done, but I agree that they should have just scrolled normally on mobile instead of capturing the scroll to advance the animation.

throw7 7y ago

The Navy has a serious, serious leadership problem. Their seals have also been fucking up left and right. This would all be comical if it wasn't so tragic.

galaxyLogic 7y ago

I wonder was the system coded in Ada?