6 ms·

I've worked in defense as well. Older software seemed to be tested more, or otherwise more reliable. Sure, it could be outdated, but generally it was usable. A

by 0xfeba 5y ago

I've worked in defense as well. Older software seemed to be tested more, or otherwise more reliable. Sure, it could be outdated, but generally it was usable. As they moved away from embedded to more networked and newer tech stacks things got terrible. The software world of today is not compatible with the needs of the defense industry. Plus all the red tape and misdirection. The software need comes from an organizational budget that has to use money to justify receiving money. Then it funnels down into political buckets, who use the dollar amount given to determine needs/wants. Then it goes back up the chain to the government. Then to a contractor, then a flurry of subcontractors. Then finally a developer. Rarely is the warfighter who will actually use the system consulted. Even then, rarely will the programmer meet that person using the software. Yet they all call themselves "agile" now.

jwithington 5y ago
I don't think all of those problems are unique to government software. Some (a lot?) enterprise software is atrocious and disliked by end users. That's because the people making the purchasing decisions != end users. But it's flabbergasting just how bad VMS is considering it's the navigation system for these national assets. It's not like an HR system with poor UX--navigation is essential to the safety and effectiveness of the whole platform!
- wolverine876 5y ago
  > Some (a lot?) enterprise software is atrocious and disliked by end users. That's because the people making the purchasing decisions != end users. It's because the people making decisions care about putting money where it earns the most return and have more on their plate than they can address. Making software more pleasant for internal users rarely makes it to the top of the list. > navigation is essential to the safety and effectiveness of the whole platform! Subs crash very rarely, so the software seems to be sufficient.
  - jkhdigital 5y ago
    > Subs crash very rarely, so the software seems to be sufficient. Subs also operated for many decades without any kind of computer-assisted navigation, and training to handle the loss of such systems is routine, so your statement is pretty vacuous. The software at a minimum should allow sailors to operate as safely as they did without it, but one would assume that the entire point of software assistance is to improve safety and/or operational effectiveness.
dn3500 5y ago
I was a developer at a large contracter doing work for the Navy in the 1970s. Compared to today, there was almost zero code review, but an insane amount of testing, from the lowest module level to the highest integration level, both against the requirements and in simulation. We did not subcontract software. I often wished I could talk to the warfighter who was going to use my stuff but it never happened. Worse I never got any feedback on whether my stuff was even used. We did have quite a few ex-Navy people in our dev group and that did help.
- wolverine876 5y ago
  How efficient was that? How fast could you develop solutions?
  - virtue3 5y ago
    https://news.usni.org/2019/08/09/navy-reverting-ddgs-back-to-physical-throttles-after-fleet-rejects-touchscreen-controls https://news.usni.org/2019/08/09/navy-reverting-ddgs-back-to... Fucking bad from what I've read. It's a really bad problem still and the USN needs to correct it by having a tighter integration with the HW/SW people and the actual soldiers/sailors.
  - dkdk8283 5y ago
    Hopefully very slowly, you do not want fast development. You’re developing a safety critical system and should prioritize stability over features.
  - dn3500 5y ago
    In a way it was awful. It was sort of like anti-agile. There was absolutely no dialog between the devs and the customer. If the requirements said the throttle had to be on the touchscreen (to use an example from another comment), that's what we did, even if we strongly suspected this was a really bad idea. I was too far down the food chain to know where these requirements came from, it was like they were handed down to Moses on stone tablets. How fast could we develop? In addition to the requirements, we were also handed a schedule (a PERT chart, if anyone still knows what that is). It was laughably padded. Padded so much that it would have been impossible to fall behind schedule. I almost always finished my work for the day before lunch. Sometimes I would then go for a three martini lunch with the sales guys, or spend the afternoon in the library or flying the F-14 simulator. Sometimes I would spend the afternoon working on tools to make debugging easier. It was a fun job, but ultimately unsatisfying and I left after two years.
VBprogrammer 5y ago
I suspect any kind of sustained warfare in the modern era would result in the immediate obsolescence of much of the equipment designed in peacetime.
- noja 5y ago
  Warfare would not even be required https://en.wikipedia.org/wiki/Carrington_Event https://en.wikipedia.org/wiki/Carrington_Event
jcadam 5y ago
The only time I had contact with the end user as a defense contractor was when I was working on-site in an R&D shop, where we were creating software for users that were all in the same "integrated" team with us. That was the only job I've had in the industry I could describe as "fast-paced."
zentiggr 5y ago
Twenty years ago, when VMS just got introduced, it behaved like this... I can't believe that after all this time, it's still completely borked, in all the same ways. Somebody at NAVSEA needs to be dragged by their nostrils out on a deployment, take notes, and go back to their office with their pride in a garbage bag. Proud as hell to have served, and angry as hell that our current crews are still dealing with the same ---procurement--- failures.
- Jtsummers 5y ago
  That's from the time when everything was still (and everything still being planned for the replacements were also) big-bang releases. 5-10 year projects to replace the broken system that still lack the necessary capabilities. I've never worked on a Navy project, but I've seen several spectacular USAF failures of similar magnitude. The most hilarious (in an absurdist sense) was having 3 generations of a system running concurrently because none of them had all the necessary features, when I left they were working on number 4.
  - Goety 5y ago
    Well... What is your solution for this?
    - rbanffy 5y ago
      Mine would be shorter development cycles and incremental deployment of improvements as soon as basic functionality is achieved. Work in an accurate simulator with an experienced crew from day one would also help.
    - Jtsummers 5y ago
      Don't plan projects out 5+ years and fail to revisit the plan. That is, don't do Waterfall. Work with the actual user/operators (this I did see a lot of, but not on new systems, only on systems in "maintenance") to get regular feedback. One of the best jobs of this I saw brought in both current and retired operators. Current ones rotated through helping evaluate the system and the requirements, retired ones were hired on to be part of test/QA. Switch from fixed contracts to more level of effort contracts. Generally shift towards continuous delivery where possible (with information systems much easier than with embedded systems, but doable to an extent with embedded systems). Fix the official release test phase which is often 6-24 months at the end, most of it waiting. This is part of the program's overall process which is very Waterfall in structure for most systems. I've seen work that was completed and put on a shelf for over a year before being evaluated for flight readiness. Well, it was useless to operators by that point [0] and delayed vital feedback for the next iteration. That's partly a logistics problem, but addressable. Fix the lack of trust the operational test teams have for development teams. There was a complete lack of trust for the teams delivering the product to be tested, but the test plans for flight testing were often so bad that they were a joke (for the particular delivered systems, not flight test plans in general). If you don't have the trust, gain the trust. Take the existing operational test plans and back port them to your own test infrastructure and test efforts. Deliver that report along with the product, eventually they may trust your test reports and can start speeding things along. As with all software, get as much testing done as early as you can. The longer you wait, the more costly it is to address discovered issues, particularly validation issues. Finally, get experienced software engineers into program offices. Most of the time their "software engineers" are a recent college graduate, or someone from a totally unrelated field that wrote some Matlab code once. They lack sufficient experience to properly manage or guide the management of the software portion of a project. The worst I saw with this was that system I mentioned in my previous comment, the program office lead for the software portion was a newly pinned on 1st LT history major. Zero technical experience (had been in some finance role before), they trusted the contractors too much and made decisions more on their input than anyone actually using the system. [0] For context, this is software that spent 2-4 years being developed and then would be shelved for 1-2 years before being tested. Which meant it still had 1-2 years for fielding. So from conception to release you're talking 4-8 years. That is an awful pace.
denkmoon 5y ago
As a defence contractor I have implemented many incredibly stupid or pointless things merely to fulfill someone's idea of "agile". Fortunately all the projects I have worked on aren't related to combat or anything where actual people could be hurt.

by 0xfeba 5y ago

jwithington 5y ago

I don't think all of those problems are unique to government software. Some (a lot?) enterprise software is atrocious and disliked by end users. That's because the people making the purchasing decisions != end users. But it's flabbergasting just how bad VMS is considering it's the navigation system for these national assets. It's not like an HR system with poor UX--navigation is essential to the safety and effectiveness of the whole platform!

wolverine876 5y ago
> Some (a lot?) enterprise software is atrocious and disliked by end users. That's because the people making the purchasing decisions != end users. It's because the people making decisions care about putting money where it earns the most return and have more on their plate than they can address. Making software more pleasant for internal users rarely makes it to the top of the list. > navigation is essential to the safety and effectiveness of the whole platform! Subs crash very rarely, so the software seems to be sufficient.
- jkhdigital 5y ago
  > Subs crash very rarely, so the software seems to be sufficient. Subs also operated for many decades without any kind of computer-assisted navigation, and training to handle the loss of such systems is routine, so your statement is pretty vacuous. The software at a minimum should allow sailors to operate as safely as they did without it, but one would assume that the entire point of software assistance is to improve safety and/or operational effectiveness.

dn3500 5y ago

I was a developer at a large contracter doing work for the Navy in the 1970s. Compared to today, there was almost zero code review, but an insane amount of testing, from the lowest module level to the highest integration level, both against the requirements and in simulation. We did not subcontract software. I often wished I could talk to the warfighter who was going to use my stuff but it never happened. Worse I never got any feedback on whether my stuff was even used. We did have quite a few ex-Navy people in our dev group and that did help.

wolverine876 5y ago
How efficient was that? How fast could you develop solutions?
- virtue3 5y ago
  https://news.usni.org/2019/08/09/navy-reverting-ddgs-back-to-physical-throttles-after-fleet-rejects-touchscreen-controls https://news.usni.org/2019/08/09/navy-reverting-ddgs-back-to... Fucking bad from what I've read. It's a really bad problem still and the USN needs to correct it by having a tighter integration with the HW/SW people and the actual soldiers/sailors.
- dkdk8283 5y ago
  Hopefully very slowly, you do not want fast development. You’re developing a safety critical system and should prioritize stability over features.
- dn3500 5y ago
  In a way it was awful. It was sort of like anti-agile. There was absolutely no dialog between the devs and the customer. If the requirements said the throttle had to be on the touchscreen (to use an example from another comment), that's what we did, even if we strongly suspected this was a really bad idea. I was too far down the food chain to know where these requirements came from, it was like they were handed down to Moses on stone tablets. How fast could we develop? In addition to the requirements, we were also handed a schedule (a PERT chart, if anyone still knows what that is). It was laughably padded. Padded so much that it would have been impossible to fall behind schedule. I almost always finished my work for the day before lunch. Sometimes I would then go for a three martini lunch with the sales guys, or spend the afternoon in the library or flying the F-14 simulator. Sometimes I would spend the afternoon working on tools to make debugging easier. It was a fun job, but ultimately unsatisfying and I left after two years.

VBprogrammer 5y ago

I suspect any kind of sustained warfare in the modern era would result in the immediate obsolescence of much of the equipment designed in peacetime.

noja 5y ago
Warfare would not even be required https://en.wikipedia.org/wiki/Carrington_Event https://en.wikipedia.org/wiki/Carrington_Event

jcadam 5y ago

The only time I had contact with the end user as a defense contractor was when I was working on-site in an R&D shop, where we were creating software for users that were all in the same "integrated" team with us. That was the only job I've had in the industry I could describe as "fast-paced."

zentiggr 5y ago

Twenty years ago, when VMS just got introduced, it behaved like this... I can't believe that after all this time, it's still completely borked, in all the same ways. Somebody at NAVSEA needs to be dragged by their nostrils out on a deployment, take notes, and go back to their office with their pride in a garbage bag. Proud as hell to have served, and angry as hell that our current crews are still dealing with the same ---procurement--- failures.

Jtsummers 5y ago
That's from the time when everything was still (and everything still being planned for the replacements were also) big-bang releases. 5-10 year projects to replace the broken system that still lack the necessary capabilities. I've never worked on a Navy project, but I've seen several spectacular USAF failures of similar magnitude. The most hilarious (in an absurdist sense) was having 3 generations of a system running concurrently because none of them had all the necessary features, when I left they were working on number 4.
- Goety 5y ago
  Well... What is your solution for this?
  - rbanffy 5y ago
    Mine would be shorter development cycles and incremental deployment of improvements as soon as basic functionality is achieved. Work in an accurate simulator with an experienced crew from day one would also help.
  - Jtsummers 5y ago
    Don't plan projects out 5+ years and fail to revisit the plan. That is, don't do Waterfall. Work with the actual user/operators (this I did see a lot of, but not on new systems, only on systems in "maintenance") to get regular feedback. One of the best jobs of this I saw brought in both current and retired operators. Current ones rotated through helping evaluate the system and the requirements, retired ones were hired on to be part of test/QA. Switch from fixed contracts to more level of effort contracts. Generally shift towards continuous delivery where possible (with information systems much easier than with embedded systems, but doable to an extent with embedded systems). Fix the official release test phase which is often 6-24 months at the end, most of it waiting. This is part of the program's overall process which is very Waterfall in structure for most systems. I've seen work that was completed and put on a shelf for over a year before being evaluated for flight readiness. Well, it was useless to operators by that point [0] and delayed vital feedback for the next iteration. That's partly a logistics problem, but addressable. Fix the lack of trust the operational test teams have for development teams. There was a complete lack of trust for the teams delivering the product to be tested, but the test plans for flight testing were often so bad that they were a joke (for the particular delivered systems, not flight test plans in general). If you don't have the trust, gain the trust. Take the existing operational test plans and back port them to your own test infrastructure and test efforts. Deliver that report along with the product, eventually they may trust your test reports and can start speeding things along. As with all software, get as much testing done as early as you can. The longer you wait, the more costly it is to address discovered issues, particularly validation issues. Finally, get experienced software engineers into program offices. Most of the time their "software engineers" are a recent college graduate, or someone from a totally unrelated field that wrote some Matlab code once. They lack sufficient experience to properly manage or guide the management of the software portion of a project. The worst I saw with this was that system I mentioned in my previous comment, the program office lead for the software portion was a newly pinned on 1st LT history major. Zero technical experience (had been in some finance role before), they trusted the contractors too much and made decisions more on their input than anyone actually using the system. [0] For context, this is software that spent 2-4 years being developed and then would be shelved for 1-2 years before being tested. Which meant it still had 1-2 years for fielding. So from conception to release you're talking 4-8 years. That is an awful pace.

denkmoon 5y ago

As a defence contractor I have implemented many incredibly stupid or pointless things merely to fulfill someone's idea of "agile". Fortunately all the projects I have worked on aren't related to combat or anything where actual people could be hurt.