5 ms·

I had the pleasure of being involved in a software project where the contractor was one of the usual suspects (Raytheon, Lockheed, etc.). Prior to this, I had

I had the pleasure of being involved in a software project where the contractor was one of the usual suspects (Raytheon, Lockheed, etc.). Prior to this, I had no idea why defense projects were so expensive and slow. You have to understand, there are so many levels of negotiation, over the smallest changes imaginable. There's no such thing as "open source code" in these projects, everything is either modified off-the-shelf, or completely boutique stuff, and of course implemented in such a way that the clients (that would be us) can't just access the code themselves, or that clients can just build upon or make wrappers. The contractor will send you their own techs or engineers - even if it's across the world - to fix the most trivial bugs. And it will take months or planning, budgeting, and what not. In once instance, one contractor tech spotted a bug in the code, which some user had reported about - it was a one character fix, just a typo, really. So he fixed it on the spot, along with the rest of the stuff he came for. A couple of weeks later he comes back, and reverses the bug. He had told his manager about the bug fix, and was berated for fixing in. In order to fix that bug, it had to be reported correctly, go through software update meetings, go through budgets, and the whole nine yards. The bug was reported the correct way, and many months later he came back to fix it (again). The whole process of specs and updates was like pulling teeth.

giancarlostoro 3y ago
> A couple of weeks later he comes back, and reverses the bug. He had told his manager about the bug fix, and was berated for fixing in. In order to fix that bug, it had to be reported correctly, go through software update meetings, go through budgets, and the whole nine yards. What do you mean you went and just fixed it!? We need all of middle management to charge for the fix!
vegetablepotpie 3y ago
My experience has been that you can use open source software, but security considers that to have a high risk and you need to make a business case for it. What isn’t a security risk? A closed source vendor supplied application that has its headquarters in the US. You can sometimes find pockets of innovation in the defense sector, but it’s mostly filled with bureaucrats with engineer in their title.
- ajmurmann 3y ago
  To be fair, securing the OSS supply chain is a tough task, especially when pulling in new versions.
- xxpor 3y ago
  And yet, things like the Patriot missile system get built. It's pretty wild what an 800 billion dollar budget will paper over.
david422 3y ago
So I worked for a company that had loads of process for even the smallest changes. It was frustrating, but what I came to realize was that the company was not interested in fixing bugs or building features - they were primarily interested in not breaking things/keeping the status quo. This was an established business that was concerned with keeping the business running, not chasing the latest and greatest. Perhaps a lot of it was just CYA, but also means it was pretty hard to "move fast and break things".
- DropInIn 3y ago
  That sort of business sounds like what we want in milsec and government suppliers, no? Reliable even down to the bug behaviors...
  - deelly 3y ago
    Not only. Also some software for financial sector, for suppliers, for maintenance too. Basically, anything that do not crash a system could be a feature. And sometime even have logical and good reason to exists.
  - ajmurmann 3y ago
    The reliable bug behavior is an underappreciated point and can be hard to understand. I work on software that gets deployed on-prem by customers and had very powerful capabilities for deploying your own code to it etc. There have been a few discussions about some bug fixes and in particular security fixes being appropriate in a patch release when following SemVer. Something might be unintentional or insecure, but customers might be building on the broken behavior and now deploying a patch breaks their apps. The line can become quite blurry.
    - pc86 3y ago
      This sounds like more a deployment/release process problem than anything else. If the customers are essentially deploying untested code straight to production, they're doing something wrong. It might not be their fault, maybe they don't know they should have a staging environment first, or maybe your pricing model makes it prohibitively expensive. It's not as simple as "your customers are doing it wrong," but if you upgrading a package breaks their production environment, someone somewhere is - or more likely, many people are - definitely doing something wrong.
numpad0 3y ago
But you'd be just as upset as his employer if you'd come across a binary release built from one commit ahead the GitHub repo. The absurdity is in the amount of overhead, not the fact that there are overheads at all.
2OEH8eoCRo0 3y ago
I worked on a radar system that could see precipitation- think doppler radar map view. The ability to do this was in the contract but there was no way to view it. We had Matlab scripts so we the devs could view it and the weather people wanted them. We weren't allowed to hand them over without a contract. They'd pop in multiple times a day to view our doppler. The adversarial nature that defense companies treat our military is appalling. I'm convinced that refactoring a lot of this BS could get us way more with less $
- mortify 3y ago
  If the viewing software was provided, the support would be included and that will cost money in the long term. That is something that the company didn't agree to provide nor the government to pay for. In this case, the adversary wasn't the contractor, it was the users in the government and the government employees holding the purse strings. No contractor will turn down the chance to extend the contract.
- JamesBarney 3y ago
  A lot of this adversarial nature comes down to how projects are awarded. Government employees have very little discretion to choose projects, so they always go with the cheapest bid. But no large software project ever has had 0 change orders. So the only profitable way to win these contracts is offer the cheapest bid by delivering as little as possible while being in spec, and then pillaging on the change orders.
  - FooBarBizBazz 3y ago
    What about the "new defense" companies, who use VC money to build demos? Surely they eventually want to build moats and sell razor blades, but for now what they're doing seems different?
jliptzin 3y ago
I have encountered a situation a few times where a bug that went unnoticed for a while was actually keeping another, more serious bug from surfacing. So when the first bug was fixed, it caused more serious problems elsewhere in the system. I could see where they're coming from where any kind of "fix" needs to trigger a careful reevaluation of the whole system. That being said, I don't see a need for the Kafkaesque months of planning and budgeting that follow.
- sneak 3y ago
  > That being said, I don't see a need for the Kafkaesque months of planning and budgeting that follow. It's a method for transferring billions of dollars from public tax revenues to private companies which are cozy with the state. It has nothing to do with safety or reliability. All other methods of stealing tax money are illegal.
  - jliptzin 3y ago
    Would be nice if we spent some time cracking down on that kind of tax theft rather than wasting time on drag shows and the 3 dozen teens in the whole country on puberty blockers
    - red-iron-pine 3y ago
      you got it mon ami. that's the whole point of making a kerfuffle about drag shows. or gay marriage. or even abortion. ain't no one talking about tax cuts, or getting after the rich, cuz of that.
- syntheticcorp 3y ago
  I’ve also encountered that a few times where a fairly anodyne bug in a codepath prevents a serious security bug from being reachable. With my attacker hat on it is very tempting to just report the first one…
bannedbybros 3y ago
[dead]
bentxt 3y ago
Do you have an opinion whether this kafkaesk bureaucratism is especially well established in the domain of software or is it a broader pattern? Where does this come from?
renewiltord 3y ago
Remember: Regulations are written in blood.
AndrewKemendo 3y ago
It’s a broken system for sure… HOWEVER when your acquisition system is designed to ensure that a nuclear detonator and a LAMP application go through the same process, these kinds of “overkill” or “nonsense” requirement responses happen. We’re long overdue for fixing the FAR and federal acquisition process, which is the root problem. The legal system and processes built over the last 50 years, and as a result the people inside it, doesn’t allow for simple and efficient differentiation between projects legally so you end up with the same hammer meets mail approach.
asylteltine 3y ago
[dead]
Eddy_Viscosity2 3y ago
Modern defense contracting has become, first and foremost, a process to transfer government funds to private hands. The fact that any useful products or services happen is purely incidental.
- quantum_state 3y ago
  you can say that again
jasmer 3y ago
[dead]
deleted 3y ago
[deleted]
stonogo 3y ago
Part of this stems from government accountability issues. If you fix that bug, and you don't go through the process, no you didn't. At any time the government can ask for a contract performance review, and doing things the easy way means lots of things that should be tracked don't get tracked... which means they never make it into the contract performance review. They mean well, and I understand how hard it can be for government agents to hold massive contractors accountable, but the purpose of a system is what it does... so we wind up with entrenched bureaucracy, and the players who don't subscribe to these approaches wind up getting outcompeted because their on-paper performance doesn't measure up. Even if they're in fact a million times easier to work with and do a great job.
d-man 3y ago
s/defense/regulated industry/ I found a bug in my company’s software product once. The code was using the user’s display name in a lookup rather than the username. I grabbed a QA engineer, he agreed it was a bug, and he put the wheels in motion. I don’t think that fix ever made it into a release. Our software was part of the stack that our regulated customers were using to document their compliance. Every year or two, they would audit our development process to insure that untoward code wasn’t slipping into the release. Getting the fix into a release would have required that the bug be prioritized, fixed and qa’ed. Never became important enough. It burned me at first, but I’m not sure it was the wrong outcome.

giancarlostoro 3y ago

> A couple of weeks later he comes back, and reverses the bug. He had told his manager about the bug fix, and was berated for fixing in. In order to fix that bug, it had to be reported correctly, go through software update meetings, go through budgets, and the whole nine yards. What do you mean you went and just fixed it!? We need all of middle management to charge for the fix!

vegetablepotpie 3y ago

My experience has been that you can use open source software, but security considers that to have a high risk and you need to make a business case for it. What isn’t a security risk? A closed source vendor supplied application that has its headquarters in the US. You can sometimes find pockets of innovation in the defense sector, but it’s mostly filled with bureaucrats with engineer in their title.

ajmurmann 3y ago
To be fair, securing the OSS supply chain is a tough task, especially when pulling in new versions.
xxpor 3y ago
And yet, things like the Patriot missile system get built. It's pretty wild what an 800 billion dollar budget will paper over.

david422 3y ago

So I worked for a company that had loads of process for even the smallest changes. It was frustrating, but what I came to realize was that the company was not interested in fixing bugs or building features - they were primarily interested in not breaking things/keeping the status quo. This was an established business that was concerned with keeping the business running, not chasing the latest and greatest. Perhaps a lot of it was just CYA, but also means it was pretty hard to "move fast and break things".

DropInIn 3y ago
That sort of business sounds like what we want in milsec and government suppliers, no? Reliable even down to the bug behaviors...
- deelly 3y ago
  Not only. Also some software for financial sector, for suppliers, for maintenance too. Basically, anything that do not crash a system could be a feature. And sometime even have logical and good reason to exists.
- ajmurmann 3y ago
  The reliable bug behavior is an underappreciated point and can be hard to understand. I work on software that gets deployed on-prem by customers and had very powerful capabilities for deploying your own code to it etc. There have been a few discussions about some bug fixes and in particular security fixes being appropriate in a patch release when following SemVer. Something might be unintentional or insecure, but customers might be building on the broken behavior and now deploying a patch breaks their apps. The line can become quite blurry.
  - pc86 3y ago
    This sounds like more a deployment/release process problem than anything else. If the customers are essentially deploying untested code straight to production, they're doing something wrong. It might not be their fault, maybe they don't know they should have a staging environment first, or maybe your pricing model makes it prohibitively expensive. It's not as simple as "your customers are doing it wrong," but if you upgrading a package breaks their production environment, someone somewhere is - or more likely, many people are - definitely doing something wrong.

numpad0 3y ago

But you'd be just as upset as his employer if you'd come across a binary release built from one commit ahead the GitHub repo. The absurdity is in the amount of overhead, not the fact that there are overheads at all.

2OEH8eoCRo0 3y ago

I worked on a radar system that could see precipitation- think doppler radar map view. The ability to do this was in the contract but there was no way to view it. We had Matlab scripts so we the devs could view it and the weather people wanted them. We weren't allowed to hand them over without a contract. They'd pop in multiple times a day to view our doppler. The adversarial nature that defense companies treat our military is appalling. I'm convinced that refactoring a lot of this BS could get us way more with less $

mortify 3y ago
If the viewing software was provided, the support would be included and that will cost money in the long term. That is something that the company didn't agree to provide nor the government to pay for. In this case, the adversary wasn't the contractor, it was the users in the government and the government employees holding the purse strings. No contractor will turn down the chance to extend the contract.
JamesBarney 3y ago
A lot of this adversarial nature comes down to how projects are awarded. Government employees have very little discretion to choose projects, so they always go with the cheapest bid. But no large software project ever has had 0 change orders. So the only profitable way to win these contracts is offer the cheapest bid by delivering as little as possible while being in spec, and then pillaging on the change orders.
- FooBarBizBazz 3y ago
  What about the "new defense" companies, who use VC money to build demos? Surely they eventually want to build moats and sell razor blades, but for now what they're doing seems different?

jliptzin 3y ago

I have encountered a situation a few times where a bug that went unnoticed for a while was actually keeping another, more serious bug from surfacing. So when the first bug was fixed, it caused more serious problems elsewhere in the system. I could see where they're coming from where any kind of "fix" needs to trigger a careful reevaluation of the whole system. That being said, I don't see a need for the Kafkaesque months of planning and budgeting that follow.

sneak 3y ago
> That being said, I don't see a need for the Kafkaesque months of planning and budgeting that follow. It's a method for transferring billions of dollars from public tax revenues to private companies which are cozy with the state. It has nothing to do with safety or reliability. All other methods of stealing tax money are illegal.
- jliptzin 3y ago
  Would be nice if we spent some time cracking down on that kind of tax theft rather than wasting time on drag shows and the 3 dozen teens in the whole country on puberty blockers
  - red-iron-pine 3y ago
    you got it mon ami. that's the whole point of making a kerfuffle about drag shows. or gay marriage. or even abortion. ain't no one talking about tax cuts, or getting after the rich, cuz of that.
syntheticcorp 3y ago
I’ve also encountered that a few times where a fairly anodyne bug in a codepath prevents a serious security bug from being reachable. With my attacker hat on it is very tempting to just report the first one…

bannedbybros 3y ago

[dead]

bentxt 3y ago

Do you have an opinion whether this kafkaesk bureaucratism is especially well established in the domain of software or is it a broader pattern? Where does this come from?

renewiltord 3y ago

Remember: Regulations are written in blood.

AndrewKemendo 3y ago

It’s a broken system for sure… HOWEVER when your acquisition system is designed to ensure that a nuclear detonator and a LAMP application go through the same process, these kinds of “overkill” or “nonsense” requirement responses happen. We’re long overdue for fixing the FAR and federal acquisition process, which is the root problem. The legal system and processes built over the last 50 years, and as a result the people inside it, doesn’t allow for simple and efficient differentiation between projects legally so you end up with the same hammer meets mail approach.

asylteltine 3y ago

[dead]

Eddy_Viscosity2 3y ago

Modern defense contracting has become, first and foremost, a process to transfer government funds to private hands. The fact that any useful products or services happen is purely incidental.

quantum_state 3y ago
you can say that again

jasmer 3y ago

[dead]

deleted 3y ago

[deleted]

stonogo 3y ago

Part of this stems from government accountability issues. If you fix that bug, and you don't go through the process, no you didn't. At any time the government can ask for a contract performance review, and doing things the easy way means lots of things that should be tracked don't get tracked... which means they never make it into the contract performance review. They mean well, and I understand how hard it can be for government agents to hold massive contractors accountable, but the purpose of a system is what it does... so we wind up with entrenched bureaucracy, and the players who don't subscribe to these approaches wind up getting outcompeted because their on-paper performance doesn't measure up. Even if they're in fact a million times easier to work with and do a great job.

d-man 3y ago

s/defense/regulated industry/ I found a bug in my company’s software product once. The code was using the user’s display name in a lookup rather than the username. I grabbed a QA engineer, he agreed it was a bug, and he put the wheels in motion. I don’t think that fix ever made it into a release. Our software was part of the stack that our regulated customers were using to document their compliance. Every year or two, they would audit our development process to insure that untoward code wasn’t slipping into the release. Getting the fix into a release would have required that the bug be prioritized, fixed and qa’ed. Never became important enough. It burned me at first, but I’m not sure it was the wrong outcome.