8 ms·

Worked with a guy ten years ago that would obsessively read these reports from the Pentagon. Here's a couple of his stories about the matter: https://www.reuter

Worked with a guy ten years ago that would obsessively read these reports from the Pentagon. Here's a couple of his stories about the matter: https://www.reuters.com/investigates/pentagon/ https://www.reuters.com/investigates/pentagon/ and https://www.reuters.com/investigates/special-report/usa-marines-audit https://www.reuters.com/investigates/special-report/usa-mari... (very old and I apologize for my horrific javascript at the time.) It sounds like nothing has changed. Relevant to this crowd, maybe: "Most of the Cobol code the Pentagon uses for payroll and accounting was written in the 1960s, according to 2006 congressional testimony by Zack Gaddy, director of DFAS from May 2004 to September 2008." "Wallace, the Army assistant deputy chief of staff, says the system has "seven million lines of Cobol code that hasn't been updated" in more than a dozen years, and significant parts of the code have been "corrupted." The older it gets, the harder it is to maintain. As DFAS itself said: "As time passes, the pool of Cobol expertise dwindles.""

samus 3y ago
If it ain't broke, don't fix it, they say. But if it is already broken in significant ways, there is little point in not replacing it.
wslh 3y ago
I would love to perform static security code analysis to that code.
- tandr 3y ago
  Are there any SA tools, that would understand COBOL?
  - Jtsummers 3y ago
    https://www.sonarsource.com/knowledge/languages/cobol/ https://www.sonarsource.com/knowledge/languages/cobol/ Never used it (and hopefully never will), but yes.
- xeromal 3y ago
  Not all the RAM on earth would be enough to process that shit
bogota 3y ago
Start paying 250k a year for cobol developers and the problem fixes itself really fast. That or start migrating the system. Or more likely they keep using it until it completely breaks and then run to the government for a big handout to fix it. Easy to be irresponsible with other peoples money.
- lancepioch 3y ago
  Don't they have caps and levels for federal employee salaries? I agree that raising the salary would decrease the issues. I think that they'd need exceptions to account for these increases though.
  - Jtsummers 3y ago
    Yes, but not for contractors. Most software sustainment ultimately ends up in the hands of contractors with government supervision in the form of program offices. Not all, though, many programs are also "organic", primarily or fully staffed by government employees and managed by government employees.
    - randmeerkat 3y ago
      > Yes, but not for contractors. Sure, but it’s not like the consulting firms are paying their “contractors” more, they just siphon up the difference for their shareholders.
      - ska 3y ago
        That's not really how it works. They are more than happy to have "market forces" drive up the hourly rates, so long as they get to keep their overhead % fixed, and that's probably how the contract is structured. Win-win (and perhaps -lose for taxpayers).
        randmeerkat 3y ago
        > That's not really how it works. They are more than happy to have "market forces" drive up the hourly rates That’s exactly how it works, look at contract government salaries compared to anything in the private sector. They charge the government more as rates “go up”, but that certainly isn’t passed along. If large contracting companies really offered value to the government _and_ kept up with market rates for their employees, the state of federal software wouldn’t be what it is today.
uticus 3y ago
Of course, devil's advocate would point out COBOL in 60s was written with a lot more focus on lasting in general. They didn't largely have the same general attitude of 'we iterate quickly with our script kiddies' as is prevalent today. Not that all software from then was wonderful. Or that COBOL magically guarantees great code. But, on the flip side, you can't say the code base isn't battle-hardened (to various literal degrees). * edit: corrected sp
- SteveNuts 3y ago
  It'd be battle hardened to 1960s standards, payroll has changed a _lot_ since then. And if they can't accurately describe exactly how certain calculations are made (or have to do some of those manually outside of the program after it runs), that would be a huge problem in an audit. This is why big companies use Oracle or SAP for their ERP, auditors are very familiar with it and the way it functions is well-known and understood.
- monocasa 3y ago
  Also throwing out there that COBOL was a language designed by the DOD specifically for managing it's bureaucracy. It was a naval research project headed by Admiral Grace Hopper to program with words rather than formulas like earlier "high-level" "autocoders", enabling secretaries to transfer their business logic into computers. It actually intentionally has a lot of the same structure as a recipe, hoping that this would allow easier uptake by secretaries.
- ebiester 3y ago
  I can build you a team using today's (quality: average B2B startup) programmers and today's methodologies that will write software for 40 years. It will just cost twice as much as the team that iterates quickly. It will cost more because I will spend more money on business analysis up front. I will spend more money writing automated tests. Most importantly, I will maintain my own toolchain and libraries or pay someone to do so. Every piece of software deployed will have someone accountable to my business to maintain it. It turns out that this is expensive. It's no more expensive than maintaining the COBOL code, mind you, and would likely give you better results than the equivalent COBOL code. Most companies will take their chances with less expensive software because that's what their competitors do. About a decade ago, I saw one of the major systems you talk about up close. While they spent that amount of money for mission-critical systems, they didn't do nearly as good of a job on the average software in the system. I'm not convinced that any business is ready to go back to the battle hardened days.
encoderer 3y ago
I can imagine a future where new software is, mainly, written by humans while legacy software is maintained by ai. It can take months for a new developer to understand a legacy code base but an LLM with a big enough context window would be instantly productive.
- Yahivin 3y ago
  I can imagine the opposite.
  - encoderer 3y ago
    Cool tell me more.
    - deleted 3y ago
      [deleted]
    - SteveNuts 3y ago
      Not OP but I could see some shops pushing AI generated code to production, then when changes need to be made, they can't get the AI to modify the existing code in just the way they need, so a human has to intervene.
    - bastawhiz 3y ago
      I can't get Copilot to generate Python that adds numbers together correctly sometimes. Getting an LLM to generate correct, working code for a language that hardly anybody writes anymore is almost assuredly going to lead to failure.
      - encoderer 3y ago
        yeah I agree but when you look at the slope not the y-intercept it’s getting obviously better. one advantage the government would have is training/fine-tuning on a hundred million lines of domain specific cobol.
deleted 3y ago
[deleted]
TurkishPoptart 3y ago
Is the problem really related to COBOL, or is the fact that this institution is basically unaccountable to taxpayers?
chatmasta 3y ago
See also, "John Stewart Questions Defense Deputy Secretary on Budget:" https://youtube.com/watch?v=50MusF365U0 https://youtube.com/watch?v=50MusF365U0

samus 3y ago

If it ain't broke, don't fix it, they say. But if it is already broken in significant ways, there is little point in not replacing it.

wslh 3y ago

I would love to perform static security code analysis to that code.

tandr 3y ago
Are there any SA tools, that would understand COBOL?
- Jtsummers 3y ago
  https://www.sonarsource.com/knowledge/languages/cobol/ https://www.sonarsource.com/knowledge/languages/cobol/ Never used it (and hopefully never will), but yes.
xeromal 3y ago
Not all the RAM on earth would be enough to process that shit

bogota 3y ago

Start paying 250k a year for cobol developers and the problem fixes itself really fast. That or start migrating the system. Or more likely they keep using it until it completely breaks and then run to the government for a big handout to fix it. Easy to be irresponsible with other peoples money.

lancepioch 3y ago
Don't they have caps and levels for federal employee salaries? I agree that raising the salary would decrease the issues. I think that they'd need exceptions to account for these increases though.
- Jtsummers 3y ago
  Yes, but not for contractors. Most software sustainment ultimately ends up in the hands of contractors with government supervision in the form of program offices. Not all, though, many programs are also "organic", primarily or fully staffed by government employees and managed by government employees.
  - randmeerkat 3y ago
    > Yes, but not for contractors. Sure, but it’s not like the consulting firms are paying their “contractors” more, they just siphon up the difference for their shareholders.
    - ska 3y ago
      That's not really how it works. They are more than happy to have "market forces" drive up the hourly rates, so long as they get to keep their overhead % fixed, and that's probably how the contract is structured. Win-win (and perhaps -lose for taxpayers).
      - randmeerkat 3y ago
        > That's not really how it works. They are more than happy to have "market forces" drive up the hourly rates That’s exactly how it works, look at contract government salaries compared to anything in the private sector. They charge the government more as rates “go up”, but that certainly isn’t passed along. If large contracting companies really offered value to the government _and_ kept up with market rates for their employees, the state of federal software wouldn’t be what it is today.

uticus 3y ago

Of course, devil's advocate would point out COBOL in 60s was written with a lot more focus on lasting in general. They didn't largely have the same general attitude of 'we iterate quickly with our script kiddies' as is prevalent today. Not that all software from then was wonderful. Or that COBOL magically guarantees great code. But, on the flip side, you can't say the code base isn't battle-hardened (to various literal degrees). * edit: corrected sp

SteveNuts 3y ago
It'd be battle hardened to 1960s standards, payroll has changed a _lot_ since then. And if they can't accurately describe exactly how certain calculations are made (or have to do some of those manually outside of the program after it runs), that would be a huge problem in an audit. This is why big companies use Oracle or SAP for their ERP, auditors are very familiar with it and the way it functions is well-known and understood.
monocasa 3y ago
Also throwing out there that COBOL was a language designed by the DOD specifically for managing it's bureaucracy. It was a naval research project headed by Admiral Grace Hopper to program with words rather than formulas like earlier "high-level" "autocoders", enabling secretaries to transfer their business logic into computers. It actually intentionally has a lot of the same structure as a recipe, hoping that this would allow easier uptake by secretaries.
ebiester 3y ago
I can build you a team using today's (quality: average B2B startup) programmers and today's methodologies that will write software for 40 years. It will just cost twice as much as the team that iterates quickly. It will cost more because I will spend more money on business analysis up front. I will spend more money writing automated tests. Most importantly, I will maintain my own toolchain and libraries or pay someone to do so. Every piece of software deployed will have someone accountable to my business to maintain it. It turns out that this is expensive. It's no more expensive than maintaining the COBOL code, mind you, and would likely give you better results than the equivalent COBOL code. Most companies will take their chances with less expensive software because that's what their competitors do. About a decade ago, I saw one of the major systems you talk about up close. While they spent that amount of money for mission-critical systems, they didn't do nearly as good of a job on the average software in the system. I'm not convinced that any business is ready to go back to the battle hardened days.

encoderer 3y ago

I can imagine a future where new software is, mainly, written by humans while legacy software is maintained by ai. It can take months for a new developer to understand a legacy code base but an LLM with a big enough context window would be instantly productive.

Yahivin 3y ago
I can imagine the opposite.
- encoderer 3y ago
  Cool tell me more.
  - deleted 3y ago
    [deleted]
  - SteveNuts 3y ago
    Not OP but I could see some shops pushing AI generated code to production, then when changes need to be made, they can't get the AI to modify the existing code in just the way they need, so a human has to intervene.
  - bastawhiz 3y ago
    I can't get Copilot to generate Python that adds numbers together correctly sometimes. Getting an LLM to generate correct, working code for a language that hardly anybody writes anymore is almost assuredly going to lead to failure.
    - encoderer 3y ago
      yeah I agree but when you look at the slope not the y-intercept it’s getting obviously better. one advantage the government would have is training/fine-tuning on a hundred million lines of domain specific cobol.

deleted 3y ago

[deleted]

TurkishPoptart 3y ago

Is the problem really related to COBOL, or is the fact that this institution is basically unaccountable to taxpayers?

chatmasta 3y ago

See also, "John Stewart Questions Defense Deputy Secretary on Budget:" https://youtube.com/watch?v=50MusF365U0 https://youtube.com/watch?v=50MusF365U0