5 ms·

Start paying 250k a year for cobol developers and the problem fixes itself really fast. That or start migrating the system. Or more likely they keep using it un

by bogota 3y ago

Start paying 250k a year for cobol developers and the problem fixes itself really fast. That or start migrating the system. Or more likely they keep using it until it completely breaks and then run to the government for a big handout to fix it. Easy to be irresponsible with other peoples money.

lancepioch 3y ago
Don't they have caps and levels for federal employee salaries? I agree that raising the salary would decrease the issues. I think that they'd need exceptions to account for these increases though.
- Jtsummers 3y ago
  Yes, but not for contractors. Most software sustainment ultimately ends up in the hands of contractors with government supervision in the form of program offices. Not all, though, many programs are also "organic", primarily or fully staffed by government employees and managed by government employees.
  - randmeerkat 3y ago
    > Yes, but not for contractors. Sure, but it’s not like the consulting firms are paying their “contractors” more, they just siphon up the difference for their shareholders.
    - ska 3y ago
      That's not really how it works. They are more than happy to have "market forces" drive up the hourly rates, so long as they get to keep their overhead % fixed, and that's probably how the contract is structured. Win-win (and perhaps -lose for taxpayers).
      - randmeerkat 3y ago
        > That's not really how it works. They are more than happy to have "market forces" drive up the hourly rates That’s exactly how it works, look at contract government salaries compared to anything in the private sector. They charge the government more as rates “go up”, but that certainly isn’t passed along. If large contracting companies really offered value to the government _and_ kept up with market rates for their employees, the state of federal software wouldn’t be what it is today.
        ska 3y ago
        I didn't say that your rate at a govt focussed consultancy would be identical to your rate elsewhere. I probably should have been more explicit. I rejected the idea that the consultancy would get a rate increase based on "market rates being higher" and then just capture it all - in my (admittedly somewhat limited and path dependent) experience that just isn't how it works. It's more like we pay randmeerkat $X, we bill them out at $X * factor + overhead. "Market forces" mean we have to go Y > X in renewal or we will lose randmeerkat & friends, so now they get $Y and we bill $Y * factor + overhead. It's of course usually more complicated in general, and overhead especially likely isn't that simple. Nowhere in there is the assertion that X or Y is what randmeerkat would get on the open market. Importantly, their market isn't really "programmers", but "programmers that work in govt + contracting halo". Which is part of why the idea: I could get $N more in SF tech may be compelling for you, but isn't compelling for them (unless too many people actually make that change, instead of just talk about it). Also there are many other ways they can raise there rates, but if the claim is that it is due to market on the developer salaries that are going to be a line item, there is going to be at least first pass look at a) is that true (find some market data and wave hands) b) did it actually get spent that way (may come up in an audit). So the real answer seems to be not "They get more and I get nothing" but "I get a bit more, and they get more scaled by what I get", i.e. "win-win". There is lots wrong with the system of contracting, but I don't think criticizing a cartoony straw-man of it gets anywhere useful.
nickpeterson 3y ago
Part of the problem is, you tend to need cobol experts to actually have any hope of successfully transitioning to another technology, and if we had enough cobol experts would look at the cost of migrating and say, “who needs to, we have a bunch of cobol developers?”
- adamc 3y ago
  Hire experts and have it be reported as "waste" and hear complaints in Congress about government waste. We want government to be cheap. You get what you pay for.
toasted-subs 3y ago
Starting pay for cobol in finance was around 350k last time I glanced. Nobody wants to rewrite a working system which will likely be much more expensive than finding one person to maintain the project.
- saiya-jin 3y ago
  With rewrite comes uncertainty and risk too, how many new bugs compared to code that works for decades?
MrMetlHed 3y ago
In the last part of the series from 2013 we have a chart that shows the Pentagon had spent over $10b modernizing their accounting systems (that was through 2013, no idea how much that number has grown.) I can't imagine anyone would miss a few million of that going to COBOL developers. The money scale for Pentagon projects is mind-boggling to me.
thelastgallon 3y ago
Incentives only work on low-skilled labor that is quantifiable. Eg. more money for flipping more burgers. It is not a settled science paying people a lot more will automagically solve problems. People crave Autonomy, Mastery & Purpose: https://www.youtube.com/watch?v=rbR2V1UeB_A https://www.youtube.com/watch?v=rbR2V1UeB_A What is definitely going to happen is you will attract the cowboys who have convinced themselves and can convince you to pay top dollar. They will come up with lets do kubernetes + service-mesh + multi-cloud hybrid + blockchain + crypto + AI/MI on this COBOL ... They will take top dollar, make a bigger pile of mess, sell it as a success story and move on make bigger messes.
- whatshisface 3y ago
  250k is not top dollar, it is average for experienced engineers.
  - Kranar 3y ago
    250k is not top dollar for programmers, but it is top dollar in absolute terms. The point is that you don't incentivize better work by paying top dollar. Taking a mediocre developer and paying them more doesn't make them a better developer or even motivate them to become better. It can attract people who are already good software developers absolutely, but the second point being made is that you also attract a lot of charlatans as well and so just throwing more money at the problem isn't exactly a great solution. You need a solid culture to go with the money.
  - ramilefu 3y ago
    Depends on where you live. An “experienced engineer” is going to make a lot less if they live in Mississippi than if they were in California. 250k for the former is likely “top dollar”.
    - 1323portloo 3y ago
      If you are a small shop, sure. If you are a major employer you will either meet market rate and get top candidates (willing to live in your area) or you will get whatever candidates decided to live in Mississippi.
curiousllama 3y ago
Cobol cowboys are a thing. They make substantially more than 250k. They’re competing with Wall St though, so hard to just pay more. Fed govt migrations are ALWAYS shitshows. The army alone has like 20 individual email services. The pentagon is to org complexity what big tech is to technical complexity. It’s turtles all the way down. This is literally one of the largest bureaucratic challenges on earth. There’s no simple fixes
- mschuster91 3y ago
  The solution is simple: you need someone with absolute authority over everyone including the generals to lead the effort. There's a time and place for democracy - but large IT projects are not. Do a thorough need analysis, compare with what's reasonably possible using COTS software, and adapt or discard what is not possible.
  - deleted 3y ago
    [deleted]
  - count 3y ago
    I can't tell if you're being sarcastic, or earnest. This is the funniest possible response. The 'thorough need analysis' is the reason we are where we are now! The thoroughness is so thorough that the needs change faster than the analysis can be completed, obviating the analysis in the first place.
  - vajrabum 3y ago
    That's exactly why DoD kept bringing Grace Hopper out of retirement first in 1967. They eventually gave her a star in the 80s, and along the way she was as responsible as anyone for the standardization efforts for COBOL and Fortran including the design of standardized compiler validation suites. Testing and interoperability before testing or interoperability were a thing! One of the later major projects she worked on was modernizing the Navy payroll system. Even junior sailors at the time knew that she and her team had made sure they got their paychecks. She retired from the Navy at age 79 in 1986. Nobody messed with Grace Hopper.
ska 3y ago
> Start paying 250k a year for cobol developers and the problem fixes itself really fast. Do you think they aren't? I don't know any now, but the last one I did was making more than that 15 years ago. Old crufty legacy systems in a big bureaucracy are many peoples idea of hell. Empirically speaking, good pay and job security aren't enough to have people knocking down your door for that kind of work if they have other decent options.
wubrr 3y ago
That could work if the existing code is actually up-to-date, correct, and if you have knowledgeable users of the systems backed by this code. The part about 'code corruption' makes me thing this is not the case. Worked at a big bank migrating some old code to new stack one time: - Original developers are gone. - Existing users don't understand the systems outside of their own narrow use-cases. - Code which was committed to repositories didn't match the actual running code, because people would go into production servers, manually edit the code there, in place, not document or commit the changes to repo. There were like 20 years of adhoc changes made in this way. We had some business analysts which were supposed to talk to the users and gather requirements - they very quickly simply gave up and asked the developers to 'reverse engineer' everything. Anyway, it was a hilarious shit-show, but kinda fun at the same time (if you're able to ignore incompetent leadership pressure - a skill I mastered during this time).
- obviouslynotme 3y ago
  >Code which was committed to repositories didn't match the actual running code I remember the first time I saw this at a small company. The head programmer was the only one with the actual production source, despite a working source control system. He complained about how people didn't appreciate how hard it is to make builds work. I should have left when we lost three man months of code to losing email. I only left after screaming at the head programmer for refusing to stop making the repository unbuildable at all.
ravenstine 3y ago
This is a problem not just in software. Today, everyone wants a senior <insert job role> for the price of a junior <insert job role>. So many problems would be solved orders of magnitude more soon if people were properly compensated. Instead, most things in the United States are running on economic fumes.

by bogota 3y ago

lancepioch 3y ago

Don't they have caps and levels for federal employee salaries? I agree that raising the salary would decrease the issues. I think that they'd need exceptions to account for these increases though.

Jtsummers 3y ago
Yes, but not for contractors. Most software sustainment ultimately ends up in the hands of contractors with government supervision in the form of program offices. Not all, though, many programs are also "organic", primarily or fully staffed by government employees and managed by government employees.
- randmeerkat 3y ago
  > Yes, but not for contractors. Sure, but it’s not like the consulting firms are paying their “contractors” more, they just siphon up the difference for their shareholders.
  - ska 3y ago
    That's not really how it works. They are more than happy to have "market forces" drive up the hourly rates, so long as they get to keep their overhead % fixed, and that's probably how the contract is structured. Win-win (and perhaps -lose for taxpayers).
    - randmeerkat 3y ago
      > That's not really how it works. They are more than happy to have "market forces" drive up the hourly rates That’s exactly how it works, look at contract government salaries compared to anything in the private sector. They charge the government more as rates “go up”, but that certainly isn’t passed along. If large contracting companies really offered value to the government _and_ kept up with market rates for their employees, the state of federal software wouldn’t be what it is today.
      - ska 3y ago
        I didn't say that your rate at a govt focussed consultancy would be identical to your rate elsewhere. I probably should have been more explicit. I rejected the idea that the consultancy would get a rate increase based on "market rates being higher" and then just capture it all - in my (admittedly somewhat limited and path dependent) experience that just isn't how it works. It's more like we pay randmeerkat $X, we bill them out at $X * factor + overhead. "Market forces" mean we have to go Y > X in renewal or we will lose randmeerkat & friends, so now they get $Y and we bill $Y * factor + overhead. It's of course usually more complicated in general, and overhead especially likely isn't that simple. Nowhere in there is the assertion that X or Y is what randmeerkat would get on the open market. Importantly, their market isn't really "programmers", but "programmers that work in govt + contracting halo". Which is part of why the idea: I could get $N more in SF tech may be compelling for you, but isn't compelling for them (unless too many people actually make that change, instead of just talk about it). Also there are many other ways they can raise there rates, but if the claim is that it is due to market on the developer salaries that are going to be a line item, there is going to be at least first pass look at a) is that true (find some market data and wave hands) b) did it actually get spent that way (may come up in an audit). So the real answer seems to be not "They get more and I get nothing" but "I get a bit more, and they get more scaled by what I get", i.e. "win-win". There is lots wrong with the system of contracting, but I don't think criticizing a cartoony straw-man of it gets anywhere useful.

nickpeterson 3y ago

Part of the problem is, you tend to need cobol experts to actually have any hope of successfully transitioning to another technology, and if we had enough cobol experts would look at the cost of migrating and say, “who needs to, we have a bunch of cobol developers?”

adamc 3y ago
Hire experts and have it be reported as "waste" and hear complaints in Congress about government waste. We want government to be cheap. You get what you pay for.

toasted-subs 3y ago

Starting pay for cobol in finance was around 350k last time I glanced. Nobody wants to rewrite a working system which will likely be much more expensive than finding one person to maintain the project.

saiya-jin 3y ago
With rewrite comes uncertainty and risk too, how many new bugs compared to code that works for decades?

MrMetlHed 3y ago

In the last part of the series from 2013 we have a chart that shows the Pentagon had spent over $10b modernizing their accounting systems (that was through 2013, no idea how much that number has grown.) I can't imagine anyone would miss a few million of that going to COBOL developers. The money scale for Pentagon projects is mind-boggling to me.

thelastgallon 3y ago

Incentives only work on low-skilled labor that is quantifiable. Eg. more money for flipping more burgers. It is not a settled science paying people a lot more will automagically solve problems. People crave Autonomy, Mastery & Purpose: https://www.youtube.com/watch?v=rbR2V1UeB_A https://www.youtube.com/watch?v=rbR2V1UeB_A What is definitely going to happen is you will attract the cowboys who have convinced themselves and can convince you to pay top dollar. They will come up with lets do kubernetes + service-mesh + multi-cloud hybrid + blockchain + crypto + AI/MI on this COBOL ... They will take top dollar, make a bigger pile of mess, sell it as a success story and move on make bigger messes.

whatshisface 3y ago
250k is not top dollar, it is average for experienced engineers.
- Kranar 3y ago
  250k is not top dollar for programmers, but it is top dollar in absolute terms. The point is that you don't incentivize better work by paying top dollar. Taking a mediocre developer and paying them more doesn't make them a better developer or even motivate them to become better. It can attract people who are already good software developers absolutely, but the second point being made is that you also attract a lot of charlatans as well and so just throwing more money at the problem isn't exactly a great solution. You need a solid culture to go with the money.
- ramilefu 3y ago
  Depends on where you live. An “experienced engineer” is going to make a lot less if they live in Mississippi than if they were in California. 250k for the former is likely “top dollar”.
  - 1323portloo 3y ago
    If you are a small shop, sure. If you are a major employer you will either meet market rate and get top candidates (willing to live in your area) or you will get whatever candidates decided to live in Mississippi.

curiousllama 3y ago

Cobol cowboys are a thing. They make substantially more than 250k. They’re competing with Wall St though, so hard to just pay more. Fed govt migrations are ALWAYS shitshows. The army alone has like 20 individual email services. The pentagon is to org complexity what big tech is to technical complexity. It’s turtles all the way down. This is literally one of the largest bureaucratic challenges on earth. There’s no simple fixes

mschuster91 3y ago
The solution is simple: you need someone with absolute authority over everyone including the generals to lead the effort. There's a time and place for democracy - but large IT projects are not. Do a thorough need analysis, compare with what's reasonably possible using COTS software, and adapt or discard what is not possible.
- deleted 3y ago
  [deleted]
- count 3y ago
  I can't tell if you're being sarcastic, or earnest. This is the funniest possible response. The 'thorough need analysis' is the reason we are where we are now! The thoroughness is so thorough that the needs change faster than the analysis can be completed, obviating the analysis in the first place.
- vajrabum 3y ago
  That's exactly why DoD kept bringing Grace Hopper out of retirement first in 1967. They eventually gave her a star in the 80s, and along the way she was as responsible as anyone for the standardization efforts for COBOL and Fortran including the design of standardized compiler validation suites. Testing and interoperability before testing or interoperability were a thing! One of the later major projects she worked on was modernizing the Navy payroll system. Even junior sailors at the time knew that she and her team had made sure they got their paychecks. She retired from the Navy at age 79 in 1986. Nobody messed with Grace Hopper.

ska 3y ago

> Start paying 250k a year for cobol developers and the problem fixes itself really fast. Do you think they aren't? I don't know any now, but the last one I did was making more than that 15 years ago. Old crufty legacy systems in a big bureaucracy are many peoples idea of hell. Empirically speaking, good pay and job security aren't enough to have people knocking down your door for that kind of work if they have other decent options.

wubrr 3y ago

That could work if the existing code is actually up-to-date, correct, and if you have knowledgeable users of the systems backed by this code. The part about 'code corruption' makes me thing this is not the case. Worked at a big bank migrating some old code to new stack one time: - Original developers are gone. - Existing users don't understand the systems outside of their own narrow use-cases. - Code which was committed to repositories didn't match the actual running code, because people would go into production servers, manually edit the code there, in place, not document or commit the changes to repo. There were like 20 years of adhoc changes made in this way. We had some business analysts which were supposed to talk to the users and gather requirements - they very quickly simply gave up and asked the developers to 'reverse engineer' everything. Anyway, it was a hilarious shit-show, but kinda fun at the same time (if you're able to ignore incompetent leadership pressure - a skill I mastered during this time).

obviouslynotme 3y ago
>Code which was committed to repositories didn't match the actual running code I remember the first time I saw this at a small company. The head programmer was the only one with the actual production source, despite a working source control system. He complained about how people didn't appreciate how hard it is to make builds work. I should have left when we lost three man months of code to losing email. I only left after screaming at the head programmer for refusing to stop making the repository unbuildable at all.

ravenstine 3y ago

This is a problem not just in software. Today, everyone wants a senior <insert job role> for the price of a junior <insert job role>. So many problems would be solved orders of magnitude more soon if people were properly compensated. Instead, most things in the United States are running on economic fumes.