5 ms·

almost all major financial institutions, utilities, gov't agencies, etc still rely heavily on COBOL today. If it ain't (extremely) broken, don't fix it? COBOL

by mbloom1915 2y ago

almost all major financial institutions, utilities, gov't agencies, etc still rely heavily on COBOL today. If it ain't (extremely) broken, don't fix it? COBOL developers are literally dying out which has made for a competitive market for remaining talent. I've heard of some large consultants charging over $500/hr to their clients for a COBOL developer!

jeremyjh 2y ago
I think the moat that COBOL developers have is not just their knowledge of the language, but knowledge of the mainframe programming and operating environment. Its just so alien to developers familiar with Windows/Linux, and there is really no way to get experience with the environment that I know of, other than to be employed doing it. But yeah that stuff is never going away as far as I can tell. Its just too risky to rewrite those core systems and many a boondoggle has tried and failed.
- rodgerd 2y ago
  About a decade ago I looked into moving some COBOL components off-mainframe (either as COBOL-on-Linux or a rewrite into Java, which itself is really COBOL Mk II at this point), and your point about the operating environment is one of the key elements, but not all of it; there's also the fact that the first big shift to automation, via mainframe assembler and COBOL, is when companies sacked a lot of the folks who knew how and why the pre-automation processes worked - that knowledge exists in the mainframe code and the heads of the people who work(ed) on it, and nowhere else. A rewrite or a replatform is very, very hard and risky as a result; the system is now defined by how the mainframe runs the processes, to a very large degree. The third is that COBOL is only the tip of the iceberg. As soon as I spent time learning about the code I was being asked to look at, you get into decades of evolving programming practises. Modern COBOL is multithreaded, probably uses DB2 and relational datamodels. COBOL from thirty years ago is probably single-threaded, only runs right on high-clocked single-execution models, cuts down to hand-written s390 assembler regularly, and uses VSAM files with non-relational data. Older code still will be sharing data simply by banging it into memory regions for other code to read out of, because that's how you got performance back in the day. Trying to identify how you'd pull a function out of that and move it off is somewhere between extremely difficult and impossible. It's usually so complicated and expensive it's easier to try and hire people who want to apprentice as mainframe programmers and keep the current codebase running.
  - mschuster91 2y ago
    > A rewrite or a replatform is very, very hard and risky as a result; the system is now defined by how the mainframe runs the processes, to a very large degree. And that's why so many neo-banks/fintechs are eating the lunch of the established banks left and right, same for insurance. The "old guard" is unwilling to pay the costs of not just upgrading off of mainframes (aka the rewrite work itself)... but of changing their processes. That is where the real cost is at: When you have 213.000 employees like BoA has and everyone needs to have at least 10 hours of training and 2 weeks until they're familiar with the new system enough to be fully productive, that's like 2 million man-hours just for training and 16 million hours in lost productivity, so assuming $50/h average salary it's around 900 million dollars in cost. Unfortunately for the dinosaurs, the demands of both the customers and (at least in Europe) regulatory agencies especially for real-time financial transfers just push the old mainframe stuff to limits, while at the same time banks don't want to cede more and more of that cake to Paypal and friends that charge quite the sum for (effectively) lending money to banks. In contrast, all the newcomers start with greenfield IT, most likely some sort of more-or-less standard SAP. That one actually supports running unit and integration tests automatically, drastically reducing the chance of fuck-ups that might draw in unwanted regulatory attention.
    - jeremyjh 2y ago
      BOA doesn't train the vast, vast majority of its workforce on mainframe systems these days. No one working in a branch or call center is looking at green screens anymore. The mainframe systems are simply used as back-ends connected through web services (yes, even in CICS!) or MQ Series and the like to web GUIs. Source: worked there for many years, and built some of those integration systems.
    - panopticon 2y ago
      Eh, I think the tech stack is less important than the legal and regulatory structure. Most fintechs aren't banks and partner with a Real Bank™ to provide the actual bank accounts. Fintechs are under much less regulatory scrutiny (for now—that may be changing with recent, high-profile screwups) and can move with much more freedom regardless of the tech stack they've chosen.
- psunavy03 2y ago
  Migrations are still a thing, with various approaches and success rates.
amelius 2y ago
Can't we just apply a bunch of correctness preserving translations towards a modern PL, perhaps aided by an LLM to keep the source as human readable as possible, while (I'm stressing this) preserving correctness?
- exhilaration 2y ago
  IBM offers just such a service under the WatsonX branding, it's an LLM to convert COBOL to Java: https://www.ibm.com/products/watsonx-code-assistant-z https://www.ibm.com/products/watsonx-code-assistant-z I work at a company with a large COBOL codebase and this has been mentioned in a few presentations about our modernization efforts.
  - refneb 2y ago
    How did that go? My employer is going to try snd evaluate watsonx product. Have you had any luck converting large/complex COBOL modules ?
    - artificialLimbs 2y ago
      Don’t know about COBOL, but we recently threw some ancient RPG 3 programs into ChatGPT for conversion to both ‘plain English’ and PHP and it got pretty close - maybe 80-90%.
  - grammie 2y ago
    You should take a look at my company. Heirloom Computing. Heirloom.cc We have migrated many mainframe application and millions of lines of cobol and pl1 into Java and deployed it into production on prem and into the cloud.
  - russfink 2y ago
    But is the conversion maintainable by a human? I’ve seen Fortran to C translators that end up encoding state transition machines that are impossible to read.
- Muromec 2y ago
akavi 2y ago
I feel like every time COBOL is mentioned we get these stories about crazy high comp for COBOL developers, but anecdotally my aunt worked on COBOL projects in the mid 2010s and was paid a much more modest 45 $/hr. Good money for small town middle America where she lives, but nowhere close to what a decent JS dev can get.
- ghosty141 2y ago
  Similar experience with a friend of mine. I feel like these high salaries only apply to people who habe worked at one of these companies for a looong time.
  - adastra22 2y ago
    High salaries are relative. $90k is a high salary for most people in the world, even for tech workers outside of Silicon Valley.
  - Ekaros 2y ago
    High salaries are when a grey beard consultant is bought in for a few months to fix something or implement some new regulation. And I don't think it was that high was ever as high as financialised tech companies managed.
- chucksmash 2y ago
  There's also the difference between what a consulting company bills for the COBOL developer and what they pay the developer. Not every consultant is the captain of their own ship. My first job after college was a software shop organized in a "services" model, where clients would have to sponsor teams to do feature dev or support beyond initial onboarding. It's been a long time and my memory is hazy, but as I recall I was expected to bill ~40 hours a week to clients and if I only worked 40 hours that week (being OT exempt, this was always the goal), my hourly pay came out to between 10-20% of what the company billed the client. So $500/hr on the bill and $45/hr on the paycheck both manage to sound plausible, even at the same company.
IshKebab 2y ago
That seems like a myth to me. I actually looked up COBOL salaries and they were a bit higher (like 20%) but definitely not enough to make them tempting.
- francisofascii 2y ago
  There is typically a big difference between a consultant's hourly rate and a full time salary hourly rate.
  - IshKebab 2y ago
    Yeah exactly. I was comparing like for like (contracts or full time). The difference due to the fact that it was COBOL was definitely not enough to make me want to learn COBOL.
psunavy03 2y ago
In COBOL implementations, it's generally not just knowledge of the language that makes you valuable, it's knowledge of the implementation at that particular organization. I'm not a COBOL dev myself, but I work with them, and part of the challenge is that everything is so uber-customized, tightly coupled, and there's 40+ years of undocumented business logic buried in the code. It's like the old joke about the engineer being asked for an itemized bill: "Chalk mark: $1. Knowing where to put it: $4,999."
the_af 2y ago
COBOL jobs are not particularly well paid in my country. In any case, they would have to pay well by a large margin to justify working on dead boring legacy systems, too.
Muromec 2y ago
It is very much broken and said institutions don’t like it
bespokedevelopr 2y ago
I work for a major utility and they used to run everything on mainframe and cobol but that went away long before I started programming. My coworker is nearing retirement, around 30 years here, and he started on cobol and worked on transitioning off. He has some really fun stories but my point being, the tales of cobol prevalence are very exaggerated. Maybe some parts of finance are still using it, not my area.
Spooky23 2y ago
That’s for specialists for the mainframe or specific skill. Generalists are usually offshored and are cheap.
jillesvangurp 2y ago
Not a bad gig to take if you can swallow your pride a bit. I bet LLMs can make working with COBOL a lot easier and more fun than it ever was. I bet that's true for a lot of legacy stuff.
- the_af 2y ago
  Working with COBOL was never fun, so that's a low bar. Like others have said, what's valuable is an understanding of the business and legacy cruft that comes with spending time working at this kind of companies/banks/etc rather than knowledge of COBOL.

jeremyjh 2y ago

I think the moat that COBOL developers have is not just their knowledge of the language, but knowledge of the mainframe programming and operating environment. Its just so alien to developers familiar with Windows/Linux, and there is really no way to get experience with the environment that I know of, other than to be employed doing it. But yeah that stuff is never going away as far as I can tell. Its just too risky to rewrite those core systems and many a boondoggle has tried and failed.

rodgerd 2y ago
About a decade ago I looked into moving some COBOL components off-mainframe (either as COBOL-on-Linux or a rewrite into Java, which itself is really COBOL Mk II at this point), and your point about the operating environment is one of the key elements, but not all of it; there's also the fact that the first big shift to automation, via mainframe assembler and COBOL, is when companies sacked a lot of the folks who knew how and why the pre-automation processes worked - that knowledge exists in the mainframe code and the heads of the people who work(ed) on it, and nowhere else. A rewrite or a replatform is very, very hard and risky as a result; the system is now defined by how the mainframe runs the processes, to a very large degree. The third is that COBOL is only the tip of the iceberg. As soon as I spent time learning about the code I was being asked to look at, you get into decades of evolving programming practises. Modern COBOL is multithreaded, probably uses DB2 and relational datamodels. COBOL from thirty years ago is probably single-threaded, only runs right on high-clocked single-execution models, cuts down to hand-written s390 assembler regularly, and uses VSAM files with non-relational data. Older code still will be sharing data simply by banging it into memory regions for other code to read out of, because that's how you got performance back in the day. Trying to identify how you'd pull a function out of that and move it off is somewhere between extremely difficult and impossible. It's usually so complicated and expensive it's easier to try and hire people who want to apprentice as mainframe programmers and keep the current codebase running.
- mschuster91 2y ago
  > A rewrite or a replatform is very, very hard and risky as a result; the system is now defined by how the mainframe runs the processes, to a very large degree. And that's why so many neo-banks/fintechs are eating the lunch of the established banks left and right, same for insurance. The "old guard" is unwilling to pay the costs of not just upgrading off of mainframes (aka the rewrite work itself)... but of changing their processes. That is where the real cost is at: When you have 213.000 employees like BoA has and everyone needs to have at least 10 hours of training and 2 weeks until they're familiar with the new system enough to be fully productive, that's like 2 million man-hours just for training and 16 million hours in lost productivity, so assuming $50/h average salary it's around 900 million dollars in cost. Unfortunately for the dinosaurs, the demands of both the customers and (at least in Europe) regulatory agencies especially for real-time financial transfers just push the old mainframe stuff to limits, while at the same time banks don't want to cede more and more of that cake to Paypal and friends that charge quite the sum for (effectively) lending money to banks. In contrast, all the newcomers start with greenfield IT, most likely some sort of more-or-less standard SAP. That one actually supports running unit and integration tests automatically, drastically reducing the chance of fuck-ups that might draw in unwanted regulatory attention.
  - jeremyjh 2y ago
    BOA doesn't train the vast, vast majority of its workforce on mainframe systems these days. No one working in a branch or call center is looking at green screens anymore. The mainframe systems are simply used as back-ends connected through web services (yes, even in CICS!) or MQ Series and the like to web GUIs. Source: worked there for many years, and built some of those integration systems.
  - panopticon 2y ago
    Eh, I think the tech stack is less important than the legal and regulatory structure. Most fintechs aren't banks and partner with a Real Bank™ to provide the actual bank accounts. Fintechs are under much less regulatory scrutiny (for now—that may be changing with recent, high-profile screwups) and can move with much more freedom regardless of the tech stack they've chosen.
psunavy03 2y ago
Migrations are still a thing, with various approaches and success rates.

amelius 2y ago

Can't we just apply a bunch of correctness preserving translations towards a modern PL, perhaps aided by an LLM to keep the source as human readable as possible, while (I'm stressing this) preserving correctness?

exhilaration 2y ago
IBM offers just such a service under the WatsonX branding, it's an LLM to convert COBOL to Java: https://www.ibm.com/products/watsonx-code-assistant-z https://www.ibm.com/products/watsonx-code-assistant-z I work at a company with a large COBOL codebase and this has been mentioned in a few presentations about our modernization efforts.
- refneb 2y ago
  How did that go? My employer is going to try snd evaluate watsonx product. Have you had any luck converting large/complex COBOL modules ?
  - artificialLimbs 2y ago
    Don’t know about COBOL, but we recently threw some ancient RPG 3 programs into ChatGPT for conversion to both ‘plain English’ and PHP and it got pretty close - maybe 80-90%.
- grammie 2y ago
  You should take a look at my company. Heirloom Computing. Heirloom.cc We have migrated many mainframe application and millions of lines of cobol and pl1 into Java and deployed it into production on prem and into the cloud.
- russfink 2y ago
  But is the conversion maintainable by a human? I’ve seen Fortran to C translators that end up encoding state transition machines that are impossible to read.
Muromec 2y ago

akavi 2y ago

I feel like every time COBOL is mentioned we get these stories about crazy high comp for COBOL developers, but anecdotally my aunt worked on COBOL projects in the mid 2010s and was paid a much more modest 45 $/hr. Good money for small town middle America where she lives, but nowhere close to what a decent JS dev can get.

ghosty141 2y ago
Similar experience with a friend of mine. I feel like these high salaries only apply to people who habe worked at one of these companies for a looong time.
- adastra22 2y ago
  High salaries are relative. $90k is a high salary for most people in the world, even for tech workers outside of Silicon Valley.
- Ekaros 2y ago
  High salaries are when a grey beard consultant is bought in for a few months to fix something or implement some new regulation. And I don't think it was that high was ever as high as financialised tech companies managed.
chucksmash 2y ago
There's also the difference between what a consulting company bills for the COBOL developer and what they pay the developer. Not every consultant is the captain of their own ship. My first job after college was a software shop organized in a "services" model, where clients would have to sponsor teams to do feature dev or support beyond initial onboarding. It's been a long time and my memory is hazy, but as I recall I was expected to bill ~40 hours a week to clients and if I only worked 40 hours that week (being OT exempt, this was always the goal), my hourly pay came out to between 10-20% of what the company billed the client. So $500/hr on the bill and $45/hr on the paycheck both manage to sound plausible, even at the same company.

IshKebab 2y ago

That seems like a myth to me. I actually looked up COBOL salaries and they were a bit higher (like 20%) but definitely not enough to make them tempting.

francisofascii 2y ago
There is typically a big difference between a consultant's hourly rate and a full time salary hourly rate.
- IshKebab 2y ago
  Yeah exactly. I was comparing like for like (contracts or full time). The difference due to the fact that it was COBOL was definitely not enough to make me want to learn COBOL.

psunavy03 2y ago

In COBOL implementations, it's generally not just knowledge of the language that makes you valuable, it's knowledge of the implementation at that particular organization. I'm not a COBOL dev myself, but I work with them, and part of the challenge is that everything is so uber-customized, tightly coupled, and there's 40+ years of undocumented business logic buried in the code. It's like the old joke about the engineer being asked for an itemized bill: "Chalk mark: $1. Knowing where to put it: $4,999."

the_af 2y ago

COBOL jobs are not particularly well paid in my country. In any case, they would have to pay well by a large margin to justify working on dead boring legacy systems, too.

Muromec 2y ago

It is very much broken and said institutions don’t like it

bespokedevelopr 2y ago

I work for a major utility and they used to run everything on mainframe and cobol but that went away long before I started programming. My coworker is nearing retirement, around 30 years here, and he started on cobol and worked on transitioning off. He has some really fun stories but my point being, the tales of cobol prevalence are very exaggerated. Maybe some parts of finance are still using it, not my area.

Spooky23 2y ago

That’s for specialists for the mainframe or specific skill. Generalists are usually offshored and are cheap.

jillesvangurp 2y ago

Not a bad gig to take if you can swallow your pride a bit. I bet LLMs can make working with COBOL a lot easier and more fun than it ever was. I bet that's true for a lot of legacy stuff.

the_af 2y ago
Working with COBOL was never fun, so that's a low bar. Like others have said, what's valuable is an understanding of the business and legacy cruft that comes with spending time working at this kind of companies/banks/etc rather than knowledge of COBOL.