6 ms·

Our Government Runs on a 60-Year-Old Coding Language, and Now Its Falling Apart

20 points by gHeadphone 7y ago

jerzyt 7y ago
What is far more scary than the old stuff our government is running in COBOL, is the new stuff running in Excel. Excel became a de facto Common Business Oriented Language (COBOL), accessibly to anybody. Old COBOL programs were still developed with some software engineering discipline. Current Excel spreadsheets have none of that. And, this doesn't apply to the government only - private enterprise runs bazillions(?) untested, unverified spreadsheets. Wish I could share what I've seen.
- gHeadphone 7y ago
  That’s true. I used to work in one of the biggest banks in the world. More than half the divisions were dependent on Excel. Every time there was an audit, a huge bunch of spreadsheets were automated. By the time the automation was complete, another hundred spreadsheets would have popped up. It was like trying to spray weeds. I remember one spreadsheet that contained the only formula for calculating risk in an almost billion dollar portfolio. Genuinely scary stuff.
sullyj3 7y ago
Not a great headline really, C is nearly 50, but most people wouldn't consider it a disaster for everything to be written in C. The problem isn't that COBOL is old, it's that it's largely obsolete.
- TomMarius 7y ago
  Who would not consider a information system application written in C a disaster? Can't think of anyone who who would want to work on that. I agree with your point though.
  - jatone 7y ago
    no, because literally every application except the most trivial touch c. every major RBMS is c or c++. you don't write frontend's with c/c++ but if you're interacting with a datastore most are c. the problem with c and c++'s age are legacy behaviors related to them we've since determined are not beneficial to actual programs and make reasoning about them harder.
    - jjeaff 7y ago
      Of course everything touches c. But it's mostly all open source stuff that is maintained by a large community of coders and is mostly underlying dependencies. When you are developing software, you just want that stuff to work. You don't want to have to dig around in the internals to figure things out. That's why scripting languages have become so popular.
    - TomMarius 7y ago
      I don't know anyone who would consider calling a C library or other software "programming in C".
otabdeveloper4 7y ago
> coding language
theincredulousk 7y ago
COBOL isn’t falling apart, the systems are exceeding their design limits. “Oh if it were just written in a modern language everything would be fine”. Not true - we create modern systems every day that would choke and die if they had to scale several orders of magnitude instantly - in fact it happens about weekly when an unsuspecting link hits the HN front page. Also, I would wager 90%+ of software written today is 10-100x more fragile over time than whatever COBOL software is on these machines. What do you think the chances are of those 100 fancifully orchestrated micro-services running 10 abstraction layers up on 3rd party infrastructure are of running at all a decade from now, never mind within their original design constraints. Old doesn’t necessarily mean worse and newer doesn’t necessarily mean better, but that logical foundation is easy to forget in a tech culture so often myopically focused on innovation(tm) and short-term profit or austerity above robustness. The relative rarity of COBOL programmers aside, this is a just an garden variety f* up of keeping legacy systems around too long because they aren’t “broken”. And like technical debt, the money and effort to fix them, despite many warnings and pleas, is called a waste because there is no immediate benefit or profit. Then something like this happens, and everyone forgets the shortsighted management or decision makers that ignored the risk and collected bonuses and praise for saving the Corp/taxpayers money. Those who warned about the debt coming due get nothing except being able to say “I told you so” for a few weeks until the whole thing blows over.
- gregjor 7y ago
  I agree with what you wrote, this is not a problem with the COBOL language, except in the narrow sense that it’s not taught anymore and probably 95% of programmers working today have never seen COBOL. A professional programmer with some experience can learn COBOL in a few weeks. It’s not Linear A. The cost of fixing technical debt has to outweigh the cost and risks of replacing legacy systems. Most ground-up rewrites fail, and that’s even more of a problem in the government sector. A working system with problems is still better than an imaginary system that will probably never go live. Google IRS, California DMV, Oregon utilities, etc. etc. for examples of how these rewrites actually work out.
  - theincredulousk 7y ago
_bxg1 7y ago
*60-Year-Old software systems
smcphile 7y ago
My first computing job, many years ago, was maintaining a large, COBOL program for an insurance company. I doubt the problem the American government is now facing has much to do with COBOL in itself. It probably has much more to do with the fact that these COBOL programs were designed to run on IBM mainframe computers and to interact with various proprietary IBM subsystems (with names like CICS, VSAM, VTAM, DB2, etc.) These subsystems are non-trivial to learn and there’s been little/no incentive for anyone new to learn them for the last (at least) twenty years. Finally, back in the 80’s and 90’s, some programs were written in IBM assembler language instead of COBOL, for example when there was a need for greater speed. Understanding and maintaining (or porting these programs to a non-mainframe environment) is also non-trivial. I left the IBM mainframe world completely 20 years ago, switched over to working only on Unix/Linux systems, did that for 20 years, and am now happily retired. However I’m sure there are still lots of older IBM mainframe people around to handle these problems, they just cost money.
- gregjor 7y ago
  A lot of the problem stems from the databases. COBOL, and these legacy systems, pre-date relational databases. They work with proprietary and custom file formats, which COBOL was designed to work well with. Regardless of technical debt or scaling problems, the data in these systems has tremendous value, and converting it to a modern relational structure is a bigger job than fixing some bugs in COBOL code.
jcrawfordor 7y ago
Attributing these problems to COBOL is short sighted. While COBOL is obsolete for good reasons, it does provide high performance and makes it relatively easy to construct new batch-processing tasks quickly, due to its built-in data model (a concept which is largely absent from modern programming languages but does pop up from time to time in forms like MS LINQ - historically it was far more common, see also MUMPS). Because COBOL applications run predominantly in batch-mode they are also fairly robust and easy to troubleshoot compared to many modern systems, isolating a problem being a matter of running the batch jobs individually and finding the bad intermediary state, and recovering from a failure being a matter of just restarting the job. Intermediate state is almost always persisted to storage so it's easy to work back to where the problem occurred, unlike in e.g. a microservice architecture where doing so can require a dedicated tracing system and still be quite difficult. Consider that we could write a headline like "Amazon runs on a 50-year-old operating system, and now it's falling apart." The sentence is not entirely untrue but rather misses the point. The problem is this: these systems are old and, more significantly, have received extremely limited maintenance and enhancement over their lifetimes. This means that there are few people familiar with each system (a "bus factor" problem) and, more broadly, that the talent-base of COBOL programmers has been allowed to recede to near-zero. While a common modernization (or "recapitalization," a term preferred in government contracting) route in the tech industry might be to take a legacy system written in e.g. Java and progressively reimplement it in e.g. Python with a simplified architecture, a common recap route in government is to take a legacy system written in COBOL or s/370 assembly and transition it to an s/370 emulator running on HPUX on NonStop. I've been involved in such processes a couple of times. It has the advantage of maintaining the reliability properties of the legacy system at relatively low-cost since very little software work is involved. It has the tremendous disadvantage of allowing the organization to continue to not touch the software, with much of the work often outsourced to a vendor who need not be very familiar with the codebase either You certainly wouldn't write anything in COBOL in 2020, but COBOL itself does have a robust and even fairly maintainable design. The problem is that any application written 20 years ago or more, regardless of language, is going to be tremendously difficult to modify to meet changing requirements and increased demand. Unemployment insurance applications reaching six million per week, for example, would strain a system designed using the most modern methodology which was specified for a few hundred thousand. The difference is that, with a newer system, there are likely to be in-house engineers or vendors who are very familiar with the system and will be able to expand its capacity more rapidly. Another underlying problem is that legacy systems designed using a mainframe methodology are intended for vertical scaling - to process more records you buy a bigger machine (or expand the LPAR or what have you). While horizontal scaling has its own disadvantages there's a reason it's so popular today - it's a lot faster to buy more capacity from a cloud provider than to requisition a larger mainframe^w midcomputer^w emulation host. There are few vendors and government acquisitions are slow at the best of times, even under emergency sole-source rules. In a way the whole thing is ironic, because a lot of COBOL application interfaces are faster than any software we use today. From the perspective of a human operator they can be absolutely outstanding once you get over a learning curve. However, the lack of any enhancements made to these systems mean that requirements changes are often "patched in peopleware" if you will, and you end up with clerks that can enter data an order of magnitude more efficiently than with a likely modern solution, but they then have to send the records to a virtual printer to be ingested into a Java EE solution... this kind of "enterprise patching" rapidly adds up to an extremely inefficient system.
throwaway888abc 7y ago
Informative comments. Thanks!
m463 7y ago
It seems like the age of something is the argument. I starting thinking of this and thought: - Much modern AI software depends on 63-year old fortran numerical methods code. - the Brooklyn Bridge is 137 years old. - the White House is 220 years old - Aristotle was using logical reasoning over 2300 years ago.
cafard 7y ago
I think that The Onion should run an article: "Coders Shocked to Find Accounting Systems Written in a Language Designed for Accounting Systems."
GnarfGnarf 7y ago
The problem with rewriting the COBOL code base in a newer, sexier language, is that this new language will itself become obsolete in ten or twenty years. Then you're back to square one re-writing the app in another newer language. So businesses faced with this problem have a choice: accept that their code is written in an old language, and train programmers to learn it. Or pay big money to convert their code into... a language that will soon be obsolete, in an endless treadmill. Isn't it better to be obsolete and not pay conversion costs, rather than be obsolete and pay perpetual conversion costs? Yes, programmers are more productive in newer languages. However, you are never going to get rid of the old stuff, so you will always need some programmers versed in every language, at every stage of your conversion hamster wheel.
gregjor 7y ago
Now I know how to reply when someone posts “I have a week with nothing to do. What language should I learn next?” COBOL. Seriously, anyone with some programming experience can learn COBOL or any other language in a few weeks. Maybe with so many hip startups for concierge cat sitting closing down more programmers with time on their hands and no paycheck will crack the McCracken book (the one I learned COBOL from ages ago).
- non-entity 7y ago
  If i could get hired with a good rate after just learning the COBOL language maybe. Unfortunately, I don't have a mainframe for me to learn the rest of the environment in.
  - lboc 7y ago
    IBM give away free year-long z/OS accounts: https://news.ycombinator.com/item?id=20873531 https://news.ycombinator.com/item?id=20873531
- ornornor 7y ago
  What are the real world prospects for cobol developers? I’m seriously asking. I’m tired of running the JavaScript treadmill, I’d much rather invest the time to learn cobol (which I bet doesn’t change everything about how things are done every 6 months) if I can be reasonably sure it comes with a steady amount of work and a good rate. But so far all I’ve seen is people saying “welp others should learn cobol because there aren’t many programmers still in the workforce who know this language and it pays big bucks” but I could never find hard evidence. Where are the job ads? Where are the typical rates?

20 points by gHeadphone 7y ago

What is far more scary than the old stuff our government is running in COBOL, is the new stuff running in Excel. Excel became a de facto Common Business Oriented Language (COBOL), accessibly to anybody. Old COBOL programs were still developed with some software engineering discipline. Current Excel spreadsheets have none of that. And, this doesn't apply to the government only - private enterprise runs bazillions(?) untested, unverified spreadsheets. Wish I could share what I've seen.

gHeadphone 7y ago
That’s true. I used to work in one of the biggest banks in the world. More than half the divisions were dependent on Excel. Every time there was an audit, a huge bunch of spreadsheets were automated. By the time the automation was complete, another hundred spreadsheets would have popped up. It was like trying to spray weeds. I remember one spreadsheet that contained the only formula for calculating risk in an almost billion dollar portfolio. Genuinely scary stuff.

sullyj3 7y ago

Not a great headline really, C is nearly 50, but most people wouldn't consider it a disaster for everything to be written in C. The problem isn't that COBOL is old, it's that it's largely obsolete.

TomMarius 7y ago
Who would not consider a information system application written in C a disaster? Can't think of anyone who who would want to work on that. I agree with your point though.
- jatone 7y ago
  no, because literally every application except the most trivial touch c. every major RBMS is c or c++. you don't write frontend's with c/c++ but if you're interacting with a datastore most are c. the problem with c and c++'s age are legacy behaviors related to them we've since determined are not beneficial to actual programs and make reasoning about them harder.
  - jjeaff 7y ago
    Of course everything touches c. But it's mostly all open source stuff that is maintained by a large community of coders and is mostly underlying dependencies. When you are developing software, you just want that stuff to work. You don't want to have to dig around in the internals to figure things out. That's why scripting languages have become so popular.
  - TomMarius 7y ago
    I don't know anyone who would consider calling a C library or other software "programming in C".

otabdeveloper4 7y ago

> coding language

theincredulousk 7y ago

COBOL isn’t falling apart, the systems are exceeding their design limits. “Oh if it were just written in a modern language everything would be fine”. Not true - we create modern systems every day that would choke and die if they had to scale several orders of magnitude instantly - in fact it happens about weekly when an unsuspecting link hits the HN front page. Also, I would wager 90%+ of software written today is 10-100x more fragile over time than whatever COBOL software is on these machines. What do you think the chances are of those 100 fancifully orchestrated micro-services running 10 abstraction layers up on 3rd party infrastructure are of running at all a decade from now, never mind within their original design constraints. Old doesn’t necessarily mean worse and newer doesn’t necessarily mean better, but that logical foundation is easy to forget in a tech culture so often myopically focused on innovation(tm) and short-term profit or austerity above robustness. The relative rarity of COBOL programmers aside, this is a just an garden variety f* up of keeping legacy systems around too long because they aren’t “broken”. And like technical debt, the money and effort to fix them, despite many warnings and pleas, is called a waste because there is no immediate benefit or profit. Then something like this happens, and everyone forgets the shortsighted management or decision makers that ignored the risk and collected bonuses and praise for saving the Corp/taxpayers money. Those who warned about the debt coming due get nothing except being able to say “I told you so” for a few weeks until the whole thing blows over.

gregjor 7y ago
I agree with what you wrote, this is not a problem with the COBOL language, except in the narrow sense that it’s not taught anymore and probably 95% of programmers working today have never seen COBOL. A professional programmer with some experience can learn COBOL in a few weeks. It’s not Linear A. The cost of fixing technical debt has to outweigh the cost and risks of replacing legacy systems. Most ground-up rewrites fail, and that’s even more of a problem in the government sector. A working system with problems is still better than an imaginary system that will probably never go live. Google IRS, California DMV, Oregon utilities, etc. etc. for examples of how these rewrites actually work out.
- theincredulousk 7y ago

_bxg1 7y ago

*60-Year-Old software systems

smcphile 7y ago

My first computing job, many years ago, was maintaining a large, COBOL program for an insurance company. I doubt the problem the American government is now facing has much to do with COBOL in itself. It probably has much more to do with the fact that these COBOL programs were designed to run on IBM mainframe computers and to interact with various proprietary IBM subsystems (with names like CICS, VSAM, VTAM, DB2, etc.) These subsystems are non-trivial to learn and there’s been little/no incentive for anyone new to learn them for the last (at least) twenty years. Finally, back in the 80’s and 90’s, some programs were written in IBM assembler language instead of COBOL, for example when there was a need for greater speed. Understanding and maintaining (or porting these programs to a non-mainframe environment) is also non-trivial. I left the IBM mainframe world completely 20 years ago, switched over to working only on Unix/Linux systems, did that for 20 years, and am now happily retired. However I’m sure there are still lots of older IBM mainframe people around to handle these problems, they just cost money.

gregjor 7y ago
A lot of the problem stems from the databases. COBOL, and these legacy systems, pre-date relational databases. They work with proprietary and custom file formats, which COBOL was designed to work well with. Regardless of technical debt or scaling problems, the data in these systems has tremendous value, and converting it to a modern relational structure is a bigger job than fixing some bugs in COBOL code.

jcrawfordor 7y ago

Attributing these problems to COBOL is short sighted. While COBOL is obsolete for good reasons, it does provide high performance and makes it relatively easy to construct new batch-processing tasks quickly, due to its built-in data model (a concept which is largely absent from modern programming languages but does pop up from time to time in forms like MS LINQ - historically it was far more common, see also MUMPS). Because COBOL applications run predominantly in batch-mode they are also fairly robust and easy to troubleshoot compared to many modern systems, isolating a problem being a matter of running the batch jobs individually and finding the bad intermediary state, and recovering from a failure being a matter of just restarting the job. Intermediate state is almost always persisted to storage so it's easy to work back to where the problem occurred, unlike in e.g. a microservice architecture where doing so can require a dedicated tracing system and still be quite difficult. Consider that we could write a headline like "Amazon runs on a 50-year-old operating system, and now it's falling apart." The sentence is not entirely untrue but rather misses the point. The problem is this: these systems are old and, more significantly, have received extremely limited maintenance and enhancement over their lifetimes. This means that there are few people familiar with each system (a "bus factor" problem) and, more broadly, that the talent-base of COBOL programmers has been allowed to recede to near-zero. While a common modernization (or "recapitalization," a term preferred in government contracting) route in the tech industry might be to take a legacy system written in e.g. Java and progressively reimplement it in e.g. Python with a simplified architecture, a common recap route in government is to take a legacy system written in COBOL or s/370 assembly and transition it to an s/370 emulator running on HPUX on NonStop. I've been involved in such processes a couple of times. It has the advantage of maintaining the reliability properties of the legacy system at relatively low-cost since very little software work is involved. It has the tremendous disadvantage of allowing the organization to continue to not touch the software, with much of the work often outsourced to a vendor who need not be very familiar with the codebase either You certainly wouldn't write anything in COBOL in 2020, but COBOL itself does have a robust and even fairly maintainable design. The problem is that any application written 20 years ago or more, regardless of language, is going to be tremendously difficult to modify to meet changing requirements and increased demand. Unemployment insurance applications reaching six million per week, for example, would strain a system designed using the most modern methodology which was specified for a few hundred thousand. The difference is that, with a newer system, there are likely to be in-house engineers or vendors who are very familiar with the system and will be able to expand its capacity more rapidly. Another underlying problem is that legacy systems designed using a mainframe methodology are intended for vertical scaling - to process more records you buy a bigger machine (or expand the LPAR or what have you). While horizontal scaling has its own disadvantages there's a reason it's so popular today - it's a lot faster to buy more capacity from a cloud provider than to requisition a larger mainframe^w midcomputer^w emulation host. There are few vendors and government acquisitions are slow at the best of times, even under emergency sole-source rules. In a way the whole thing is ironic, because a lot of COBOL application interfaces are faster than any software we use today. From the perspective of a human operator they can be absolutely outstanding once you get over a learning curve. However, the lack of any enhancements made to these systems mean that requirements changes are often "patched in peopleware" if you will, and you end up with clerks that can enter data an order of magnitude more efficiently than with a likely modern solution, but they then have to send the records to a virtual printer to be ingested into a Java EE solution... this kind of "enterprise patching" rapidly adds up to an extremely inefficient system.

throwaway888abc 7y ago

Informative comments. Thanks!

m463 7y ago

It seems like the age of something is the argument. I starting thinking of this and thought: - Much modern AI software depends on 63-year old fortran numerical methods code. - the Brooklyn Bridge is 137 years old. - the White House is 220 years old - Aristotle was using logical reasoning over 2300 years ago.

cafard 7y ago

I think that The Onion should run an article: "Coders Shocked to Find Accounting Systems Written in a Language Designed for Accounting Systems."

GnarfGnarf 7y ago

The problem with rewriting the COBOL code base in a newer, sexier language, is that this new language will itself become obsolete in ten or twenty years. Then you're back to square one re-writing the app in another newer language. So businesses faced with this problem have a choice: accept that their code is written in an old language, and train programmers to learn it. Or pay big money to convert their code into... a language that will soon be obsolete, in an endless treadmill. Isn't it better to be obsolete and not pay conversion costs, rather than be obsolete and pay perpetual conversion costs? Yes, programmers are more productive in newer languages. However, you are never going to get rid of the old stuff, so you will always need some programmers versed in every language, at every stage of your conversion hamster wheel.

gregjor 7y ago

Now I know how to reply when someone posts “I have a week with nothing to do. What language should I learn next?” COBOL. Seriously, anyone with some programming experience can learn COBOL or any other language in a few weeks. Maybe with so many hip startups for concierge cat sitting closing down more programmers with time on their hands and no paycheck will crack the McCracken book (the one I learned COBOL from ages ago).

non-entity 7y ago
If i could get hired with a good rate after just learning the COBOL language maybe. Unfortunately, I don't have a mainframe for me to learn the rest of the environment in.
- lboc 7y ago
  IBM give away free year-long z/OS accounts: https://news.ycombinator.com/item?id=20873531 https://news.ycombinator.com/item?id=20873531
ornornor 7y ago
What are the real world prospects for cobol developers? I’m seriously asking. I’m tired of running the JavaScript treadmill, I’d much rather invest the time to learn cobol (which I bet doesn’t change everything about how things are done every 6 months) if I can be reasonably sure it comes with a steady amount of work and a good rate. But so far all I’ve seen is people saying “welp others should learn cobol because there aren’t many programmers still in the workforce who know this language and it pays big bucks” but I could never find hard evidence. Where are the job ads? Where are the typical rates?