6 ms·

I have to admit that I don’t fully understand why COBOL is so difficult to replace. If the systems are so vital surely there is a detailed specification and wri

I have to admit that I don’t fully understand why COBOL is so difficult to replace. If the systems are so vital surely there is a detailed specification and writing in an easier to maintain language would repay the effort? Is it just that there is so much COBOL and it’s not worth the effort? Edit : thanks for all the great replies - HN at its best!

mywittyname 5y ago
I bet there were numerous failed attempt at modernization with these platforms over the years, and after wasting enough money/resources on them, management made a blanket decision of No More.
nzach 5y ago
>If the systems are so vital surely there is a detailed specification In my experience those things not always go together. I've seen "mission critical" software with basically zero formal specification. Maybe this also happens with COBOL?
- wongarsu 5y ago
  I imagine this happened especially in COBOL, a language designed to be easy to learn so that business people can write their own subroutines instead of solely relying on programmers.
  - rbanffy 5y ago
    It was the same idea of DSL's. If you bring the language very close to the business domain, the program and the spec are very close to one another. Didn't quite happen in COBOL's case.
ahofmann 5y ago
I think you greatly overestimate how well software is documented. Just one example: When the Y2K problem was discussed in the late 90s, one of the big problems was that almost nobody understood the decades old code base. They hired experts, of course, to make sure there were no difficulties when they switched to the year 2000. They made such hair-raising discoveries in some code trees that a lot of the variables had completely useless names, like "Mickey Mouse".
- SteveMoody73 5y ago
  In the late 90s I was involved in replacing a software for a financial system for a company that was written in a basic of all languages. It was complex and had almost no documentation. All knowledge of the system was in the head of the programmer of that system and he wasn't happy that his code was being replaced so he was reluctant to help. The little document that did exist was hopelessly out of date. A lot of business was done on that software
- jazzyjackson 5y ago
  hah, sounds like a memory palace trick, I think I’ve heard a mathemagician use mickey mouse as a kind of variable name when doing mental math out loud.
wongarsu 5y ago
A lot of knowledge is probably institutional knowledge, especially about how the different systems interact (over the decades everything starts relying on implementation details and bugs of everything else). And the people who hold that institutional knowledge either left at some point in the last 60 years or have no interest in undermining their job security by telling you
rbanffy 5y ago
> writing in an easier to maintain language would repay the effort COBOL is very easy to learn, but, as was pointed out in [1], it's not only COBOL, but COBOL and CICS. CICS is an extremely optimized transaction manager that's been continuously improved and has been coevolving with IBM hardware since the 60's. There is also the workflows around the very non-Unixy nature of zOS. IBM has been doing an excellent job at introducing good operations practices and adding tools and functionality to facilitate that, but it's a long process. Also, remember these machines also run Linux incredibly well. They are unbelievably fast. 1- https://news.ycombinator.com/item?id=29229525 https://news.ycombinator.com/item?id=29229525
ihumanable 5y ago
I have retired some COBOL in my life, I can’t say that my experience is universal, but here’s what happened during my time consulting for a Fortune 500 company sunsetting COBOL into Java (1.4). > If the systems are so vital surely there is a detailed specification This is hilarious, and wrong. There probably is a specification somewhere, but there is drift between the implementation and the specification. When that 4am bug pops up and stops processing a billion dollars worth of transactions because of a weird corner case, someone fixed that to get the system running again, did they update the master specification, maybe. Half of the job is figuring out how the system really works, all the bugs, all the quirks, all the things that other systems have come to expect. Sunsetting is often a bug-for-bug type of conversion, because the output of COBOL programs are similar to database tables. Other programs are built on top of this data and that consumer code introduces a bunch of expectations on how the output of your replacement must behave to keep everything working. The number of consumer programs that have some expectation on the data can range from dozens to thousands. So you can either read and understand and fix all the consumers as you make your replacement behave to spec instead of behaving identically, or you do what most sunset teams do and you make it produce identical output. This now means that you are reading the target code base and understanding hundreds of thousands of lines of COBOL, you are unearthing decades of extensions, bug fixes, hacks, and mistakes. Your job then is to make sure the replacement behaves exactly the same way over decades of input. This is the best test of course, learn JCL and beg the mainframe operators to let you run a massive job that will process historical records with the target and replacement programs and compare the results. Your days begin to look like the following. Get in, get the report from last nights run which tells you all the discrepencies. For each one you can run your code and the cobol locally with some work. Formulate a theory for why some figure is wrong, fix it, move onto the next. Do that for 8 hours. At the end of the day kick off your massive JCL and do it all again tomorrow. Just do that everyday for a few weeks or months or however long it takes until your replacement program behaves the same as the COBOL program and then you can swap them without breaking 1000 other applications. Then you get the long tail of bug reports. The zipjack flim-florp report isn’t working. So now you dig into that, the flim-florps are processed by the COBOL you replaced, so maybe the zipjack system had a requirement you didn’t realize. Dig dig dig, days go by, your will to live eroding, but finally you find it, deep in the CVS history bug-1376892 looks like the target program accidentally added an extra hour to the business day in Seoul on leap years because the time zone data is wonky and they have special code expecting this wonkiness and correcting on their side because the mainframe programmers in 1994 decided it was too much work to fix on the mainframe side. Our program uses a different timezone database and it knows how to properly calculate this, so just add a data transform at the end to detect this scenario and fudge the numbers back to what the zipjack (and who knows how many other systems) expects. The COBOL systems are rarely used directly, they are batch processors taking in data and pumping out data. Applications are then written on top of their output and bind the behavior of the COBOL system with all their expectations. And this is where the complexity lies, the specification for how the system works is possibly written down somewhere, but the actual specification is in the COBOL code and the hundreds or thousands of consumer programs and all their expectations. To do the job well you have to write something that replaces the actual real-world specification encoded in all that other software, not the one jotted down in wordstar a few decades ago.
tkahnoski 5y ago
I had a brief exposure to a COBOL application and started to understand some of the technical hurdles that make this hard and largely lead to 'big system replacement' type projects which are extremely high-risk. If you've ever worked a VisualBasic Application, you'll have some inkling of what COBOL is like. One of the core things I saw with COBOl was strong coupling of application and data persistence. Further, where modern apps take a stateless approach for say a single API call, many COBOL application predate this concept. So not only are application and persistence coupled, the applications are VERY stateful. This makes replacing a 'piece' of the application extremely difficult as there is a lot of shared state to tease out of the system. Nor can you start building a second system that owns some piece of the data model, as COBOL's persistence models aren't built for shared write activity that a normal DB facilitates. You are forced to still write through the COBOl data model. Clearly not impossible, just difficult.
rdtsc 5y ago
In many cases these system may process millions or billions of dollars of transactions a day. They may be a specification or there may not be and there are probably a dozen corner cases that are not specified. Nobody wants to be the one who decided to replace it with a shiny kube cluster running node js or something and then have it all go up in smoke and losing money, customer, reputation.
jsmith45 5y ago
Realistically If there were detailed specifications, they were hard copy print-outs from the 80s that are probably no longer readable, and undoubtedly were thrown away in the intervening decades. Many of these old systems rarely require changes in most modules. (As in, of the hundreds of thousands of lines in the system, many subsections just never get touched). This means for the the parts that never get touched, there is a negative return on investment for rewriting them, so there has to be a correspondingly larger positive return on investment in the higher churn parts to make a rewrite worthwhile. Most of the times when this is studied, the return on investment from the rewrite is either tiny or net negative from this effect. Ah you say, then just rewrite only the parts that tend to change a lot, and leave the other stuff as COBOL. That sounds good right? Except that generally the cross language interop code needed becomes an absolute nightmare. I've dealt with this in an near ideal enviroment, and it was still a royal pain. This near ideal environment was: COBOL code restricted to just a subset used by the company, which was being translated to C code by the companies own compiler, which allowed for a C-compatible ABI to be defined allowing for surprisingly simple and direct C to Cobol interop. Yet even then the code code was still a royal pain, just due to how COBOL data structures differ so much from the idiomatic data structures for C or C++ code. My code was actually C# code interacting via C++/CLI with some C++ code that interacted with the the COBOL. But the interop code was still some of the worst code to touch.
- tablespoon 5y ago
  >> I have to admit that I don’t fully understand why COBOL is so difficult to replace. If the systems are so vital surely there is a detailed specification and writing in an easier to maintain language would repay the effort? > Realistically If there were detailed specifications, they were hard copy print-outs from the 80s that are probably no longer readable, and undoubtedly were thrown away in the intervening decades. Or not kept up to date. My employer has some mainframe systems, and when they were built they had technical writers and created pretty good documentation, organized in binders. For the last two decades, at least, these systems have been "legacy" and the company stopped employing technical writers. During that time, they were still pretty critical, but I'm pretty sure no one kept that documentation up to date with whatever changes were made. Also, about a year before the pandemic, we had building reorganization, and a lot of those binders were thrown out during the move. They were kept on several large, neglected shared bookshelves, and I think anything anyone didn't specifically claim was discarded by facilities. The thing with legacy systems is they can be vital, but they also can be neglected. The gamble most businesses seem to take is: "as long as it works, starve it and hope it doesn't blow up on my watch."
MichaelRazum 5y ago
Basically it very difficult, because no one truly no one understands the whole business logic implemented in Cobol. Take life insurance. You have contracts running for decades. You just don’t want to break something. There are no tests and so on. So if you would start from scratch you would be faster. So it’s just very very expensive with a lot of downside risks and little upsides.
- chasil 5y ago
  Even if you understand your business logic with extreme clarity, if you don't know how the math is implemented, then your migration will fail. Too many people learn this specific COBOL fact the hard way. https://medium.com/the-technical-archaeologist/is-cobol-holding-you-hostage-with-math-5498c0eb428b https://medium.com/the-technical-archaeologist/is-cobol-hold...
  - mrjin 5y ago
    Oh well, isn't understanding how data types represented one of the first a few thing to begin with a programming language?
    - lokedhs 5y ago
      Except in COBOL, data types are decimal, which means that whatever datatypes you choose to use in a modern language will not work the same as they do in COBOL. Also consider that the COBOL datatypes are directly represented in the underlying storage, so whatever format you decide to represent a value in in the code is the same format that will be used for external communication. If you want to retain compatibility with COBOL, you're probably going to need a COBOL-compatible numeric library in whatever language you're using. at least if you want to just reimplement the original code.
    - chasil 5y ago
      What this article is trying to tell you is that COBOL uses fixed-point arithmetic, often BCD. It was standardized two decades before ieee-754, that we now take for granted. The math is wildly different.
AnimalMuppet 5y ago
> If the systems are so vital surely there is a detailed specification and writing in an easier to maintain language would repay the effort? As others have said, no, there's probably not a complete specification. But even if there is, it's not nearly complete enough. That code may have 50 years of bug fixes and corner case fixes in it. How many times was the spec changed to reflect those fixes? Probably not always. If you rewrite it from the spec, how many of those issues are you going to re-introduce? And, how many new bugs are you going to introduce? The old code has 50 years of "getting the bugs out". It may be hard to maintain, but for what it does, it probably does it pretty solidly. Will the new code be as solid? Re-writing isn't as easy as it looks...
hindsightbias 5y ago
Here's the original spec in Interleaf v5 format and a few thousand pages of notes and emails. We might have a 25 year old PC we can boot to print that. That said, all that code has been running for 3 or 4 decades and anything that gets ported will have a shelf life of < 10 years.
worewood 5y ago
IMO One of the reasons is the tight integration with other IBM Products and the operating system. In short you can do distributed, networked transactions seamlessly with COBOL, involving files, database and messaging all at once, without the programmer even knowing what distributed transactions are and how to implement them "manually" - few do.
- marcodave 5y ago
  Holy moly, it's like COBOL was one of the original PaaS , with lock-in as a feature
- klelatti 5y ago
  So it’s not really COBOL lock in - it’s IBM lock in?

mywittyname 5y ago

I bet there were numerous failed attempt at modernization with these platforms over the years, and after wasting enough money/resources on them, management made a blanket decision of No More.

nzach 5y ago

>If the systems are so vital surely there is a detailed specification In my experience those things not always go together. I've seen "mission critical" software with basically zero formal specification. Maybe this also happens with COBOL?

wongarsu 5y ago
I imagine this happened especially in COBOL, a language designed to be easy to learn so that business people can write their own subroutines instead of solely relying on programmers.
- rbanffy 5y ago
  It was the same idea of DSL's. If you bring the language very close to the business domain, the program and the spec are very close to one another. Didn't quite happen in COBOL's case.

ahofmann 5y ago

I think you greatly overestimate how well software is documented. Just one example: When the Y2K problem was discussed in the late 90s, one of the big problems was that almost nobody understood the decades old code base. They hired experts, of course, to make sure there were no difficulties when they switched to the year 2000. They made such hair-raising discoveries in some code trees that a lot of the variables had completely useless names, like "Mickey Mouse".

SteveMoody73 5y ago
In the late 90s I was involved in replacing a software for a financial system for a company that was written in a basic of all languages. It was complex and had almost no documentation. All knowledge of the system was in the head of the programmer of that system and he wasn't happy that his code was being replaced so he was reluctant to help. The little document that did exist was hopelessly out of date. A lot of business was done on that software
jazzyjackson 5y ago
hah, sounds like a memory palace trick, I think I’ve heard a mathemagician use mickey mouse as a kind of variable name when doing mental math out loud.

wongarsu 5y ago

A lot of knowledge is probably institutional knowledge, especially about how the different systems interact (over the decades everything starts relying on implementation details and bugs of everything else). And the people who hold that institutional knowledge either left at some point in the last 60 years or have no interest in undermining their job security by telling you

rbanffy 5y ago

> writing in an easier to maintain language would repay the effort COBOL is very easy to learn, but, as was pointed out in [1], it's not only COBOL, but COBOL and CICS. CICS is an extremely optimized transaction manager that's been continuously improved and has been coevolving with IBM hardware since the 60's. There is also the workflows around the very non-Unixy nature of zOS. IBM has been doing an excellent job at introducing good operations practices and adding tools and functionality to facilitate that, but it's a long process. Also, remember these machines also run Linux incredibly well. They are unbelievably fast. 1- https://news.ycombinator.com/item?id=29229525 https://news.ycombinator.com/item?id=29229525

ihumanable 5y ago

I have retired some COBOL in my life, I can’t say that my experience is universal, but here’s what happened during my time consulting for a Fortune 500 company sunsetting COBOL into Java (1.4). > If the systems are so vital surely there is a detailed specification This is hilarious, and wrong. There probably is a specification somewhere, but there is drift between the implementation and the specification. When that 4am bug pops up and stops processing a billion dollars worth of transactions because of a weird corner case, someone fixed that to get the system running again, did they update the master specification, maybe. Half of the job is figuring out how the system really works, all the bugs, all the quirks, all the things that other systems have come to expect. Sunsetting is often a bug-for-bug type of conversion, because the output of COBOL programs are similar to database tables. Other programs are built on top of this data and that consumer code introduces a bunch of expectations on how the output of your replacement must behave to keep everything working. The number of consumer programs that have some expectation on the data can range from dozens to thousands. So you can either read and understand and fix all the consumers as you make your replacement behave to spec instead of behaving identically, or you do what most sunset teams do and you make it produce identical output. This now means that you are reading the target code base and understanding hundreds of thousands of lines of COBOL, you are unearthing decades of extensions, bug fixes, hacks, and mistakes. Your job then is to make sure the replacement behaves exactly the same way over decades of input. This is the best test of course, learn JCL and beg the mainframe operators to let you run a massive job that will process historical records with the target and replacement programs and compare the results. Your days begin to look like the following. Get in, get the report from last nights run which tells you all the discrepencies. For each one you can run your code and the cobol locally with some work. Formulate a theory for why some figure is wrong, fix it, move onto the next. Do that for 8 hours. At the end of the day kick off your massive JCL and do it all again tomorrow. Just do that everyday for a few weeks or months or however long it takes until your replacement program behaves the same as the COBOL program and then you can swap them without breaking 1000 other applications. Then you get the long tail of bug reports. The zipjack flim-florp report isn’t working. So now you dig into that, the flim-florps are processed by the COBOL you replaced, so maybe the zipjack system had a requirement you didn’t realize. Dig dig dig, days go by, your will to live eroding, but finally you find it, deep in the CVS history bug-1376892 looks like the target program accidentally added an extra hour to the business day in Seoul on leap years because the time zone data is wonky and they have special code expecting this wonkiness and correcting on their side because the mainframe programmers in 1994 decided it was too much work to fix on the mainframe side. Our program uses a different timezone database and it knows how to properly calculate this, so just add a data transform at the end to detect this scenario and fudge the numbers back to what the zipjack (and who knows how many other systems) expects. The COBOL systems are rarely used directly, they are batch processors taking in data and pumping out data. Applications are then written on top of their output and bind the behavior of the COBOL system with all their expectations. And this is where the complexity lies, the specification for how the system works is possibly written down somewhere, but the actual specification is in the COBOL code and the hundreds or thousands of consumer programs and all their expectations. To do the job well you have to write something that replaces the actual real-world specification encoded in all that other software, not the one jotted down in wordstar a few decades ago.

tkahnoski 5y ago

I had a brief exposure to a COBOL application and started to understand some of the technical hurdles that make this hard and largely lead to 'big system replacement' type projects which are extremely high-risk. If you've ever worked a VisualBasic Application, you'll have some inkling of what COBOL is like. One of the core things I saw with COBOl was strong coupling of application and data persistence. Further, where modern apps take a stateless approach for say a single API call, many COBOL application predate this concept. So not only are application and persistence coupled, the applications are VERY stateful. This makes replacing a 'piece' of the application extremely difficult as there is a lot of shared state to tease out of the system. Nor can you start building a second system that owns some piece of the data model, as COBOL's persistence models aren't built for shared write activity that a normal DB facilitates. You are forced to still write through the COBOl data model. Clearly not impossible, just difficult.

rdtsc 5y ago

In many cases these system may process millions or billions of dollars of transactions a day. They may be a specification or there may not be and there are probably a dozen corner cases that are not specified. Nobody wants to be the one who decided to replace it with a shiny kube cluster running node js or something and then have it all go up in smoke and losing money, customer, reputation.

jsmith45 5y ago

Realistically If there were detailed specifications, they were hard copy print-outs from the 80s that are probably no longer readable, and undoubtedly were thrown away in the intervening decades. Many of these old systems rarely require changes in most modules. (As in, of the hundreds of thousands of lines in the system, many subsections just never get touched). This means for the the parts that never get touched, there is a negative return on investment for rewriting them, so there has to be a correspondingly larger positive return on investment in the higher churn parts to make a rewrite worthwhile. Most of the times when this is studied, the return on investment from the rewrite is either tiny or net negative from this effect. Ah you say, then just rewrite only the parts that tend to change a lot, and leave the other stuff as COBOL. That sounds good right? Except that generally the cross language interop code needed becomes an absolute nightmare. I've dealt with this in an near ideal enviroment, and it was still a royal pain. This near ideal environment was: COBOL code restricted to just a subset used by the company, which was being translated to C code by the companies own compiler, which allowed for a C-compatible ABI to be defined allowing for surprisingly simple and direct C to Cobol interop. Yet even then the code code was still a royal pain, just due to how COBOL data structures differ so much from the idiomatic data structures for C or C++ code. My code was actually C# code interacting via C++/CLI with some C++ code that interacted with the the COBOL. But the interop code was still some of the worst code to touch.

tablespoon 5y ago
>> I have to admit that I don’t fully understand why COBOL is so difficult to replace. If the systems are so vital surely there is a detailed specification and writing in an easier to maintain language would repay the effort? > Realistically If there were detailed specifications, they were hard copy print-outs from the 80s that are probably no longer readable, and undoubtedly were thrown away in the intervening decades. Or not kept up to date. My employer has some mainframe systems, and when they were built they had technical writers and created pretty good documentation, organized in binders. For the last two decades, at least, these systems have been "legacy" and the company stopped employing technical writers. During that time, they were still pretty critical, but I'm pretty sure no one kept that documentation up to date with whatever changes were made. Also, about a year before the pandemic, we had building reorganization, and a lot of those binders were thrown out during the move. They were kept on several large, neglected shared bookshelves, and I think anything anyone didn't specifically claim was discarded by facilities. The thing with legacy systems is they can be vital, but they also can be neglected. The gamble most businesses seem to take is: "as long as it works, starve it and hope it doesn't blow up on my watch."

MichaelRazum 5y ago

Basically it very difficult, because no one truly no one understands the whole business logic implemented in Cobol. Take life insurance. You have contracts running for decades. You just don’t want to break something. There are no tests and so on. So if you would start from scratch you would be faster. So it’s just very very expensive with a lot of downside risks and little upsides.

chasil 5y ago
Even if you understand your business logic with extreme clarity, if you don't know how the math is implemented, then your migration will fail. Too many people learn this specific COBOL fact the hard way. https://medium.com/the-technical-archaeologist/is-cobol-holding-you-hostage-with-math-5498c0eb428b https://medium.com/the-technical-archaeologist/is-cobol-hold...
- mrjin 5y ago
  Oh well, isn't understanding how data types represented one of the first a few thing to begin with a programming language?
  - lokedhs 5y ago
    Except in COBOL, data types are decimal, which means that whatever datatypes you choose to use in a modern language will not work the same as they do in COBOL. Also consider that the COBOL datatypes are directly represented in the underlying storage, so whatever format you decide to represent a value in in the code is the same format that will be used for external communication. If you want to retain compatibility with COBOL, you're probably going to need a COBOL-compatible numeric library in whatever language you're using. at least if you want to just reimplement the original code.
  - chasil 5y ago
    What this article is trying to tell you is that COBOL uses fixed-point arithmetic, often BCD. It was standardized two decades before ieee-754, that we now take for granted. The math is wildly different.

AnimalMuppet 5y ago

> If the systems are so vital surely there is a detailed specification and writing in an easier to maintain language would repay the effort? As others have said, no, there's probably not a complete specification. But even if there is, it's not nearly complete enough. That code may have 50 years of bug fixes and corner case fixes in it. How many times was the spec changed to reflect those fixes? Probably not always. If you rewrite it from the spec, how many of those issues are you going to re-introduce? And, how many new bugs are you going to introduce? The old code has 50 years of "getting the bugs out". It may be hard to maintain, but for what it does, it probably does it pretty solidly. Will the new code be as solid? Re-writing isn't as easy as it looks...

hindsightbias 5y ago

Here's the original spec in Interleaf v5 format and a few thousand pages of notes and emails. We might have a 25 year old PC we can boot to print that. That said, all that code has been running for 3 or 4 decades and anything that gets ported will have a shelf life of < 10 years.

worewood 5y ago

IMO One of the reasons is the tight integration with other IBM Products and the operating system. In short you can do distributed, networked transactions seamlessly with COBOL, involving files, database and messaging all at once, without the programmer even knowing what distributed transactions are and how to implement them "manually" - few do.

marcodave 5y ago
Holy moly, it's like COBOL was one of the original PaaS , with lock-in as a feature
klelatti 5y ago
So it’s not really COBOL lock in - it’s IBM lock in?