4 ms·
I have retired some COBOL in my life, I can’t say that my experience is universal, but here’s what happened during my time consulting for a Fortune 500 company
by ihumanable 5y ago
I have retired some COBOL in my life, I can’t say that my experience is universal, but here’s what happened during my time consulting for a Fortune 500 company sunsetting COBOL into Java (1.4).
> If the systems are so vital surely there is a detailed specification
This is hilarious, and wrong. There probably is a specification somewhere, but there is drift between the implementation and the specification. When that 4am bug pops up and stops processing a billion dollars worth of transactions because of a weird corner case, someone fixed that to get the system running again, did they update the master specification, maybe. Half of the job is figuring out how the system really works, all the bugs, all the quirks, all the things that other systems have come to expect.
Sunsetting is often a bug-for-bug type of conversion, because the output of COBOL programs are similar to database tables. Other programs are built on top of this data and that consumer code introduces a bunch of expectations on how the output of your replacement must behave to keep everything working. The number of consumer programs that have some expectation on the data can range from dozens to thousands.
So you can either read and understand and fix all the consumers as you make your replacement behave to spec instead of behaving identically, or you do what most sunset teams do and you make it produce identical output.
This now means that you are reading the target code base and understanding hundreds of thousands of lines of COBOL, you are unearthing decades of extensions, bug fixes, hacks, and mistakes.
Your job then is to make sure the replacement behaves exactly the same way over decades of input. This is the best test of course, learn JCL and beg the mainframe operators to let you run a massive job that will process historical records with the target and replacement programs and compare the results.
Your days begin to look like the following. Get in, get the report from last nights run which tells you all the discrepencies. For each one you can run your code and the cobol locally with some work. Formulate a theory for why some figure is wrong, fix it, move onto the next. Do that for 8 hours. At the end of the day kick off your massive JCL and do it all again tomorrow.
Just do that everyday for a few weeks or months or however long it takes until your replacement program behaves the same as the COBOL program and then you can swap them without breaking 1000 other applications.
Then you get the long tail of bug reports. The zipjack flim-florp report isn’t working. So now you dig into that, the flim-florps are processed by the COBOL you replaced, so maybe the zipjack system had a requirement you didn’t realize. Dig dig dig, days go by, your will to live eroding, but finally you find it, deep in the CVS history bug-1376892 looks like the target program accidentally added an extra hour to the business day in Seoul on leap years because the time zone data is wonky and they have special code expecting this wonkiness and correcting on their side because the mainframe programmers in 1994 decided it was too much work to fix on the mainframe side.
Our program uses a different timezone database and it knows how to properly calculate this, so just add a data transform at the end to detect this scenario and fudge the numbers back to what the zipjack (and who knows how many other systems) expects.
The COBOL systems are rarely used directly, they are batch processors taking in data and pumping out data. Applications are then written on top of their output and bind the behavior of the COBOL system with all their expectations. And this is where the complexity lies, the specification for how the system works is possibly written down somewhere, but the actual specification is in the COBOL code and the hundreds or thousands of consumer programs and all their expectations. To do the job well you have to write something that replaces the actual real-world specification encoded in all that other software, not the one jotted down in wordstar a few decades ago.
- hydroxideOH- 5y agoThis is hilarious and accurate, at least fron my impression, though thankfully not experience, of talking with people who worked with a COBOL codebase at a large insurance company. At that company, they said 10 years ago that they'd replace the COBOL in 10 years, and they are still a decade away from replacing it.
- klelatti 5y agoThanks for the great reply. I had the pleasure of spending a year replacing an APL program on an IBM mainframe may years ago so understand a lot of the points you’re making. I think that the missing element in my initial analysis is that many firms don’t have much in-house expertise left on these systems. They’ve cut to the bone or outsourced. And for management replacing is huge risk for limited upside - even if the economics make sense.
- grvdrm 5y ago>Sunsetting is often a bug-for-bug type of conversion, because the output of COBOL programs are similar to database tables. Other programs are built on top of this data and that consumer code introduces a bunch of expectations on how the output of your replacement must behave to keep everything working. The number of consumer programs that have some expectation on the data can range from dozens to thousands. > So you can either read and understand and fix all the consumers as you make your replacement behave to spec instead of behaving identically, or you do what most sunset teams do and you make it produce identical output. So on point. And this problem exists not just in legacy systems built in COBOL. Also in much newer systems built with SQL in my line of work. The horror show I regularly encounter is a SQL function used in a production system that requires 50+ variables as input. 50+ fields as output. Any change in the order of the fields screws everything up. Makes me crazy.
- fma 5y agoI've rewritten an old application that was a rewrite of a mainframe (prob COBOL I guess). There were quirks that we still had to carry over. Luckily there were some old hands that can sometimes tell you why and we didn't always have to look into the CVS history and find bug-1376892. For testing you'd run a few weeks of data through and make sure they match outputs exactly. In prod we'd run them in parallel a bit for each report before decommissioning it. There'd always be something like..at the end of 3rd quarter if the last day is a Friday, some other junk happened and now your data looks different. After a few we got a hang of which things were likely to break and was able to have certain parts driven by configuration and many could be fixed via an updating a config if a bug was found.