7 ms·

It persists in many places due to the deeply ingrained belief that, for one reason or another, it would be technically, practically, or economically impossible

It persists in many places due to the deeply ingrained belief that, for one reason or another, it would be technically, practically, or economically impossible to migrate functionality off the mainframe. Somehow the people in charge of these systems have managed to convince large enterprises of this for decades. And now we are in a situation where their long held beliefs have become true because nobody is around that understands how or why the software works the way it does. It's a disgrace.

ghaff 3y ago
Or the people in charge are of the belief that things work and they're a tiny part of the overall organization's costs--and any massive migration will run over schedule and budget, won't necessarily be better at the end, and will be unnecessary and disruptive. They may or may not be correct but it's not an irrational belief whatever those advocating for moving everything to a distributed platform think.
ElectricalUnion 3y ago
> It persists in many places due to the deeply ingrained belief that, for one reason or another, it would be technically, practically, or economically impossible to migrate functionality off the mainframe. It is never technical or pratical reasons, always economic reasons. To use contemporary vocabulary, you "just" need to replicate a "multi tenant HA cloud environment" to migrate from a mainframe at great cost. > And now we are in a situation where their long held beliefs have become true because nobody is around that understands how or why the software works the way it does. Statistically unlikely, most of the business software that goes onto said mainframe is in fact very simple and straightforward and can be easly ported out, however, the real deal is all the integration and high availability logic built-in into "the mainframe platform". You're not porting that out without replicating said "multi tenant HA cloud environment" at great cost.
- ojhughes 3y ago
  I'm not sure how much water the multi tenant HA argument holds, these institutions are already doing that on AWS / Azure / k8s for the more modern apps. Rewriting all the old crusty COBOL code might be a different story.
  - dralley 3y ago
    Getting the same performance out of AWS as a local mainframe is neither trivial nor cheap for a lot of the workloads these systems are running.
    - jeffbee 3y ago
      Getting stupefying amounts of I/O on AWS is as simple as turning a knob and getting out your credit card. The kind of throughput quoted in the article—48GB/s—is achievable on AWS for $250k/year, list price.
      - ElectricalUnion 3y ago
        I thought such throughput would cost around 30 274 560 USD/year (60 * 60 * 24 * 365 * 48 * 0.02) on data transit fees alone.
        jeffbee 3y ago
        Obviously it would be very foolish to use an AWS storage backend for an application that was not also in AWS. That said, it is not at all clear to me that the TCO of a maxed-out IBM DS8900F is less than $30 million/year.
      - dralley 3y ago
        But let's say you want that data compressed and/or encrypted, you're going to need an awful lot of compute to exploit that 48GB/s if it's possible at all. Whereas on mainframe the service processors can deal with it transparently to the CPU using specialized hardware. And let's say it's financial transactions - fixed-point decimal arithmetic. On mainframe, that can be hardware accelerated too.
        jeffbee 3y ago
        Yes, and it's interesting how Intel is moving in that direction with their Xeon line, integrating the QAT peripheral for compression and encryption offloading. Unfortunately these things are unicorns and that aren't generally available in the cloud, either.
- Zigurd 3y ago
  The risk in a reengineering project that comprehensively replaces the mainframe is enough to discourage most people making that decision. It could be a $3M project that pays pack in 4 years after scrapping the mainframe and running out the maintenance contract, or it could go over budget and take forever to pay back. Your IBM rep has undoubtedly run the numbers.
- wredue 3y ago
  >most of the business software that goes into said mainframe is in fact very simple and straightforward Yeah no. I worked on mainframe code for 15 years of the last 20 and can tell you that you’re wrong. Even reasonably simple business lines have cobbled together insane cobol, batch and cics screens that would make spaghetti blush. When you open up a 100,000 line batch program that’s just 1 of 30 cobol programs in that run which is a mess of gotos, global mutable state and basically every single other generally frowned upon software practice we know about, you start to understand the issues. Moving off of mainframe is practically impossible because of it. That other person being voted down is actually correct, nobody in the organization actually knows how any of it works. When organizations try to transition away, what ends up happening is they get these insane promises of “5 years” from vendors (like IBM. Ask me how I know!), but then it takes 10 years to transition single systems. Nothing is documented. So you have business people who have no clue how anything works providing high level work flows. But then there’s always 50 random edge cases or interactions they never considered cause the 40 year old code just does it. IT contains most of the business knowledge, but the mainframe grey beards are… stubborn. The business hangs on to these people because they’re seen as insanely valuable, not realizing that the grey beards have actually been actively suppressing knowledge to only their hidden documents, and they’ve got 8 years till retirement and are going to make sure that their job exists at least until then by actively sabotaging efforts to change. But also, for some reason, the business manager love the saboteurs. It’s insane to watch.
  - pmontra 3y ago
    What happens when there are new requirements? An example, fiscal legislation changes and programs dealing with money have to adapt. Of course a way out is to lobby for changes that don't touch the software, but what happens when the software must be touched?
    - wredue 3y ago
      The estimate for carbon tax legislation was 700 hours and it took 1600 hours and didn’t work correctly for 9 months following. Edit: and for the record, at that point, we were already splitting off carbon transactions for their own processing because, surprise, energy has lots of regulation.
- TheRealDunkirk 3y ago
  >> It persists in many places due to the deeply ingrained belief that, for one reason or another, it would be technically, practically, or economically impossible to migrate functionality off the mainframe. > It is never technical or pratical reasons, always economic reasons. To use contemporary vocabulary, you "just" need to replicate a "multi tenant HA cloud environment" to migrate from a mainframe at great cost. When your business processes are so ossified by a mainframe that you have, say, groups preparing Excel spreadsheets -- with data formatted from one mainframe screen in 3 columns -- for use by another group to input back into the mainframe -- with the data reformatted in 4 columns -- you know you're never escaping the gravity well of the black hole at the center of IT that is "The Mainframe." This is just how it's going to be until the company is so technically irrelevant that the remains are picked up in a fire sale by some PE to sell off to some other, tangentially-related company, solely for the IP assets.
jmartrican 3y ago
Before we even get to the costs of a migration, what is the reason for the migration?
- ghaff 3y ago
  It's what would be cool and would look good on my resume. (Only half /s)
  - jmartrican 3y ago
    Nailed it ^^^
  - TheRealDunkirk 3y ago
    Over my 30 years, mostly in Fortune 250's, I've seen 3 mainframe migrations fail. If you're part of the team, the trick is to get another job before people realize -- in year 4 of your "5-year" migration -- that it was never going to work, because managers in other parts of the business will never allow their headcount to be reduced by better IT workflows.
    - tracker1 3y ago
      Having worked adjacent to some of these kinds of projects, can't upvote this enough. Not to mention that business doesn't take the suggested 1-2 years to simply document and define test cases for the old/new system to ensure such a new system works, and simply start working on the new system based on what the business/da types drew up. Missing literally tens of thousands of legal edge cases defined in the legacy codebase.
- mberning 3y ago
  People may assume that a lot of this mainframe software is "running just fine" but nothing could be farther from the truth. Often you will find that this software was fine when it was written 40-50 years ago, but now it has a lot of deferred maintenance and upgrades which are now a "big problem". Is it a problem that an application that can move millions of dollars is only secured by a strictly 8 character alphanumeric password? You be the judge.
zitterbewegung 3y ago
The greatest hurdle for a project to be replaced is that if its works. I assure you that the users of IBM mainframes would be looking at at least a multi million dollar project to perform a decommission and if even approved it would be first on the chopping block to pause / eliminate the project.
- josho 3y ago
  You are correct. There was a time when companies ran projects to replace their mainframes. They all largely failed. The trend now is to integrate around the mainframe. Build new functionality on modern systems and leverage the mainframe where it makes sense. I worked at one company that is trying to replace their mainframe. The project is now in its second decade. The company has the burden of supporting the mainframe application and now the 'modern' application. The project is still a few years away from being able to completely shut off the mainframe. Oh and the modern application is also showing its age as the business has changed and the modern system built assumption in from the mainframe system and in many ways is just as inflexible to change as the mainframe app was. This is why companies aren't scrambling to replace their mainframe systems. They are happy to continue paying for the high cost of compute on these systems because its still cheaper and less risky than replacing the system.
  - technofiend 3y ago
    I've been on a couple of successful migration projects, both of which moved things to unices. One was motivated by the $5 million/yr costs to maintain the mainframe, which was replaced by a handful of lowest cost bidder unix boxes. The second was regulatory-agency-mandated cleanup after a financial fallout, and unlike the previous two or three times that company had tried and failed to consolidate their risk reporting onto a single system, this time it worked. Because it had to. The threat of an external regulator was sufficient to break down the previous barriers of balkanized system owners protecting their turf.
  - elzbardico 3y ago
    I bet that the "modern" application, given its 20 years vintage, is probably a J2EE application running on whatever is the modern incarnation of IBM Websphere and it was written as a mind-boggling soup of EJB 2.x Entity and Session beans.
adamc 3y ago
I have worked on an RFP for migration. It persists because the cost for migrating can be quite high -- many mainframe environments have many applications with complex sets of dependencies (after decades, not surprising). No one doubts that it could be migrated, but it is not a trivial undertaking. When folks have replaced significant enough chunks with other apps, or otherwise made the prospect less daunting, then it may happen. It just comes down to cost and risk management.
ibiza 3y ago
Migrating a suite of programs written w/ COBOL, CICS, DB2 from z/OS to Linux is fairly straightforward. MicroFocus COBOL compiles IBM COBOL, DB2 runs on Linux or you can swap out another RDBMS. The CICS part requires a simple runtime easily implemented in C plus a new scanner to replace the calls in your programs. I was part of a 4 person team that knocked this out in under a year. 1,000's of mainframe programs running w/o change (at the time) on Unix. This was in the 90s.
- stcroixx 3y ago
  What replaces JCL?
  - redandblack 3y ago
    perl scripts. sorry
  - ibiza 3y ago
    The env I helped set up was online only, no batch jobs. JCL is essentially a declarative shell language, so it's not that hard to map to Bourne shell et. al.
- hermitdev 3y ago
  > DB2 runs on Linux The DBA's I interned with would yell at you for this claim. They'd say UDB runs on Linux, but DB2 only runs on z/OS. I worked with both DB2 & UDB during my internship 2 decades ago, and while largely compatible with core SQL functionality, that changed quickly after you got off the "normal" path. UDB was released at a much faster cadence, and if you developed to the latest features of UDB, you'd often find yourself unable to deploy against DB2. A few of my last projects as an intern: 1) storing BLOB objects in DB2 from a .Net app and 2) porting an "interactive" batch-processed COBOL app to ASP.Net form 3) getting DB2 Connect clustering on windows working (I never did get this working despite spending an entire summer on the phone with IBM support). I've some fond memories of working with the mainframe. Like the time I managed to crash the entire development LPAR with a specific SQL query that broke the DB2 query optimizer... The phone call from the NOC was immediate and went something along the lines of "I don't know what you did, but don't do it again, you just took down all of development". Development was something like +10k users...
KennyBlanken 3y ago
Notice they didn't mention how incredibly outdated all these applications are? Our financial money transfer system is an ancient joke. The IRS has systems that are hopelessly out of date and limited in capabilities. It's really easy and safe to sign a contract for a newer mainframe; porting a legacy app is fraught. Middle-upper management goes for what's safe, not what could torpedo their career. The hardware might be whiz-bang but the legacy applications are slow, outdated, and extremely hampered in functionality. In my state, the birth records system is electronic, and so is our RMV. But in order to get a RealID, I had to get a paper copy printed out to bring to the RMV because they can not, or will not, integrate the two systems. Meanwhile other countries have things like electronic ID cards with personal certs so you can electronically sign documents and identify yourself. The whole thing is a giant puff piece for IBM, reading like a sales presentation transcript.
tracker1 3y ago
Have you ever migrated a decades old application off of a mainframe? Most of these kinds of systems have decades of codified knowledge that applies the legal requirements of many disparate jurisdictions in these systems. Laws vary from city to city, county to county and state to state, and that's only for the US. Some of these systems implement code that applies legal requirements for dozens of nations. In general, you're going to have to spend at least a year just to document the existing system. Another six months to a year generating test cases and scripts for validation. And that's before even starting to build the new system. And that is ALL optimistic, assuming everything goes right. I had to once write a relatively simple application for "vacation bidding" where in employees will get a position to request certain weeks off during the year. This was a legal requirement and happened to involve the company, two other companies that had been merged in and three unions. Each with different rules regarding how seniority is calculated. The contractual specifics were over two thousand pages long. That's just one very small piece of software in one of these very large companies. It took the better part of a year to implement. Now imagine taking that a couple decades later after it's evolved and migrating to an entirely new platform and paradigm. Now imagine the cost to do that.

ghaff 3y ago

Or the people in charge are of the belief that things work and they're a tiny part of the overall organization's costs--and any massive migration will run over schedule and budget, won't necessarily be better at the end, and will be unnecessary and disruptive. They may or may not be correct but it's not an irrational belief whatever those advocating for moving everything to a distributed platform think.

ElectricalUnion 3y ago

> It persists in many places due to the deeply ingrained belief that, for one reason or another, it would be technically, practically, or economically impossible to migrate functionality off the mainframe. It is never technical or pratical reasons, always economic reasons. To use contemporary vocabulary, you "just" need to replicate a "multi tenant HA cloud environment" to migrate from a mainframe at great cost. > And now we are in a situation where their long held beliefs have become true because nobody is around that understands how or why the software works the way it does. Statistically unlikely, most of the business software that goes onto said mainframe is in fact very simple and straightforward and can be easly ported out, however, the real deal is all the integration and high availability logic built-in into "the mainframe platform". You're not porting that out without replicating said "multi tenant HA cloud environment" at great cost.

ojhughes 3y ago
I'm not sure how much water the multi tenant HA argument holds, these institutions are already doing that on AWS / Azure / k8s for the more modern apps. Rewriting all the old crusty COBOL code might be a different story.
- dralley 3y ago
  Getting the same performance out of AWS as a local mainframe is neither trivial nor cheap for a lot of the workloads these systems are running.
  - jeffbee 3y ago
    Getting stupefying amounts of I/O on AWS is as simple as turning a knob and getting out your credit card. The kind of throughput quoted in the article—48GB/s—is achievable on AWS for $250k/year, list price.
    - ElectricalUnion 3y ago
      I thought such throughput would cost around 30 274 560 USD/year (60 * 60 * 24 * 365 * 48 * 0.02) on data transit fees alone.
      - jeffbee 3y ago
        Obviously it would be very foolish to use an AWS storage backend for an application that was not also in AWS. That said, it is not at all clear to me that the TCO of a maxed-out IBM DS8900F is less than $30 million/year.
    - dralley 3y ago
      But let's say you want that data compressed and/or encrypted, you're going to need an awful lot of compute to exploit that 48GB/s if it's possible at all. Whereas on mainframe the service processors can deal with it transparently to the CPU using specialized hardware. And let's say it's financial transactions - fixed-point decimal arithmetic. On mainframe, that can be hardware accelerated too.
      - jeffbee 3y ago
        Yes, and it's interesting how Intel is moving in that direction with their Xeon line, integrating the QAT peripheral for compression and encryption offloading. Unfortunately these things are unicorns and that aren't generally available in the cloud, either.
Zigurd 3y ago
The risk in a reengineering project that comprehensively replaces the mainframe is enough to discourage most people making that decision. It could be a $3M project that pays pack in 4 years after scrapping the mainframe and running out the maintenance contract, or it could go over budget and take forever to pay back. Your IBM rep has undoubtedly run the numbers.
wredue 3y ago
>most of the business software that goes into said mainframe is in fact very simple and straightforward Yeah no. I worked on mainframe code for 15 years of the last 20 and can tell you that you’re wrong. Even reasonably simple business lines have cobbled together insane cobol, batch and cics screens that would make spaghetti blush. When you open up a 100,000 line batch program that’s just 1 of 30 cobol programs in that run which is a mess of gotos, global mutable state and basically every single other generally frowned upon software practice we know about, you start to understand the issues. Moving off of mainframe is practically impossible because of it. That other person being voted down is actually correct, nobody in the organization actually knows how any of it works. When organizations try to transition away, what ends up happening is they get these insane promises of “5 years” from vendors (like IBM. Ask me how I know!), but then it takes 10 years to transition single systems. Nothing is documented. So you have business people who have no clue how anything works providing high level work flows. But then there’s always 50 random edge cases or interactions they never considered cause the 40 year old code just does it. IT contains most of the business knowledge, but the mainframe grey beards are… stubborn. The business hangs on to these people because they’re seen as insanely valuable, not realizing that the grey beards have actually been actively suppressing knowledge to only their hidden documents, and they’ve got 8 years till retirement and are going to make sure that their job exists at least until then by actively sabotaging efforts to change. But also, for some reason, the business manager love the saboteurs. It’s insane to watch.
- pmontra 3y ago
  What happens when there are new requirements? An example, fiscal legislation changes and programs dealing with money have to adapt. Of course a way out is to lobby for changes that don't touch the software, but what happens when the software must be touched?
  - wredue 3y ago
    The estimate for carbon tax legislation was 700 hours and it took 1600 hours and didn’t work correctly for 9 months following. Edit: and for the record, at that point, we were already splitting off carbon transactions for their own processing because, surprise, energy has lots of regulation.
TheRealDunkirk 3y ago
>> It persists in many places due to the deeply ingrained belief that, for one reason or another, it would be technically, practically, or economically impossible to migrate functionality off the mainframe. > It is never technical or pratical reasons, always economic reasons. To use contemporary vocabulary, you "just" need to replicate a "multi tenant HA cloud environment" to migrate from a mainframe at great cost. When your business processes are so ossified by a mainframe that you have, say, groups preparing Excel spreadsheets -- with data formatted from one mainframe screen in 3 columns -- for use by another group to input back into the mainframe -- with the data reformatted in 4 columns -- you know you're never escaping the gravity well of the black hole at the center of IT that is "The Mainframe." This is just how it's going to be until the company is so technically irrelevant that the remains are picked up in a fire sale by some PE to sell off to some other, tangentially-related company, solely for the IP assets.

jmartrican 3y ago

Before we even get to the costs of a migration, what is the reason for the migration?

ghaff 3y ago
It's what would be cool and would look good on my resume. (Only half /s)
- jmartrican 3y ago
  Nailed it ^^^
- TheRealDunkirk 3y ago
  Over my 30 years, mostly in Fortune 250's, I've seen 3 mainframe migrations fail. If you're part of the team, the trick is to get another job before people realize -- in year 4 of your "5-year" migration -- that it was never going to work, because managers in other parts of the business will never allow their headcount to be reduced by better IT workflows.
  - tracker1 3y ago
    Having worked adjacent to some of these kinds of projects, can't upvote this enough. Not to mention that business doesn't take the suggested 1-2 years to simply document and define test cases for the old/new system to ensure such a new system works, and simply start working on the new system based on what the business/da types drew up. Missing literally tens of thousands of legal edge cases defined in the legacy codebase.
mberning 3y ago
People may assume that a lot of this mainframe software is "running just fine" but nothing could be farther from the truth. Often you will find that this software was fine when it was written 40-50 years ago, but now it has a lot of deferred maintenance and upgrades which are now a "big problem". Is it a problem that an application that can move millions of dollars is only secured by a strictly 8 character alphanumeric password? You be the judge.

zitterbewegung 3y ago

The greatest hurdle for a project to be replaced is that if its works. I assure you that the users of IBM mainframes would be looking at at least a multi million dollar project to perform a decommission and if even approved it would be first on the chopping block to pause / eliminate the project.

josho 3y ago
You are correct. There was a time when companies ran projects to replace their mainframes. They all largely failed. The trend now is to integrate around the mainframe. Build new functionality on modern systems and leverage the mainframe where it makes sense. I worked at one company that is trying to replace their mainframe. The project is now in its second decade. The company has the burden of supporting the mainframe application and now the 'modern' application. The project is still a few years away from being able to completely shut off the mainframe. Oh and the modern application is also showing its age as the business has changed and the modern system built assumption in from the mainframe system and in many ways is just as inflexible to change as the mainframe app was. This is why companies aren't scrambling to replace their mainframe systems. They are happy to continue paying for the high cost of compute on these systems because its still cheaper and less risky than replacing the system.
- technofiend 3y ago
  I've been on a couple of successful migration projects, both of which moved things to unices. One was motivated by the $5 million/yr costs to maintain the mainframe, which was replaced by a handful of lowest cost bidder unix boxes. The second was regulatory-agency-mandated cleanup after a financial fallout, and unlike the previous two or three times that company had tried and failed to consolidate their risk reporting onto a single system, this time it worked. Because it had to. The threat of an external regulator was sufficient to break down the previous barriers of balkanized system owners protecting their turf.
- elzbardico 3y ago
  I bet that the "modern" application, given its 20 years vintage, is probably a J2EE application running on whatever is the modern incarnation of IBM Websphere and it was written as a mind-boggling soup of EJB 2.x Entity and Session beans.

adamc 3y ago

I have worked on an RFP for migration. It persists because the cost for migrating can be quite high -- many mainframe environments have many applications with complex sets of dependencies (after decades, not surprising). No one doubts that it could be migrated, but it is not a trivial undertaking. When folks have replaced significant enough chunks with other apps, or otherwise made the prospect less daunting, then it may happen. It just comes down to cost and risk management.

ibiza 3y ago

Migrating a suite of programs written w/ COBOL, CICS, DB2 from z/OS to Linux is fairly straightforward. MicroFocus COBOL compiles IBM COBOL, DB2 runs on Linux or you can swap out another RDBMS. The CICS part requires a simple runtime easily implemented in C plus a new scanner to replace the calls in your programs. I was part of a 4 person team that knocked this out in under a year. 1,000's of mainframe programs running w/o change (at the time) on Unix. This was in the 90s.

stcroixx 3y ago
What replaces JCL?
- redandblack 3y ago
  perl scripts. sorry
- ibiza 3y ago
  The env I helped set up was online only, no batch jobs. JCL is essentially a declarative shell language, so it's not that hard to map to Bourne shell et. al.
hermitdev 3y ago
> DB2 runs on Linux The DBA's I interned with would yell at you for this claim. They'd say UDB runs on Linux, but DB2 only runs on z/OS. I worked with both DB2 & UDB during my internship 2 decades ago, and while largely compatible with core SQL functionality, that changed quickly after you got off the "normal" path. UDB was released at a much faster cadence, and if you developed to the latest features of UDB, you'd often find yourself unable to deploy against DB2. A few of my last projects as an intern: 1) storing BLOB objects in DB2 from a .Net app and 2) porting an "interactive" batch-processed COBOL app to ASP.Net form 3) getting DB2 Connect clustering on windows working (I never did get this working despite spending an entire summer on the phone with IBM support). I've some fond memories of working with the mainframe. Like the time I managed to crash the entire development LPAR with a specific SQL query that broke the DB2 query optimizer... The phone call from the NOC was immediate and went something along the lines of "I don't know what you did, but don't do it again, you just took down all of development". Development was something like +10k users...

KennyBlanken 3y ago

Notice they didn't mention how incredibly outdated all these applications are? Our financial money transfer system is an ancient joke. The IRS has systems that are hopelessly out of date and limited in capabilities. It's really easy and safe to sign a contract for a newer mainframe; porting a legacy app is fraught. Middle-upper management goes for what's safe, not what could torpedo their career. The hardware might be whiz-bang but the legacy applications are slow, outdated, and extremely hampered in functionality. In my state, the birth records system is electronic, and so is our RMV. But in order to get a RealID, I had to get a paper copy printed out to bring to the RMV because they can not, or will not, integrate the two systems. Meanwhile other countries have things like electronic ID cards with personal certs so you can electronically sign documents and identify yourself. The whole thing is a giant puff piece for IBM, reading like a sales presentation transcript.

tracker1 3y ago

Have you ever migrated a decades old application off of a mainframe? Most of these kinds of systems have decades of codified knowledge that applies the legal requirements of many disparate jurisdictions in these systems. Laws vary from city to city, county to county and state to state, and that's only for the US. Some of these systems implement code that applies legal requirements for dozens of nations. In general, you're going to have to spend at least a year just to document the existing system. Another six months to a year generating test cases and scripts for validation. And that's before even starting to build the new system. And that is ALL optimistic, assuming everything goes right. I had to once write a relatively simple application for "vacation bidding" where in employees will get a position to request certain weeks off during the year. This was a legal requirement and happened to involve the company, two other companies that had been merged in and three unions. Each with different rules regarding how seniority is calculated. The contractual specifics were over two thousand pages long. That's just one very small piece of software in one of these very large companies. It took the better part of a year to implement. Now imagine taking that a couple decades later after it's evolved and migrating to an entirely new platform and paradigm. Now imagine the cost to do that.