12 ms·

New development on the mainframe started tapering off in the mid 1990's. So here's my take on IBM's list: - 45 of the top 50 banks - 8 of the top 10 insurers

New development on the mainframe started tapering off in the mid 1990's. So here's my take on IBM's list: - 45 of the top 50 banks - 8 of the top 10 insurers - 8 of the top 10 telcos - 7 of the top 10 retailers - 4 of the top 5 airlines All had significant operations prior to the mid 1990's and so have extensive mainframe investments that are very difficult and costly to migrate from. I know. I work for one of the top electrical utilities in the United States and at the heart of our customer operations lies an IBM mainframe. We're in the midst of launching a multi-year, billion dollar program to replace that mainframe. We've been talking about doing this for years and are just now pulling the trigger on it. There's still time to tell whether we actually go through with it or not, or get $100 million into the project and realize we're looking at a $1.5 billion - $2 billion effort and abandon the project. I would not call this state of affairs thriving. Is the mainframe alive? Yes. Is it thriving? No. Most CIOs are looking for a means for retiring those assets.

Pet_Ant 6y ago
I'd be interested in when was the last time an organisation with no existing mainframes adopted them. Do they have a net-new value proposition?
- closeparen 6y ago
  I’m too young to have used one, but from what I’ve read, it’s the programming model of a single machine with the scale and resilience of a distributed system.
  - dralley 6y ago
    A mainframe is effectively a mail-order on-premise cloud, from the hardware up, developed 3 to 4 decades before things like OpenStack or Kubernetes. Mainframes supported hardware-accelerated virtualization way back in the early 1970s. You could have thousands or tens of thousands of VMs running on a single machine (granted, a machine the size of a room). One of my older greybeard coworkers tells stories of how sometimes an IBM employee would show up at 6 AM to replace a water pump (or some other component) that had phoned home the night before to report that it was beginning to fail. The hardware is super reliable and resilient but also heavily instrumented to detect & resolve failures before they caused problems. And they would send their techs out to pre-empt those problems before the customer ever noticed anything. Not that it would have been too much a problem if it had failed - because there was so much hardware redundancy built into those machines. One of his other stories, is a tech demo that IBM did for his company, where they put a mainframe in the back of a trailer and would drive it around to customer sites. The demo consisted of starting a huge computation on the machine, and pulling up a screen that showed the available hardware resources, then a tech would reach into the machine and start pulling out CPUs and RAM, and you'd watch live as it detected that the hardware had disappeared and shuffled the jobs around to other hardware resources without crashing. And then the tech would start swapping hardware back in, and it would add the resources back into the pool and start doing work with them once again. All handled by the hardware and the OS.
    - rbanffy 6y ago
      One of the reasons for that instrumentation is that they used to be operated in a cloud-like environment where the machine is sliced into smaller ones and time on them is billed much like cloud computing is billed today.
      - tablespoon 6y ago
        > One of the reasons for that instrumentation is that they used to be operated in a cloud-like environment where the machine is sliced into smaller ones and time on them is billed much like cloud computing is billed today. IIRC, that's still true. I think IBM mainframes are typically leased, and the customer is billed on usage. There's some complexity to this, where I think the metering only applies to programs in the "traditional" mainframe environment, while things running on the mainframe Linux environment are exempt from metered billing.
        rbanffy 6y ago
        They still pay for licensed capacity when running things like z/OS. The machine is shipped "full" but the hardware has to be activated by paying a license. One of the interesting features I heard when they launched z15 was that, if one needed to restart a partition, it could tap unlicensed CPUs to start faster and then go back to its licensed capacity.
        ghaff 6y ago
        I don't know the details of billing these days; it's been a while since I followed the space. But when Linux came out on Z, it ran on something called IFLs. These were not AFAIK physically different from the processors running more traditional IBM workloads. They were simply priced at a lower level. I don't remember if there was capacity on demand for Linux workloads or not.
    - nn3 6y ago
      I worked at a place that had a little mainframe (essentially a large server) from IBM to port some software to. Most of the people didn't know anything about mainframes. On a hot day they had an air condition problem in the data center and had to turn off a lot of machines. They tried to turn off the mainframe using the big button, but it just kept running. Then they decided to pull the power plug. System still kept running. It turned out it had an built-in UPS. After some more attempts they finally managed to turn it off.
  - zozbot234 6y ago
    > it’s the programming model of a single machine with the scale and resilience of a distributed system. Single-system-image clustering delivers on that promise with no dependence on proprietary platforms. Linux is now slowly getting the underlying features needed to enable this (e.g. process migration and checkpointing, namespacing of all system resources), largely as a side-effect of earlier developments such as cgroups. You can already experiment with Linux-CRIU to see what the basic tech is like, it's not fully mainlined yet but it's slowly getting there.
  - epc 6y ago
    For most of their history, IBM mainframes were used for two tasks: batch workloads and interactive users. Batch workloads were things like billing: you're a utility and you need to do the accounting and billing for the million customers you have. You're MCI and you've spent a year advertising Friends and Family discounts and now need to do custom billing every month that figures out all those intersections of friends & family and sends custom billing. Not rocket science, just a lot of boring data crunching you have to do over and over again. Interactive was everything you do today on personal systems: stores had terminal displays that hooked to mainframes for inventory control so salespeople could put together an order or cash registers would transmit sales near real time to a central mainframe and pump into/out of your ERP system. Depending on your company's view of the mainframe world, you might have an MVS system doing all of the batch processing and a VM/TSO system providing interactive. And the MVS system could run under the VM system as a guest O/S, or you could partition the single mainframe to have MVS running on some CECs and VM running on others (e.g. you might have more interactive users during the daytime, then tune down the systems so that the MVS systems had more capacity overnight for batch operations). Mainframes systems could be clustered together in a single data center and you could move workloads back and forth across systems while the workload was in progress. Eventually (circa 1994) you could link mainframe complexes together across DCs to provide redundancy (if a failure occurred in the mainframe equivalent of an AWS AZ the workload could, theoretically, transfer to another AZ with minimal interruption or reconfiguration). On the interactive side…you could have thousands of systems connected to a single mainframe via various forms of I/O systems. All of your retail stores, all of the printers at those stores, frequently all of the security systems, would be set up using APPC or SNA to connect to your regional mainframe over one or more leased POTS lines. The 3270 protocol that drove those green screens was screen oriented, if you changed two fields on the screen and hit ENTER, only the changed data from the fields would be sent, so having a bunch of terminals share a whopping 33kbps line wasn't outrageous. From a programming and systems management perspective, the systems were designed to run as close to maximum utilization as possible. You just spent 10-15-20 million setting up this complex, you want to run it at 90-95% utilization 24 hours a day. So there's a whole set of workload management tools that developed by IBM, CA, and others to tweak applications and the operating environment to run it as hot as possible. There's all sorts of accounting built into the operating system(s) and language environments, going back to when you set literal budgets for how much a given department or program could spend. So, this is the environment that ran from effectively the 1950s through the 1980s into the 1990s. The S/360 line launched in the early 1960s but offered some backwards compatibility to some IBM systems of the 1950s. COBOL gets tagged as the most popular mainframe language but PL/1 (or PL/I) was frequently used (IBM had an internal variant called PL/X which was mostly used for systems software). C was used a lot as well inside IBM but usually for smaller products running inside VM/CMS. I tried writing a web server for MVS in my spare time circa 1995–1996 and even though I was working with people inside MVS development we had a hard time getting it to work with what turned out to be a pretty crappy TCP/IP stack. The batch oriented nature of MVS wasn't the problem, IIRC the TCP/IP stack just could not handle the set up/tear down of the typical HTTP/0.9 and 1.0 connections. When a web server was eventually developed for MVS it was under the posix flavored "OpenEdition". Net: It's just a different model of computing, with a different character set (EBCDIC, still) and languages. It's weird to see the industry kind of / sort of return to the classic shared computing resource model of the cloud with containerized applications and services. IBM in the 1990s was just as convinced at the corporate level that mainframes were dead as the consensus public opinion was at that time, which in retrospect is a shame since IBM chose to fire 1000s of people with the skills it would have needed to leverage to build a competitive cloud offering.
- taylodl 6y ago
  Given the cost of mainframe operations vs. cost of cloud, no, I can't imagine anyone adopting a mainframe. The mainframe is extremely expensive compared to say, AWS - especially if you're running AWS' auto-managed services. Let me put it this way - we're looking at a multi-year, $1 billion project and we're able to justify it. Keep in mind too that we're a regulated utility and the regulators agree with our analysis. Trust me, whatever your AWS bill is it's a pittance compared to what your IBM bill would be.
  - shrubble 6y ago
    What are you licensing that is so expensive? The hardware itself can be purchased, right?
    - taylodl 6y ago
      No. The hardware is leased - you can't buy an IBM mainframe. All the software is priced per MIP per CPU core for your installation - regardless of how much you actually use the software. For all intents and purposes that software is leased as well. There's also retroactive charges - if you bump of the capacity of your mainframe mid-cycle on your license then you have to retroactively pay for the upgrade. We had a situation where that happened with one of our vendors and somehow it got missed for a few years and we ended up owing millions of dollars. So imagine an environment where you can't buy your hardware, you can't buy your software, and when you've made an upgrade to your hardware your software gets even more expensive. That's a mainframe. Our mainframe, which is mostly used for customer billing (electrical utility billing is slightly complex due to all the different tariffs, localities and taxes), and customer relationship & service management, is costing us $15 million - $20 million per year. On the one hand you'd say holy smokes that's expensive! But on the other you could say that's only $3-$4 per customer per year. Thing is you could do the same amount of processing in AWS for under $250,000 per month or $3 million per year, making the mainframe at least 5-7 times more expensive than AWS.
      - shrubble 6y ago
        Pretty sure that you can buy an IBM mainframe. I know individuals who own them. So how does saving $17 million per year justify spending $1 billion?
- KptMarchewa 6y ago
  Just imagine if cloud provider gave you same problems, and moving from GCP to AWS ment 5 years and billion dollars project.
monocasa 6y ago
Out of curiosity, given the nine figure migration costs you're throwing around, why bother? How is it hurting you day to day (to the tune of a billion dollars) enough to migrate. Obvs all else being equal there's no way you'd start on mainframe today, but now that you have it why jump off?
- taylodl 6y ago
  That's a good question! The reason given is it's getting difficult to find COBOL programmers and the platform isn't as agile as other platforms. I have to admit that IBM is working to make the platform more agile and you can do more than run COBOL - you can run Java and Linux on Z. I've been in IBM training sessions where I've been running Linux on Z and then running Docker on top of that and unless you had told me otherwise the experience was normal: you're running bash, running Docker and everything runs as you'd expect. BUT - the problem is the core of our system is written in COBOL. There's still CICS, JCL, Copybooks and DB/2 and those who can work with those technologies are now within 10 years of retiring and the new generation doesn't know them or have much of a desire to learn them. Heck, I'm part of the first generation of PC developers from the mid-1980's and I've never done any mainframe development! The thinking then is we'd better jump off before all these people retire and you end up having to pay $300+/hr for resources capable of working with these legacy technologies. Pay now and buy yourself a new technology set and new capabilities or pay later and just keep the system running.
  - xirbeosbwo1234 6y ago
    Why ditch the mainframe? It sounds like the software is the problem here, not the hardware.
  - OldHand2018 6y ago
    > the new generation doesn't know them or have much of a desire to learn them On one hand, I'd say that this is true. But on the other, I'm not so sure. 15 years ago, I interviewed for a job that turned out to be for building software on OpenVMS running on Alpha. I had honestly never heard of either thing. The interviewer, who would be my manager, said that isn't a problem, it's just a computer that works a little different than others. Do you want to learn it? I said ok. Over the years, the hiring process changed, and there is no way that someone like me would even be sitting at that interview where they could ask if I wanted to learn. So companies can go ahead and spend all that money on legacy consultants and all that money on migrating their systems.
twunde 6y ago
To add on to this, this results in an odd job market where demand for COBOL programmers is pretty low most of the time since there isn't a ton of work, but when a company does need to hire COBOL programmers its much more difficult to find them and hire them. Which leads to these common stories about COBOL programmers being in demand, when the reality is that most have a hard time finding new COBOL jobs, especially without needing to move. Off-topic but do you work for Eversource? An ex-coworker of mine found a bug that he immediately recognized as a COBOL bug while working with Eversource CT's EDI
meepmorp 6y ago
>We're in the midst of launching a multi-year, billion dollar program to replace that mainframe. We've been talking about doing this for years and are just now pulling the trigger on it. There's still time to tell whether we actually go through with it or not, or get $100 million into the project and realize we're looking at a $1.5 billion - $2 billion effort and abandon the project. So, you're looking at a high risk, multi-year, multi-billion dollar project to remove your own mainframe dependency. And the business isn't fully committed to it. > Most CIOs are looking for a means for retiring those assets. And finding extremely expensive, high risk, long duration software projects are the usual alternative, per your own experience. Why waste prestige and budget on that when you can keep all the critical LOB software running exactly as it is (or faster) forever? Mainframes are the OG backward compatibility and virtualization platforms. Given that, why do you think the mainframe business isn't thriving, especially given the billions in revenue that the companies you list in your post undoubtedly bring in for IBM? It's not sexy, and they're probably not winning new customers. But they're making a shit ton of money on a long lived, successful product and continuing to fund research and development on the platform.
- jameshart 6y ago
  Yes, the mainframe business is 'thriving' in that environment. Much like how SARS-CoV-2 has been 'thriving' in a non-immunized population who don't like wearing masks...
- marcosdumay 6y ago
  > Why waste prestige and budget on that when you can keep all the critical LOB software running exactly as it is (or faster) forever? Hum... Probably (and that was the case at my workplace) because business as usual has higher risks and ongoing costs than the migration. It's not like mainframes are cheap to own, or guaranteed to stay in existence in the future. > why do you think the mainframe business isn't thriving "Thriving" on my lexical implies on growing or at least sustainable. A market where new players don't want to buy at any cost and old players are all trying to get out is not this.
  - meepmorp 6y ago
agumonkey 6y ago
Do you think it's technically sounds to migrate ? or is it just a trend to get away from old tech ?
tablespoon 6y ago
> We're in the midst of launching a multi-year, billion dollar program to replace that mainframe. We've been talking about doing this for years and are just now pulling the trigger on it. There's still time to tell whether we actually go through with it or not, or get $100 million into the project and realize we're looking at a $1.5 billion - $2 billion effort and abandon the project. I work at a company that basically built its current business on mainframes in the 70s/80s/90s. The migration effort to get off them has been going on for more than 20 years and still isn't done yet.
- pickle-wizard 6y ago
  Sounds about like Interstate 35, by the time you're done, it is time to do it again.
  - tablespoon 6y ago
    > Sounds about like Interstate 35, by the time you're done, it is time to do it again. Yep. They're already spinning up a big cloud migration project that will eventually replace many of the re-engineered on-premise systems, and they're still not yet off the mainframe (they're close, but that last 20% is always the hardest).
kjs3 6y ago
Anecdata. I work for a top 10 bank. The mainframe isn't going away here. We're investing in new development, new tools, and just (last 2 years or so) had a hiring spree to support those efforts. I am also very familiar with strategy at several other peer banks (they're doing the same thing), a number of the telcos (same) and 2 of the major airlines (who will look at you like you're insane if you suggest they get rid of their mainframe). I've been hearing "mainframes are dead" since the 1980s. The market for them has dramatically changed, but by the same token there's just more people in IT who have no clue what the mainframes do from a business perspective, so "obviously" it's a dead end.
- paol 6y ago
  That doesn't make it thriving though. It's a niche that continues to get smaller, albeit at a very slow rate.
  - meepmorp 6y ago
    Their anecdata beats your unsourced generalization.
    - kjs3 6y ago
      No, mine is anecdata too, just more of it. And neither wins because there's actual data out there about the size and velocity of the mainframe market. I just don't much care to look it up every time someone makes an unfounded declaration that mainframes are dead. And, since we're keeping score, zero content snark is bottom of the heap.
      - meepmorp 6y ago
        I was saying your anecdata beats the other guy. I definitely agree the mainframe ain't dead. > And, since we're keeping score, zero content snark is bottom of the heap. That's just hurtful, man.
  - Guest42 6y ago
    I think the difference is that the mainframe isn’t going anywhere and the large companies themselves have been thriving and will likely continue to do so. My employer has a number of them and they work great although it’s challenging to find the developers. If it was my decision to migrate I’d be hesitant because it seems as though infrastructure is priced low enough to convince people to switch but then will likely creep up over time in either prices or lock-in.
deleted 6y ago
[deleted]
NotSammyHagar 6y ago
Why don't people migrate the systems to vms, then you could run them on vastly cheaper hardware? You have all this data across the complex storage system, but all of that could be virtualized if they wanted. The vms I worked on in the 1980s had so many different os variants, cics etc but if I was working on this approach I'd try to run things in vms, eventually teeze out the pieces I could into new services. But the parts that are too hard could just keep running in vms. You have to virtualize your data sources and provide apis to read those old disk drives or virtualized tape drives etc. But if you are spending a billion dollars, how much code do you have? Also, on the lack of programmers, just pay them more and you'll get a lot more applicants.
tyingq 6y ago
Some of that list is at risk. "4 of the top 5 airlines" for example. Amadeus finally shuttered their TPF mainframe, so Sabre can't be that far behind.
fallingknife 6y ago
What does the mainframe do that makes it take years and cost $1B to replace? I'm guessing that it didn't take that much time and money to set up the mainframe in the first place. So why does it cost that much to build an equivalent system today?
- zozbot234 6y ago
  In most cases, it's running millions of lines of legacy code that interacts with all sorts of crufty, decades-old proprietary software components. (iow, it's not COBOL itself that blocks a migration, it's the fact that real-world COBOL code tends to rely on this terrible proprietary stuff. I mean, it's nothing that couldn't be dealt with cleanly if a gigadollar was literally on the line, but I suspect that such figures are somewhat exaggerated.)
shrubble 6y ago
Have you examined whether the Hercules emulator can play a part in your organization? Could it be used to offload certain parts of the infrastructure?
ksec 6y ago
>No. Most CIOs are looking for a means for retiring those assets. May I ask Why? If they are working perfectly well with a decent roadmap and upgrade path and quality assurance from IBM?

Pet_Ant 6y ago

I'd be interested in when was the last time an organisation with no existing mainframes adopted them. Do they have a net-new value proposition?

closeparen 6y ago
I’m too young to have used one, but from what I’ve read, it’s the programming model of a single machine with the scale and resilience of a distributed system.
- dralley 6y ago
  A mainframe is effectively a mail-order on-premise cloud, from the hardware up, developed 3 to 4 decades before things like OpenStack or Kubernetes. Mainframes supported hardware-accelerated virtualization way back in the early 1970s. You could have thousands or tens of thousands of VMs running on a single machine (granted, a machine the size of a room). One of my older greybeard coworkers tells stories of how sometimes an IBM employee would show up at 6 AM to replace a water pump (or some other component) that had phoned home the night before to report that it was beginning to fail. The hardware is super reliable and resilient but also heavily instrumented to detect & resolve failures before they caused problems. And they would send their techs out to pre-empt those problems before the customer ever noticed anything. Not that it would have been too much a problem if it had failed - because there was so much hardware redundancy built into those machines. One of his other stories, is a tech demo that IBM did for his company, where they put a mainframe in the back of a trailer and would drive it around to customer sites. The demo consisted of starting a huge computation on the machine, and pulling up a screen that showed the available hardware resources, then a tech would reach into the machine and start pulling out CPUs and RAM, and you'd watch live as it detected that the hardware had disappeared and shuffled the jobs around to other hardware resources without crashing. And then the tech would start swapping hardware back in, and it would add the resources back into the pool and start doing work with them once again. All handled by the hardware and the OS.
  - rbanffy 6y ago
    One of the reasons for that instrumentation is that they used to be operated in a cloud-like environment where the machine is sliced into smaller ones and time on them is billed much like cloud computing is billed today.
    - tablespoon 6y ago
      > One of the reasons for that instrumentation is that they used to be operated in a cloud-like environment where the machine is sliced into smaller ones and time on them is billed much like cloud computing is billed today. IIRC, that's still true. I think IBM mainframes are typically leased, and the customer is billed on usage. There's some complexity to this, where I think the metering only applies to programs in the "traditional" mainframe environment, while things running on the mainframe Linux environment are exempt from metered billing.
      - rbanffy 6y ago
        They still pay for licensed capacity when running things like z/OS. The machine is shipped "full" but the hardware has to be activated by paying a license. One of the interesting features I heard when they launched z15 was that, if one needed to restart a partition, it could tap unlicensed CPUs to start faster and then go back to its licensed capacity.
      - ghaff 6y ago
        I don't know the details of billing these days; it's been a while since I followed the space. But when Linux came out on Z, it ran on something called IFLs. These were not AFAIK physically different from the processors running more traditional IBM workloads. They were simply priced at a lower level. I don't remember if there was capacity on demand for Linux workloads or not.
  - nn3 6y ago
    I worked at a place that had a little mainframe (essentially a large server) from IBM to port some software to. Most of the people didn't know anything about mainframes. On a hot day they had an air condition problem in the data center and had to turn off a lot of machines. They tried to turn off the mainframe using the big button, but it just kept running. Then they decided to pull the power plug. System still kept running. It turned out it had an built-in UPS. After some more attempts they finally managed to turn it off.
- zozbot234 6y ago
  > it’s the programming model of a single machine with the scale and resilience of a distributed system. Single-system-image clustering delivers on that promise with no dependence on proprietary platforms. Linux is now slowly getting the underlying features needed to enable this (e.g. process migration and checkpointing, namespacing of all system resources), largely as a side-effect of earlier developments such as cgroups. You can already experiment with Linux-CRIU to see what the basic tech is like, it's not fully mainlined yet but it's slowly getting there.
- epc 6y ago
  For most of their history, IBM mainframes were used for two tasks: batch workloads and interactive users. Batch workloads were things like billing: you're a utility and you need to do the accounting and billing for the million customers you have. You're MCI and you've spent a year advertising Friends and Family discounts and now need to do custom billing every month that figures out all those intersections of friends & family and sends custom billing. Not rocket science, just a lot of boring data crunching you have to do over and over again. Interactive was everything you do today on personal systems: stores had terminal displays that hooked to mainframes for inventory control so salespeople could put together an order or cash registers would transmit sales near real time to a central mainframe and pump into/out of your ERP system. Depending on your company's view of the mainframe world, you might have an MVS system doing all of the batch processing and a VM/TSO system providing interactive. And the MVS system could run under the VM system as a guest O/S, or you could partition the single mainframe to have MVS running on some CECs and VM running on others (e.g. you might have more interactive users during the daytime, then tune down the systems so that the MVS systems had more capacity overnight for batch operations). Mainframes systems could be clustered together in a single data center and you could move workloads back and forth across systems while the workload was in progress. Eventually (circa 1994) you could link mainframe complexes together across DCs to provide redundancy (if a failure occurred in the mainframe equivalent of an AWS AZ the workload could, theoretically, transfer to another AZ with minimal interruption or reconfiguration). On the interactive side…you could have thousands of systems connected to a single mainframe via various forms of I/O systems. All of your retail stores, all of the printers at those stores, frequently all of the security systems, would be set up using APPC or SNA to connect to your regional mainframe over one or more leased POTS lines. The 3270 protocol that drove those green screens was screen oriented, if you changed two fields on the screen and hit ENTER, only the changed data from the fields would be sent, so having a bunch of terminals share a whopping 33kbps line wasn't outrageous. From a programming and systems management perspective, the systems were designed to run as close to maximum utilization as possible. You just spent 10-15-20 million setting up this complex, you want to run it at 90-95% utilization 24 hours a day. So there's a whole set of workload management tools that developed by IBM, CA, and others to tweak applications and the operating environment to run it as hot as possible. There's all sorts of accounting built into the operating system(s) and language environments, going back to when you set literal budgets for how much a given department or program could spend. So, this is the environment that ran from effectively the 1950s through the 1980s into the 1990s. The S/360 line launched in the early 1960s but offered some backwards compatibility to some IBM systems of the 1950s. COBOL gets tagged as the most popular mainframe language but PL/1 (or PL/I) was frequently used (IBM had an internal variant called PL/X which was mostly used for systems software). C was used a lot as well inside IBM but usually for smaller products running inside VM/CMS. I tried writing a web server for MVS in my spare time circa 1995–1996 and even though I was working with people inside MVS development we had a hard time getting it to work with what turned out to be a pretty crappy TCP/IP stack. The batch oriented nature of MVS wasn't the problem, IIRC the TCP/IP stack just could not handle the set up/tear down of the typical HTTP/0.9 and 1.0 connections. When a web server was eventually developed for MVS it was under the posix flavored "OpenEdition". Net: It's just a different model of computing, with a different character set (EBCDIC, still) and languages. It's weird to see the industry kind of / sort of return to the classic shared computing resource model of the cloud with containerized applications and services. IBM in the 1990s was just as convinced at the corporate level that mainframes were dead as the consensus public opinion was at that time, which in retrospect is a shame since IBM chose to fire 1000s of people with the skills it would have needed to leverage to build a competitive cloud offering.
taylodl 6y ago
Given the cost of mainframe operations vs. cost of cloud, no, I can't imagine anyone adopting a mainframe. The mainframe is extremely expensive compared to say, AWS - especially if you're running AWS' auto-managed services. Let me put it this way - we're looking at a multi-year, $1 billion project and we're able to justify it. Keep in mind too that we're a regulated utility and the regulators agree with our analysis. Trust me, whatever your AWS bill is it's a pittance compared to what your IBM bill would be.
- shrubble 6y ago
  What are you licensing that is so expensive? The hardware itself can be purchased, right?
  - taylodl 6y ago
    No. The hardware is leased - you can't buy an IBM mainframe. All the software is priced per MIP per CPU core for your installation - regardless of how much you actually use the software. For all intents and purposes that software is leased as well. There's also retroactive charges - if you bump of the capacity of your mainframe mid-cycle on your license then you have to retroactively pay for the upgrade. We had a situation where that happened with one of our vendors and somehow it got missed for a few years and we ended up owing millions of dollars. So imagine an environment where you can't buy your hardware, you can't buy your software, and when you've made an upgrade to your hardware your software gets even more expensive. That's a mainframe. Our mainframe, which is mostly used for customer billing (electrical utility billing is slightly complex due to all the different tariffs, localities and taxes), and customer relationship & service management, is costing us $15 million - $20 million per year. On the one hand you'd say holy smokes that's expensive! But on the other you could say that's only $3-$4 per customer per year. Thing is you could do the same amount of processing in AWS for under $250,000 per month or $3 million per year, making the mainframe at least 5-7 times more expensive than AWS.
    - shrubble 6y ago
      Pretty sure that you can buy an IBM mainframe. I know individuals who own them. So how does saving $17 million per year justify spending $1 billion?
KptMarchewa 6y ago
Just imagine if cloud provider gave you same problems, and moving from GCP to AWS ment 5 years and billion dollars project.

monocasa 6y ago

Out of curiosity, given the nine figure migration costs you're throwing around, why bother? How is it hurting you day to day (to the tune of a billion dollars) enough to migrate. Obvs all else being equal there's no way you'd start on mainframe today, but now that you have it why jump off?

taylodl 6y ago
That's a good question! The reason given is it's getting difficult to find COBOL programmers and the platform isn't as agile as other platforms. I have to admit that IBM is working to make the platform more agile and you can do more than run COBOL - you can run Java and Linux on Z. I've been in IBM training sessions where I've been running Linux on Z and then running Docker on top of that and unless you had told me otherwise the experience was normal: you're running bash, running Docker and everything runs as you'd expect. BUT - the problem is the core of our system is written in COBOL. There's still CICS, JCL, Copybooks and DB/2 and those who can work with those technologies are now within 10 years of retiring and the new generation doesn't know them or have much of a desire to learn them. Heck, I'm part of the first generation of PC developers from the mid-1980's and I've never done any mainframe development! The thinking then is we'd better jump off before all these people retire and you end up having to pay $300+/hr for resources capable of working with these legacy technologies. Pay now and buy yourself a new technology set and new capabilities or pay later and just keep the system running.
- xirbeosbwo1234 6y ago
  Why ditch the mainframe? It sounds like the software is the problem here, not the hardware.
- OldHand2018 6y ago
  > the new generation doesn't know them or have much of a desire to learn them On one hand, I'd say that this is true. But on the other, I'm not so sure. 15 years ago, I interviewed for a job that turned out to be for building software on OpenVMS running on Alpha. I had honestly never heard of either thing. The interviewer, who would be my manager, said that isn't a problem, it's just a computer that works a little different than others. Do you want to learn it? I said ok. Over the years, the hiring process changed, and there is no way that someone like me would even be sitting at that interview where they could ask if I wanted to learn. So companies can go ahead and spend all that money on legacy consultants and all that money on migrating their systems.

twunde 6y ago

To add on to this, this results in an odd job market where demand for COBOL programmers is pretty low most of the time since there isn't a ton of work, but when a company does need to hire COBOL programmers its much more difficult to find them and hire them. Which leads to these common stories about COBOL programmers being in demand, when the reality is that most have a hard time finding new COBOL jobs, especially without needing to move. Off-topic but do you work for Eversource? An ex-coworker of mine found a bug that he immediately recognized as a COBOL bug while working with Eversource CT's EDI

meepmorp 6y ago

>We're in the midst of launching a multi-year, billion dollar program to replace that mainframe. We've been talking about doing this for years and are just now pulling the trigger on it. There's still time to tell whether we actually go through with it or not, or get $100 million into the project and realize we're looking at a $1.5 billion - $2 billion effort and abandon the project. So, you're looking at a high risk, multi-year, multi-billion dollar project to remove your own mainframe dependency. And the business isn't fully committed to it. > Most CIOs are looking for a means for retiring those assets. And finding extremely expensive, high risk, long duration software projects are the usual alternative, per your own experience. Why waste prestige and budget on that when you can keep all the critical LOB software running exactly as it is (or faster) forever? Mainframes are the OG backward compatibility and virtualization platforms. Given that, why do you think the mainframe business isn't thriving, especially given the billions in revenue that the companies you list in your post undoubtedly bring in for IBM? It's not sexy, and they're probably not winning new customers. But they're making a shit ton of money on a long lived, successful product and continuing to fund research and development on the platform.

jameshart 6y ago
Yes, the mainframe business is 'thriving' in that environment. Much like how SARS-CoV-2 has been 'thriving' in a non-immunized population who don't like wearing masks...
marcosdumay 6y ago
> Why waste prestige and budget on that when you can keep all the critical LOB software running exactly as it is (or faster) forever? Hum... Probably (and that was the case at my workplace) because business as usual has higher risks and ongoing costs than the migration. It's not like mainframes are cheap to own, or guaranteed to stay in existence in the future. > why do you think the mainframe business isn't thriving "Thriving" on my lexical implies on growing or at least sustainable. A market where new players don't want to buy at any cost and old players are all trying to get out is not this.
- meepmorp 6y ago

agumonkey 6y ago

Do you think it's technically sounds to migrate ? or is it just a trend to get away from old tech ?

tablespoon 6y ago

pickle-wizard 6y ago
Sounds about like Interstate 35, by the time you're done, it is time to do it again.
- tablespoon 6y ago
  > Sounds about like Interstate 35, by the time you're done, it is time to do it again. Yep. They're already spinning up a big cloud migration project that will eventually replace many of the re-engineered on-premise systems, and they're still not yet off the mainframe (they're close, but that last 20% is always the hardest).

kjs3 6y ago

Anecdata. I work for a top 10 bank. The mainframe isn't going away here. We're investing in new development, new tools, and just (last 2 years or so) had a hiring spree to support those efforts. I am also very familiar with strategy at several other peer banks (they're doing the same thing), a number of the telcos (same) and 2 of the major airlines (who will look at you like you're insane if you suggest they get rid of their mainframe). I've been hearing "mainframes are dead" since the 1980s. The market for them has dramatically changed, but by the same token there's just more people in IT who have no clue what the mainframes do from a business perspective, so "obviously" it's a dead end.

paol 6y ago
That doesn't make it thriving though. It's a niche that continues to get smaller, albeit at a very slow rate.
- meepmorp 6y ago
  Their anecdata beats your unsourced generalization.
  - kjs3 6y ago
    No, mine is anecdata too, just more of it. And neither wins because there's actual data out there about the size and velocity of the mainframe market. I just don't much care to look it up every time someone makes an unfounded declaration that mainframes are dead. And, since we're keeping score, zero content snark is bottom of the heap.
    - meepmorp 6y ago
      I was saying your anecdata beats the other guy. I definitely agree the mainframe ain't dead. > And, since we're keeping score, zero content snark is bottom of the heap. That's just hurtful, man.
- Guest42 6y ago
  I think the difference is that the mainframe isn’t going anywhere and the large companies themselves have been thriving and will likely continue to do so. My employer has a number of them and they work great although it’s challenging to find the developers. If it was my decision to migrate I’d be hesitant because it seems as though infrastructure is priced low enough to convince people to switch but then will likely creep up over time in either prices or lock-in.

deleted 6y ago

[deleted]

NotSammyHagar 6y ago

Why don't people migrate the systems to vms, then you could run them on vastly cheaper hardware? You have all this data across the complex storage system, but all of that could be virtualized if they wanted. The vms I worked on in the 1980s had so many different os variants, cics etc but if I was working on this approach I'd try to run things in vms, eventually teeze out the pieces I could into new services. But the parts that are too hard could just keep running in vms. You have to virtualize your data sources and provide apis to read those old disk drives or virtualized tape drives etc. But if you are spending a billion dollars, how much code do you have? Also, on the lack of programmers, just pay them more and you'll get a lot more applicants.

tyingq 6y ago

Some of that list is at risk. "4 of the top 5 airlines" for example. Amadeus finally shuttered their TPF mainframe, so Sabre can't be that far behind.

fallingknife 6y ago

What does the mainframe do that makes it take years and cost $1B to replace? I'm guessing that it didn't take that much time and money to set up the mainframe in the first place. So why does it cost that much to build an equivalent system today?

zozbot234 6y ago
In most cases, it's running millions of lines of legacy code that interacts with all sorts of crufty, decades-old proprietary software components. (iow, it's not COBOL itself that blocks a migration, it's the fact that real-world COBOL code tends to rely on this terrible proprietary stuff. I mean, it's nothing that couldn't be dealt with cleanly if a gigadollar was literally on the line, but I suspect that such figures are somewhat exaggerated.)

shrubble 6y ago

Have you examined whether the Hercules emulator can play a part in your organization? Could it be used to offload certain parts of the infrastructure?

ksec 6y ago

>No. Most CIOs are looking for a means for retiring those assets. May I ask Why? If they are working perfectly well with a decent roadmap and upgrade path and quality assurance from IBM?