9 ms·

There's no pressing reason for a company to toss its legacy mainframes. What VP would sign off on replacing core systems that run an international corporation?

There's no pressing reason for a company to toss its legacy mainframes. What VP would sign off on replacing core systems that run an international corporation? The downsides are infinite! I mean, are you insane? Just throw money at an "expensive" COBOL programmer and get out of my office! I grilled an employee at a Major Package Shipping Company a few years ago about their infrastructure, and it's pretty stereotypical: a few mainframes that were coded decades ago in COBOL and an expensive team of legacy programmers performing surgical tweaks. They're porting unimportant side systems to modern hardware + languages (ostensibly so the mainframe programmers can focus on the major systems), but there are no plans to replace the core. Ever. Replacement systems just aren't good enough to consider switching the heart of their company. And so it will continue until there's a compelling storyline for dropping mainframes

mey 16y ago
These corporations may not see it, because they do not understand it, but having such legacy systems a risk to their operations. The longer they wait to migrate, the more expensive it will be, and the harder it will be to attract talent to the project. There may come a point where they can no longer find anyone to support their hardware or software, and it'll force them to stagnate against their competition who may be able to provide better or different offerings because they are not tied to a system they can't modify. My understanding of business is that you should never stagnate, and always keep advancing. People that I've talked to who have worked with such systems indicated that they built giant shims around the systems in other languages to allow them to be adaptable. In theory this gives them a clear interface definition (it may be huge, thousands of method signatures, and possible direct schema access, but it can still be a clear interface) to boot strap a core system replacement in another language and hardware system.
- jacquesm 16y ago
  The laws of supply and demand will make sure that there will always be people available to maintain those systems. The price will go up until some 'modern' programmer is tempted enough to open a 3 decades old book on COBOL and JCL. Your understanding of business is flawed, businesses are advancing by marketing, not by shacking up with every new hot tech that comes along in favour of their old, trusted and reliable tech. The larger, older and more profitable a business is the more likely that they are going to be ultra-conservative about using new technology at the core of their business. Eventually we'll get rid of this stuff but I wouldn't bet on it happening in the next two decades, unless someone finds a way to emulate a mainframe on a cluster with similar reliability. And then we still need to get rid of the software.
  - logicalmind 16y ago
    You don't think that there is room for a new high-tech bank to step in and eat the lunch of the big banks? If you started a bank from scratch, built with modern technology, you could provide all kinds of valuable services and adapt to change much more quickly. Pretty much every service provided, particularly on the web, is some application that screenscrapes CICS screens or reads data from VSAM. These applications a tricky to build and take much more time than they should.
    - jacquesm 16y ago
      Starting a bank from scratch will have you worry about other things than the technology of your IT department, mostly to do with solvency and staying clear of SO. Sure, at some level the technology will come in to play but nobody will 'start a high tech bank'. For instance, ING direct (one of the efforts of the banking industry to leverage the use of technology) was jump started from the legacy ING systems and as far as I know still ties in to them for all of their back-end stuff. When I was the 'systems administrator' for the corporate division of a bank (in 1986 or so) we had daily rounds of getting the signatures of the various people responsible that the totals for the night batch runs were correct. That's the sort of responsibility you're talking about, I can't imagine that an MD5 or some digital signature would be sufficient today, it's all about procedures and accountability, emphatically not about technology. I got one of the first PCs the bank bought on account of our project manager being very forward thinking but nobody ever thought of it as more than a curiosity and something that might be a handy replacement for the terminals. That attitude has changed a bit but not by much and that's 25 years.
      - logicalmind 16y ago
        Well, nobody would have built another search engine either, but google did it. Part of the problem is the legacy mindset of the higher ups at financial institutions combined with their extremely conservative nature. I currently work in a fairly large financial institution, giving them a web presence for various services and the backend system is no longer a mainframe but an AS/400. Though it still runs COBOL-based apps, so it's really no different than the standard mainframe.
- rbranson 16y ago
  He's talking about FedEx. They understand that it's a risk to their business continuity. The guy who hijacked a FedEx plane planned to fly it into their datacenter near the airport, which houses the primary mainframe that runs FedEx's most critical business software. He wasn't an IT employee either, so it's fairly common knowledge. AFAIK, they are now running in a hybrid mainframe / client+server model. Most of the package transactions are processed simultaneously. Apparently all of the tracking on FedEx.com uses the client+server system to back it. Obviously they are taking it slowly and carefully (it's like a decade-long project) to avoid any screwups.
  - Hoff 16y ago
    When you're working at this scale, geographic redundancy of DCs is commonplace. I know of several cases where entire DCs were destroyed, and production continued unabated. There are commercial platforms that are explicitly designed for this sort of DC redundancy over 800+ km spans. Where you have 400 km between various volumes in your RAIDset.
    - mkr-hn 16y ago
      Do you have links? That's a scenario I've always been curious about, and would like to read about situations where it actually happened.
      - Hoff 16y ago
        http://h71000.www7.hp.com/openvms/brochures/commerzbank/commerzbank.pdf http://h71000.www7.hp.com/openvms/brochures/commerzbank/comm... A moderately technical article on some of the issues that arise, and (for your question) see the end of the following: http://h71000.www7.hp.com/openvms/journal/v1/disastertol.pdf http://h71000.www7.hp.com/openvms/journal/v1/disastertol.pdf
        wallflower 16y ago
        Fascinating. "At 2 a.m. on Dec. 29, 1999, an active stock market trading day, the audio alert on a UPS system alarmed a security guard on his first day on the job. He pressed the emergency power-off switch, taking down the entire datacenter. Because a disaster-tolerant OpenVMS cluster was in place, the cluster continued to run at the opposite site, with no disruption. The brokerage ran through that stock-trading day on the one site alone, and performed a shadow full-copy operation to restore redundancy in the evening, after trading hours. They procured a replacement for the failed security guard by the next day." The last sentence.
- Hoff 16y ago
  Don't presume that you're operating with sufficient domain knowledge to make those judgements, and don't presume that the folks that are making those judgements are ignorant of the options and alternatives. When you're operating at the juggernaught-scale of business or with industrial-scale production, the rules and the requirements and the competitive landscapes are different. And the risks can come not from the existing environments - which generally work - but from making the changes to the environment, and from the risks of outages and downtime. Every vendor that can make even a tangental case for a platform or application or process migration or update or new hardware is also certainly pressing that, too; that's an automatic education for the folks involved. If you can't process customer orders for a week or a month or can't ship orders or your outage slags a whole production line, you could well end up out of business. I know of places that have had similar meltdowns during updates and during migrations, and it's Not Pretty.
- fmora 16y ago
  You have to remember that they use these systems to run their business and they've been tested for thousands or millions of hours so the system is extremely reliable. It is naive to suggest to switch to a newer platform or language. Those things are always changing. Also, once you are a seasoned software developer in multiple languages how long do you really think it will take you to learn cobol? Not much I can assure you. You'll probably be familiar with the basics in a couple of days, and maybe even enough to be dangerous. Switching to a platform just for the sake of switching is extremely dangerous and potentially quite costly. Especially when from the customers point of view it really makes no difference whether behind the scenes the business is using cobol, fortran, java, ruby or whatever new fad of the day. If the system is going to behave exactly the same whether you use java or cobol then what is the point of switching?
  - jacquesm 16y ago
    A million hours is more than a century, I think that is somewhat unlikely.
    - fmora 16y ago
      No is not. Thousands if not millions of people use their systems every year. Every person that uses it counts as time testing the system. Notice how quickly the hours add up if you look at it this way.
      - jacquesm 16y ago
        Ok, I was thinking in terms of the run-time of the software, not in terms of the time people spent using the system. But that's a valid way to look at it.
      - whatusername 16y ago
        You could say the same about Windows XP.
hcho 16y ago
The rules of the free market game says someone using cutting edge technology will come up and sweep all. Which doesn't seem to happen. So maybe, as long as it works, it doesn't matter if it's Cobol and the expensive Cobol programmer is being paid peanuts when compared to the money being made.
- jacquesm 16y ago
  Or compared to the money that could be lost in case of a business interruption. Some of these companies carry a downtime penalty on the order of hundreds of dollars per second. You really don't want to mess up when there is that kind of money on the line.
- SandB0x 16y ago
  Technology is irrelevant. It doesn't happen because there are so many requirements and regulations that even if you had the world's smartest hackers writing in the purest Clojure/Ruby/etc, your software would still have to contain all this horribly complicated business logic and would take years to develop.
  - deleted 16y ago
    [deleted]
- Super_Jambo 16y ago
  By 'the rules of the free market game' I presume you mean the Efficient Market Hypothesis. The EMH supposes that every actor has total and correct information, every actor is rational and products are fungible. Whilst we might approach this on say a stock exchange to imagine most companies compete in an efficient market. When so much of modern business is about branding and advertising (destroying fungiblity and corrupting actors information and rationality). Is, I suggest, somewhat naive. Essentially modern businesses do not generally compete on product, they compete on their ability to distort the markets they're in.

mey 16y ago

These corporations may not see it, because they do not understand it, but having such legacy systems a risk to their operations. The longer they wait to migrate, the more expensive it will be, and the harder it will be to attract talent to the project. There may come a point where they can no longer find anyone to support their hardware or software, and it'll force them to stagnate against their competition who may be able to provide better or different offerings because they are not tied to a system they can't modify. My understanding of business is that you should never stagnate, and always keep advancing. People that I've talked to who have worked with such systems indicated that they built giant shims around the systems in other languages to allow them to be adaptable. In theory this gives them a clear interface definition (it may be huge, thousands of method signatures, and possible direct schema access, but it can still be a clear interface) to boot strap a core system replacement in another language and hardware system.

jacquesm 16y ago
The laws of supply and demand will make sure that there will always be people available to maintain those systems. The price will go up until some 'modern' programmer is tempted enough to open a 3 decades old book on COBOL and JCL. Your understanding of business is flawed, businesses are advancing by marketing, not by shacking up with every new hot tech that comes along in favour of their old, trusted and reliable tech. The larger, older and more profitable a business is the more likely that they are going to be ultra-conservative about using new technology at the core of their business. Eventually we'll get rid of this stuff but I wouldn't bet on it happening in the next two decades, unless someone finds a way to emulate a mainframe on a cluster with similar reliability. And then we still need to get rid of the software.
- logicalmind 16y ago
  You don't think that there is room for a new high-tech bank to step in and eat the lunch of the big banks? If you started a bank from scratch, built with modern technology, you could provide all kinds of valuable services and adapt to change much more quickly. Pretty much every service provided, particularly on the web, is some application that screenscrapes CICS screens or reads data from VSAM. These applications a tricky to build and take much more time than they should.
  - jacquesm 16y ago
    Starting a bank from scratch will have you worry about other things than the technology of your IT department, mostly to do with solvency and staying clear of SO. Sure, at some level the technology will come in to play but nobody will 'start a high tech bank'. For instance, ING direct (one of the efforts of the banking industry to leverage the use of technology) was jump started from the legacy ING systems and as far as I know still ties in to them for all of their back-end stuff. When I was the 'systems administrator' for the corporate division of a bank (in 1986 or so) we had daily rounds of getting the signatures of the various people responsible that the totals for the night batch runs were correct. That's the sort of responsibility you're talking about, I can't imagine that an MD5 or some digital signature would be sufficient today, it's all about procedures and accountability, emphatically not about technology. I got one of the first PCs the bank bought on account of our project manager being very forward thinking but nobody ever thought of it as more than a curiosity and something that might be a handy replacement for the terminals. That attitude has changed a bit but not by much and that's 25 years.
    - logicalmind 16y ago
      Well, nobody would have built another search engine either, but google did it. Part of the problem is the legacy mindset of the higher ups at financial institutions combined with their extremely conservative nature. I currently work in a fairly large financial institution, giving them a web presence for various services and the backend system is no longer a mainframe but an AS/400. Though it still runs COBOL-based apps, so it's really no different than the standard mainframe.
rbranson 16y ago
He's talking about FedEx. They understand that it's a risk to their business continuity. The guy who hijacked a FedEx plane planned to fly it into their datacenter near the airport, which houses the primary mainframe that runs FedEx's most critical business software. He wasn't an IT employee either, so it's fairly common knowledge. AFAIK, they are now running in a hybrid mainframe / client+server model. Most of the package transactions are processed simultaneously. Apparently all of the tracking on FedEx.com uses the client+server system to back it. Obviously they are taking it slowly and carefully (it's like a decade-long project) to avoid any screwups.
- Hoff 16y ago
  When you're working at this scale, geographic redundancy of DCs is commonplace. I know of several cases where entire DCs were destroyed, and production continued unabated. There are commercial platforms that are explicitly designed for this sort of DC redundancy over 800+ km spans. Where you have 400 km between various volumes in your RAIDset.
  - mkr-hn 16y ago
    Do you have links? That's a scenario I've always been curious about, and would like to read about situations where it actually happened.
    - Hoff 16y ago
      http://h71000.www7.hp.com/openvms/brochures/commerzbank/commerzbank.pdf http://h71000.www7.hp.com/openvms/brochures/commerzbank/comm... A moderately technical article on some of the issues that arise, and (for your question) see the end of the following: http://h71000.www7.hp.com/openvms/journal/v1/disastertol.pdf http://h71000.www7.hp.com/openvms/journal/v1/disastertol.pdf
      - wallflower 16y ago
        Fascinating. "At 2 a.m. on Dec. 29, 1999, an active stock market trading day, the audio alert on a UPS system alarmed a security guard on his first day on the job. He pressed the emergency power-off switch, taking down the entire datacenter. Because a disaster-tolerant OpenVMS cluster was in place, the cluster continued to run at the opposite site, with no disruption. The brokerage ran through that stock-trading day on the one site alone, and performed a shadow full-copy operation to restore redundancy in the evening, after trading hours. They procured a replacement for the failed security guard by the next day." The last sentence.
Hoff 16y ago
Don't presume that you're operating with sufficient domain knowledge to make those judgements, and don't presume that the folks that are making those judgements are ignorant of the options and alternatives. When you're operating at the juggernaught-scale of business or with industrial-scale production, the rules and the requirements and the competitive landscapes are different. And the risks can come not from the existing environments - which generally work - but from making the changes to the environment, and from the risks of outages and downtime. Every vendor that can make even a tangental case for a platform or application or process migration or update or new hardware is also certainly pressing that, too; that's an automatic education for the folks involved. If you can't process customer orders for a week or a month or can't ship orders or your outage slags a whole production line, you could well end up out of business. I know of places that have had similar meltdowns during updates and during migrations, and it's Not Pretty.
fmora 16y ago
You have to remember that they use these systems to run their business and they've been tested for thousands or millions of hours so the system is extremely reliable. It is naive to suggest to switch to a newer platform or language. Those things are always changing. Also, once you are a seasoned software developer in multiple languages how long do you really think it will take you to learn cobol? Not much I can assure you. You'll probably be familiar with the basics in a couple of days, and maybe even enough to be dangerous. Switching to a platform just for the sake of switching is extremely dangerous and potentially quite costly. Especially when from the customers point of view it really makes no difference whether behind the scenes the business is using cobol, fortran, java, ruby or whatever new fad of the day. If the system is going to behave exactly the same whether you use java or cobol then what is the point of switching?
- jacquesm 16y ago
  A million hours is more than a century, I think that is somewhat unlikely.
  - fmora 16y ago
    No is not. Thousands if not millions of people use their systems every year. Every person that uses it counts as time testing the system. Notice how quickly the hours add up if you look at it this way.
    - jacquesm 16y ago
      Ok, I was thinking in terms of the run-time of the software, not in terms of the time people spent using the system. But that's a valid way to look at it.
    - whatusername 16y ago
      You could say the same about Windows XP.

hcho 16y ago

The rules of the free market game says someone using cutting edge technology will come up and sweep all. Which doesn't seem to happen. So maybe, as long as it works, it doesn't matter if it's Cobol and the expensive Cobol programmer is being paid peanuts when compared to the money being made.

jacquesm 16y ago
Or compared to the money that could be lost in case of a business interruption. Some of these companies carry a downtime penalty on the order of hundreds of dollars per second. You really don't want to mess up when there is that kind of money on the line.
SandB0x 16y ago
Technology is irrelevant. It doesn't happen because there are so many requirements and regulations that even if you had the world's smartest hackers writing in the purest Clojure/Ruby/etc, your software would still have to contain all this horribly complicated business logic and would take years to develop.
- deleted 16y ago
  [deleted]
Super_Jambo 16y ago
By 'the rules of the free market game' I presume you mean the Efficient Market Hypothesis. The EMH supposes that every actor has total and correct information, every actor is rational and products are fungible. Whilst we might approach this on say a stock exchange to imagine most companies compete in an efficient market. When so much of modern business is about branding and advertising (destroying fungiblity and corrupting actors information and rationality). Is, I suggest, somewhat naive. Essentially modern businesses do not generally compete on product, they compete on their ability to distort the markets they're in.