10 ms·

Is it really common for banks to run legacy applications without sources? How would they know or trust the behavior?

by rjeli 7y ago

jtulkki 7y ago
Because it's been running a certain way for decades?
- stjohnswarts 7y ago
  Yep at a known fixed cost. Corps like knowns and hate unknowns, especially stalwarts like the various big banks. It's kind of like buying stocks rather than shorting stocks. Can you handle the unknown of potentially unlimited losses many times your original "investment" in the venture?
mathattack 7y ago
Not just banks. Any long standing company has this problem.
- lanstin 7y ago
  I have worked with a dot com that linked to commercial C libraries for which they never had the source. Did finally reimplement but not till 2016 or something.
tyingq 7y ago
I can't speak for banks specifically, but for sure lots of old stodgy Fortune 500 companies have critical software running for which they have lost the source code. Similarly, critical closed source software for which the vendor no longer exists. And things like servers running that they are afraid to shut down...they have no idea if the output is used by anyone. It's a mess out there :)
- mdorazio 7y ago
  Can confirm. The last two clients I worked with (both Fortune 500) used mainframes for business critical operations. The most recent one didn't actually have source for some components of their business ops software stack - they've just been working as-is for about 30 years and no one will touch them. I was told they did an assessment a year or two ago to price out what it would take to move the business completely off mainframe and onto an x86 stack. It was in the hundreds of millions of dollars to do so because so much other software has been built to interact with and rely on the mainframe over the years that switching off it would be a multi-year effort across every department in the company. So of course that ROI calculation was pretty damn easy and the mainframe isn't going anywhere.
  - benj111 7y ago
    If you were dependent, to the tune of 100s of millions of $ on certain hardware, wouldn't it make sense to start the process anyway? You don't necessarily need to move the $new project off the legacy hardware, just write it in such a way that you can do easily later. I'm eliding many details here, but the principle stands.
    - mdorazio 7y ago
      Not as long as IBM continues to make mainframes and components, which they do. If you were unable to source replacement hardware to keep it running indefinitely, then yes, but until then it's better to keep paying a couple million a year to keep your business running as-is than to spend hundreds of millions to effectively rebuild the entire infrastructure from the ground up in parallel to the running infrastructure. Remember that these are large, public businesses. Explaining to shareholders that profits are going to take a noticeable hit for years because of IT investment that isn't strictly necessary is effectively a non-starter.
      - benj111 7y ago
        "rebuild the entire infrastructure from the ground up in parallel" I wasn't thinking that. I was thinking more like when you add a new feature, fit it into an API that is portable. Or add a translation layer so the feature can be written how you would like system to be in the future, but it works on your hardware today.
- solotronics 7y ago
  This is awesome in a terrifying way. The code itself outlived any employee that knows what it does.
  - nickserv 7y ago
    Hi, pedant here. That's the original meaning of the word: to inspire awe, and there is always a bit of terror in awe. A hurricane is awesome, as is a $deity that sends one. But alas, language has changed and these kids won't get off my lawn.
    - skrebbel 7y ago
      This is the kind of pedantry I come to HN for, thanks.
      - stjohnswarts 7y ago
        This is the kind of pedantry that makes me thankful for the [-] box
    - rdiddly 7y ago
      Speaking of words that changed: sublime used to mean this. Something powerful and dangerous that inspired terror and a sort of admiration was said to be sublime. Here's Edmund Burke writing in 1757: ‘Whatever is fitted in any sort to excite the ideas of pain, and danger, that is to say, whatever is in any sort terrible, or is conversant about terrible objects, or operates in a manner analogous to terror, is a source of the sublime; that is, it is productive of the strongest emotion which the mind is capable of feeling.’
- wglb 7y ago
  And this was particularly exciting in the runup to Y2K
njharman 7y ago
With few exceptions, every non open source software is. Without sources. Not every legacy but critical app was in-house custom software.
- jeremyjh 7y ago
  A lot of vendor packages included source code licenses actually. I'm not sure distributing binaries was even practical in all cases since the code would have to be built with and linked to specific versions of systems libraries and keeping track of which customer is on which system would be a nightmare. I'm familiar with several loan accounting systems and financial authorization systems that are either nowadays totally maintained by the bank or co-developed with the vendor in a shared source model.
dejv 7y ago
Even having access to the source code might not be enough: I’ve seen cases of weirdly forked repos with custom additions and half baked backports that relied on hacked system libraries and weird mix of specific legacy packages in some symbiotic way that just one specific laptop of some long gone developer was able to compile it and virtual machine of this Windows XP was passed around.
tgsovlerkhgsel 7y ago
> How would they know or trust the behavior? It has worked for the past 30 years and hasn't been touched since 20.
rodgerd 7y ago
As someone who works for a bank... no, not in my experience. What is a problem is that the last time a lot of code was touched may be 5-10 years ago. That code may have started being written 20-40 years ago. It may well have been maintained by people who think "if it was hard to write, it should be hard to read" or "documentation is for the weak". It definitely will have been written when the cost of a gig of memory and storage was many orders of magnitude higher than today (indeed, last time I priced memory for a mainframe, a Z10, it ran to $10,000 a gig); hence terseness in everything from table and column names through stored data and everything else was prized. Dropping from COBOL into assembler is not uncommon for critical path performance. Making any changes will be a week of coding and three months of working out the what and why of the code, because the last person who worked on it retired a couple of years ago.

Is it really common for banks to run legacy applications without sources? How would they know or trust the behavior?

by rjeli 7y ago

Is it really common for banks to run legacy applications without sources? How would they know or trust the behavior?

jtulkki 7y ago

Because it's been running a certain way for decades?

stjohnswarts 7y ago
Yep at a known fixed cost. Corps like knowns and hate unknowns, especially stalwarts like the various big banks. It's kind of like buying stocks rather than shorting stocks. Can you handle the unknown of potentially unlimited losses many times your original "investment" in the venture?

mathattack 7y ago

Not just banks. Any long standing company has this problem.

lanstin 7y ago
I have worked with a dot com that linked to commercial C libraries for which they never had the source. Did finally reimplement but not till 2016 or something.

tyingq 7y ago

I can't speak for banks specifically, but for sure lots of old stodgy Fortune 500 companies have critical software running for which they have lost the source code. Similarly, critical closed source software for which the vendor no longer exists. And things like servers running that they are afraid to shut down...they have no idea if the output is used by anyone. It's a mess out there :)

mdorazio 7y ago
Can confirm. The last two clients I worked with (both Fortune 500) used mainframes for business critical operations. The most recent one didn't actually have source for some components of their business ops software stack - they've just been working as-is for about 30 years and no one will touch them. I was told they did an assessment a year or two ago to price out what it would take to move the business completely off mainframe and onto an x86 stack. It was in the hundreds of millions of dollars to do so because so much other software has been built to interact with and rely on the mainframe over the years that switching off it would be a multi-year effort across every department in the company. So of course that ROI calculation was pretty damn easy and the mainframe isn't going anywhere.
- benj111 7y ago
  If you were dependent, to the tune of 100s of millions of $ on certain hardware, wouldn't it make sense to start the process anyway? You don't necessarily need to move the $new project off the legacy hardware, just write it in such a way that you can do easily later. I'm eliding many details here, but the principle stands.
  - mdorazio 7y ago
    Not as long as IBM continues to make mainframes and components, which they do. If you were unable to source replacement hardware to keep it running indefinitely, then yes, but until then it's better to keep paying a couple million a year to keep your business running as-is than to spend hundreds of millions to effectively rebuild the entire infrastructure from the ground up in parallel to the running infrastructure. Remember that these are large, public businesses. Explaining to shareholders that profits are going to take a noticeable hit for years because of IT investment that isn't strictly necessary is effectively a non-starter.
    - benj111 7y ago
      "rebuild the entire infrastructure from the ground up in parallel" I wasn't thinking that. I was thinking more like when you add a new feature, fit it into an API that is portable. Or add a translation layer so the feature can be written how you would like system to be in the future, but it works on your hardware today.
solotronics 7y ago
This is awesome in a terrifying way. The code itself outlived any employee that knows what it does.
- nickserv 7y ago
  Hi, pedant here. That's the original meaning of the word: to inspire awe, and there is always a bit of terror in awe. A hurricane is awesome, as is a $deity that sends one. But alas, language has changed and these kids won't get off my lawn.
  - skrebbel 7y ago
    This is the kind of pedantry I come to HN for, thanks.
    - stjohnswarts 7y ago
      This is the kind of pedantry that makes me thankful for the [-] box
  - rdiddly 7y ago
    Speaking of words that changed: sublime used to mean this. Something powerful and dangerous that inspired terror and a sort of admiration was said to be sublime. Here's Edmund Burke writing in 1757: ‘Whatever is fitted in any sort to excite the ideas of pain, and danger, that is to say, whatever is in any sort terrible, or is conversant about terrible objects, or operates in a manner analogous to terror, is a source of the sublime; that is, it is productive of the strongest emotion which the mind is capable of feeling.’
wglb 7y ago
And this was particularly exciting in the runup to Y2K

njharman 7y ago

With few exceptions, every non open source software is. Without sources. Not every legacy but critical app was in-house custom software.

jeremyjh 7y ago
A lot of vendor packages included source code licenses actually. I'm not sure distributing binaries was even practical in all cases since the code would have to be built with and linked to specific versions of systems libraries and keeping track of which customer is on which system would be a nightmare. I'm familiar with several loan accounting systems and financial authorization systems that are either nowadays totally maintained by the bank or co-developed with the vendor in a shared source model.

dejv 7y ago

Even having access to the source code might not be enough: I’ve seen cases of weirdly forked repos with custom additions and half baked backports that relied on hacked system libraries and weird mix of specific legacy packages in some symbiotic way that just one specific laptop of some long gone developer was able to compile it and virtual machine of this Windows XP was passed around.

tgsovlerkhgsel 7y ago

> How would they know or trust the behavior? It has worked for the past 30 years and hasn't been touched since 20.

rodgerd 7y ago

As someone who works for a bank... no, not in my experience. What is a problem is that the last time a lot of code was touched may be 5-10 years ago. That code may have started being written 20-40 years ago. It may well have been maintained by people who think "if it was hard to write, it should be hard to read" or "documentation is for the weak". It definitely will have been written when the cost of a gig of memory and storage was many orders of magnitude higher than today (indeed, last time I priced memory for a mainframe, a Z10, it ran to $10,000 a gig); hence terseness in everything from table and column names through stored data and everything else was prized. Dropping from COBOL into assembler is not uncommon for critical path performance. Making any changes will be a week of coding and three months of working out the what and why of the code, because the last person who worked on it retired a couple of years ago.