8 ms·

Seeing stories like this makes me wonder if we'll ever see a company actually go bust based purely on the instability of its decades-old, duct-taped architectur

Seeing stories like this makes me wonder if we'll ever see a company actually go bust based purely on the instability of its decades-old, duct-taped architecture. I have a relative who works for a large credit card processor running on mainframe systems and I hear over and over again that they're all retiring and unable to fill the newly opened roles. This problem is compounded by the fact that their offshore contracting firms are cutting back on providing COBOL resources because there's a lot more money in supplying java/.net/etc. devs. The work culture there sounds so unsustainable and yet they're one of the largest payment processors in the country. If they were have widespread stability issues like this, things could get interesting.

silisili 5y ago
This COBOL shortage myth has been around since I was a kid. It wasn't true then, and isn't now. The entire problem is that they don't pay well for what is supposedly a rare resource. And nobody wants to learn a dead language for peanuts. Offer 500k/yr, and me and about 100 million other people will be learning COBOL starting tomorrow.
- treespace88 5y ago
  Also there are a ton of working programers that have some experience in cobol from school. But everyone knows it’s a dead end. So yeah they need to pay very well to get the fixed they need.
- smsm42 5y ago
  Right. COBOL is not some mythical ancient knowledge that can't be learned today. I'm pretty sure anyone who can learn Haskell can learn COBOL too, it's just a question of making them want to.
- drstewart 5y ago
  Sounds like it isn't a myth - you yourself admit there aren't people who know it, just that would OFFER to learn it given the appropriate compensation. By that logic, there isn't a shortage of anything if you have enough money.
  - silisili 5y ago
    The 'myth' since the early 90s is that you should definitely study COBOL because you'll be set for life because of the big shortage.
- bluedino 5y ago
  500? Try 250. Heck, 150 if we can work from home The problem is now you have a bunch of D-level cobol coders without any experience, domain knowledge, etc Which is where we are with most Java dev shops.
  - rad_gruchalski 5y ago
    > The problem is now you have a bunch of D-level cobol coders without any experience, domain knowledge, etc Right? So don’t wait too long because all of those who can pass this knowledge will be gone!
dilyevsky 5y ago
Not exactly but https://en.m.wikipedia.org/wiki/Knight_Capital_Group https://en.m.wikipedia.org/wiki/Knight_Capital_Group
sushid 5y ago
This problem isn't occurring due to software issues.
mberning 5y ago
It is maddening. Even if the stated goal is to “get off the mainframe” everybody deep down knows it isn’t going anywhere for 10 years. We recently had a discussion about porting from one mainframe “module” to another due to cost. I asked why we don’t just continue paying for the current module since the goal is to sunset mainframe. Everybody on the call laughed. So we will go ahead and “invest” several man years and god knows how much money into a platform everybody agrees we should get off of.
- tdeck 5y ago
  I've been on the other side of this and seen too many teams neglect to fix even the simplest and most frustrating bugs in a system because they're going to rewrite it "in the next six months". Usually the rewrite is still going years later, and the actual code used in production is left to rot, holding back other teams as well.
  - wyldfire 5y ago
    Can you blame them? These are the same teams who typically refuse to adopt practices that would allow them to create tests and we'll encapsulated designs. So inevitably the team is once bitten twice shy about breaking production. Why be the poor bastard who introduced anything but the tiniest bugfix?
    - tdeck 5y ago
      I definitely agree it's a matter of incentives. Even in an environment with a now healthier quality culture, fixing bugs to bring an old system from 70% functionality to 100% is typically a career dead end. It's much better to leave it broken and build a new system with 105% functionality so you can take credit for the whole 105%. And typically, optimists that we are, it seems like completely reimplementing the original functionality will be "trivial" compared to merely trying to understand how the existing solution works.
  - trixie_ 5y ago
    It happens when you have a lot of new people who refuse to work with legacy code (which is anything over 2 years old), trivialize the complexity of the existing system (because they don’t understand a tenth of what it does), go create a sub par replacement with tons of people and years of work (that ends up coupled to the legacy system anyways), and now the company has two systems to maintain requiring double the resources as before. When everyone gets fed up and leaves, the cycle will start again. Our system is about four levels deep now. It’s the f’ing matrix.
riffic 5y ago
> decades-old, duct-taped architecture The way the survivorship bias works out, this is generally a strength, not a weakness. You want to get good money, you learn how stuff works under the hood and get familiar with your predecessor architectures, emulate them if you need to.
- Daneel_ 5y ago
  I have to disagree. These systems aren't "survivors" - they're held together with a fragile weave of scripts, hacks, unsupported hardware and outdated knowledge. Often treated as a black box, they become an object that people have to hand-nurse through issues that would never trouble modern systems. The amount of effort and money that's expended to maintain these systems is typically far more than the upfront cost would be to replace them every decade with a more modern one, but that would look bad in the short term financials, so we keep babysitting the tower of glass and praying it doesn't fall.
  - khuey 5y ago
    And yet the flip side where companies set piles of cash on fire trying to replace that legacy stuff is also common. e.g. https://www.computerweekly.com/news/252446965/Lidl-dumps-500m-SAP-project https://www.computerweekly.com/news/252446965/Lidl-dumps-500...
  - nimih 5y ago
    I spent multiple years of my life working on a (at the time I started) decade+ project to retire such a system (i.e. a duct-taped hodgepodge of poorly documented COBOL and mainframe assembler that was understood and maintained by staff who was entering retirement age at an astounding rate), at a "midsize" financial company (i.e. between 3k and 4k employees, all told). While the cost of maintaining the old system was certainly high, it worked astonishingly well (provided you didn't need to add new features, such as a "graphical interface"), and the cost and difficulty in replacing it was truly enormous. I think you're drastically underselling the difficulty of re-implementing a company's entire suite of LOB software, complete with migrations and training, and likely with very little in the way of flexibility to change the core business rules/logic. Furthermore, > they're held together with a fragile weave of scripts, hacks, unsupported hardware and outdated knowledge Swapping out COBOL for typescript and mainframes for linux boxes and a pile of YAML is not going to change any of that. People can and will build garbage with any set of tools you give them. Not ever software problem is like use-after-free where you can fix it forever with a sufficiently clever piece of tech.
_8j50 5y ago
Right... they can't train someone on COBOL? I call b.s. on that. It's an HR issue, you need blah blah degrees and yadi yadi yada experience. Big corps don't adopt to job economy changes over time like this unless their core business is built to adopt to economic changes. Inability to make exemptions and place the right first line managers over teams is what kills big companies, or forces them into an endless loop of unoriginal aquisitions and failures imo.
- coliveira 5y ago
  Banks are genetically incapable of doing anything useful in terms of technology. Their mainframe architecture was outdated 40 years ago! They will forever maintain their hacks and buggy software.
  - listenallyall 5y ago
    Please define "buggy software." Can you point to a single confirmed case in the US of a person's bank account balance disappearing, or even a deposit not appearing, due to a software bug? I doubt it. Banks' core software is rock solid or they would be out of business very, very quickly.
    - novok 5y ago
      Transaction reversibility, fear of law enforcement and the ability to react to things is the reason why the bank tire fire doesn't collapse on itself.
      - rrix2 5y ago
        im sure glad the cryptocurrency folks seem intent on throwing all three of those away at once.
    - scollet 5y ago
      It is not, based on the loopholes one has to jump through for basic support.
    - 5y ago
41209 5y ago
If they want to pay they'll find someone. I wouldn't mind taking a 250$ an hour position with on the job training. Odds are they're getting outbid by easier jobs. If I can write Javascript for 130k or COBOL for 130, I'm doing JS.
dustintrex 5y ago
Decades-old, duct-taped architecture is endemic across the airline industry though, and AFAIK Southwest isn't even particularly bad on this front. American Airlines's original reservation system Sabre, still a major GDS player today, was first put together in the 1950s and is by some reckonings the oldest commercial (as in, non-government/military/research) computer system out there. https://en.wikipedia.org/wiki/Sabre_(computer_system) https://en.wikipedia.org/wiki/Sabre_(computer_system)
- namdnay 5y ago
  Southwest are better than most NA carriers in this regard, they switched to Amadeus a few years ago. Which is "only" 10-30 years old (depending on the components).
tawayffee 5y ago
You're falling for misdirection. This isn't IT related.
- pferde 5y ago
  Your comment is being downvoted not because people disagree with you, but because it is not bringing anything constructive to the discussion. Perhaps if you bothered to write about what you think the meltdown really is related to, your commend would have some value.
  - tawayffee2 5y ago
    The original comment from the union only mentioned IT issues from June. "Our operation and our frontline employees have endured continuous and unending disruptions since the first time our airline made headlines in early June due to widespread IT failures." That's what makes it misdirection, mentioning an issue from June, and very specifically not talking about a current IT issue. If there were a current issue, they would have called it out. The union has also been pretty clear in other statements that operational issues over the past few months were due to staffing shortages, not IT.
soylentnewsorg 5y ago
Decades-old does not mean duct-taped. It means every edge case has been hit, it's rock solid, and it works. There are many patches on top of patches, and the system is hard to comprehend for someone new. But it's a heck of a lot more stable that something redesigned. The real issue here, is during bad weather and understaffed ATC, for some reason the airforce decided to do some training in airspace used by commercial flights. This military training part in the article is mentioned quickly, then completely ignored. This was the US Military causing delays, cancellations, and people getting stuck overnight at airports. This was the US Military causing huge operational and financial issues for one of the most customer-friendly airlines out there. And the did this while eating up millions of dollars that are taken out of our paychecks - that they used to screw us. Time for congress to remove the funds that allowed them to conduct this training at a time when it caused problems for the general population paying their salaries.
- creato 5y ago
  If the cause is the military, why is southwest so disproportionately affected, and being affected in so many cities? > “The problems weren’t concentrated in just one particular region. The cancelations affected the entire Southwest network. People living in cities where Southwest has a big operation, like Phoenix, Denver, Las Vegas, Los Angeles, and where I am in the San Francisco Bay Area, saw cancelations in the double-digits.”
  - soylentnewsorg 5y ago
    an airport is a hub for an airline, but not for other airlines. if southwest has 50 planes at an airport, and delta has 10 southwest is going to be impacted. when your plane takeoffs are cancelled at your hub, every single flight that was going to use that plane later in the day, or the following day, is now missing a plane. if Chicago O'hare is shut down, United is done for the day in half the country, because half its planes are stuck at O'hare. if Chicago Midway is shut down, only Southwest is screwed. if Atlanta shuts down, only Delta is screwed. Yes, I'm sure there are other cities that also had planes. Like Phoenix. Except the plane that was supposed to land in Phoenix is stuck in Florida. Or the pilot piloting the plane already in Phoenix is stuck in Florida. Planes go from one city to another. If you introduce a blockage along the way, you've now taken that plane, or a crew out of commission.

silisili 5y ago

This COBOL shortage myth has been around since I was a kid. It wasn't true then, and isn't now. The entire problem is that they don't pay well for what is supposedly a rare resource. And nobody wants to learn a dead language for peanuts. Offer 500k/yr, and me and about 100 million other people will be learning COBOL starting tomorrow.

treespace88 5y ago
Also there are a ton of working programers that have some experience in cobol from school. But everyone knows it’s a dead end. So yeah they need to pay very well to get the fixed they need.
smsm42 5y ago
Right. COBOL is not some mythical ancient knowledge that can't be learned today. I'm pretty sure anyone who can learn Haskell can learn COBOL too, it's just a question of making them want to.
drstewart 5y ago
Sounds like it isn't a myth - you yourself admit there aren't people who know it, just that would OFFER to learn it given the appropriate compensation. By that logic, there isn't a shortage of anything if you have enough money.
- silisili 5y ago
  The 'myth' since the early 90s is that you should definitely study COBOL because you'll be set for life because of the big shortage.
bluedino 5y ago
500? Try 250. Heck, 150 if we can work from home The problem is now you have a bunch of D-level cobol coders without any experience, domain knowledge, etc Which is where we are with most Java dev shops.
- rad_gruchalski 5y ago
  > The problem is now you have a bunch of D-level cobol coders without any experience, domain knowledge, etc Right? So don’t wait too long because all of those who can pass this knowledge will be gone!

dilyevsky 5y ago

Not exactly but https://en.m.wikipedia.org/wiki/Knight_Capital_Group https://en.m.wikipedia.org/wiki/Knight_Capital_Group

sushid 5y ago

This problem isn't occurring due to software issues.

mberning 5y ago

It is maddening. Even if the stated goal is to “get off the mainframe” everybody deep down knows it isn’t going anywhere for 10 years. We recently had a discussion about porting from one mainframe “module” to another due to cost. I asked why we don’t just continue paying for the current module since the goal is to sunset mainframe. Everybody on the call laughed. So we will go ahead and “invest” several man years and god knows how much money into a platform everybody agrees we should get off of.

tdeck 5y ago
I've been on the other side of this and seen too many teams neglect to fix even the simplest and most frustrating bugs in a system because they're going to rewrite it "in the next six months". Usually the rewrite is still going years later, and the actual code used in production is left to rot, holding back other teams as well.
- wyldfire 5y ago
  Can you blame them? These are the same teams who typically refuse to adopt practices that would allow them to create tests and we'll encapsulated designs. So inevitably the team is once bitten twice shy about breaking production. Why be the poor bastard who introduced anything but the tiniest bugfix?
  - tdeck 5y ago
    I definitely agree it's a matter of incentives. Even in an environment with a now healthier quality culture, fixing bugs to bring an old system from 70% functionality to 100% is typically a career dead end. It's much better to leave it broken and build a new system with 105% functionality so you can take credit for the whole 105%. And typically, optimists that we are, it seems like completely reimplementing the original functionality will be "trivial" compared to merely trying to understand how the existing solution works.
- trixie_ 5y ago
  It happens when you have a lot of new people who refuse to work with legacy code (which is anything over 2 years old), trivialize the complexity of the existing system (because they don’t understand a tenth of what it does), go create a sub par replacement with tons of people and years of work (that ends up coupled to the legacy system anyways), and now the company has two systems to maintain requiring double the resources as before. When everyone gets fed up and leaves, the cycle will start again. Our system is about four levels deep now. It’s the f’ing matrix.

riffic 5y ago

> decades-old, duct-taped architecture The way the survivorship bias works out, this is generally a strength, not a weakness. You want to get good money, you learn how stuff works under the hood and get familiar with your predecessor architectures, emulate them if you need to.

Daneel_ 5y ago
I have to disagree. These systems aren't "survivors" - they're held together with a fragile weave of scripts, hacks, unsupported hardware and outdated knowledge. Often treated as a black box, they become an object that people have to hand-nurse through issues that would never trouble modern systems. The amount of effort and money that's expended to maintain these systems is typically far more than the upfront cost would be to replace them every decade with a more modern one, but that would look bad in the short term financials, so we keep babysitting the tower of glass and praying it doesn't fall.
- khuey 5y ago
  And yet the flip side where companies set piles of cash on fire trying to replace that legacy stuff is also common. e.g. https://www.computerweekly.com/news/252446965/Lidl-dumps-500m-SAP-project https://www.computerweekly.com/news/252446965/Lidl-dumps-500...
- nimih 5y ago
  I spent multiple years of my life working on a (at the time I started) decade+ project to retire such a system (i.e. a duct-taped hodgepodge of poorly documented COBOL and mainframe assembler that was understood and maintained by staff who was entering retirement age at an astounding rate), at a "midsize" financial company (i.e. between 3k and 4k employees, all told). While the cost of maintaining the old system was certainly high, it worked astonishingly well (provided you didn't need to add new features, such as a "graphical interface"), and the cost and difficulty in replacing it was truly enormous. I think you're drastically underselling the difficulty of re-implementing a company's entire suite of LOB software, complete with migrations and training, and likely with very little in the way of flexibility to change the core business rules/logic. Furthermore, > they're held together with a fragile weave of scripts, hacks, unsupported hardware and outdated knowledge Swapping out COBOL for typescript and mainframes for linux boxes and a pile of YAML is not going to change any of that. People can and will build garbage with any set of tools you give them. Not ever software problem is like use-after-free where you can fix it forever with a sufficiently clever piece of tech.

_8j50 5y ago

Right... they can't train someone on COBOL? I call b.s. on that. It's an HR issue, you need blah blah degrees and yadi yadi yada experience. Big corps don't adopt to job economy changes over time like this unless their core business is built to adopt to economic changes. Inability to make exemptions and place the right first line managers over teams is what kills big companies, or forces them into an endless loop of unoriginal aquisitions and failures imo.

coliveira 5y ago
Banks are genetically incapable of doing anything useful in terms of technology. Their mainframe architecture was outdated 40 years ago! They will forever maintain their hacks and buggy software.
- listenallyall 5y ago
  Please define "buggy software." Can you point to a single confirmed case in the US of a person's bank account balance disappearing, or even a deposit not appearing, due to a software bug? I doubt it. Banks' core software is rock solid or they would be out of business very, very quickly.
  - novok 5y ago
    Transaction reversibility, fear of law enforcement and the ability to react to things is the reason why the bank tire fire doesn't collapse on itself.
    - rrix2 5y ago
      im sure glad the cryptocurrency folks seem intent on throwing all three of those away at once.
  - scollet 5y ago
    It is not, based on the loopholes one has to jump through for basic support.
  - 5y ago

41209 5y ago

If they want to pay they'll find someone. I wouldn't mind taking a 250$ an hour position with on the job training. Odds are they're getting outbid by easier jobs. If I can write Javascript for 130k or COBOL for 130, I'm doing JS.

dustintrex 5y ago

Decades-old, duct-taped architecture is endemic across the airline industry though, and AFAIK Southwest isn't even particularly bad on this front. American Airlines's original reservation system Sabre, still a major GDS player today, was first put together in the 1950s and is by some reckonings the oldest commercial (as in, non-government/military/research) computer system out there. https://en.wikipedia.org/wiki/Sabre_(computer_system) https://en.wikipedia.org/wiki/Sabre_(computer_system)

namdnay 5y ago
Southwest are better than most NA carriers in this regard, they switched to Amadeus a few years ago. Which is "only" 10-30 years old (depending on the components).

tawayffee 5y ago

You're falling for misdirection. This isn't IT related.

pferde 5y ago
Your comment is being downvoted not because people disagree with you, but because it is not bringing anything constructive to the discussion. Perhaps if you bothered to write about what you think the meltdown really is related to, your commend would have some value.
- tawayffee2 5y ago
  The original comment from the union only mentioned IT issues from June. "Our operation and our frontline employees have endured continuous and unending disruptions since the first time our airline made headlines in early June due to widespread IT failures." That's what makes it misdirection, mentioning an issue from June, and very specifically not talking about a current IT issue. If there were a current issue, they would have called it out. The union has also been pretty clear in other statements that operational issues over the past few months were due to staffing shortages, not IT.

soylentnewsorg 5y ago

Decades-old does not mean duct-taped. It means every edge case has been hit, it's rock solid, and it works. There are many patches on top of patches, and the system is hard to comprehend for someone new. But it's a heck of a lot more stable that something redesigned. The real issue here, is during bad weather and understaffed ATC, for some reason the airforce decided to do some training in airspace used by commercial flights. This military training part in the article is mentioned quickly, then completely ignored. This was the US Military causing delays, cancellations, and people getting stuck overnight at airports. This was the US Military causing huge operational and financial issues for one of the most customer-friendly airlines out there. And the did this while eating up millions of dollars that are taken out of our paychecks - that they used to screw us. Time for congress to remove the funds that allowed them to conduct this training at a time when it caused problems for the general population paying their salaries.

creato 5y ago
If the cause is the military, why is southwest so disproportionately affected, and being affected in so many cities? > “The problems weren’t concentrated in just one particular region. The cancelations affected the entire Southwest network. People living in cities where Southwest has a big operation, like Phoenix, Denver, Las Vegas, Los Angeles, and where I am in the San Francisco Bay Area, saw cancelations in the double-digits.”
- soylentnewsorg 5y ago
  an airport is a hub for an airline, but not for other airlines. if southwest has 50 planes at an airport, and delta has 10 southwest is going to be impacted. when your plane takeoffs are cancelled at your hub, every single flight that was going to use that plane later in the day, or the following day, is now missing a plane. if Chicago O'hare is shut down, United is done for the day in half the country, because half its planes are stuck at O'hare. if Chicago Midway is shut down, only Southwest is screwed. if Atlanta shuts down, only Delta is screwed. Yes, I'm sure there are other cities that also had planes. Like Phoenix. Except the plane that was supposed to land in Phoenix is stuck in Florida. Or the pilot piloting the plane already in Phoenix is stuck in Florida. Planes go from one city to another. If you introduce a blockage along the way, you've now taken that plane, or a crew out of commission.