6 ms·

Right... they can't train someone on COBOL? I call b.s. on that. It's an HR issue, you need blah blah degrees and yadi yadi yada experience. Big corps don't ado

by _8j50 5y ago

Right... they can't train someone on COBOL? I call b.s. on that. It's an HR issue, you need blah blah degrees and yadi yadi yada experience. Big corps don't adopt to job economy changes over time like this unless their core business is built to adopt to economic changes. Inability to make exemptions and place the right first line managers over teams is what kills big companies, or forces them into an endless loop of unoriginal aquisitions and failures imo.

coliveira 5y ago
Banks are genetically incapable of doing anything useful in terms of technology. Their mainframe architecture was outdated 40 years ago! They will forever maintain their hacks and buggy software.
- listenallyall 5y ago
  Please define "buggy software." Can you point to a single confirmed case in the US of a person's bank account balance disappearing, or even a deposit not appearing, due to a software bug? I doubt it. Banks' core software is rock solid or they would be out of business very, very quickly.
  - novok 5y ago
    Transaction reversibility, fear of law enforcement and the ability to react to things is the reason why the bank tire fire doesn't collapse on itself.
    - rrix2 5y ago
      im sure glad the cryptocurrency folks seem intent on throwing all three of those away at once.
  - scollet 5y ago
    It is not, based on the loopholes one has to jump through for basic support.
  - manigandham 5y ago
    Banks have procedures and regulations that protect customers from those issues. In fact there's a lot of of secondary checks and offline reconciliation precisely to catch potential software errors.
  - rurounijones 5y ago
    Not the US, but in the UK which I think is comparable: * https://www.theguardian.com/money/2015/aug/28/many-hsbc-customers-facing-payday-without-pay https://www.theguardian.com/money/2015/aug/28/many-hsbc-cust... (I believe this one was caused by firing greybeards who understood a certain very complicated batch processing system and offshoring to undertrained staff in India who then make a mistake and basically brough the entire system down as a knock-on effect) * https://www.theguardian.com/business/2019/nov/19/tsb-it-meltdown-report-computer-failure-accounts https://www.theguardian.com/business/2019/nov/19/tsb-it-melt... (Insufficiently tested software upgrade although to be fair this was a major upgrade) * https://scottishfinancialnews.com/article/rbs-it-failure-prevents-customer-access-to-accounts https://scottishfinancialnews.com/article/rbs-it-failure-pre... (Not sure on root cause. Other interesting stats at the bottom of the article regarding failure counts) Now, in all these cases I believe the outages lasted a few days at least where people had limited or no access to bank accounts, salaries not paid on time, incorrectly charged overdraft fees, automated payments not happening etc. It can probably be argued what is "core" here and as far as I know no money actually disappeared into the ether but, in the UK at least, I am not sure I would be so confident in my declaration of Rock Solid software.
    - listenallyall 5y ago
      Thanks, these examples confirm my point. The old, ancient, "outdated" (as the original poster claimed) software worked fine for years/decades. The failures that you cited were introduced during an upgrade, a rewrite/outsourcing, or due to a 3rd party outage.
      - snovv_crash 5y ago
        But requirements change, so upgrades are a normal part of business, due to at the bare minimum legal reporting requirements changing, even if there weren't all sorts of new things like internet banking making their old APIs inadequate. If the software isn't extendable or changeable then it isn't 'working fine', because if you can't offer internet banking or do FATCA reporting you go out of business or get shut down.
  - chefandy 5y ago
    You've cited two comparatively minor failures confined to a very small subset of what banks do. There's a whole lot of critical software systems in banks that don't touch personal bank accounts. If you've ever worked on bank systems before, it was certainly in a very limited capacity. I did code-level support for a large internal data management system for the federal reserve bank in the mid aughts. I'm not really interested in disclosing the specific bland but critically important data we moved/managed. It involved a giant amalgam of legacy commercial AIX and HP-UX software with some newer Java stuff in a Websphere environment. It was all duct-tape-and-bubble-gummed together with huge ksh88 scripts that were in constant development by devs and support people like me who'd just ssh into production systems to make changes. None of the internal software had dev/test systems, there was definitely no code review, and there was no version control. Yep. The system on a whole required constant monitoring and intervention to keep it plodding along. We did periodically see data loss that was unrecoverable from a technological standpoint but recovered with physical media kept around from their necessarily belt-and-suspenders business practices and tons of available staff for data entry. I reckon tens-of-thousands of folks outside the bank would have been affected per incident.
    - listenallyall 5y ago
      Money disappearing from a bank account is a "comparatively minor failure"? Sorry, I stopped reading at that point.
      - craftinator 5y ago
        There have been major bank tech failures too, if you wanted to see major vs minor side by side...
      - chefandy 5y ago
        If you're committed to not knowing something then I can't stop you. But compared to, say, tens of thousands of investment transactions for pension and retirement funds evaporating, disbursements to a hospital's operations budget being delayed, sending confidential historical balance records in bulk to a large corporate customer's competitor, having an online investment system down during a period of extreme market volatility, or even large-scale customer data breaches, then yes. 'Money missing from a bank account' is in every significant way a comparatively minor problem probably fixable with a phone call and a couple hours of investigation on the bank's part, even with a significant sum of money. The bank would almost certainly catch it themselves during an automated audit. It's exactly the sort of problem that bookkeeping organizations are designed to mitigate and root out if they do happen. The others could affect many, many more people's lives for much longer— potentially permanently.
- jcims 5y ago
  What do you mean by 'useful' here? Facebook goes down for half a day and everybody laughs. Take 'banks' down for the same amount of time and see what happens.
  - jimmydorry 5y ago
    Multi-hour outages are already relatively common and getting more frequent by the year for banks. About 10 days ago there was a massive Bank of America outage.
  - stickfigure 5y ago
    It's pretty standard for banks to "go down" outside the hours of 9am-5pm. Nearly everything meaningful that a bank does is processed in batch, often with days-long delays. Operationally, Facebook is a vastly harder problem.
  - coliveira 5y ago
    You certainly don't use banks often, because their systems being offline for several hours is a relative common event.
ctdonath 5y ago
Not if they can’t find anyone willing to learn COBOL. There’s huge demand for developers on modern systems. Seriously, how much are you going to pay a developer to do mainframe COBOL - and is that developer going to produce commensurate value, really able to hold together a crumbling antique infrastructure?
- _8j50 5y ago
  Plenty of people working helpdesk that know a bit of python who can learn COBOL if they knew a job offer was on the other end of the investment in learning the language. There are hoards of people learning python and powershell just so they can make more money, trust me when I say they (the people I see learning python for career prospects) have no sentimental attachment to the language.
  - noduerme 5y ago
    I don't think it's learning COBOL language that's the hard part. An inexperienced programmer who hasn't worked on financial systems is going to make security mistakes and write buggy code for awhile, both of which can be deadly to a bank. But it would be better to have that person rewrite the infrastructure than try to fix bugs. The problem with these ancient systems is trying to understand how bugs arise from subtle unintended interactions within millions of lines of code. I've been coding many languages for 25 years and I would consider it extremely daunting to try to tackle a bank bug knowing that one wrong assumption about one line of code could crash whole systems or cost millions of dollars.
    - mandelbrotwurst 5y ago
      Eh, someone's got to do it. Letting thing stagnate in perpetuity isn't cheap either. Move slowly and write tests.
      - noduerme 5y ago
        Sure, but a veteran coder at least understands that tampering with something may affect some other subsystem you don't even know exists until it breaks... which you wouldn't even think to test. The idea of throwing the phone staff into COBOL instead of them learning Python just obfuscates the real problem which isn't that the language is uncool and dying but that the systems are like the rules of the NFL. They stopped making sense a long time ago to anyone who wasn't immersed and paying attention.

by _8j50 5y ago

coliveira 5y ago

Banks are genetically incapable of doing anything useful in terms of technology. Their mainframe architecture was outdated 40 years ago! They will forever maintain their hacks and buggy software.

listenallyall 5y ago
Please define "buggy software." Can you point to a single confirmed case in the US of a person's bank account balance disappearing, or even a deposit not appearing, due to a software bug? I doubt it. Banks' core software is rock solid or they would be out of business very, very quickly.
- novok 5y ago
  Transaction reversibility, fear of law enforcement and the ability to react to things is the reason why the bank tire fire doesn't collapse on itself.
  - rrix2 5y ago
    im sure glad the cryptocurrency folks seem intent on throwing all three of those away at once.
- scollet 5y ago
  It is not, based on the loopholes one has to jump through for basic support.
- manigandham 5y ago
  Banks have procedures and regulations that protect customers from those issues. In fact there's a lot of of secondary checks and offline reconciliation precisely to catch potential software errors.
- rurounijones 5y ago
  Not the US, but in the UK which I think is comparable: * https://www.theguardian.com/money/2015/aug/28/many-hsbc-customers-facing-payday-without-pay https://www.theguardian.com/money/2015/aug/28/many-hsbc-cust... (I believe this one was caused by firing greybeards who understood a certain very complicated batch processing system and offshoring to undertrained staff in India who then make a mistake and basically brough the entire system down as a knock-on effect) * https://www.theguardian.com/business/2019/nov/19/tsb-it-meltdown-report-computer-failure-accounts https://www.theguardian.com/business/2019/nov/19/tsb-it-melt... (Insufficiently tested software upgrade although to be fair this was a major upgrade) * https://scottishfinancialnews.com/article/rbs-it-failure-prevents-customer-access-to-accounts https://scottishfinancialnews.com/article/rbs-it-failure-pre... (Not sure on root cause. Other interesting stats at the bottom of the article regarding failure counts) Now, in all these cases I believe the outages lasted a few days at least where people had limited or no access to bank accounts, salaries not paid on time, incorrectly charged overdraft fees, automated payments not happening etc. It can probably be argued what is "core" here and as far as I know no money actually disappeared into the ether but, in the UK at least, I am not sure I would be so confident in my declaration of Rock Solid software.
  - listenallyall 5y ago
    Thanks, these examples confirm my point. The old, ancient, "outdated" (as the original poster claimed) software worked fine for years/decades. The failures that you cited were introduced during an upgrade, a rewrite/outsourcing, or due to a 3rd party outage.
    - snovv_crash 5y ago
      But requirements change, so upgrades are a normal part of business, due to at the bare minimum legal reporting requirements changing, even if there weren't all sorts of new things like internet banking making their old APIs inadequate. If the software isn't extendable or changeable then it isn't 'working fine', because if you can't offer internet banking or do FATCA reporting you go out of business or get shut down.
- chefandy 5y ago
  You've cited two comparatively minor failures confined to a very small subset of what banks do. There's a whole lot of critical software systems in banks that don't touch personal bank accounts. If you've ever worked on bank systems before, it was certainly in a very limited capacity. I did code-level support for a large internal data management system for the federal reserve bank in the mid aughts. I'm not really interested in disclosing the specific bland but critically important data we moved/managed. It involved a giant amalgam of legacy commercial AIX and HP-UX software with some newer Java stuff in a Websphere environment. It was all duct-tape-and-bubble-gummed together with huge ksh88 scripts that were in constant development by devs and support people like me who'd just ssh into production systems to make changes. None of the internal software had dev/test systems, there was definitely no code review, and there was no version control. Yep. The system on a whole required constant monitoring and intervention to keep it plodding along. We did periodically see data loss that was unrecoverable from a technological standpoint but recovered with physical media kept around from their necessarily belt-and-suspenders business practices and tons of available staff for data entry. I reckon tens-of-thousands of folks outside the bank would have been affected per incident.
  - listenallyall 5y ago
    Money disappearing from a bank account is a "comparatively minor failure"? Sorry, I stopped reading at that point.
    - craftinator 5y ago
      There have been major bank tech failures too, if you wanted to see major vs minor side by side...
    - chefandy 5y ago
      If you're committed to not knowing something then I can't stop you. But compared to, say, tens of thousands of investment transactions for pension and retirement funds evaporating, disbursements to a hospital's operations budget being delayed, sending confidential historical balance records in bulk to a large corporate customer's competitor, having an online investment system down during a period of extreme market volatility, or even large-scale customer data breaches, then yes. 'Money missing from a bank account' is in every significant way a comparatively minor problem probably fixable with a phone call and a couple hours of investigation on the bank's part, even with a significant sum of money. The bank would almost certainly catch it themselves during an automated audit. It's exactly the sort of problem that bookkeeping organizations are designed to mitigate and root out if they do happen. The others could affect many, many more people's lives for much longer— potentially permanently.
jcims 5y ago
What do you mean by 'useful' here? Facebook goes down for half a day and everybody laughs. Take 'banks' down for the same amount of time and see what happens.
- jimmydorry 5y ago
  Multi-hour outages are already relatively common and getting more frequent by the year for banks. About 10 days ago there was a massive Bank of America outage.
- stickfigure 5y ago
  It's pretty standard for banks to "go down" outside the hours of 9am-5pm. Nearly everything meaningful that a bank does is processed in batch, often with days-long delays. Operationally, Facebook is a vastly harder problem.
- coliveira 5y ago
  You certainly don't use banks often, because their systems being offline for several hours is a relative common event.

ctdonath 5y ago

Not if they can’t find anyone willing to learn COBOL. There’s huge demand for developers on modern systems. Seriously, how much are you going to pay a developer to do mainframe COBOL - and is that developer going to produce commensurate value, really able to hold together a crumbling antique infrastructure?

_8j50 5y ago
Plenty of people working helpdesk that know a bit of python who can learn COBOL if they knew a job offer was on the other end of the investment in learning the language. There are hoards of people learning python and powershell just so they can make more money, trust me when I say they (the people I see learning python for career prospects) have no sentimental attachment to the language.
- noduerme 5y ago
  I don't think it's learning COBOL language that's the hard part. An inexperienced programmer who hasn't worked on financial systems is going to make security mistakes and write buggy code for awhile, both of which can be deadly to a bank. But it would be better to have that person rewrite the infrastructure than try to fix bugs. The problem with these ancient systems is trying to understand how bugs arise from subtle unintended interactions within millions of lines of code. I've been coding many languages for 25 years and I would consider it extremely daunting to try to tackle a bank bug knowing that one wrong assumption about one line of code could crash whole systems or cost millions of dollars.
  - mandelbrotwurst 5y ago
    Eh, someone's got to do it. Letting thing stagnate in perpetuity isn't cheap either. Move slowly and write tests.
    - noduerme 5y ago
      Sure, but a veteran coder at least understands that tampering with something may affect some other subsystem you don't even know exists until it breaks... which you wouldn't even think to test. The idea of throwing the phone staff into COBOL instead of them learning Python just obfuscates the real problem which isn't that the language is uncool and dying but that the systems are like the rules of the NFL. They stopped making sense a long time ago to anyone who wasn't immersed and paying attention.