3 ms·

Please define "buggy software." Can you point to a single confirmed case in the US of a person's bank account balance disappearing, or even a deposit not appear

Please define "buggy software." Can you point to a single confirmed case in the US of a person's bank account balance disappearing, or even a deposit not appearing, due to a software bug? I doubt it. Banks' core software is rock solid or they would be out of business very, very quickly.

novok 5y ago
Transaction reversibility, fear of law enforcement and the ability to react to things is the reason why the bank tire fire doesn't collapse on itself.
- rrix2 5y ago
  im sure glad the cryptocurrency folks seem intent on throwing all three of those away at once.
scollet 5y ago
It is not, based on the loopholes one has to jump through for basic support.
manigandham 5y ago
Banks have procedures and regulations that protect customers from those issues. In fact there's a lot of of secondary checks and offline reconciliation precisely to catch potential software errors.
rurounijones 5y ago
Not the US, but in the UK which I think is comparable: * https://www.theguardian.com/money/2015/aug/28/many-hsbc-customers-facing-payday-without-pay https://www.theguardian.com/money/2015/aug/28/many-hsbc-cust... (I believe this one was caused by firing greybeards who understood a certain very complicated batch processing system and offshoring to undertrained staff in India who then make a mistake and basically brough the entire system down as a knock-on effect) * https://www.theguardian.com/business/2019/nov/19/tsb-it-meltdown-report-computer-failure-accounts https://www.theguardian.com/business/2019/nov/19/tsb-it-melt... (Insufficiently tested software upgrade although to be fair this was a major upgrade) * https://scottishfinancialnews.com/article/rbs-it-failure-prevents-customer-access-to-accounts https://scottishfinancialnews.com/article/rbs-it-failure-pre... (Not sure on root cause. Other interesting stats at the bottom of the article regarding failure counts) Now, in all these cases I believe the outages lasted a few days at least where people had limited or no access to bank accounts, salaries not paid on time, incorrectly charged overdraft fees, automated payments not happening etc. It can probably be argued what is "core" here and as far as I know no money actually disappeared into the ether but, in the UK at least, I am not sure I would be so confident in my declaration of Rock Solid software.
- listenallyall 5y ago
  Thanks, these examples confirm my point. The old, ancient, "outdated" (as the original poster claimed) software worked fine for years/decades. The failures that you cited were introduced during an upgrade, a rewrite/outsourcing, or due to a 3rd party outage.
  - snovv_crash 5y ago
    But requirements change, so upgrades are a normal part of business, due to at the bare minimum legal reporting requirements changing, even if there weren't all sorts of new things like internet banking making their old APIs inadequate. If the software isn't extendable or changeable then it isn't 'working fine', because if you can't offer internet banking or do FATCA reporting you go out of business or get shut down.
chefandy 5y ago
You've cited two comparatively minor failures confined to a very small subset of what banks do. There's a whole lot of critical software systems in banks that don't touch personal bank accounts. If you've ever worked on bank systems before, it was certainly in a very limited capacity. I did code-level support for a large internal data management system for the federal reserve bank in the mid aughts. I'm not really interested in disclosing the specific bland but critically important data we moved/managed. It involved a giant amalgam of legacy commercial AIX and HP-UX software with some newer Java stuff in a Websphere environment. It was all duct-tape-and-bubble-gummed together with huge ksh88 scripts that were in constant development by devs and support people like me who'd just ssh into production systems to make changes. None of the internal software had dev/test systems, there was definitely no code review, and there was no version control. Yep. The system on a whole required constant monitoring and intervention to keep it plodding along. We did periodically see data loss that was unrecoverable from a technological standpoint but recovered with physical media kept around from their necessarily belt-and-suspenders business practices and tons of available staff for data entry. I reckon tens-of-thousands of folks outside the bank would have been affected per incident.
- listenallyall 5y ago
  Money disappearing from a bank account is a "comparatively minor failure"? Sorry, I stopped reading at that point.
  - craftinator 5y ago
    There have been major bank tech failures too, if you wanted to see major vs minor side by side...
  - chefandy 5y ago
    If you're committed to not knowing something then I can't stop you. But compared to, say, tens of thousands of investment transactions for pension and retirement funds evaporating, disbursements to a hospital's operations budget being delayed, sending confidential historical balance records in bulk to a large corporate customer's competitor, having an online investment system down during a period of extreme market volatility, or even large-scale customer data breaches, then yes. 'Money missing from a bank account' is in every significant way a comparatively minor problem probably fixable with a phone call and a couple hours of investigation on the bank's part, even with a significant sum of money. The bank would almost certainly catch it themselves during an automated audit. It's exactly the sort of problem that bookkeeping organizations are designed to mitigate and root out if they do happen. The others could affect many, many more people's lives for much longer— potentially permanently.

novok 5y ago

Transaction reversibility, fear of law enforcement and the ability to react to things is the reason why the bank tire fire doesn't collapse on itself.

rrix2 5y ago
im sure glad the cryptocurrency folks seem intent on throwing all three of those away at once.

scollet 5y ago

It is not, based on the loopholes one has to jump through for basic support.

manigandham 5y ago

Banks have procedures and regulations that protect customers from those issues. In fact there's a lot of of secondary checks and offline reconciliation precisely to catch potential software errors.

rurounijones 5y ago

Not the US, but in the UK which I think is comparable: * https://www.theguardian.com/money/2015/aug/28/many-hsbc-customers-facing-payday-without-pay https://www.theguardian.com/money/2015/aug/28/many-hsbc-cust... (I believe this one was caused by firing greybeards who understood a certain very complicated batch processing system and offshoring to undertrained staff in India who then make a mistake and basically brough the entire system down as a knock-on effect) * https://www.theguardian.com/business/2019/nov/19/tsb-it-meltdown-report-computer-failure-accounts https://www.theguardian.com/business/2019/nov/19/tsb-it-melt... (Insufficiently tested software upgrade although to be fair this was a major upgrade) * https://scottishfinancialnews.com/article/rbs-it-failure-prevents-customer-access-to-accounts https://scottishfinancialnews.com/article/rbs-it-failure-pre... (Not sure on root cause. Other interesting stats at the bottom of the article regarding failure counts) Now, in all these cases I believe the outages lasted a few days at least where people had limited or no access to bank accounts, salaries not paid on time, incorrectly charged overdraft fees, automated payments not happening etc. It can probably be argued what is "core" here and as far as I know no money actually disappeared into the ether but, in the UK at least, I am not sure I would be so confident in my declaration of Rock Solid software.

listenallyall 5y ago
Thanks, these examples confirm my point. The old, ancient, "outdated" (as the original poster claimed) software worked fine for years/decades. The failures that you cited were introduced during an upgrade, a rewrite/outsourcing, or due to a 3rd party outage.
- snovv_crash 5y ago
  But requirements change, so upgrades are a normal part of business, due to at the bare minimum legal reporting requirements changing, even if there weren't all sorts of new things like internet banking making their old APIs inadequate. If the software isn't extendable or changeable then it isn't 'working fine', because if you can't offer internet banking or do FATCA reporting you go out of business or get shut down.

chefandy 5y ago

You've cited two comparatively minor failures confined to a very small subset of what banks do. There's a whole lot of critical software systems in banks that don't touch personal bank accounts. If you've ever worked on bank systems before, it was certainly in a very limited capacity. I did code-level support for a large internal data management system for the federal reserve bank in the mid aughts. I'm not really interested in disclosing the specific bland but critically important data we moved/managed. It involved a giant amalgam of legacy commercial AIX and HP-UX software with some newer Java stuff in a Websphere environment. It was all duct-tape-and-bubble-gummed together with huge ksh88 scripts that were in constant development by devs and support people like me who'd just ssh into production systems to make changes. None of the internal software had dev/test systems, there was definitely no code review, and there was no version control. Yep. The system on a whole required constant monitoring and intervention to keep it plodding along. We did periodically see data loss that was unrecoverable from a technological standpoint but recovered with physical media kept around from their necessarily belt-and-suspenders business practices and tons of available staff for data entry. I reckon tens-of-thousands of folks outside the bank would have been affected per incident.

listenallyall 5y ago
Money disappearing from a bank account is a "comparatively minor failure"? Sorry, I stopped reading at that point.
- craftinator 5y ago
  There have been major bank tech failures too, if you wanted to see major vs minor side by side...
- chefandy 5y ago
  If you're committed to not knowing something then I can't stop you. But compared to, say, tens of thousands of investment transactions for pension and retirement funds evaporating, disbursements to a hospital's operations budget being delayed, sending confidential historical balance records in bulk to a large corporate customer's competitor, having an online investment system down during a period of extreme market volatility, or even large-scale customer data breaches, then yes. 'Money missing from a bank account' is in every significant way a comparatively minor problem probably fixable with a phone call and a couple hours of investigation on the bank's part, even with a significant sum of money. The bank would almost certainly catch it themselves during an automated audit. It's exactly the sort of problem that bookkeeping organizations are designed to mitigate and root out if they do happen. The others could affect many, many more people's lives for much longer— potentially permanently.