7 ms·
Citi’s $900M Misfire Happened During Software Switch
- neonate 6y agohttps://archive.is/Wwicp https://archive.is/Wwicp
- jakub_g 6y agoThe title might suggest that the rollout of new software was the issue, but the article states the very contrary: it was the old software that was the culprit: > An internal review at the bank found humans manually operating the old software were ultimately at fault
- PaulAJ 6y ago> An internal review at the bank found humans manually operating the old software were ultimately at fault Which is of course an entirely bogus cop-out. If a mistake can be made in a manual operation then sooner or later it will be. Lower down the article says that manual checks that were supposed to catch this error failed to do so. Ineffective checks are a management responsibility, and that responsibility goes all the way up to the CEO.
- sharken 6y agoAt least Citibank is trying to upgrade their ancient systems, but it sure looks as if previous or current CEOs failed to exercise due diligence. Doesn't look like good risk management at all.
- HenryBemis 6y agoCiti is one of them banks that spend a lot of their $$$$$$ in IT. They jokingly say that they are an IT company with a banking license. Anyone related can please pitch in with a TA account. How bad/frequent are their Software Errors?
- leokennis 6y agoI mean...if this were any other business, someone clicking the wrong button and then someone else not catching this error would maybe mean someone's account was accidentally closed, or someone receiving a free pizza, or whatever. The IT systems within banks are more or less the same as IT systems anywhere. Just as advanced, just as crappy. The difference is that if there is a human error with banking software, you're not sending free pizza, you accidentally pay ouy $900,000,000.
- deleted 6y ago[deleted]
- tumetab1 6y agoActually, banks risk management is easy. Most in-bank or between banks transfers are reversible and usually a non-issue. That why the risk management probably says something like this: Risk: Incorrect transfer of funds to customer in another bank Mitigation: Manual review of all funds transfer above 5 million dollars Mitigation: Besides litigation issues, lost funds are easily recovered by asking the receiving bank Status: Risk accepted Edit: Clarified "Mitigation: Besides litigation issues bank transfers are reversible" into "Mitigation: Besides litigation issues, lost funds are easily recovered by asking the receiving bank"
- user5994461 6y agoBank transfers between banks are not reversible.
- netsharc 6y agoDo you mean when the mortal clients of the banks are doing them, or when the banks are doing them? I imagine bank CEOs know each other and can even call each other and say "Oh sorry old chap, that was a mistake!".
- pieno 6y agoMost bank transfers are actually not reversible, except for some limited retail client (including small companies) operations where specific terms & conditions allow the bank to reverse payments to the extent possible, which they really prefer avoid using as it looks really bad for a bank whose most important asset is the confidence of its clients and counterparties. Reversal may also no longer be possible if the money has already gone out in a system that does not allow reversal, or if the client is bankrupt in the meantime (depending on local banking and bankruptcy laws and circumstances). For any other payment system for larger sums / corporate and institutional parties, settlement finality is a huge thing that is the subject of all sorts of specific legislation, as it would be a real issue for the health of the financial system if a settled payment can simply be reversed, as it would have a lot of unintended consequences further down the line. So banks actually do have strict risk management policies to avoid wrong payments, but there are so many complex transactions for which ultimately a human (actually at least 2 due to 4-eyes principles) must confirm whether conditions for payment are satisfied and whether payment details are correct, and humans are always prone to making mistakes once in a while.
- LordAtlas 6y agoThis is the key part: "But the employee didn’t select the correct system options -- instead allowing the loan to be repaid in full with interest. Colleagues who are supposed to catch such errors didn’t." Saved you a click.
- HenryBemis 6y agoFrom experience in investigating mishaps like that: 1) no maker-checker control, 2) no imposed limits (with forced maker-checkers - more than one checker)($900m with one click???? what the actual ....), 3) lack of training, 4a) pressure to do this NOW NOW NOW NOW (sorry for the caps), 4b) overworked/tired (matching point 6 below), if that person is "stuck" at home with two screaming kids aged 2-6 for the past five months, I feel for them. 5) toxic environnment that did not allow the employee to spend 2 extra mins to think twice before clicking, 6) in these COVID times not having someone next to him/her and/or was too afraid to ping someone to ask "hey dude, just to make sure, am I using MenuOption1 or MenuOption2 for this almost $1b thingie?" (again, inadequate training & toxic env.)(easier to tap someone in the bag and ask them to look at your screen that get on a Lync call, share screen. Absolute controls in place would be limits & maker-checker. And this is the point, when I browse the "jobs" HN, I NEVER see any on audit/controls/GRC.. as if DevOps are the gods of everythinig and auditors are useless and not needed.. sigh I know there are other (better?) websites when it comes to looking for Audit/Sec work, but I feel that things like that should be taken care of in the development cycle, not the post-mortem of a mishap.
- toomuchtodo 6y agoWhile the jobs weren't on HN, Hashicorp has a very strong security/governance/compliance team based on my interview there. Credit where credit due.
- cosmie 6y ago> And this is the point, when I browse the "jobs" HN, I NEVER see any on audit/controls/GRC.. as if DevOps are the gods of everythinig and auditors are useless and not needed.. sigh The roles that get posted to HN are almost exclusively development related or development adjacent (such as PM roles). If you're not looking for those roles, it tends to not be very helpful directly. But if can be useful to look through, identify companies that appear to be doing interesting things, and then looking up their full job board to see _all_ of the roles they're hiring for.
- scott31 6y agoExpected, someone should have caught it in code review. Switch statements are generally harder to follow than if/else chain and fallthrough etc make it even more complicated.
- deleted 6y ago[deleted]
- testplzignore 6y agoI think you misread the article. This is about the Nintendo Switch. They accidentally bought 350 million copies of Pizza Bar Tycoon.
- jordache 6y agohow did you summarize this to be a coding issue?
- yesplorer 6y agoby not reading the article itself.
- londons_explore 6y agoI assume financial software has the concept of a set of atomic transactions - ie. "debit bob $X and credit mary $X". Given that, presumably all buttons an operator clicks should generate a set of atomic transactions between customers and the bank. An automated system can then check that the total loss to the bank after these transactions have been executed isn't too big. I can't really imagine how any bit of software didn't have those checks in place...
- yellowstuff 6y agoI don't fully understand what happened, but it sounds like essentially someone was supposed to update a payment from "repay full amount" to "pay interest only", and didn't do it. The system correctly executed the instructions it was given. The only way this would show up as a "loss" to the bank is if some part of the system expected to pay only the interest, and it sounds like that wasn't the case. It also sounds like there was another human who was supposed to catch the error and didn't. My guess is that the volume of transactions is high enough and errors of this type are rare enough that most people wouldn't be able to catch it manually.
- sukilot 6y agoThe transaction wasn't too big in any general sense. It was just incorrect for the specific intended action that day.
- noahmbarr 6y agoI can only imagine a guy like larry ellison reacting to this article.
- fooyc 6y agoThe title is misleading, the switch doesn’t appear to have anything to do with the misfire.
- wwright 6y agoYeah, but how many lay people care about causation in an RCA? ;)
- massaman 6y agoFrom the blameful-postmortem: Q. Why did we click `Send $900M?` A. Not sure. Tom felt all clicky-clicky, so he clicked it. Q. Why did we hire Tom? A. Also not sure. [Action Item: Fire Tom]
- massaman 6y agoSomewhere in your company hiring queue is a resume with: Strengths: Architecting risk management systems Weaknesses: Sometimes I click on things to see what happens
- sukilot 6y agoThose are both strengths. If Citi had that person they'd have learned from a lot of $1M messes before having a $900M mess.
- totaldude87 6y agoreminds me of that missile alerting system gif.. https://giphy.com/gifs/emibob-ads-missile-warning-system-xULW8wq0KtPBfXPRpC https://giphy.com/gifs/emibob-ads-missile-warning-system-xUL...
- kmarc 6y agoI'm put on a banking project (as external) which already flushed down the toilet around ~$500M. Based on my experiences of the meetings and meetings about meetings, I totally understand how the incompetence lead to this clusterfuck. My question to my boss was rather: "but _where_ do these banks get this huge amount of money from? I guess it's not from the $5 account fees." He answered that although he is in the banking business for decades, he still doesn't know. These 100s of Millions of losses are not necessarily threatening core business. I find it amusing.
- howeyc 6y ago> My question to my boss was rather: "but _where_ do these banks get this huge amount of money from? I guess it's not from the $5 account fees." They create it. https://www.bankofengland.co.uk/quarterly-bulletin/2014/q1/money-creation-in-the-modern-economy https://www.bankofengland.co.uk/quarterly-bulletin/2014/q1/m...
- retube 6y agoThis is a very misunderstood article. The money they "create", i.e loaned or paid out, has to be funded by a deposit or similar borrowing. Making sure they can fund all their commitments is what liquidity managers and treasury departments do, it's why regulators subject banks to annual ILAAPs (Internal Liquidity Adequacy Assessment Process), it's why banks have liquidity risk and modelling teams to manage any "gap" risk banks are running in this respect. If banks could simply create money then they'd never go bust. The only exception is the Central Bank, which can create new money that is it uses to buy assets of the same value, supporting prices and improving liquidity in the financial system.
- syndacks 6y agohttps://en.m.wikipedia.org/wiki/Fractional-reserve_banking https://en.m.wikipedia.org/wiki/Fractional-reserve_banking Banks do create money, by loaining other people's. I think you're saying the same thing?
- jordache 6y agostupid article. Reveals nothing of what contributed to the human error. They had been using the legacy software for many many years, w/o significant issues around human errors.
- brooklyndude 6y agoI worked at Citi for a very short time way back when. We were doing some things I thought were a bit “sketchy”, and was wondering if we were breaking the law. The response from my boss: we’re only breaking the law if we get caught, so theoretically we’re not actually breaking the law since no one has “caught us.” Guess there was some logic there. Of course this was a very long time ago. And sure they follow those pesky banking rules now , never, ever “breaking the law.”