14 ms·
Citigroup fined $7m after legit transactions mistaken for test data for 15 years
- twunde 10y agoI think the most impressive part is that apparently the same system and code had been running for 15 years as the number of transactions reported on had increased exponentially
- mherkender 10y agoI think it's the bad kind of impressive, one where ATMs still run COBOL.
- Jgrubb 10y agoi think the bad kind of impressive would be replacing a system that's had thousands of man-years of real world user testing without a really really really critical reason.
- wil421 10y agoI'd have to disagree on that part. The ATMs that the company I work for run Windows. We are the largest manufacturer, until the other 2 big ones finish merging. Besides, C is 44 years old and used on a ton of stuff.
- trentnelson 10y agoInteresting. Windows CE? Are there any glaring technical issues you've run into over the years? What processors are the ATMs using?
- eru 10y agoI guess these machines are not connected to the Internet, and have a very limited user interface surface. Flaws are less important.
- wil421 10y agoIn the US, we have private leased lines to connect with some of the major banks. But the banks most likely use something else to connect to the ATMs. Most attacks are physically breaking into the cash safe but there were some attacks a couple years ago where people were plugging into the USB port and getting money or something.
- breakingcups 10y agoI'm not GP, but if I had to guess it would be Windows Embedded, I know of a number of ATMs that use it.
- lorenzhs 10y agoI took this in the UK two years ago: https://4z2.de/atm_windows.jpg https://4z2.de/atm_windows.jpg - I don't know enough about how the various versions of Windows look but maybe this helps. It also doesn't look very trust-inspiring.
- wil421 10y agoYou need to trust the bank's network security more than you trust the ATM itself. There were some high profile breaches at some retailers in the past couple years that exploited some 0days. How can you defend your POS/ATM against a 0day if the retailer/bank has bad network security practices?
- colechristensen 10y agoWhy do you think it's necessary to constantly re-write code using the latest popular trends? Programmers reinvent the universe every few years, but really nobody else does.
- Twirrim 10y agoWhat advantages do you anticipate from a complete rewrite?
- PeCaN 10y agoThe bad kind of impressive would be getting running ATMs on NodeJS. ducks
- ryanlol 10y agoThis is a thing.
- AtheistOfFail 10y agoMay I introduce you to the "Lamassu Bitcoin ATM"?
- jonknee 10y agoThey did, it just left out a few transactions (26,810 transactions over 15 years is a very small number).
- nstart 10y agoGood point. Just under 5 transactions a day really. I think the point that's really annoyed the sec is the number of years they have to go back and correct now that they have the right data.
- trentnelson 10y agoAnd these systems clear around 1.5-2.5 million transactions a day at the moment.
- jonknee 10y agoThey didn't lose anything, they were FINED $7m for submitting incorrect reports. With the number of mergers that they have gone through it's not surprising they have trouble with company wide reporting.
- adamnemecek 10y agoEdited.
- josu 10y agoThis is the gist of it: "But in 1998, the company started using alphanumeric branch codes as it expanded its business. Among them were the codes 10B, 10C and so on, which the system treated as being within the excluded range, and so their transactions were removed from any reports sent to the SEC."
- jonknee 10y agoThe original title stated said "Citi lost $7m after legit transactions mistaken for test data". That was what the Register used too, but it's very misleading.
- deleted 10y ago[deleted]
- azurezyq 10y agoSeems to be something like "089" < x < "100", should never use "range" to handle string ids.
- jmtulloss 10y agoThere's probably a comment above it // XXX: too general, revisit when we have more than 100 branches
- bbcbasic 10y agoShould have written TODO
- smcl 10y agoI've seen the XXX suffix in the CPython source code - what does it mean compared to TODO?
- brianshaler 10y agoI would assume searchability. A non-case-sensitive non-regex search of a codebase could yet false positives for things like "autoDot"
- Sacho 10y agoArbitrary short and easy to remember string presumed to be unlikely to generate false positives when searched for. TODO can generate false positives if your system actually deals with todo items, etc.
- BallinBige 10y agowhat were these transactions for?
- mjcl 10y agoTrades (stock, etc) the bank and/or their clients made.
- andyjdavis 10y agoThere is mention of trading data which suggests to me that it might be to do with traders employed by Citigroup ie staff members whose job it is to use a block of the bank's money or outside investor's money to buy and sell shares, commodities and pretty much anything where they believe they can buy and sell to turn a profit.
- fredfoobar42 10y ago$7m is pocket change for Citigroup. No wonder the big banks keep shitting the bed. There's never any real consequences for their fuckups.
- rtpg 10y ago> Citigroup failing to send information on 26,810 transactions in over 2,300 such requests. 26810 requests, do you think they made more than $7m on this? It's about proportional fines. Just because they're a giant company doesn't mean we should find then $1b for forgetting to put a handicapped parking space at one of their offices.
- eli 10y agoThere is a school of thought that punitive fines should be a proportion of company value/earnings rather than an absolute dollar figure to exact an equal amount of discomfort. I think Finnish speeding fines are a percent of income.
- insickness 10y agoEven if fines were in proportion to company value/earnings, they would also need to be in proportion to the violation. Not every violation is equally as egregious.
- Guvante 10y agoToo punitive of a fine would result in Citigroup not reporting.
- ubernostrum 10y agoThere is a school of thought that punitive fines should be a proportion of company value/earnings rather than an absolute dollar figure to exact an equal amount of discomfort. In the infamous McDonald's coffee lawsuit, this was actually the motivation behind the initial large damage award. The jury attempted to award punitive damages equal to two days' worth of McDonald's coffee revenue. (obligatory note here for the many people who have heard false information about that case: the coffee spilled in a car, yes, but the car was motionless, in a parking space, and the person who spilled it was not the driver, and was found partially at fault for the spill; she suffered severe burns requiring hospitalization and skin grafts, which is not generally what one expects from coffee; it was found McDonald's served its coffee significantly hotter than other chains, in a temperature range making burns more likely, and was aware of the fact that it could cause severe burns because this wasn't the first case, and in fact McDonald's was aware of hundreds of cases of burns resulting from its coffee; the initial damage award was significantly reduced by the judge; search for Liebeck v. McDonald's for more details)
- jswny 10y agoImportant to note that they only left these transactions out of report data sent to the SEC. It's not like they were not honoring the transactions or that some people were missing money.
- bboreham 10y agoBut if those trades included insider trading or market manipulation then they were hidden from the regulators' investigations. Barings was brought down by trades in a special "error account" 88888 that was excluded from (different) reporting.
- argelius 10y agoThis makes me feel more comfortable with the reality that some broken code I wrote a long time ago might still be running somewhere... :)
- comp1927 10y ago//Try this in your console. function should_not_exclude_10B_10C(x) { if ("089" < x < "100") { console.log('excluding ' + x + ' in report.') } else { console.log(x + ' is normal.') } } should_not_exclude_10B_10C("10B") should_not_exclude_10B_10C("10C")
- rfurmani 10y agoActually, no: ascii 10B>100. The bug above is that you can't do chained inequalities like that in Javascript. Try: should_not_exclude_10B_10C("200")
- AWildDHHAppears 10y agoThese could be IBM Mainframes that are EBCDIC based.
- comp1927 10y agoB > 0 assumption is broken in their code. Read Page 4, footnote no.4: https://www.sec.gov/litigation/admin/2016/34-78291.pdf https://www.sec.gov/litigation/admin/2016/34-78291.pdf This was not caused by Ascii confusion...
- kazinator 10y agoSynopsis: SEC sends clear message to tech people in finance: shut the fuck up if you find something, silently fix it, and sweep the remaining crumbs under the rug, or else your company will be fined millions.
- AdeptusAquinas 10y agoIf they did that and the SEC found out, they probably wouldn't stop at just fining you. Which is probably what Citigroup guessed and why, after realising the extent of the issue, they reported it fully within a month
- DugFin 10y agoIndeed, Martha Stewart is a prime example of how NOT to deal with the SEC. Turns out she wasn't guilty of insider trading, but she still went to jail for lying to the SEC about the trades in the first place. They take that very seriously.
- stickfigure 10y agoThis particular example seems to reinforce the argument for saying nothing at all.
- deleted 10y ago[deleted]
- simonh 10y agoDirectors of companies have legal responsibilities to provide these reports to the government. So instead of the company getting a fine for making a mistake even though you tried to comply, now you are personally choosing not to fulfil your legal obligations as a director. Smart move.
- mattlutze 10y agoThe corporate veil is pretty tough, but that's one good way to give the government reason to pierce it.
- sf56 10y agoTotally unrelated, but I am not surprised. I once logged into my Citi credit card account and was granted access to another user's account. Certain places were off limits but I was able to view a lot of details. Pretty scary! I never heard back after reporting the issue.
- Sami_Lehtinen 10y agoDid you collect proper snapshot? Did you report it to the right official authorities?
- sf56 10y agoI did grab a few screens and contacted the only available "support" link I could find easily on their site, but after a few weeks with no response I eventually deleted the files.
- jsprogrammer 10y agoYawn. This company died eight years ago.
- ComodoHacker 10y agoPart of responsibility here is on SEC too IMO. When designing that "blueprint" format and data exchange protocols, they didn't implement mechanisms to verify correctness and completeness of data received.
- trequartista 10y agoI have done bluesheet reporting code for another large bank and it is one of the most tedious things you can ever do in software. It's a report of all the trades that the bank does and the SEC can come back and ask for historical data at any point of time. So there's a huge database with feeds coming in from multiple trading systems. Usually these feeds have to be enriched with the right account numbers and so on. This needs lookups to other reference data systems, which frequently are changing because of changing regulations, growing businesses etc. I am not surprised this bug remained undetected for more than 15 years. The guys who coded this initially are probably long gone and nobody did knowledge transition of the fact that some account ranges are not test accounts, even though they look like test accounts.
- gravypod 10y agoI've never understood how these fines are meant to benefit anyone. If no one is affected, then why is there a fine? Who is this money going to for damages to be repaid? Also, why are you not allowed to use real data for testing purposes?
- XorNot 10y agoLying to the SEC is a crime. "It was an accident" doesn't really cut it with financial institutions data handling and development practices.
- bagacrap 10y agoThe SEC was not impressed and said in a statement announcing the fine that the "failure to discover the coding error and to produce the missing data for many years potentially impacted numerous Commission investigations."
- critium 10y agoI suspect the bugs like this come about because of a patch not because of the original development. Devs and dev teams tend to get sloppy after the first push and budgets tend to shrink dramatically. One time I found a bug that was running for a few years and the result of it was the company was under reporting by millions of dollars per quarter (the running total was near to $100mm, and im sure it crossed it after my contract ended). This was VERY well tested software in the beginning (one of the best test suites i've seen actually) and audited up to high heaven. The problem started when the patches rolled in and those, are not tested anywhere near as much.