7 ms·
Knight Capital Says Trading Glitch Cost It $440 Million
- brianbreslin 14y agoMy initial skepticism makes me want to think this was maliciously orchestrated. Where one firm stands to lose, someone else would undoubtedly gain.
- fr0sty 14y agoInteresting theory, but there is no implicit parity between # of winners and # of losers. Whereas 1 firm lost $440M there were thousands (maybe tens of thousands) of firms and individuals that made hundreds or thousands of dollars.
- jonknee 14y agoUnless someone comes in and scoops up Knight for cheap--the stock is trading down 65% or so over night.
- justincormack 14y agoThey are out of cash apparently. Likely to be sold or file for bankruptcy shortly it seems.
- sp332 14y agoDark Knight Rises spoiler alert: Isn't it really odd that a company called "Knight" makes all these nonsensical trades just a couple weeks after the movie where a similar thing bankrupted a company?
- salman89 14y agoI don't understand how it went on for 45 minutes without human intervention. Do they really not have a live person at the very least monitoring trades?
- amykhar 14y agoThank heaven I didn't write that code.
- rabidsnail 14y agoOr deploy it. Imagine being the last one in the push queue that day.
- AznHisoka 14y agoon the other hand, it's a good story to tell your grand kids one day...
- jagermo 14y agoI'm not sure, if it was a technical failure, but here is a great talk from Def Con 19 about security and high trading systems https://www.youtube.com/watch?v=ncpm6vRi9F4 https://www.youtube.com/watch?v=ncpm6vRi9F4
- NelsonMinar 14y agoIs there anything good written about failsafe programming? I try to code defensively so terrible bugs like this can't snowball. A heuristic like "only make 1000 trades a minute" could have saved Knight $400 million! I try to build global failsafes into my code, particularly network code. For instance, a recent Twitter project I wrote has various loops calling their API. That code has a global counter that says "this program can only ever make 25 calls" that's enforced in the bottom level HTTP calling function. If the rest of my code is correct I'll never trip that limit, I have application-specific logic that means I should only ever need to make 10 calls. But I appreciate having the failsafe in case that more complex logic fails.
- fr0sty 14y ago> A heuristic like "only make 1000 trades a minute" could have saved Knight $400 million! A heuristic like that would not allow Knight to function. Knight is an electronic market-maker (one of the biggest) and is probably making markets in thousands of symbols and likely updates those quotes several times a second. Their normal quoting rate is on the order of millions of messages/minute and in volatile times may spike 10x or more and still be within 'normal' limits.
- justincormack 14y agoA heuristic like "do not lose more than $100m a day" would have saved the company.
- pmb 14y agoThese sorts of problems will occur in any distributed system which runs at a high rate without humans in the loop. There is no chaos monkey for the stock market, and transaction rollback is available only in the most extreme circumstances. I have yet to see a convincing rationale for how high frequency trading adds value to the system - it certainly doesn't seem to add pricing stability. Because this is hacker news, I suspect a few people reading this either work in the industry and/or have strong opinions about it, and I would love to hear why I am wrong. Because I do like it when clever computer scientists make money, and that seems to be pretty much the only social benefit of HFT, at the cost of flash crashes and things like the example above.
- fr0sty 14y agoThe 'Does HFT add value?' question gets flogged to death every time one of these articles gets posted. http://news.ycombinator.com/item?id=3894302 http://news.ycombinator.com/item?id=3894302 http://news.ycombinator.com/item?id=3852341 http://news.ycombinator.com/item?id=3852341 http://news.ycombinator.com/item?id=2828538 http://news.ycombinator.com/item?id=2828538
- patio11 14y agoThese sorts of problems will occur in any distributed system which runs at a high rate without humans in the loop. This implies that humans are actually better than machines at making data-driven decisions at high rates. They're not. They're astoundingly not once you compute the cost of humans versus the cost of machines on a per-decision basis. Liquidity in the market used to be provided by large groups of sweaty, overpaid alpha males yelling at each other. We replaced them with dueling robots. The robots can provide liquidity for a fraction of the price. The alpha males really, really hate competing with robots, because the alpha males invariably lose, so they complain that robots are stealing the money that the alpha males used to extract from their customers by right of being the one with a license to be shouting and sweaty at a particular physical location. Occasionally a robot blows up. Not a problem -- robots are easy to replace. Besides, humans blow up all the time. We only ignore their ridiculous strictly-inferior-in-every-way-at-this-task nature because they look more like us than the robots do, and because these particular humans being displaced used to be rich, whereas e.g. telephone operators tasked with manually doing call routing (also clearly inferior to highly reliable distributed algorithms) were poor.
- codegeek 14y agoI wonder whats going on with the IT executives/CIOs of Knight at this point . They must be hanging by a thread?
- debacle 14y agoThey're probably very close to hanging from a thick rope.
- rcavezza 14y ago"Trading Glitch" = a very good excuse for very bad trades. I'm not implying this happened, but if I was an analyst at Knight Capital and we just lost $440 million, I'd blame it on a "trading glitch".
- eli 14y agoWhy is admitting you have buggy trading software better than admitting you have traders who made bad trades?
- fr0sty 14y agoGiven that there was suspicious activity in ~150 names of ~45 minutes "trading glitch" is much more plausible than "bad analysis".
- incision 14y agoYep. They're calling it a "glitch" to suggest something unavoidable, a problem that could have happened to anyone. It's same vein of bullshit we heard in 2008 "We couldn't possibly have known!" and more recently with the string of "rogue traders". In every case, its just a bunch of folks who don't want to take responsibility for their gambles when they lose.
- jwoah12 14y agoYou are completely wrong. Try looking into the circumstances before you make some unfounded accusation about what caused this. Comparing a 45 minute isolated incident at a firm with an otherwise-excellent reputation to the housing crisis or rogue traders is moronic.
- incision 14y agoI love it. Looks like someone's HFT system backfired. I'm looking forward to the Nanex analysis of this one. Nanex on the 2010/5/6 'Flash Crash': http://www.nanex.net/FlashCrashFinal/FlashCrashAnalysis_Theory.html http://www.nanex.net/FlashCrashFinal/FlashCrashAnalysis_Theo...
- joezydeco 14y agoSecond that. Nanex has done some great work. To date, afaik, they're the only ones to try and figure out the Facebook IPO trainwreck from the technical side. And it's a doozy. http://www.nanex.net/aqck/3099.html http://www.nanex.net/aqck/3099.html
- fr0sty 14y ago> I'm looking forward to the Nanex analysis of this one Already posted: http://www.nanex.net/aqck2/3522.html http://www.nanex.net/aqck2/3522.html
- joezydeco 14y agoWow, you can almost see the bug happening as you read this. Amazing stuff.
- samstave 14y ago$chadenfreude
- itsprofitbaron 14y agoIt seems that Knight Capital's ended up buying at the offer and selling at the bid. In order words, Knight Capital essentially collected the opposite of a rebate by paying someone for no reason aside from attempting to create liquidity in the market which resulted in these losses. As a result, they were pushing stocks higher and higher which causes a huge incentive for a system like Knight Capital have. Again, this explains not only some volume surge which was seen on the market but also massive moves in certain stocks like China Cord Blood Corporation (CO) [1] who rose several hundred percent, until someone finally stepped in & the important thing here to note that there isn't a reason under the current SEC order cancellation methodology to bail out Knight Capital and its algorithm. Similarly, they were trading Exelon Corporation (EXC) [2] and were losing $0.15 on every single trade and when its being done 2.4k times per minute, its very easy to lose money. [1] http://finance.yahoo.com/q?s=co&ql=1 http://finance.yahoo.com/q?s=co&ql=1 [2] http://finance.yahoo.com/q?s=EXC&ql=0 http://finance.yahoo.com/q?s=EXC&ql=0
- samstave 14y agoWTH is "China Cord Blood Corp" - they collect cord blood for stem cells I suppose - id be very wary of any such product like that from china, both for quality and humanitarian reasons. Take a look at the Vice documentary on Tiger part harvesting. China states that tigers are endangered, yet there are these secret farms that breed thousands of them and the make tiger dick wine, all sorts of things with their other parts as well. I wouldn't think any biological products being produced in china on the level.
- bhurt 14y agoI think this talk is relevant: https://ocaml.janestreet.com/?q=node/61 https://ocaml.janestreet.com/?q=node/61
- create_account 14y agoWhy? You might be far less likely to blow up the way Knight did, but you're hardly immune.
- sseveran 14y agoIt is reasonable to use technology to make the universe of potential errors significantly smaller.
- jberryman 14y agoeasy come, easy go.
- dhyasama 14y agoMy first job out of college was at a small HFT firm. The night before rolling out my first changes is forever imprinted in my memory. Lots of tossing and turning and images of exactly this kind of disaster.
- danso 14y agoHow did its testing/security practices measure up to other places you've worked? I understand that the tech division of a financial firms might be top notch, but the culture of the moneymaking dept. may mismanage them.
- veyron 14y agoSEC/FINRA have required practices. The risk controls have to be firewalled from the trading desks precisely so that this type of blowup is contained.
- dhyasama 14y agoIs that new? Back then it was two devs and two traders in a room. See a problem? Press the big red button. Deploy change. Press the big green button. Cowboy style. I STRONGLY recommend against doing it this way. I was young and stupid and I'm pretty sure it took years off my life.
- veyron 14y agoThe issue regards naked access. At that time the broker had requirements. Now that two-dev shop has to worry about risk etc.
- veyron 14y agoThis is incredibly strange. You get back an acknowledgement with each fill. There are risk checks in place that should check whether the order was a take or add liquidity fill (as per rule 15c3-5 amongst others) This is a risk check that should have been handled by a component which appears not to exist. They could lose FINRA and SEC authorization if it is as bad as it sounds ...
- eps 14y agoI'd love to see what the patch looked like.
- swalsh 14y agoIs it possible to purchase a ridiculously expensive insurance policy protecting yourself against the risk of dangerous bugs like this?
- justincormack 14y agoNo. Buying the insurance would probably increase the risk as checks would get more lax. And you can't hedge it.
- manishsharan 14y agoI am highly skeptical of this being a software glitch and here is why: I have been working for banks' brokerage and capital markets groups for years and the amount of checks and reviews processes that they have put in place is just incredible. One does not simply deploy a new software into production without there being several small pilot runs, staging tests etc. etc. There are committees with onerous questionnaires and test results compiled by QA that need to be documented and approved before a new deployment is approved. Therefore I don't see this being a software issue. If the checks were not in place then it seems that the management was lax in safeguarding its capital.
- romey 14y agoI've always wondered, what happens to the actual programmer[s] responsible for a bug like this? Assuming they find the exact issue, and then go back through the version control logs to the exact commit, how responsible, if at all, is the individual programmer? Or the QA team, for that matter. Incidents like this make me think that I'd have a panic attack with every commit, if I worked in the financial industry.