12 ms·

Knight Capital lost $465 million in 45 minutes caused (at least in part) by technical debt and poor development practices. Summary: http://pythonsweetness.tumb

by pards 10y ago

Knight Capital lost $465 million in 45 minutes caused (at least in part) by technical debt and poor development practices. Summary: http://pythonsweetness.tumblr.com/post/64740079543/how-to-lose-172222-a-second-for-45-minutes http://pythonsweetness.tumblr.com/post/64740079543/how-to-lo...

robtaylor 10y ago
Wow. "The consequences of the failures were substantial. For the 212 incoming parent orders that were processed by the defective Power Peg code, SMARS sent millions of child orders, resulting in 4 million executions in 154 stocks for more than 397 million shares in approximately 45 minutes. Knight inadvertently assumed an approximately $3.5 billion net long position in 80 stocks and an approximately $3.15 billion net short position in 74 stocks. Ultimately, Knight realized a $460 million loss on these positions. " https://www.sec.gov/litigation/admin/2013/34-70694.pdf https://www.sec.gov/litigation/admin/2013/34-70694.pdf
dx034 10y ago
I'd say it wasn't due to technical debt, more a start-up like development approach to a company that trades millions within seconds in full automation. It sounds like the deployment process wasn't that complicated for a company of that size, but it was deployed without a single check by a second person. If you're trading automatically, you'll need a very, very solid deployment and audit process, even if you're just a small company. The reason banks are so slow in deploying software is because most of them lost a few millions at some point due to some bug. Startups that think they can act faster than banks just haven't had that bug yet. That's also why I'm rather negative on the whole Fintech scene at the moment.
- kafkaesq 10y ago
  Call it "process debt", or "management debt" (i.e. the lack of investment in proper management and the culture that goes a long with it -- in favor of a "STFU and just add that feature now! I need it yesterday!" mentality). Either way, part of the same boat, basically.
- beat 10y ago
  That's not a startup approach. That's an enterprise approach. Believe me, I've fought tons of resistance in automating deployment operations in the enterprise. There's a perception that automation is dangerous, and you need human checkpoints. In practice, I've worked on projects much, much larger than Knight Capital, where the deployment process was driven by a huge spreadsheet, and orchestrated by non-technical overseers telling techs what commands to run based on the spreadsheet in front of them. It's incredibly vulnerable to human error like "Oops, forgot to deploy to one of the eight servers in the cluster". In the enterprise, this is called "mature" and is a sign of great sophistication.
  - bpchaps 10y ago
    Yeah, the amount of resistance towards automation at some larger companies is a total mindfuck. My first ops job back in 2008 was at a large exchange's NOC where we shut down and clean the application environment every day. Every Friday, we would have to take a backup of the ~20 or so production databases - by hand, in an ancient CDE based UI with a . Right click -> menu -> submenu -> backup database. Very little room for error, and you weren't allowed to do it without somebody else watching you. Throughout the weekend, customers would then run tests against the production databases. Once testing was done, we'd restore the prod databases back to their original state to wipe out test data. At one point, I asked my boss if it was alright if I automated it after showing him a POC and was rejected because, "We don't trust automation to do it accurately every single time." Mind boggling. In mild fairness, in the 15 or so years they were doing that, I don't think anyone did it wrong.. which is an enormous miracle in itself. (That was a strange company. My boss was a JW who'd worked there for 30 years regularly tried to convert me and would spend four hours a day on spreadsheets for his church. We'd also manually kick off stock split processing from a ~10" CRT monitor from the early nineties.)
tboyd47 10y ago
> During the deployment of the new code, however, one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Was the issue technical debt or a sloppy deployment?
- mhluongo 10y ago
  Which was probably due to technical debt. I can't think of another reason you'd manually copy code to 8 servers...
throwaway2016a 10y ago
It is surprising there wasn't a circuit breaker here.
corecoder 10y ago
To me, the single line that stood out was: "The new RLP code also repurposed a flag". I've never seen a flag repurposed without catastrophic effects.
- pards 10y ago
  ^^^ That's the part of the disaster that I see as Technical Debt
- kevinpet 10y ago
  To be fair, the only time you hear about someone repurposing a flag is when it has catastrophic effects.

by pards 10y ago

robtaylor 10y ago

Wow. "The consequences of the failures were substantial. For the 212 incoming parent orders that were processed by the defective Power Peg code, SMARS sent millions of child orders, resulting in 4 million executions in 154 stocks for more than 397 million shares in approximately 45 minutes. Knight inadvertently assumed an approximately $3.5 billion net long position in 80 stocks and an approximately $3.15 billion net short position in 74 stocks. Ultimately, Knight realized a $460 million loss on these positions. " https://www.sec.gov/litigation/admin/2013/34-70694.pdf https://www.sec.gov/litigation/admin/2013/34-70694.pdf

dx034 10y ago

I'd say it wasn't due to technical debt, more a start-up like development approach to a company that trades millions within seconds in full automation. It sounds like the deployment process wasn't that complicated for a company of that size, but it was deployed without a single check by a second person. If you're trading automatically, you'll need a very, very solid deployment and audit process, even if you're just a small company. The reason banks are so slow in deploying software is because most of them lost a few millions at some point due to some bug. Startups that think they can act faster than banks just haven't had that bug yet. That's also why I'm rather negative on the whole Fintech scene at the moment.

kafkaesq 10y ago
Call it "process debt", or "management debt" (i.e. the lack of investment in proper management and the culture that goes a long with it -- in favor of a "STFU and just add that feature now! I need it yesterday!" mentality). Either way, part of the same boat, basically.
beat 10y ago
That's not a startup approach. That's an enterprise approach. Believe me, I've fought tons of resistance in automating deployment operations in the enterprise. There's a perception that automation is dangerous, and you need human checkpoints. In practice, I've worked on projects much, much larger than Knight Capital, where the deployment process was driven by a huge spreadsheet, and orchestrated by non-technical overseers telling techs what commands to run based on the spreadsheet in front of them. It's incredibly vulnerable to human error like "Oops, forgot to deploy to one of the eight servers in the cluster". In the enterprise, this is called "mature" and is a sign of great sophistication.
- bpchaps 10y ago
  Yeah, the amount of resistance towards automation at some larger companies is a total mindfuck. My first ops job back in 2008 was at a large exchange's NOC where we shut down and clean the application environment every day. Every Friday, we would have to take a backup of the ~20 or so production databases - by hand, in an ancient CDE based UI with a . Right click -> menu -> submenu -> backup database. Very little room for error, and you weren't allowed to do it without somebody else watching you. Throughout the weekend, customers would then run tests against the production databases. Once testing was done, we'd restore the prod databases back to their original state to wipe out test data. At one point, I asked my boss if it was alright if I automated it after showing him a POC and was rejected because, "We don't trust automation to do it accurately every single time." Mind boggling. In mild fairness, in the 15 or so years they were doing that, I don't think anyone did it wrong.. which is an enormous miracle in itself. (That was a strange company. My boss was a JW who'd worked there for 30 years regularly tried to convert me and would spend four hours a day on spreadsheets for his church. We'd also manually kick off stock split processing from a ~10" CRT monitor from the early nineties.)

tboyd47 10y ago

> During the deployment of the new code, however, one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Was the issue technical debt or a sloppy deployment?

mhluongo 10y ago
Which was probably due to technical debt. I can't think of another reason you'd manually copy code to 8 servers...

throwaway2016a 10y ago

It is surprising there wasn't a circuit breaker here.

corecoder 10y ago

To me, the single line that stood out was: "The new RLP code also repurposed a flag". I've never seen a flag repurposed without catastrophic effects.

pards 10y ago
^^^ That's the part of the disaster that I see as Technical Debt
kevinpet 10y ago
To be fair, the only time you hear about someone repurposing a flag is when it has catastrophic effects.