4 ms·

How Power Peg Brought Down a Knight

18 points by minimax 12y ago

PhantomGremlin 12y ago
This is a little long, but it's a must read for anyone interested in real-life consequences of computer bugs. Basically, new software was pushed to seven of eight servers at Knight, one of the biggest players on Wall Street. A flag activating the new software was repurposed from very old inactive software that was still on the servers. The new software ran fine on seven servers, but the flag activated the old software on the eighth server, which promptly started doing something completely insane as far as financial markets are concerned. A few minutes later people removed the new software. But didn't change the flag. So then: (the old software) jumped in to fill that void and was now running on all eight servers simultaneously, buying on the offer, selling on the bid. Knight lost over $400 million in under an hour, and was forced to eventually sell out to one of their biggest competitors.
- dclusin 12y ago
  45 minutes to be precise: http://www.sec.gov/litigation/admin/2013/34-70694.pdf http://www.sec.gov/litigation/admin/2013/34-70694.pdf
rmason 12y ago
A little surprising to me that when you're trading hundreds of millions of dollars an hour that the software wasn't built with a NASA style of redundancy built in that would have caught the mistake.
- kasey_junk 12y ago
  NASA style redundancy comes with a significant cost, that is delivery times. The opportunity costs of time to markets in electronic trading are huge. Some opportunities disappear in weeks (though these systems were not targeting those sorts of opportunities). Also, redundancies/qa are hard to quantify from a cost benefit perspective and trading firms are very short term P&L driven (pay is frequently heavily bonus based).
  - PhantomGremlin 12y ago
    > The opportunity costs of time to markets in electronic trading are huge. That's probably 90% of it right there. But perhaps the remaining 10% is that the "big swinging dicks" that run the brokerages have nothing but contempt for anyone who isn't a broker or a trader. IT is a cost center, nothing more. Even after the debacle, one of the executives said: “The IT guys were arrogant.” Well, maybe they were. And, maybe, because of the arrogant attitude the executives had, they couldn't attract any non-arrogant IT employees?

18 points by minimax 12y ago

This is a little long, but it's a must read for anyone interested in real-life consequences of computer bugs. Basically, new software was pushed to seven of eight servers at Knight, one of the biggest players on Wall Street. A flag activating the new software was repurposed from very old inactive software that was still on the servers. The new software ran fine on seven servers, but the flag activated the old software on the eighth server, which promptly started doing something completely insane as far as financial markets are concerned. A few minutes later people removed the new software. But didn't change the flag. So then: (the old software) jumped in to fill that void and was now running on all eight servers simultaneously, buying on the offer, selling on the bid. Knight lost over $400 million in under an hour, and was forced to eventually sell out to one of their biggest competitors.

dclusin 12y ago
45 minutes to be precise: http://www.sec.gov/litigation/admin/2013/34-70694.pdf http://www.sec.gov/litigation/admin/2013/34-70694.pdf

rmason 12y ago

A little surprising to me that when you're trading hundreds of millions of dollars an hour that the software wasn't built with a NASA style of redundancy built in that would have caught the mistake.

kasey_junk 12y ago
NASA style redundancy comes with a significant cost, that is delivery times. The opportunity costs of time to markets in electronic trading are huge. Some opportunities disappear in weeks (though these systems were not targeting those sorts of opportunities). Also, redundancies/qa are hard to quantify from a cost benefit perspective and trading firms are very short term P&L driven (pay is frequently heavily bonus based).
- PhantomGremlin 12y ago
  > The opportunity costs of time to markets in electronic trading are huge. That's probably 90% of it right there. But perhaps the remaining 10% is that the "big swinging dicks" that run the brokerages have nothing but contempt for anyone who isn't a broker or a trader. IT is a cost center, nothing more. Even after the debacle, one of the executives said: “The IT guys were arrogant.” Well, maybe they were. And, maybe, because of the arrogant attitude the executives had, they couldn't attract any non-arrogant IT employees?