4 ms·

Nice anecdote about Knight Capital: "When the market opened on Aug. 1, a new piece of trading software that Knight had just installed went haywire and started

Nice anecdote about Knight Capital: "When the market opened on Aug. 1, a new piece of trading software that Knight had just installed went haywire and started aggressively buying shares... By the end of Aug. 2, Knight had spent $440 million unwinding its trades" I'd love to know more about that particular launch. A hole in somebody's test suite!

themckman 13y ago
I think the story goes that something not ready for production made it into production. I obviously can't say anything about their test suite, but it sounds more like a mismanaged merge/deployment process.
- nissimk 13y ago
  There are some interesting links here: http://spectrum.ieee.org/riskfactor/computing/it/new-insights-into-knight-capital-trading-snafu http://spectrum.ieee.org/riskfactor/computing/it/new-insight... I had initially heard that the test program was released to production causing the system to take the wrong side of every trade, but I don't think it was ever confirmed.
joezydeco 13y ago
Nanex has lots of fun data on the Knight incident: "We believe Knight accidentally released the test software they used to verify that their new market making software functioned properly, into NYSE's live system." http://www.nanex.net/aqck2/3525.html http://www.nanex.net/aqck2/3525.html "We can fully appreciate the nightmare their team must have experienced that morning." And a whole boatload of amazing charts and data here: http://www.nanex.net/aqck2/3522.html http://www.nanex.net/aqck2/3522.html
- themckman 13y ago
  Nanex is a company that intrigues me immensely. I follow Scott (their founder, I think) on twitter and I wish I understood even a small percentage of what they produce on a daily basis. The stock market in general is something I wish I understood more of. Can anyone recommend some reading material?
  - minimax 13y ago
    Trading and Exchanges by Larry Harris is a great introduction to the structure of modern financial markets.
chollida1 13y ago
One of the prevailing theories from http://www.nanex.net/ http://www.nanex.net/ is that when Knight went live they also put their order simulator live as well. The thinking behind this theory was that a simulator would keep adding liquidity to a fake market for testing a market making algo. During the day that Knight lost their pants nanex saw that while their algo was taking liquidity someone else was constantly providing it, ie the bid ask spread didn't really open up that much and the price held relatively flat. This meant that someone else was constantly adding liquidity that Knight was taking away. The thinking was that Knights market simulator was the one sending out the live orders to add liquidity. TL/DR Knight crossed themselves many many times in a few minutes. See http://www.nanex.net/aqck2/3525.html http://www.nanex.net/aqck2/3525.html for the acutal details
- mcphilip 13y ago
  As an example of why Nanex is worth listening to, here is an article [1] describing how Nanex claimed Monday's ISM manufacturing data was leaked 15 milliseconds early, leading ultimately to an admission by Reuters that it caused the leak. Eric Hunsander, a software developer at Nanex, went so far as to hint to Reuters that they should check their NTP settings since a drift of 15 milliseconds is allowed by default [2]. Eric is very interesting to follow on Twitter [3]. He highlights mysterious HFT related events on a daily basis. My only complaint is that his articles and charts, in particular, are very difficult to interpret. As an outsider to the HFT industry, I don't know where to find the resources to learn enough to interpret most of Eric's insights. [1] http://www.cnbc.com/id/100792260 http://www.cnbc.com/id/100792260 [2] http://www.nanex.net/aqck2/4306.html http://www.nanex.net/aqck2/4306.html [3] https://twitter.com/nanexllc https://twitter.com/nanexllc
  - ansible 13y ago
    ... claimed Monday's ISM manufacturing data was leaked 15 milliseconds early ... While I'm peeved that as an adult I'm still not able to purchase an affordable flying car (though I was able to afford a tiny flying robot recently), there are days where I do think I am living in the science fictional future anticipated in my youth. Reading that CNBC article... this is one of those days.
anonu 13y ago
apparently there was a test component that took every order the model generated and copied the message 5 or 10 times - supposedly to stress test the system. Somehow this component was in production. Real orders were going out to the market and executions flowed back to an error account that was not being monitored. I imagine if you were monitoring the PnL you would have seen massive profits in the first few moments that the system went haywire - only to realize that you were simply just driving up the prices of your own positions...
- anonu 13y ago
  also - the changes were made on the same day NYSE's RLP program went live. so the guess is that these changes were made to handle RLP flow.
fmstephe 13y ago
From what I read about this crash the most important lesson was to have a kill switch. The biggest problem Knight Capital had was that they could not switch the system off quickly. It was roughly 45 minutes (according most accounts) between Knight realising something was wrong to ceasing trading. That there was a bug in some code isn't surprising. That it was so hard to stop their trading is really surprising.
- chollida1 13y ago
  In Knights defense they aren't just a high frequency hedge fund that trades their own book to make a profit. They have legitimate customers that they put orders out on behalf of and allow clients to trade through them. Part of the reason why it took so long to shut the system down is that if they had just shut the system down, then they'd be in breach of contract for those clients. So shutting down was more of a business decision that a technical one.
- brazzy 13y ago
  IIRC there were indications that they restarted the system twice in those 45 minutes, to no avail. Presumably it took that long to get someone with the authority to make a decision to switch it off entirely to decide to do so.
shrikant 13y ago
More details and related HN discussion here: https://news.ycombinator.com/item?id=4329101 https://news.ycombinator.com/item?id=4329101

themckman 13y ago

I think the story goes that something not ready for production made it into production. I obviously can't say anything about their test suite, but it sounds more like a mismanaged merge/deployment process.

nissimk 13y ago
There are some interesting links here: http://spectrum.ieee.org/riskfactor/computing/it/new-insights-into-knight-capital-trading-snafu http://spectrum.ieee.org/riskfactor/computing/it/new-insight... I had initially heard that the test program was released to production causing the system to take the wrong side of every trade, but I don't think it was ever confirmed.

joezydeco 13y ago

Nanex has lots of fun data on the Knight incident: "We believe Knight accidentally released the test software they used to verify that their new market making software functioned properly, into NYSE's live system." http://www.nanex.net/aqck2/3525.html http://www.nanex.net/aqck2/3525.html "We can fully appreciate the nightmare their team must have experienced that morning." And a whole boatload of amazing charts and data here: http://www.nanex.net/aqck2/3522.html http://www.nanex.net/aqck2/3522.html

themckman 13y ago
Nanex is a company that intrigues me immensely. I follow Scott (their founder, I think) on twitter and I wish I understood even a small percentage of what they produce on a daily basis. The stock market in general is something I wish I understood more of. Can anyone recommend some reading material?
- minimax 13y ago
  Trading and Exchanges by Larry Harris is a great introduction to the structure of modern financial markets.

chollida1 13y ago

One of the prevailing theories from http://www.nanex.net/ http://www.nanex.net/ is that when Knight went live they also put their order simulator live as well. The thinking behind this theory was that a simulator would keep adding liquidity to a fake market for testing a market making algo. During the day that Knight lost their pants nanex saw that while their algo was taking liquidity someone else was constantly providing it, ie the bid ask spread didn't really open up that much and the price held relatively flat. This meant that someone else was constantly adding liquidity that Knight was taking away. The thinking was that Knights market simulator was the one sending out the live orders to add liquidity. TL/DR Knight crossed themselves many many times in a few minutes. See http://www.nanex.net/aqck2/3525.html http://www.nanex.net/aqck2/3525.html for the acutal details

mcphilip 13y ago
As an example of why Nanex is worth listening to, here is an article [1] describing how Nanex claimed Monday's ISM manufacturing data was leaked 15 milliseconds early, leading ultimately to an admission by Reuters that it caused the leak. Eric Hunsander, a software developer at Nanex, went so far as to hint to Reuters that they should check their NTP settings since a drift of 15 milliseconds is allowed by default [2]. Eric is very interesting to follow on Twitter [3]. He highlights mysterious HFT related events on a daily basis. My only complaint is that his articles and charts, in particular, are very difficult to interpret. As an outsider to the HFT industry, I don't know where to find the resources to learn enough to interpret most of Eric's insights. [1] http://www.cnbc.com/id/100792260 http://www.cnbc.com/id/100792260 [2] http://www.nanex.net/aqck2/4306.html http://www.nanex.net/aqck2/4306.html [3] https://twitter.com/nanexllc https://twitter.com/nanexllc
- ansible 13y ago
  ... claimed Monday's ISM manufacturing data was leaked 15 milliseconds early ... While I'm peeved that as an adult I'm still not able to purchase an affordable flying car (though I was able to afford a tiny flying robot recently), there are days where I do think I am living in the science fictional future anticipated in my youth. Reading that CNBC article... this is one of those days.

anonu 13y ago

apparently there was a test component that took every order the model generated and copied the message 5 or 10 times - supposedly to stress test the system. Somehow this component was in production. Real orders were going out to the market and executions flowed back to an error account that was not being monitored. I imagine if you were monitoring the PnL you would have seen massive profits in the first few moments that the system went haywire - only to realize that you were simply just driving up the prices of your own positions...

anonu 13y ago
also - the changes were made on the same day NYSE's RLP program went live. so the guess is that these changes were made to handle RLP flow.

fmstephe 13y ago

From what I read about this crash the most important lesson was to have a kill switch. The biggest problem Knight Capital had was that they could not switch the system off quickly. It was roughly 45 minutes (according most accounts) between Knight realising something was wrong to ceasing trading. That there was a bug in some code isn't surprising. That it was so hard to stop their trading is really surprising.

chollida1 13y ago
In Knights defense they aren't just a high frequency hedge fund that trades their own book to make a profit. They have legitimate customers that they put orders out on behalf of and allow clients to trade through them. Part of the reason why it took so long to shut the system down is that if they had just shut the system down, then they'd be in breach of contract for those clients. So shutting down was more of a business decision that a technical one.
brazzy 13y ago
IIRC there were indications that they restarted the system twice in those 45 minutes, to no avail. Presumably it took that long to get someone with the authority to make a decision to switch it off entirely to decide to do so.

shrikant 13y ago

More details and related HN discussion here: https://news.ycombinator.com/item?id=4329101 https://news.ycombinator.com/item?id=4329101