4 ms·

Everytime I'm reading the story, there is one question that I've never understood: why can't the just shutdown the servers itself? There ought to be some mechan

by NhanH 12y ago

Everytime I'm reading the story, there is one question that I've never understood: why can't the just shutdown the servers itself? There ought to be some mechanism to do that. I mean, $400 millions is a lot of money to not just bash the server with a hammer. It seems like they realized the issue early on and was debugging for at least part of the 45 minutes. I know they might not have physical access to the server, but wouldn't there be any way to do a hard reboot?

jpatokal 12y ago
Violently shutting down trading also exposes the bank to significant risk, eg. massively leveraged trades on stocks that were meant to be held for 3 milliseconds suddenly hanging on until the system's back up. In hindsight, this would still have been preferable to losing $400 million, but quite obviously nobody at the time realized just how catastrophic this was going be.
- xyzzy123 12y ago
  With the benefit of hindsight I think company wide realtime dashboard would be high priority. I guess actual deployment procedures would be higher though.
  - geocar 12y ago
    They had a wire-protocol they were incrementally updating without a version-byte. That is caused by bad engineering, and they needed to do less of that. "Realtime dashboard" and "deployment procedures" are more engineering, not less.
  - zhte415 12y ago
    If the system was not counting its trades, would they show up in the dashboard? A reconciliation would be necessary, which would come from where the orders were being sent to (the exchange), but with millisecond delays a realtime dashboard seems only necessary for this case (not that it is a bad case) and while End Of Day reconciliations are needed, I'd be interested if anyone knew of exchange requirements for intra-day trading reconciliations?
jacques_chester 12y ago
In the story he points out that there was no kill switch. And, as has been found in other disasters in other industries, kill switches are hard to test.
- derefr 12y ago
  It feels more and more like the only responsible way to engineer systems is with a built-in always-on-in-production chaos monkey, to always be killing various parts of them. Normally this is done to ensure that random component failure results in no visible service interuption, but in this situation, you'd also be able to reuse the same "apoptosis" signal the chaos monkey sends to just kill everything at once.
  - narrator 12y ago
    Everything should be written crash-only[1]. That way they don't have to worry about pulling the plug at any time. http://en.wikipedia.org/wiki/Crash-only_software http://en.wikipedia.org/wiki/Crash-only_software
    - acveilleux 12y ago
      Crash-only is nice and all but you can't crash the other side of a socket... Like you couldn't crash a steel mill controller and expect the process equipment to be magically free of solidified metal. It only means the servers will come back up with a consistent state.
      - userbinator 12y ago
        At first I thought you had written "rocket" instead of "socket", which would also make much sense.
        dredmorbius 12y ago
        And as a general rule of thumb, what the other end of the rocket should be doing is pointing down.
Bluestrike2 12y ago
Given the types of transactions Knight was involved in, it's unlikely they had physical access as the servers would be in lower Manhattan to keep latency down. Couple that with the lack of any established procedures to kill their systems, and it's one hell of a nightmare. If the idea of just pulling the plug even came to them, I can't imagine how well that phone call would be received by the datacenter techs even if they believed it wasn't a hoax. And that's assuming that pulling the plug wouldn't cause other problems. But really, the 45 minutes probably flew by faster than you or I could really imagine. You're in a crisis situation, you tell yourself you just need another five minutes to fix something. Five becomes ten, becomes twenty, and before you know it, your company is looking at a $400M nightmare.
masonium 12y ago
As a designated market maker, there are probably regulatory requirements that force them to be in the market. Granted, if they'd known the full extent of the damage, they almost certainly would've pulled the plug. But, I'm guessing that was a factor.
easytiger 12y ago
Hedging.

by NhanH 12y ago

jpatokal 12y ago

Violently shutting down trading also exposes the bank to significant risk, eg. massively leveraged trades on stocks that were meant to be held for 3 milliseconds suddenly hanging on until the system's back up. In hindsight, this would still have been preferable to losing $400 million, but quite obviously nobody at the time realized just how catastrophic this was going be.

xyzzy123 12y ago
With the benefit of hindsight I think company wide realtime dashboard would be high priority. I guess actual deployment procedures would be higher though.
- geocar 12y ago
  They had a wire-protocol they were incrementally updating without a version-byte. That is caused by bad engineering, and they needed to do less of that. "Realtime dashboard" and "deployment procedures" are more engineering, not less.
- zhte415 12y ago
  If the system was not counting its trades, would they show up in the dashboard? A reconciliation would be necessary, which would come from where the orders were being sent to (the exchange), but with millisecond delays a realtime dashboard seems only necessary for this case (not that it is a bad case) and while End Of Day reconciliations are needed, I'd be interested if anyone knew of exchange requirements for intra-day trading reconciliations?

jacques_chester 12y ago

In the story he points out that there was no kill switch. And, as has been found in other disasters in other industries, kill switches are hard to test.

derefr 12y ago
It feels more and more like the only responsible way to engineer systems is with a built-in always-on-in-production chaos monkey, to always be killing various parts of them. Normally this is done to ensure that random component failure results in no visible service interuption, but in this situation, you'd also be able to reuse the same "apoptosis" signal the chaos monkey sends to just kill everything at once.
- narrator 12y ago
  Everything should be written crash-only[1]. That way they don't have to worry about pulling the plug at any time. http://en.wikipedia.org/wiki/Crash-only_software http://en.wikipedia.org/wiki/Crash-only_software
  - acveilleux 12y ago
    Crash-only is nice and all but you can't crash the other side of a socket... Like you couldn't crash a steel mill controller and expect the process equipment to be magically free of solidified metal. It only means the servers will come back up with a consistent state.
    - userbinator 12y ago
      At first I thought you had written "rocket" instead of "socket", which would also make much sense.
      - dredmorbius 12y ago
        And as a general rule of thumb, what the other end of the rocket should be doing is pointing down.

Bluestrike2 12y ago

Given the types of transactions Knight was involved in, it's unlikely they had physical access as the servers would be in lower Manhattan to keep latency down. Couple that with the lack of any established procedures to kill their systems, and it's one hell of a nightmare. If the idea of just pulling the plug even came to them, I can't imagine how well that phone call would be received by the datacenter techs even if they believed it wasn't a hoax. And that's assuming that pulling the plug wouldn't cause other problems. But really, the 45 minutes probably flew by faster than you or I could really imagine. You're in a crisis situation, you tell yourself you just need another five minutes to fix something. Five becomes ten, becomes twenty, and before you know it, your company is looking at a $400M nightmare.

masonium 12y ago

As a designated market maker, there are probably regulatory requirements that force them to be in the market. Granted, if they'd known the full extent of the damage, they almost certainly would've pulled the plug. But, I'm guessing that was a factor.

easytiger 12y ago

Hedging.