i worked in electronic trading for a few years. we weren't high frequency, though - it took us a few milliseconds to respond to price movements, which is why i

i worked in electronic trading for a few years. we weren't high frequency, though - it took us a few milliseconds to respond to price movements, which is why i left for the startup scene in 2010. exception-handling is something we took very seriously, because the consequeneces of a fuckup are so bad. in late 2008, our firm lost around 300k in ~2 seconds. one of the exchanges sent a signal that the interest rate was now 0, which, according to their spec, meant that something wasn't right. our system didnt' handle that properly, so the algos went nuts because hey money is free! the fuse system caught it a few seconds later, but not before we shot orders at everything in sight. i wrote a system that handled federation and service location, and as part of the design, i constructed four state machines which were used to handle the four possible cases in the formal model we developed. i went over the state machines many times by hand, and coded unit tests for each possible (State, Input) transition, followed by a battery of stress tests. these guys are good - but nobody's good enough to never make mistakes.