4 ms·

Is there anything good written about failsafe programming? I try to code defensively so terrible bugs like this can't snowball. A heuristic like "only make 1000

Is there anything good written about failsafe programming? I try to code defensively so terrible bugs like this can't snowball. A heuristic like "only make 1000 trades a minute" could have saved Knight $400 million! I try to build global failsafes into my code, particularly network code. For instance, a recent Twitter project I wrote has various loops calling their API. That code has a global counter that says "this program can only ever make 25 calls" that's enforced in the bottom level HTTP calling function. If the rest of my code is correct I'll never trip that limit, I have application-specific logic that means I should only ever need to make 10 calls. But I appreciate having the failsafe in case that more complex logic fails.

fr0sty 14y ago
> A heuristic like "only make 1000 trades a minute" could have saved Knight $400 million! A heuristic like that would not allow Knight to function. Knight is an electronic market-maker (one of the biggest) and is probably making markets in thousands of symbols and likely updates those quotes several times a second. Their normal quoting rate is on the order of millions of messages/minute and in volatile times may spike 10x or more and still be within 'normal' limits.
- justincormack 14y ago
  A heuristic like "do not lose more than $100m a day" would have saved the company.
  - veyron 14y ago
    PNL limits are required risk checks. Clearly Knight didn't implement them ...
    - harryh 14y ago
      More likely they thought they had implemented them but they were implemented incorrectly.
    - beagle3 14y ago
      Stop-losses are not guarantees of execution. If you're in an illiquid market, or one that moves faster than you're prepared for, then your PNL check may alert you, but it won't save you from the loss.
- NelsonMinar 14y ago
  Use your imagination to come up with a more appropriate heuristic.
javert 14y ago
Look up Jane Street Capital. They use functional programming with a very strict/smart type system (Ocaml) to help make sure that, for example, there are no logically possible situations in the code that they haven't programmed a response for. They have some pretty good eductional material online.
- create_account 14y ago
  "there are no logically possible situations in the code that they haven't programmed a response for" Until they blow up one day themselves... Call me cynical, but no type system, no matter how sophisticated, nor team of programmers, no matter how talented & experienced, is immune to this type of thing.
  - kd0amg 14y ago
    Sure, it doesn't prevent them from doing the wrong thing in that situation, but it does require that they have given the situation enough attention to write code for it, which is a serious improvement over not handling it at all. As I understand it, they play defense in depth with regard to code quality.
  - javert 14y ago
    As I said, it helps make sure that doesn't happen. Anyway, I don't think they'll blow themselves up big, like Dark Knight Capital did. First, this specific kind of thing doesn't happen often (where one firm just messed up so much that it makes world news). Second, it's clear from Jane Street's videos that they're pretty serious about quality. If I were going to bet on which market maker will screw up, it would be somebody else.
- beagle3 14y ago
  It's only as good as the specs (APIs, conventions, contracts) they are working against. In 2003, I was using a trade API that gave me confirmation about canceling an order at 10am. And then executed the same order at 1pm; Apparently, even though it did confirm cancellation, the order wasn't canceled. No matter how well your code is equipped against that - it's a legal problem, not a technical one. Similarly, when the exchange retroactively cancels ("busts") trades after a flash crash, with a rather arbitrary rule (e.g. all trades between 3:45pm and 3:54pm in AAPL with price < $500) - you're left holding the bag, regardless of how good your code is. The only winning moves in these cases is not to play - unfortunately, you can only tell retroactively.

fr0sty 14y ago

> A heuristic like "only make 1000 trades a minute" could have saved Knight $400 million! A heuristic like that would not allow Knight to function. Knight is an electronic market-maker (one of the biggest) and is probably making markets in thousands of symbols and likely updates those quotes several times a second. Their normal quoting rate is on the order of millions of messages/minute and in volatile times may spike 10x or more and still be within 'normal' limits.

justincormack 14y ago
A heuristic like "do not lose more than $100m a day" would have saved the company.
- veyron 14y ago
  PNL limits are required risk checks. Clearly Knight didn't implement them ...
  - harryh 14y ago
    More likely they thought they had implemented them but they were implemented incorrectly.
  - beagle3 14y ago
    Stop-losses are not guarantees of execution. If you're in an illiquid market, or one that moves faster than you're prepared for, then your PNL check may alert you, but it won't save you from the loss.
NelsonMinar 14y ago
Use your imagination to come up with a more appropriate heuristic.

javert 14y ago

Look up Jane Street Capital. They use functional programming with a very strict/smart type system (Ocaml) to help make sure that, for example, there are no logically possible situations in the code that they haven't programmed a response for. They have some pretty good eductional material online.

create_account 14y ago
"there are no logically possible situations in the code that they haven't programmed a response for" Until they blow up one day themselves... Call me cynical, but no type system, no matter how sophisticated, nor team of programmers, no matter how talented & experienced, is immune to this type of thing.
- kd0amg 14y ago
  Sure, it doesn't prevent them from doing the wrong thing in that situation, but it does require that they have given the situation enough attention to write code for it, which is a serious improvement over not handling it at all. As I understand it, they play defense in depth with regard to code quality.
- javert 14y ago
  As I said, it helps make sure that doesn't happen. Anyway, I don't think they'll blow themselves up big, like Dark Knight Capital did. First, this specific kind of thing doesn't happen often (where one firm just messed up so much that it makes world news). Second, it's clear from Jane Street's videos that they're pretty serious about quality. If I were going to bet on which market maker will screw up, it would be somebody else.
beagle3 14y ago
It's only as good as the specs (APIs, conventions, contracts) they are working against. In 2003, I was using a trade API that gave me confirmation about canceling an order at 10am. And then executed the same order at 1pm; Apparently, even though it did confirm cancellation, the order wasn't canceled. No matter how well your code is equipped against that - it's a legal problem, not a technical one. Similarly, when the exchange retroactively cancels ("busts") trades after a flash crash, with a rather arbitrary rule (e.g. all trades between 3:45pm and 3:54pm in AAPL with price < $500) - you're left holding the bag, regardless of how good your code is. The only winning moves in these cases is not to play - unfortunately, you can only tell retroactively.