4 ms·

I would argue the real issue was the lack of an automated system (or multiple automated systems) that would hit the kill switch if the trading activity didn’t l

by amluto 3y ago

I would argue the real issue was the lack of an automated system (or multiple automated systems) that would hit the kill switch if the trading activity didn’t look right.

distortionfield 3y ago
But how would you even start to define something as stochastic as trading activity as “not looking right”?
- mxz3000 3y ago
  spamming the market with orders for one
  - distortionfield 3y ago
    Define “spamming”, then? High frequency traders would probably look a lot like spammers. There are always two error rates.
- Jorge1o1 3y ago
  I’ve had to fill out forms for new algorithms / quant strategies with questions like: - how many orders per minute do you expect to create? - how many orders per minute do you expect to cancel/amend? - what’s your max per-ticker position? - what’s your max strategy-level GMV/NMV? Etc. Any one of those questions can be used to set up killswitches. [edited for formatting]
  - distortionfield 3y ago
    Sure, but there is always the possibility that then you shut down trading when things _arent_ broken. There are always two error rates. Defining behavior is great for retrospective analysis but would you really feel comfortable putting hard cuts into production based on the answers to those questions? I’m genuinely asking, because IME I wouldn’t be.
    - amluto 3y ago
      That last nine in a trading system uptime has exponentially low value unless you have customers who care quite a lot. Seriously, suppose you have a truly awesome system making $100B per year of revenue. If you unnecessarily shut down 0.1% of the time, that’s only $100M per year lost, and an 0.1% unnecessary shutdown rate seems pretty high.
      - distortionfield 3y ago
        > That last nine in a trading system uptime has exponentially low value IME that last 9 is where all the action happens > unless you have customers who care quite a lot All customers care about their trades. I’ve worked with these systems. You can’t treat smaller traders as less-than. > only $100M How far removed from the problem do you have to be to think one hundred million dollars is not going to effect anyone?
        Narkov 3y ago
        > How far removed from the problem do you have to be to think one hundred million dollars is not going to effect anyone? If $10b is at risk, $100m is not a lot for an insurance policy.
        amluto 3y ago
        Not all automated trading systems have customers.
    - leaflets2 3y ago
      A way to add limits when being clueless: Estimate what a real human can do in a day, and use that as the limits. Verify that the system behaves ok for some time, then scale up the desired trading volume and limits, observe, scale, repeat. But you don't do it by making a (bad) guess up front and then just leaving it at that.
- piyh 3y ago
  "If we lose 100 million dollars in 20 minutes" seems like a good one.
leaflets2 3y ago
Yes definitely, one has to assume that from time to time, bugs will reach the prod servers, no amount of tests and code review can completely prevent that. Hopefully the kill switch system is reasonably easy to code review and test :-)
joncrocks 3y ago
I think attributing blame to a single place can be difficult in complex situations. It's a set of failures in a system that all had to happen for the failure to occur. https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model

by amluto 3y ago

I would argue the real issue was the lack of an automated system (or multiple automated systems) that would hit the kill switch if the trading activity didn’t look right.

distortionfield 3y ago

But how would you even start to define something as stochastic as trading activity as “not looking right”?

mxz3000 3y ago
spamming the market with orders for one
- distortionfield 3y ago
  Define “spamming”, then? High frequency traders would probably look a lot like spammers. There are always two error rates.
Jorge1o1 3y ago
I’ve had to fill out forms for new algorithms / quant strategies with questions like: - how many orders per minute do you expect to create? - how many orders per minute do you expect to cancel/amend? - what’s your max per-ticker position? - what’s your max strategy-level GMV/NMV? Etc. Any one of those questions can be used to set up killswitches. [edited for formatting]
- distortionfield 3y ago
  Sure, but there is always the possibility that then you shut down trading when things _arent_ broken. There are always two error rates. Defining behavior is great for retrospective analysis but would you really feel comfortable putting hard cuts into production based on the answers to those questions? I’m genuinely asking, because IME I wouldn’t be.
  - amluto 3y ago
    That last nine in a trading system uptime has exponentially low value unless you have customers who care quite a lot. Seriously, suppose you have a truly awesome system making $100B per year of revenue. If you unnecessarily shut down 0.1% of the time, that’s only $100M per year lost, and an 0.1% unnecessary shutdown rate seems pretty high.
    - distortionfield 3y ago
      > That last nine in a trading system uptime has exponentially low value IME that last 9 is where all the action happens > unless you have customers who care quite a lot All customers care about their trades. I’ve worked with these systems. You can’t treat smaller traders as less-than. > only $100M How far removed from the problem do you have to be to think one hundred million dollars is not going to effect anyone?
      - Narkov 3y ago
        > How far removed from the problem do you have to be to think one hundred million dollars is not going to effect anyone? If $10b is at risk, $100m is not a lot for an insurance policy.
      - amluto 3y ago
        Not all automated trading systems have customers.
  - leaflets2 3y ago
    A way to add limits when being clueless: Estimate what a real human can do in a day, and use that as the limits. Verify that the system behaves ok for some time, then scale up the desired trading volume and limits, observe, scale, repeat. But you don't do it by making a (bad) guess up front and then just leaving it at that.
piyh 3y ago
"If we lose 100 million dollars in 20 minutes" seems like a good one.

leaflets2 3y ago

Yes definitely, one has to assume that from time to time, bugs will reach the prod servers, no amount of tests and code review can completely prevent that. Hopefully the kill switch system is reasonably easy to code review and test :-)

joncrocks 3y ago

I think attributing blame to a single place can be difficult in complex situations. It's a set of failures in a system that all had to happen for the failure to occur. https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model