5 ms·

Yes, the deployment practices were bad, but they still would have had an issue even with proper practices. The real issue was re-using an old flag. That should

Yes, the deployment practices were bad, but they still would have had an issue even with proper practices. The real issue was re-using an old flag. That should have never been thought of or approved.

notnmeyer 3y ago
this is what stood out to me reading the story. i wonder if there was a reason why they opted for this, however half-baked. it reads less to me like a case for devops as it does a case for better practices at every stage of development. how arrogant or willfully ignorant do you have to be to operate like this considering what’s at stake?
- SoftTalker 3y ago
  They probably already had a bitfield of feature flags, maybe it was a 16-bit integer and full, and someone notices "hey this one is old, we can reuse it and not have to change the datatype"
  - notnmeyer 3y ago
    ah, yeah—hadn’t considered that!
Neil44 3y ago
I can only think that it was some kind of fixed binary blob of 1/0 flags where all the positions had been used umpteen times over the years and nobody wanted to mess with the system to replace it with something better.
amluto 3y ago
I would argue the real issue was the lack of an automated system (or multiple automated systems) that would hit the kill switch if the trading activity didn’t look right.
- distortionfield 3y ago
  But how would you even start to define something as stochastic as trading activity as “not looking right”?
  - mxz3000 3y ago
    spamming the market with orders for one
    - distortionfield 3y ago
      Define “spamming”, then? High frequency traders would probably look a lot like spammers. There are always two error rates.
  - Jorge1o1 3y ago
    I’ve had to fill out forms for new algorithms / quant strategies with questions like: - how many orders per minute do you expect to create? - how many orders per minute do you expect to cancel/amend? - what’s your max per-ticker position? - what’s your max strategy-level GMV/NMV? Etc. Any one of those questions can be used to set up killswitches. [edited for formatting]
    - distortionfield 3y ago
      Sure, but there is always the possibility that then you shut down trading when things _arent_ broken. There are always two error rates. Defining behavior is great for retrospective analysis but would you really feel comfortable putting hard cuts into production based on the answers to those questions? I’m genuinely asking, because IME I wouldn’t be.
      - amluto 3y ago
        That last nine in a trading system uptime has exponentially low value unless you have customers who care quite a lot. Seriously, suppose you have a truly awesome system making $100B per year of revenue. If you unnecessarily shut down 0.1% of the time, that’s only $100M per year lost, and an 0.1% unnecessary shutdown rate seems pretty high.
rwmj 3y ago
There's definitely more to this story. Why was there a fixed number of "flags" so that they needed to be reused? I wish there was a true technical explanation.
matkoniecz 3y ago
There are multiple real root issues here. Missing manual kill switch is also one of them.

Yes, the deployment practices were bad, but they still would have had an issue even with proper practices. The real issue was re-using an old flag. That should have never been thought of or approved.

notnmeyer 3y ago

this is what stood out to me reading the story. i wonder if there was a reason why they opted for this, however half-baked. it reads less to me like a case for devops as it does a case for better practices at every stage of development. how arrogant or willfully ignorant do you have to be to operate like this considering what’s at stake?

SoftTalker 3y ago
They probably already had a bitfield of feature flags, maybe it was a 16-bit integer and full, and someone notices "hey this one is old, we can reuse it and not have to change the datatype"
- notnmeyer 3y ago
  ah, yeah—hadn’t considered that!

Neil44 3y ago

I can only think that it was some kind of fixed binary blob of 1/0 flags where all the positions had been used umpteen times over the years and nobody wanted to mess with the system to replace it with something better.

amluto 3y ago

I would argue the real issue was the lack of an automated system (or multiple automated systems) that would hit the kill switch if the trading activity didn’t look right.

distortionfield 3y ago
But how would you even start to define something as stochastic as trading activity as “not looking right”?
- mxz3000 3y ago
  spamming the market with orders for one
  - distortionfield 3y ago
    Define “spamming”, then? High frequency traders would probably look a lot like spammers. There are always two error rates.
- Jorge1o1 3y ago
  I’ve had to fill out forms for new algorithms / quant strategies with questions like: - how many orders per minute do you expect to create? - how many orders per minute do you expect to cancel/amend? - what’s your max per-ticker position? - what’s your max strategy-level GMV/NMV? Etc. Any one of those questions can be used to set up killswitches. [edited for formatting]
  - distortionfield 3y ago
    Sure, but there is always the possibility that then you shut down trading when things _arent_ broken. There are always two error rates. Defining behavior is great for retrospective analysis but would you really feel comfortable putting hard cuts into production based on the answers to those questions? I’m genuinely asking, because IME I wouldn’t be.
    - amluto 3y ago
      That last nine in a trading system uptime has exponentially low value unless you have customers who care quite a lot. Seriously, suppose you have a truly awesome system making $100B per year of revenue. If you unnecessarily shut down 0.1% of the time, that’s only $100M per year lost, and an 0.1% unnecessary shutdown rate seems pretty high.

rwmj 3y ago

There's definitely more to this story. Why was there a fixed number of "flags" so that they needed to be reused? I wish there was a true technical explanation.

matkoniecz 3y ago

There are multiple real root issues here. Missing manual kill switch is also one of them.