4 ms·

> A seemingly innocuous engineering change led to some drastic consequences. If you think that this is "drastic" you should read "Case Study 4: The $440 Milli

> A seemingly innocuous engineering change led to some drastic consequences. If you think that this is "drastic" you should read "Case Study 4: The $440 Million Software Error at Knight Capital". https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html https://www.henricodolfing.com/2019/06/project-failure-case-... tldr; At Knight, some new trading software contained a flaw that became apparent only after the software was activated when the New York Stock Exchange (NYSE) opened that day. The errant software sent Knight on a buying spree, snapping up 150 different stocks at a total cost of around $7 billion, all in the first hour of trading.

KMag 4y ago
I was developing trading systems at Goldman then. I remember that day. I got a call that evening to come in early the next day to discuss additional preventions and mitigations to implement. Scarier was the day I put out a change request shortly before the end of the day in Asia, went out to dinner and drinks with some colleagues, stopped by my office on the way home, and merged the approved change request, during early morning trading in the US. The next day (in Asia) I woke up to discover Goldman had a roughly 28 million dollar trading loss. I spent a couple of minutes proving to myself that my change couldn't possibly have been the cause, and then realized I would have been woken in the middle of the night by a phone call if there was any possibility that my change had caused the trading loss. Since then, I don't merge code at the end of the day, and try to avoid Fridays. I was already in the habit of not coding if I had consumed any alcohol, but added the habit of not merging if I've consumed any alcohol. There's nothing like a 28 million dollar panic to get your practices in shape.
cobertos 4y ago
I meant drastic for their use case. They put all this work into a piece of software specifically for streaming and it gets banned from the streaming platform. Definitely not life altering or a safety issue, but worst case for them, and I can imagine this going similarly wrong in a different piece of software where stakes actually matter

KMag 4y ago

I was developing trading systems at Goldman then. I remember that day. I got a call that evening to come in early the next day to discuss additional preventions and mitigations to implement. Scarier was the day I put out a change request shortly before the end of the day in Asia, went out to dinner and drinks with some colleagues, stopped by my office on the way home, and merged the approved change request, during early morning trading in the US. The next day (in Asia) I woke up to discover Goldman had a roughly 28 million dollar trading loss. I spent a couple of minutes proving to myself that my change couldn't possibly have been the cause, and then realized I would have been woken in the middle of the night by a phone call if there was any possibility that my change had caused the trading loss. Since then, I don't merge code at the end of the day, and try to avoid Fridays. I was already in the habit of not coding if I had consumed any alcohol, but added the habit of not merging if I've consumed any alcohol. There's nothing like a 28 million dollar panic to get your practices in shape.

cobertos 4y ago

I meant drastic for their use case. They put all this work into a piece of software specifically for streaming and it gets banned from the streaming platform. Definitely not life altering or a safety issue, but worst case for them, and I can imagine this going similarly wrong in a different piece of software where stakes actually matter