I was working on trading systems at Goldman in NYC at the time. After hours on that day, I got a call from my manager to come in early the next day to ensure o

I was working on trading systems at Goldman in NYC at the time. After hours on that day, I got a call from my manager to come in early the next day to ensure our kill switches worked properly, that our release and review processes were sufficient, and that our monitoring systems were sufficient. A few years later, I was working on trading systems at Goldman in Hong Kong. I sent a change out for review, went out for dinner and drinks with a colleague visiting from Tokyo, and swung by the office on my way home. My change had been approved by my NYC colleagues, so I merged it and went to bed. The next morning, I woke up to news that Goldman had a 100 million USD trading lost due to a software bug. Edit: This was in Goldman's Slang language, where source code is loaded from a globally-distributed eventually-consistent NoSQL DB. Most applications execute from read-only DB snapshots after extensive release testing. However, as soon as you merge your change, it's potentially instantly running in production somewhere in the world by some team you might not even know exists. It was possible my merge, maybe 45 minutes before the NYC market open, had gotten picked up by the errant system. I spent a while convincing myself that there was no way my change was the cause, and realized my phone would be ringing off the hook had my change been the cause. The guy who made the software change (let's call him Zaphod Beeblebrox since that's clearly not his name), and the guy who approved it, were both put on leave before I woke up. I found out who made the change only because I had an open chat window with Zaphod, and through several rounds of "fifth quartile" annual layoffs, knew how the chat and email systems responded when accounts got locked out. The chat system showed Zaphod's location as unknown, and a test email to him came back with the "mailbox full" message for a locked account. I walked over to the desk of one of the senior Equity Options Flow Strats in Hong Kong, and whispered "So... Zaphod Beeblebrox", and the Strat's face lit up and he whispered back "How did you know?", to which I responded "I didn't until I saw your reaction". The guy who mode the 100 million mistake was actually very very good at his job. He caught quite a few subtle bugs in other people's code that he wasn't even asked to review, but was reviewing out of curiosity. But, he was working late under time pressure, didn't test his change properly, and you only have to slip up once. As I remember, many of the trades were broken by the exchange, and the total loss came out to about 28 million USD. On the one hand, the guy didn't deserve to get fired, and I'd totally hire him for my team. On the other hand, if someone cuts corners and that results in tens of millions of USD in losses and doesn't get fired, that's very demotivating for everyone else at the firm. They did a very good job about not naming and shaming. After waking up to being momentarily scared I had made a 100 million USD error, I don't merge changes after-hours any more, and certainly never after having consumed any alcohol. If a guy like Zaphod can lose 28 million USD from a tired merge, so can I. Zaphod, if you're reading this and ever looking for a job, give me a ring.