7 ms·
How many millions in orders do they normally process per minute? Since there were no procedures in place, would you like to be the guy who pulled the plug on t
by MBCook 13y ago
How many millions in orders do they normally process per minute?
Since there were no procedures in place, would you like to be the guy who pulled the plug on the (let's guess) $100 million/minute processing system? Do you think you could get another job after that? What would the costs be for violating contracts? You could single handedly sink the company (which, in the end, this issue basically did).
I don't blame the guy for not killing all operations. He never should have been put in that situation. Proper QA, regression testing, monitoring, running a shadow copy and verifying it's output, there are tons of things that could have prevented/mitigated this.
- candydance 13y agoHave them in pairs or teams, war games style. Guy1 "Sir, we're losing money much faster than predicted, turn your key" Guy2 "It could just be market variance" Guy1 pulls out gun "Turn your key, sir"
- kops 13y agoDespite the gun brandishing, that conversation looks too polite to have ever taken place on a trading floor :-)
- paganel 13y ago> running a shadow copy Honest question, it's that really doable at this scale?
- babuskov 13y agoIt's doable. The real question is how much it would cost and does it make sense then.
- michaelt 13y agoIn this case the system was deploying new functionality, so your shadow couldn't just run last week's version or you'd get false alarms all the time. And obviously deploying the same broken version to the primary and the shadow wouldn't detect anything. So you'd need two codebases and two developer teams, coordinated enough that their code produced exactly the same output yet independent enough that they didn't make the same mistakes. With the challenges of coordination this would more than double your costs. Of course, with the benefit of hindsight, the costs might have been worthwhile...