5 ms·
An old coworker at a clearing firm once accidentally rebooted two very critical servers during a very critical time. From what I'm told, it ended up costing th
by bpchaps 8y ago
An old coworker at a clearing firm once accidentally rebooted two very critical servers during a very critical time. From what I'm told, it ended up costing the company about a billion dollars in fees and fines.
He managed to keep his job.. just with no more prod access. :)
- foota 8y agoWhat's a clearing?
- deleted 8y ago[deleted]
- koolba 8y agoClearing (and settlement) is the process of delivering securities and receiving the cash after a securities transaction (ex: stock purchase).
- gowld 8y agoOne imagines that if he were fired, he could have blown the whistle on the incompetent CIO/CTO who set up such a weak process.
- gitgud 8y agoA billion dollars in fines is an unfathomable mistake! It's on the level of a natural disaster, like a hurricane! Would love to hear more about that :)
- bpchaps 8y agoThat's all I know, really. It was one of the strangest companies I've ever worked at. Another story from the same place (won't confirm nor deny if it was the same sysadmin) - someone accidentally pushed a puppet config out that changed all hosts' timezones to CDT. A bit later, I received a random and kinda joking text from a buddy from a past job - who couldn't get an answer from my company's tech support - asking why all of their clearing reports had the wrong times. He was the only person outside of three other sysadmins who ever mentioned anything was wrong. Not even my boss brought it up! It was odd.
- hobls 8y agoSounds like someone else’s fault, unless he owned and designed the system. No single engineer should be able to cause that kind of damage. Quorum rules, etc.
- bpchaps 8y agoIt was a team of five senior level linux sysadmins who oversaw about 2k servers. Lots of arguments for and against pushing blame here or there, but at the end of the day - he fucked up big time and shoulda known better.
- hobls 8y agoI’m sure that’s true, but my point is that if you have that much money riding on a system you should have to figuratively (if not literally!) put two keys in and turn the lock at the same time to break shit. There should be systematically enforced mandatory reviews, two plus person policy for issuing commands, etc. You have to expect people to make mistakes. I’m not saying he didn’t fuck up, but if a company is down a billion dollars the story should be of multiple people making multiple mistakes.
- bpchaps 8y agoYou're preaching to the choir. There's a reason the company was bought and completely gutted. For what it's worth, a billion dollars for this company, while still a lot, isn't world ending. It was probably insured, too.