Got another long story. This one wasn't me personally, but affected the team I was in at the time. The responsible party has moved on (left for another company)

Got another long story. This one wasn't me personally, but affected the team I was in at the time. The responsible party has moved on (left for another company). Background: I work for a company making business expense software - yes, it really is as dull as it sounds. I used ot be a SQL dev, now a DevOps engineer. At the time, I was still a dev. Our software does all manner of expense tracking, including requests and procurement. As a result, the customer's individual site is usually seeded with a dump of their HR data. The team I worked for managed the importers to deal with all sorts of weird home-grown HR files. One thing the company did right was building a very powerful generic import engine - for something built in-house, it's very flexible, easy to use and works with just about every format I've ever seen. We'd define the rules for transforming the data to our systems and let the import engine chew through it. Obviously this sort of thing lends itself to automation, and we built exactly that. The system defined clients, file drop locations and the import rules to use, then would check regularly for any files to import. One of our clients spotted a bug with their HR importer, so that got switched off in the config table while we worked through it, but the client continued to send us regular HR dumps via SFTP. Not a problem. Fast forward a couple of weeks. We have a new starter who's being trained on the stuff our team manages. The person doing the training, himself the previous most recent joiner to the team (something of a tradition, and hasn't resulted in too much Chinese Whispers) demonstrated the automation engine by creating a new job. In doing so, he manages to commit the dreaded forgotten-WHERE-clause boner. He accidentally switches every single automation job ON. By coincidence, the aforementioned client is alphabetically at the top of the list. And the bug with their importer is unfixed. Nonetheless, the importer starts up, spots 2 weeks' worth of HR dumps for this client, and gets to work. 6 hours later, I spot that clients further down the alphabet haven't had their automated runs start. At the time, the import engine ran single-threaded; I'd complained about it as inefficient, but it's probably this fact that let us realise what was going on. These are big HR dump files, so the importer is taking upwards of an hour per file to process them, but it's not moved past this job. We as a team piece together what's going on behind the scenes. Unfortunately, the client had been using the site at the time, and had run through some very large/expensive procurements, of which we as a company take a cut (our business model). The bug in the HR importer? It was incorrectly closing people's accounts and trashing their records. So while the client has been merrily been ordering expensive stuff on an unaffected account, the importer was inadvertently corrupting the rest. Now, we do have very thorough backups in progress - daily full backups and 15-minute incrementals for production. And they work. Unfortunately, this is where the CTO stepped in. Rather than just roll back the site to the point six hours ago before the import ran, he directed that we fix the HR data instead. The reasoning being, the requests linked out to third parties and couldn't be copied - new requests would have to be made, stopping us replaying the valid data in those 6 hours. Personally I don't know how accurate these claims are, as I wasn't directly involved in what happened next, but the net result was that the guy who committed the booboo, another of my colleagues and my immediate boss were given orders to fix the HR data, writing scripts by hand. It took 3 developers one and a half days to write the scripts to massage the HR data back together. I have to wonder if our cut of this client's bill actually exceeds the daily cost of 3 devs. And there was much frustration along the way, when they started using an internal-only dev site to test the scripts on, only for one of the support personnel to unknowingly overwrite it while troubleshooting an unrelated issue and nuking all their work for half a day. It was a total farce, done purely to save face with the client. Rather than rolling back to a known-good state, we spent a lot of time and effort trying to take a known-bad state back to a might-be-good one. Idiocy. And of course as a result I have to sit through meetings where my boss describes how we can't let developers have direct access to tables in case this happens again, so there's now a suite of stored procedures managing all the config and triggers on the table to stop any other queries. There were no repurcussions for the guy who made the mistake because it was a genuine mistake, but strikes me as a typical knee-jerk overreaction. It did affect me indirectly, because for that day and a half, I was the only one able to work tickets!!