3 ms·
I've marked a few tens of thousands of records as not delivered to a customer, instead of as intended just one record. We had no development or test environmen
by Zobat 3y ago
I've marked a few tens of thousands of records as not delivered to a customer, instead of as intended just one record.
We had no development or test environment so development were done in production. The system lacked the necessary functions to correct errors so there were frequent updates/corrections made by writing SQL-queries directly towards the production database. The incident was when I should update a single record so the system would resend it to the customer, but ran the query before writing the WHERE-clause.
Luckily there were working backups (incremental backups that ran every 15 min, almost like someone knew they might need them!) and no one had made any changes since the last one. We did a quick restore and let out a sigh of relief.
We did eventually write a tool to do those updates (and deletes) we were doing multiple times a day and were able to hand that tool over to one of the users, saving us multiple five minute interruptions each day and boosting our productivity.
Worth noting that this was part of a business that we bought from another company and we did transfer all customers to our own product with dev environments and no need to manually updating databases.
- sowbug 3y agoA sysadmin at one of my startups once mixed up c and x in a tar backup. Our source repository suddenly teleported back in time. We polled ourselves to see who'd updated most recently, and that guy's snapshot became commit #1 in the new repo. (Unfortunately this was the age of cvs, so the local copies were just source, not the history.)