3 ms·

Several years I was a junior dev at a 3PL (3rd Party Logistics) company troubleshooting an issue with some overnight data imports. I had it all loaded up on th

Several years I was a junior dev at a 3PL (3rd Party Logistics) company troubleshooting an issue with some overnight data imports. I had it all loaded up on the test environment, then I'd delete the import, make some changes, and re-run it. Wash, rinse, repeat, trying to track down the issue. As I'm sure you can guess, at one point I executed my scirpt in the wrong window and deleted last nights import from Production. I immediately told my boss, who was very understanding with a "everyone does this kind of thing at some point" kind of shrug and went over to our DBA's office to ask him to re-load the last snapshot. But it turned out the snapshots had been broken for two weeks and no one had noticed. And it wasn't a simple issue of re-running the import. After all the orders had been imported, humans had manually assigned orders to trucks and dispatched them in the wee hours of the morning. Now there was no way to know what packages were on what trucks. I think the DBA ended up buying ApexSQL Log out of pocket to roll back the deletion with the transaction log. The result was that for several hours the delivery drivers for a national office supply company in a certain state were completely unable to use their handhelds or access their truck's inventory. That was my team-member-1 moment.

mulmen 10y ago
What were the consequences to you, the DBA, and the organization?
- keithpeter 10y ago
  I'm interested to know as well. I imagine that getting the snapshots working properly would be quite an important factor, otherwise the company would always be a typing error away from chaos.
  - lightbritefight 10y ago
    Working snapshots/backups are system engineers number 1 concern, always. They mitigate issues with every other system, of any scope. Its the first thing a new hire sysadmin/eng should ask about coming through the door.
- Pharylon 10y ago
  Nothing really happened in the end. I'd only worked there for about six months but my boss liked me and thought I did good work. He chalked it up as a learning experience (which it certainly was). Our DBA was someone who'd been with the company 15 years. I think most of the blame fell on him, but he didn't lose his job or anything like that. I think the snapshot issue was pretty trivial to fix. The only real problem was that there were no notifications going to anyone that they failed. The customer was pissed, and it was a new customer at that. They basically lost a day of deliveries out of it. But they kind of had vendor lock-in with us (the office supply company they were delivering for liked us and had basically told them to use us because we already knew how to process their data the way they wanted). Switching wouldn't have been trivial, but we nevertheless fell over ourselves to keep them happy for a few months until it blew over.
chipperyman573 10y ago
Why did you write an abbreviation if you just write what it stands for right after?

mulmen 10y ago

What were the consequences to you, the DBA, and the organization?

keithpeter 10y ago
I'm interested to know as well. I imagine that getting the snapshots working properly would be quite an important factor, otherwise the company would always be a typing error away from chaos.
- lightbritefight 10y ago
  Working snapshots/backups are system engineers number 1 concern, always. They mitigate issues with every other system, of any scope. Its the first thing a new hire sysadmin/eng should ask about coming through the door.
Pharylon 10y ago
Nothing really happened in the end. I'd only worked there for about six months but my boss liked me and thought I did good work. He chalked it up as a learning experience (which it certainly was). Our DBA was someone who'd been with the company 15 years. I think most of the blame fell on him, but he didn't lose his job or anything like that. I think the snapshot issue was pretty trivial to fix. The only real problem was that there were no notifications going to anyone that they failed. The customer was pissed, and it was a new customer at that. They basically lost a day of deliveries out of it. But they kind of had vendor lock-in with us (the office supply company they were delivering for liked us and had basically told them to use us because we already knew how to process their data the way they wanted). Switching wouldn't have been trivial, but we nevertheless fell over ourselves to keep them happy for a few months until it blew over.

chipperyman573 10y ago

Why did you write an abbreviation if you just write what it stands for right after?