3 ms·
> Well, for one thing, you must always remember the immortal words, DON'T PANIC So true. A colleague of mine managed - on his second or third day on job - to d
by krylon 11y ago
> Well, for one thing, you must always remember the immortal words, DON'T PANIC
So true. A colleague of mine managed - on his second or third day on job - to delete every single user account in our Active Directory. After an hour, we gave up trying to restore the AD (it was an SBS2008, so no AD recycle bin) and simply restored the entire DC (at the time, our domain only had the one DC) from backup. Surprisingly, most of our users took it very well and used the time to get some paperwork done or clean up their desks or something like that. Still, it was one of the most stressful days of my life. So we kind of panicked. In restrospect, I think another hour or so of research might have saved us the eight hours of restoring that server (did I mention that our backup infrastructure really, really sucked at the time?).
In smaller desasters, I've found the ability to remain calm most valuable, though. Having your boss breathing down your neck impatiently can instill a deep desire to simply do something just to show that you are working on the problem. But if you don't understand what's wrong, at best you are wasting time, and possibly making the problem even worse.
- Spooky23 11y agoAmen. I was a new DBA at a company that used a DBMS which used extent based disk allocation with a default extent size of 16kb. That was a problem on a busy table (which this was), because there was a finite limit to the number of extents that could allocated, but that limit was impossible to predict. The proactive fix was simple. Backup, restore the table in a two hour outage window. I asked for 5 hours, as the previous DBA never tested a restore. "Unacceptable", said the SVP. Fast forward two weeks. We hit the limit, and the entire company is essentially down. Between lost revenue, SLA fines and payroll, they were losing something like $5k/minute. Recovery at this point required 30 hours of full database restore, including journal recoveries from slooooow DAT tapes. There were three of us, everyone stayed calm, provided regular updates and handled a few hours of direct observation by the CEO and a board member for a few hours.
- dredmorbius 11y agoAnd the SVP?
- Spooky23 11y agoMaster politician... He made it somebody else's problem. Funny story is that he forgot to pay the phone bill for one of the call centers, and when they went to walk him out, they found him in a "compromising position" with his secretary in the office. That company was an unlimited source of material! Good times!
- krylon 11y agoOh my! And I thought our backup software took a long time to restore the server... (This whole thing happened on a Friday, so 30 hours would not have been that much of a problem for the rest of the company, but I would have had to work over the weekend and spend the night on-site...) The sad thing about such events is that afterwards, you could go "Told you so", but usually, people will not only not listen, but sometimes will still find a way to blame you for what went down. (In our case, it was our mistake, but we were lucky our CEO took it very well - he has no problem with people making mistakes as long as they are open about it and try to learn from their mistakes. What he cannot stand, though, is people trying to cover their butt and/or shift the blame onto others...)