6 ms·
We've all been there. Shit happens. That's what backup is for. OT: It's probably bad form to publicly blame someone for it, even if it's done by him. It's s
by ww520 11y ago
We've all been there. Shit happens. That's what backup is for.
OT: It's probably bad form to publicly blame someone for it, even if it's done by him. It's suffice to say, we screwed up but on our way to recovery. It's better to follow the practice of praising in public and discussing problem in private.
- roflc0ptic 11y agoHow else can they explain this outage? I guess it could be "we accidentally deleted the production database." But at that point they would just be euphemizing - clearly someone pulled the trigger. If they were naming the person, that would be pretty terrible on their part. But they're not. It seems perfectly fine to my eyes.
- ninju 11y agoThey could have initially said that we are encountering a database issue requiring a database restore. Once the system has been fully restored they can provide more details as to the "database issue"
- abluecloud 11y ago"faulty SQL queries caused us to have to restore the database"
- ww520 11y agoThe person might be under the gun from someone else to fix the prior problem, and made a mistake under pressure. Singling out the admin just absolves management from any blame. And it shows the lack of leadership and the lack of willingness to take on responsibility for the organization. And the problem is clearly a organizational problem. There's no clear backup and restore procedure. It's probably never tested for restore. There's no failover. There's no disaster recovery. Even if it's there, it has not been fire-drilled periodically. There's no clear access procedure in protecting the production servers. There's no prior spell out of definite steps to address production problems before doing them. There's no rollback procedure. There's no review. There's no approval process.
- roflc0ptic 11y agoThat is a good explanation. I see why blaming the admin is problematic. Thanks.
- steven2012 11y agoAs I mention above, it's not so much the administrators fault as it is the VP of engineering. There was clearly no disaster recovery plan which is unacceptable.
- vollmond 11y agoI worked on a team that had a list of "breakfastable offences" -- violating these rules meant you had to bring in breakfast for the whole team (donuts, bagels, whatever). One of them was "throwing someone under the bus." In conversations with anyone outside the team, you weren't allowed to single out a person as responsible for any particular bug/error/etc. Granted, this is pretty vague (depending on how many "administrators" the company has), but it's still too specific for me.
- caminante 11y agoI like the idea of "breakfastable offenses." I'm curious if you had remote teammates. If so, was there a workaround? It reminds me of my SCUBA training. The head instructor had a list of offenses. If you committed an offense (i.e. leaving goggles on forehead after surfacing), he'd say, "6-pack," obligating you to bring a 6-pack of beer. ;-)
- vollmond 11y agoWe had no remote teammates. Not sure how we would have handled that :-) Some of the other offenses: * Breaking the build (unit tests only) and then leaving for the day without fixing it or reverting your change * Walking away without logging out of your machine (very security-conscious business) * Not completing an assigned code review within a week of being assigned (within reason -- if the code review was enormous, or you were overly busy with an enormous project of your own, don't worry about it)