3 ms·

Ha, whenever some 3rd party SaaS tool is down and I am obsessively refreshing their status page I'm always enraged with the lack of updates. But then later I th

Ha, whenever some 3rd party SaaS tool is down and I am obsessively refreshing their status page I'm always enraged with the lack of updates. But then later I think about what if I were one of the engineers trying to bring things back up. There is nothing worse than being heads down trying to get a mission-critical system back online and being constantly distracted by demands for "status updates." What is the ETA on getting back online? Well, about 5 minutes longer than it would have been now that you pulled me away from what I was doing....

quesera 5y ago
In my experience, Good organizations have scheduled updates for critical outages. No one pesters anyone, and everyone knows that a five minute update is expected periodically, e.g. at the top of every hour. Great organizations instantly spin up a response team which includes a communications liaison, who is technical enough to participate in the reponse, but is not presently leading it. They assist and communicate, either to the PM or to customers directly. ...and other organizations burn through humans until they get a handle on how to be better. That said, in these days of AWS and GCE, you can go a loong time without running into the kinds of failures that used to cause critical, extended outages.
- thinkharderdev 5y ago
  Yep, there is definitely a way to do it well and having scheduled, regular updates (as opposed to ad-hoc requests) is key. But when "the big one" comes then it is hard to maintain the discipline in that process.
derekp7 5y ago
And what is really frustrating is you see an error in a tool you depend on, try something to fix it (such as rebuilding a database index), then you sit back and wait without having any idea how long that process is going to take. Or if it is even going to fix the issue. So when someone asks how much longer, you really don't know. And there is a lot of quick pounding on the keyboard, followed by a long wait period where it feels like you aren't doing anything.

quesera 5y ago

In my experience, Good organizations have scheduled updates for critical outages. No one pesters anyone, and everyone knows that a five minute update is expected periodically, e.g. at the top of every hour. Great organizations instantly spin up a response team which includes a communications liaison, who is technical enough to participate in the reponse, but is not presently leading it. They assist and communicate, either to the PM or to customers directly. ...and other organizations burn through humans until they get a handle on how to be better. That said, in these days of AWS and GCE, you can go a loong time without running into the kinds of failures that used to cause critical, extended outages.

thinkharderdev 5y ago
Yep, there is definitely a way to do it well and having scheduled, regular updates (as opposed to ad-hoc requests) is key. But when "the big one" comes then it is hard to maintain the discipline in that process.

derekp7 5y ago

And what is really frustrating is you see an error in a tool you depend on, try something to fix it (such as rebuilding a database index), then you sit back and wait without having any idea how long that process is going to take. Or if it is even going to fix the issue. So when someone asks how much longer, you really don't know. And there is a lot of quick pounding on the keyboard, followed by a long wait period where it feels like you aren't doing anything.