5 ms·
It's about time we heard from them. I understand the timing of this was unfortunate (right before a holiday), but the trust alluded to in the conclusion of that
by apeace 14y ago
It's about time we heard from them. I understand the timing of this was unfortunate (right before a holiday), but the trust alluded to in the conclusion of that post would be bolstered by faster post-mortems on major outages like this one.
- deleted 14y ago[deleted]
- nixgeek 14y agoBetter to take a few days to gather evidence and really understand the problem, than rush out a post-mortem which is inaccurate or incorrect leaving the impression that downtime is not taken seriously and investigated thoroughly. IMO.
- InclinedPlane 14y agoI disagree. Other than for curiosity needs what's the value in a faster post mortem? Especially considering that a faster post-mortem would likely be a less accurate and less complete post-mortem? What is the difference in actionability for anyone between an update in 2 days vs an update in 4 days? I see none.
- apeace 14y agoTo me, a post-mortem doesn't just satisfy curiosity, it eases fears that the problem will return and informs me of future plans which may help prevent the problem, or may bring it back. It helps me form my own plan, since I'm a user of Github. I for one spent my break hawking my email, in case further Github outages caused any of my automated deploy scripts to fail. I realize it's my own responsibility to write scripts that handle failure scenarios, but the fact is my company pays Github to host our repositories. Downtime happens, I'm understanding of that. But when it does, I want to know what's going on as soon as that information's available--especially when I'm on holiday. I don't think a blog post written after they resolved all the issues would have been less accurate. It just would have inconvenienced whoever wrote it on a holiday. Not the biggest deal in the world--it would take me a lot more than this slip-up to switch from Github. But IMO a service provider should get at least some information out faster than this.
- res0nat0r 14y agoI'd consider yourself lucky when it comes to Github post-mortems. They seem to respond very quickly when it comes to a large event. Compared to AWS write ups which can come week(s) after the event after all data has been gathered and a summary written. I don't believe any company that is smart would put out a writeup as quickly as possible if they weren't sure of the facts, that just doesn't help anyone out if the company is spreading unverified information. Plus developers need to be focused on fixing the actual problem and root causes first, pretty write ups come second. For upto the second info the status page should be more than enough.
- imbriaco 14y agoI appreciate your point of view but respectfully disagree. The post-mortem would have absolutely been less accurate if we had delivered it sooner since we did not have the details about why the MLAG failover did not happen as expected until late in the evening on Christmas Eve. We've worked as quickly as possible since then to provide a full post-mortem.
- rdl 14y agoIMO anything less than a billing cycle (minus a week or so) is pretty acceptable for a post-mortem, although a week is better.
- InclinedPlane 14y agoHow is a post-mortem provided 4 days after the event less actionable in any significant way than a post-mortem delivered sooner?