4 ms·
That was a 7 hours interplanetary outage because of Docker. All that’s left from the outage is a few messages on a GitHub issue. There was no postmortem. It
by 0xmohit 10y ago
That was a 7 hours interplanetary outage because of Docker. All
that’s left from the outage is a few messages on a GitHub
issue. There was no postmortem. It had little (none?) tech news
or press coverage, in spite of the catastrophic failure.
This is incorrect. There were github issues, HN posts, and so on. Solomon Hykes posted on Github [0] and HN [1]:
Hi everyone. I work at Docker.
First, my apologies for the outage. I consider our package
infrastructure as critical infrastructure, both for the free
and commercial versions of Docker. It's true that we offer
better support for the commercial version (it's one if its
features), but that should not apply to fundamental things like
being able to download your packages.
The team is working on the issue and will continue to give
updates here. We are taking this seriously.
Some of you pointed out that the response time and use of
communication channels seem inadequate, for example the
@dockerststus bot has not mentioned the issue when it was
detected. I share the opinion but I don't know the full story
yet; the post-mortem will tell us for sure what went wrong. At
the moment the team is focusing on fixing the issue and I don't
want to distract them from that.
Once the post-mortem identifies what went wrong, we will take
appropriate corrective action. I suspect part of it will be
better coordination between core engineers and infrastructure
engineers (2 distinct groups within Docker).
Thanks and sorry again for the inconvenience.
[0] https://github.com/docker/docker/issues/23203#issuecomment-223326996 https://github.com/docker/docker/issues/23203#issuecomment-2...
[1] https://news.ycombinator.com/item?id=11823231 https://news.ycombinator.com/item?id=11823231
- bambax 10y agoThis is not a postmortem; this is a github comment copy-and-pasted to HN, saying "sorry" and talking about a future postmortem.
- paulddraper 10y agoSo what did go wrong?
- user5994461 10y agoAs said, they pushed packages with invalid signatures. Why did this happen? There wasn't any explanation given. I'd love to see a postmortem from Docker.
- paulddraper 10y agoExactly.