6 ms·
> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because
by jph00 10y ago
> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3.
Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with confidence about how the rest of the system is designed, frankly...
- tlack 10y agoAnd apparently they had never tried rebooting some of the most important parts of that system. Just when you start to think that someone's really gotten it right you come to learn they're just fumbling around in the dark like everyone else.
- jonhohle 10y agoMy interpretation of this is that the indexing system was resilient to lost of a certain amount of capacity (probably around ⅓ + 1 host). As a guess, the indexing system probably used some form of consensus (e.g. paxos) which has had an active leader for years. Deployments stay within that capacity constraint, so while hosts have been restarted and replaced (data center migrations, hardware lease expiration, failures, upgrades, etc.), they may have not recently run into a situation where quorum wasn't available for a partition, especially at the scale of restarting the entire fleet. Since restarting the entire fleet would incur downtime of all relevant S3 operations, it's unlikely that it was something ever intentionally done in production (and they may or may not have run that scenario in other environments). Source: I used to run several large scale services at Amazon.
- pyre 10y agoTo put what @jonhohle said another way, Amazon had probably never brought up the entirety of S3 from Zero to Production-Ready on in a production environment before. I wouldn't necessarily classify this as "fumbling around in the dark." Perhaps they should have tested this in a simulated environment, but (to be fair) on a distributed fault-tolerant system, it probably wasn't a top-priority situation to test.
- joelthelion 10y ago> Perhaps they should have tested this in a simulated environment What makes you think they didn't?
- idlewords 10y agoMoreover, beyond a certain scale, it becomes really hard to simulate.
- richardlblair 10y agoAnyone who works on large complex systems will read this is and go "this was all preventable, but also completely understandable".
- frankc 10y agoI agree with that in general but having your monitoring system be dependent on the thing it monitors is a pretty big goof. It possible that the dependency was very non-obvious and many layers deep, which is more understandable, but still...its pretty fundamental.
- jedberg 10y agoThe monitoring system was not dependent on the thing it was monitoring. The website that shows the public results of the monitoring, which is updates only by humans, depended on it.
- abraves10001 10y agoAn important distinction but doesn't negate the ideas that their users should be in the dark because of something this stupid.
- kakarot 10y agoThey have a twitter account for such incidents and used it appropriately. They did not slack in relaying the outage to customers, and between that and the fact that no S3 services were operating I think the message was pretty clear: "We fucked up, give us a couple hours"
- paulddraper 10y agoI'm, I don't understand. My us-east-1 RSS feed said S3 had no incidents.
- kevinchen 10y agoI'm curious as to why their fix was to host the Service Health Dashboard on more AWS regions. It seems like the responsible thing to do is to host it entirely on a competitor's service. That way, it's very simple to know that the status page will work no matter what happens to you.
- deleted 10y ago[deleted]
- ghayes 10y agoOr have it pull from two sources, one local (S3) and one remote (GCE or whatever), and make a hard positive from either source signal "down." Otherwise the page would be down if just the remote source were down.
- ckozlowski 10y agoPrecisely. "we have changed the SHD administration console to run across multiple AWS regions."
- djsumdog 10y agoYou have to eat your own dogfood, if you're trying to sell a service product. That being said, a better solution would be to stand up a separate infrastructure just for status pages.
- ceejayoz 10y ago"Except for status pages" is a perfectly reasonable exception to "eat your own dogfood".
- pmoriarty 10y agoIf they did host their status page on a competitor's service, then they'd be reliant on that service, which might backfire if the competitor's service goes down while Amazon's own systems stay up. What they really need is failover capability, which can fire up the status page on a competitor's service (or maybe on a completely separate disaster recovery site site owned by Amazon) in case Amazon's own services go down. I'm sure Amazon's architects and engineers are more than capable of designing and implementing such a robust system and recognizing its importance. So it puzzles me as to why it wasn't done.
- avsupport 10y agoConsidering the size of AWS, the number of services one service relies on would be too many, even if a single service many layers deep depended on s3, and when s3 goes down, the service as a whole will be affected. Honestly S3, it like a blackhole, everybody stores everything in S3 these days. Its a horizontal component but used in a vertical manner. Weird but true.
- jsmeaton 10y agoHow many companies do you think running on AWS host their status page on S3?