13 ms·
I love that every time this happens, 100% of the services on https://status.aws.amazon.com https://status.aws.amazon.com are green.
by zegl 5y ago
I love that every time this happens, 100% of the services on https://status.aws.amazon.com https://status.aws.amazon.com are green.
- strictfp 5y ago"some customers may experience a slight elevation in error rates" --> everything is on fire, absolutely nothing works
- davecap1 5y agoOn top of that, the "Personalized Health Dashboard" doesn't work because I can't seem to log in to the console.
- qudat 5y agoStatus pages are hard
- jtdev 5y agoWhy? Twitter and HN can tell me that AWS is having an outage, why can’t AWS?
- AH4oFVbPT4f8 5y agoThey sent their CEO into space, I am sure they have the resources to figure it out.
- smt88 5y agoNo they're not. Step 1: deploy status checks to an external cloud.
- kube-system 5y agoI agree, but does come with increased challenges with false positives. That being said, AWS status pages are up.
- cr3ative 5y agoWhen they have too much pride in an all-green dash, sure. Allowing any engineer to declare a problem when first detected? Not so hard, but it doesn't make you look good if you have an ultra-twitchy finger. They have the balance badly wrong at the moment though.
- lukeschlather 5y agoA trigger-happy status page gives realtime feedback for anyone doing a DoS attack. Even if you published that information publicly you would probably want it on a significant delay.
- pid-1 5y agoMore like admitting failure is hard.
- wruza 5y ago“Falsehoods Programmers Believe About Status Pages”
- mrweasel 5y agoNot if you’re AWS. At this point I’m fairly sure their status page is just a static html that always show all green.
- siva7 5y agoWell, it is.
- judge2020 5y agoI don't see why they couldn't provide an error rate graph like Reddit[0] or simply make services yellow saying "increased error rate detected, investigating..." 0: https://www.redditstatus.com/#system-metrics https://www.redditstatus.com/#system-metrics
- willcipriano 5y agoA executive has a OKR around uptime and a automated system prevents him or her from having control over the messaging. Therefore any effort to create one is squashed, leaving the people requesting it confused as to why and left without any explanation. Oldest story in the book.
- jkingsman 5y agoBecause Amazon has $$$$$ in their SLOs, and it costs them through the nose every minute they're down in payments made to customers and fees refunded. I trust them and most companies not to be outright fraudulent (although I'm sure some are), but it's totally understandable they'd be reticent to push the "Downtime Alert/Cost Us a Ton of Money" button until they're sure something serious is happening.
- btilly 5y agoThis is an incentive to dishonesty, leading to fraudulent payments and false advertising of uptime to potential customers. Hopefully it results in a class action lawsuit for enough money that Amazon decides that an automated system is better than trying to supply human judgement.
- jenkinstrigger 5y agoCan someone just have a site ping all the GET endpoints on the AWS API? That is very far from "automating [their entire] system" but it's better than what they're doing.
- tynorf 5y ago
- sakopov 5y agoThose five 9s don't come easy. Sometimes you have to prop them up :)
- jjoonathan 5y agoI wonder how often outages really happen. The official page is nonsense, of course, and we only collectively notice when the outage is big enough that lots of us are affected. On AWS, I see about a 3:1 ratio of "bump in the night" outages (quickly resolved, little corroboration) to mega too-big-to-hide outages. Does that mirror others' experiences?
- Spivak 5y agoIf you count any time AWS is having a problem that impacts our production workloads then I think it's about 5:1. Dealing with "AWS is down" outages are easy because I can just sit back and grab some popcorn, it's the "dammit I know this is AWS's fault" outages that are a PITA because you count yourself lucky to even get a report in your personalized dashboard.
- jjoonathan 5y agoYep. Random aside: any chance you are related to the Calculus on Manifolds Spivak?
- kylemh 5y agohttps://aws.amazon.com/compute/sla/ https://aws.amazon.com/compute/sla/ looks like only four 9's
- queuebert 5y agoAre they lying, or just prioritizing their own services?
- _verandaguy 5y agoWilling to bet the status page gets updated by logic on us-east-1
- itsyaboi 5y agoStatus service is probably hosted in us-east-1
- AH4oFVbPT4f8 5y agoamazon.com seems to be having problems too. I get something went wrong to a new design/layout which I assume is either new or a fail safe.
- KineticLensman 5y agoLooks okay right now to this UK user of amazon.co.uk
- btilly 5y agoIt depends on which Amazon region you are being served from. It is very unlikely that Amazon would deliberately make your messages cross the Atlantic just to find an American region that is unable to serve you.
- bennyp101 5y agohttps://music.amazon.co.uk https://music.amazon.co.uk is giving me an error since about 16:30 GMT "We are experiencing an error. Our apologies – We will be back up soon."
- notreallyserio 5y agoMakes you wonder if they have to manually update the page when outages occur. That'd be a pretty bad way to go, so I'd hope not. Maybe the code to automatically update the page is in us-east-1? :)
- gromann 5y agoWord on the street is the status page is just a JPG
- wfleming 5y agoSomething like that has impacted the status page in the past. There was a severe Kinesis outage last year (https://aws.amazon.com/message/11201/ https://aws.amazon.com/message/11201/), and they couldn't update the service dashboard for quite a while because their tool to do manage the service dashboard lives in us-east-1 and depends on Kinesis.
- ta20200710 5y agoEC2 or S3 showing red in any region literally requires personal approval of the CEO of AWS.
- dia80 5y agoUnfortunately, errors don't require his approval...
- notreallyserio 5y agoIs this true or a joke? This sort of policy is how you destroy trust.
- bsedlm 5y agomaybe we gotta consider the publicly facing status pages as something other than a technical tool (e.g. marketing or PR or something like that, dunno)
- jedberg 5y agoFrom what I've heard it's mostly true. Not only the CEO but a few SVPs can approve it, but yes a human must approve the update and it must be a high level exec. Part of the reason is because their SLAs are based on that dashboard, and that dashboard going red has a financial cost to AWS, so like any financial cost, it needs approval.
- notreallyserio 5y agoI wonder how well known this is. You'd think it would be hard to hire ethical engineers with such a scheme in place and yet they have tens of thousands.
- orangepurple 5y agoBeing dishonest about SLAs seems to bear zero cost in this case?
- 5y ago
- deleted 5y ago[deleted]
- goshx 5y agoI remember the time when S3 went down and took the status page down with it
- JohnJamesRambo 5y ago> Goodhart's Law is expressed simply as: “When a measure becomes a target, it ceases to be a good measure.” It’s very frustrating. Why even have them?
- Spivak 5y agoBecause "uptime" and "nines" became a marketing term. Simple as that. But the problem is that any public-facing measure of availability becomes a defacto marketing term.
- Enginerrrd 5y agoThe older I get the more I hate marketers. The whole field stands on the back of war-time propaganda research and it sure feels like it's the cause of so much rot in society.
- hinkley 5y agoAlso 4-5 nines is virtually impossible for complex systems, so the sort of responsible people who could make 3 nines true begin to check out, and now you've getting most of your info from the delusional, and you're lucky if you manage 2 objective nines.
- whoknew1122 5y agoThe problem being that often times you can't actually update the status page. Most internal systems are down. We can't even update our product to say it's down, because accessing the product requires a process that is currently dead.
- thayne 5y agoThat's why your status page should be completely independent from the services it is monitoring (minus maybe something that automatically updates it). We use a third party to host our status page specifically so that we can update it even if all our systems are down.
- whoknew1122 5y agoI'm not saying you're wrong, or that the status page is architected properly. I'm just speaking to the current situation.
- deleted 5y ago[deleted]
- jrochkind1 5y agoEven better, when I try to go to console, I get: > AWS Management Console Home page is currently unavailable. > You can monitor status on the AWS Service Health Dashboard. "AWS Service Health Dashboard" is a link to status.aws.amazon.com... which is ALL GREEN. So... thanks for the suggestion? At this point the AWS service health dashboard is kind of famous for always been green isn't it? It's a joke to it's users. Do the folks who work on the relevant AWS internal team(s) know this, and just not have the resources to do anything about it, or what? If it's a harder problem than you'd think for interesting technical reasons, that'd be interesting to hear about.
- ballenf 5y agohttps://downdetector.com https://downdetector.com Amazing and scary to see all the unrelated services down right now.
- nightpool 5y agoI think it's pretty unlikely that both Google and Facebook are affected by this minor AWS outage, whatever DownDetector says. I even did a spot check on some of the smaller websites they report as "down", like canva.com, and didn't see any issues.
- zarkov99 5y agoYou might be right about Google and Facebook, but this isn't minor at all. Impact is widespread.
- jrockway 5y agoI wonder if the other parts of Amazon do this. Like their inventory system thinks something is in stock, but people can't find it in the warehouse, do they just simply not send it to you and hope you don't notice? AWS's culture sounds super broken. My favorite status page, though, is Slack's. You can read an article in the New York Times about how Slack was down for most of a day, and the status page is just like "some percentage of users experienced minor connectivity issues". "Some percentage" is code for "100%" and "minor" is code for "total". Good try.
- kortex 5y agoIt's like trying to get the truth out of a kid that caused some trouble. Mom: Alexa, did you break something? Alexa: No. M: Really? What's this? 500 Internal server error A: ok maybe management console is down M: Anything else? A: ... A: ... ok maybe cloudwatch logs M: Ah hah. What else? A: That's it, I swear! M: 503 ClientError A: ...well okay secretsmanager might be busted too...
- mdni007 5y agoFunny I literally just asked my Alexa. Me: Alexa, is AWS down right now? Alexa: I'd rather not answer that
- hinkley 5y agoWise robot. That's a bit like involving your kid in an argument between parents.
- hinkley 5y agoThere was a great response in r/relationship advice the other day where someone said that OP's partner forced a fight because they're planning to cheat on them, reconcile, and then will 'trickle out the truth' over the next 6 months. I'm stealing that phrase.
- PopeUrbanX 5y agoThe very expensive EC2 instance I started this morning still works. Of course now I can't shut it down.
- daniel-s 5y agoThat page is not loading for me... on which region is it hosted?
- barbazoo 5y agoIt seems they updated it ~30 minutes after your comment.
- NicoJuicy 5y agoNot right now. I think they monitor if it appears on HN too.
- john37386 5y agoIt starts to show issues now. I agree that it was a bit long before we can get real visibility on the incident.
- deleted 5y ago[deleted]
- jbavari 5y agoWell yea, it's eventually consistent ;)
- gbear0 5y agoI assume each service has its own health check that checks the service is accessible from an internal location, thus most are green. However, when Service A requires Service B to do work, but Service B is down, a simple access check on Service A clearly doesn't give a good representation of uptime. So what's a good health check actually report these days? Is it just about its own status, or should it include a breakdown of the status of external dependencies as part of its folded up status?
- 0xmohit 5y agoNo wonder IMDB <https://www.imdb.com/ https://www.imdb.com/> is down (returning 503). Sad that Amazon engineers don't implement what they teach their customers -- designing fault-tolerant and highly available systems.
- sneak 5y agoIt's widespread industry knowledge now that AWS is publicly dishonest about downtime. When the biggest cloud provider in the world is famous for gaslighting, it sets expectations for our whole industry. It's fucking disgraceful that they tolerate such a lack of integrity in their organization.
- swiftcoder 5y agoWhen I worked there it required the signoff of both your VP-level executive and the comms team to update the status page. I do not believe I ever received said signoff before the issues were resolved.
- ricardobayes 5y agoWonder why almost all Amazon frontend looks like it was written in c++