5 ms·
It's crazy to me how long it takes Amazon to notify users that there is even an issue. I think it took 15 minutes for them to acknowledge an issue, thats a lot
by willio58 4y ago
It's crazy to me how long it takes Amazon to notify users that there is even an issue. I think it took 15 minutes for them to acknowledge an issue, thats a lot of time for our services.
- socialismisok 4y agoThat's because it's a political issue inside AWS. They have the technology to report it automatically, but there's a strong pressure not to post "green-i" or yellow or red, because those things impact SLA payments. So if there's any way they can spin it as not an outage they will try not to post it.
- dilyevsky 4y agoI participated in multiple sla penalty payments requests on the customer side and never once we even discussed status page color
- socialismisok 4y agoI participated in many status page color discussions and we always discussed sla. Shrug
- dilyevsky 4y agoYou mean on the AWS/service provider side? I've also participated in internal incident response on the service provider side and we again made SLA refund decisions based on actual customer impact not our status page. But then again we diligently update our status pages so there's that.
- socialismisok 4y agoYeah, on the AWS side.
- hatware 4y agoThat's a conflict of interest and it's a shame the proper governing bodies have not penalized Amazon for such shitty behavior.
- jeffwask 4y agoThis is my understand as well. I have anecdotally heard swapping some of those dots from green to red can require up to CTO approval.
- ogn3rd 4y agoCan confirm this, it's political. It's always great seeing the "rationality" for not doing the right thing. After the 3 outages last December, I can remember a certain person in a certain global outage slack channel laughing at customers who weren't "resilient" enough to withstand the outage. Then AWS focused a resiliency campaign to document all those customers risks, as if AWS's own poorly designed system wasn't the cause for peoples inability to fail over. Glad I'm out of that place, it's super toxic these days.
- thedougd 4y agoIt would be nice if all services had a great story. The guidance for how to make Cognito and SSO multi-region is laughable.
- haloboy777 4y agoI know many services that do this. And even if you fight with customer care for RCA they go silent and pass you around to other team members to whom you need to tell the problem again and again.
- faangiq 4y agoThey are brutally political and incompetent.
- JshWright 4y agoIt was 41 minutes from the first sign of trouble in our monitoring (an SQS queue backing up because the lambda it triggers wasn't getting invoked) to AWS posting the "informational" status.
- duluca 4y agoI don’t think you should rely on Amazon to tell you anything.
- tyingq 4y agoI continue to be irritated that services that consistently return errors are characterized as "increased error rates" or "increased latency". They seem to use those phrases for any kind of outage.
- dylan604 4y agoIt's all networking issues as you're remotely access them remotely, so of course it's increased error rates or increased latency. When a piece of gear stops responding the network tries to self-heal as designed which causes those issues.
- tyingq 4y agoWell, yes, but, for example: Characterizing both a 5% error rate and a 100% error rate as "increased error rates" is not as helpful as it could be. I'm reasonably sure they know when a service isn't functional at all.
- icedchai 4y agoIt's all in the messaging. I once worked at a place that was having bandwidth problems because of an "issue with an upstream vendor." The "issue" was we hadn't paid our bill in a couple months, so they disconnected us.
- mbesto 4y agoYup, we're trying to fix this here: https://www.awareops.com/ https://www.awareops.com/