6 ms·
Ask HN: Where is the AWS outage post-mortem?
I've been checking in on this page [0] frequently since the recent us-east-1 outage.
The outage was significant and had many unexpected knock-on effects for us - despite most of our instances being in a totally separate region - so just wondering what happened.
Any idea how long these post-mortems typically take to be published?
Thanks.
[0] https://aws.amazon.com/premiumsupport/technology/pes/
- brycewray 5y agoIn this case, probably around 4:55 PM Eastern Time on Friday, December 24.
- cyberge99 5y agoI was thinking tomorrow (Friday, USA) at 4:55 PM. That’s when the slimy politicians always release bad news.
- thecrumb 5y agoTakes awhile to get the creative fictional juices flowing.
- cosmotic 5y agoI don't know; I thought "impairment of several network devices" was a pretty creative spin
- nix23 5y agoNothing happened you can go away now.
- exabrial 5y agoPro tip: Never mark your services down on the status page and you don't have to write an RCA!
- dandigangi 5y agothink_about_it_meme.jpg
- imwillofficial 5y agoRCA is in progress
- 1cvmask 5y agoCat got stuck in the server room.
- MrWiffles 5y agoJust to chime in here and maybe help explain some of the more snarky responses... If I were in your shoes, I wouldn't expect one, and if one ever does get published, be suspicious of it. VERY suspicious. Because it's likely missing critical information and/or some portion is likely fictitious to some degree. I mean, this is a company that can't even update its own system status dashboard truthfully, so don't expect any degree of honesty or accountability here.
- beebmam 5y agoFrankly, why should a company NOT lie? There's no reason to tell the truth if the truth will hurt your business.
- mapmap 5y agoCredibility?
- yunwal 5y agoI'm an infrastructure engineer that has never considered using a comparable cloud provider unless the solution I was looking for wasn't supported by AWS. It used to feel like AWS cared about it's reputation and trustworthiness, and now they clearly do not. If I was starting a project from scratch now, I'd certainly explore other cloud providers, which is not something I would've said 3 years ago. Their blatant lying is definitely going to cost them in the long run.
- ptcrash 5y agoWhen did AWS lie? On their status page? That's the last place to receive updates on service status. It sucks but it's also common knowledge. I'll give you a little insider knowledge: The SHD is updated manually. If it's any consolation, GCP does the exact same thing. I saw notifications in the AWS Personal Health Dashboard pretty very early into the issue. Did you see something different? Also, I've seen issues logging into the default console a lot, so I keep one of the backup endpoints bookmarked. Might come in handy for you too! https://us-east-2.console.aws.amazon.com/ https://us-east-2.console.aws.amazon.com/
- belter 5y agoAlso waiting to know more details. In the meanwhile...You might want to check these: "AWS uncertain about true cause of outage" https://news.ycombinator.com/item?id=29492120 https://news.ycombinator.com/item?id=29492120 "Thousand Eyes AWS Outage Analysis: December 7, 2021 " https://news.ycombinator.com/item?id=29483487 https://news.ycombinator.com/item?id=29483487 Edit: Using [0] linked above come up with the following statistics... [0] https://aws.amazon.com/premiumsupport/technology/pes/ https://aws.amazon.com/premiumsupport/technology/pes/ 15 major events of last 10 years, including the one from 7 Dec. Percentage distribution across months of the year: ---------------------------------------------- Jan: 0% Apr: 7% Jul: 7% Oct: 7% Feb: 7% May: 0% Aug: 13% Nov: 13% Mar: 0% Jun: 13% Sep: 13% Dec: 20% ----------------------------------------------- Book your vacation for Jan, Mar, and May... :-)
- dandigangi 5y agoProbably within a month. Maybe a small post about it while they get things together more seriously.
- deleted 5y ago[deleted]
- Bluecobra 5y agoMy guess is that the the person that broke Facebook for a day got hired at Amazon...
- doubled112 5y agoWhat could go wrong? Nobody would ever make a mistake like that twice! Right?
- cyounkins 5y agoConveniently most of AWS' published postmortems do not have publish dates. Some are for the same day, some say "earlier this week". Realistically, it takes time to fully dig into what happened. In a public postmortem you also want to describe your mitigation efforts, which you also need to think through fully. I expect something in the next few days.
- la6471 5y agoIt takes 5 business days.
- imwillofficial 5y agoThe COE is in progress.
- ptcrash 5y agoLet's try to remember the guidelines, be kind, and have curious conversation. Hacker News isn't the place for baseless speculation and generalization. AWS publishes postmortems for major outages here: [1] This outage was significant enough to expect a public PES but it's typical for it to take at least a few weeks for that page to get updated. At least, that's been the trend for all previous publications. If the PES is anything like previous PESes, it will have a detailed explanation of the root-cause and an explanation of what will change to prevent the issue from happening again but it will still be technologically abstract because cloud providers are very secretive of how they orchestrate resources behind the scenes. Enterprise-tier support customers can ask for RCA's but as I understand it, the account managers don't have any official wording internally yet either. But, it's only been 48 hours, so something of this significance will likely have a large chain of sign-offs it has to go through before official wording is announced. This type of quietness from AWS before their official wording gets published is standard practice for them. They have a large legal team, PR team, and executive team that will all be interested in controlling the narrative but that's not uncommon for other large companies either. If I were to take a stab in the dark, I'd say we'll see a PES in ~2 weeks. Maybe sooner, maybe later. If they don't announce a PES, I'll be really shocked because last year's Kinesis outage had arguably smaller impact but ended up getting a publication. [2] [1] https://aws.amazon.com/premiumsupport/technology/pes/ https://aws.amazon.com/premiumsupport/technology/pes/ [2] https://aws.amazon.com/message/11201/ https://aws.amazon.com/message/11201/ Edit: Typos
- herodoturtle 5y agoOP here. Just wanted to say thank you for this.
- unbanned 5y agoWhat outage? Status pages are static OK messages.
- gukov 5y agoPublic post mortems are usually done by growing companies that have something to lose. AWS is too big to “lose” anything.
- stunt 5y agoIt's not that simple. Incident responders and teams that own the pieces have to write the initial draft and review it. Perhaps two different copies. One with more details for internal and another for public. Multiple round of reviews, and then it goes to a separate team to proofread the public material. Meanwhile they also have to measure customer impact and sometimes even inform high tier customers about the incident and its impact.
- gavinegolden 5y agoHere it is: https://aws.amazon.com/message/12721/ https://aws.amazon.com/message/12721/