12 ms·
Goddamn AWS seems to be down again 4th time this month alone
- bruce343434 5y agoSo this is the 4th time an uninteresting update like this has come to the front page... Why? What can be gathered from it? Get the AWS post-mortem on here, for that I'll be piqued. HN is not a status page. I don't instinctively check HN when my service is down because why would I?
- mplanchard 5y agoAs a counterpoint, HN is one of the first places I check if a big service seems to be down, because it usually shows up here before their status pages.
- noodlesUK 5y agoThis is getting to be a massive joke. Any reputation AWS has for being robust has been wiped out over the past month or so. I feel bad for the various engineers at AWS and other companies who are having to work on their day off.
- yjftsjthsd-h 5y ago> Goddamn AWS seems to be down again 4th time this month alone I feel obligated to point out that this is a very editorialized title, which is against site guidelines. Not... that I don't sympathize, just that it's perhaps a bit out of line even for providing context.
- nefitty 5y agoYou felt obligated to the benefit of who, exactly?
- rootusrootus 5y agoThe entire HN community, presumably.
- Ostrogodsky 5y agoAWS
- collegeburner 5y agoTrue, but it's funny... perhaps next time a "Tell HN" with the link?
- pictur 5y ago
- pictur 5y ago
- kadoban 5y agoIt's not offensive, it's just not the appropriate tone for HN.
- chrisseaton 5y agoAlways remember that there are real people behind technology and services that you use.
- pictur 5y agoyou can not be serious. Is aws really run by humans?
- chrisseaton 5y agoTake a look at the reaction to your comments - other people also don't want to see snide and unkind responses to things real people work on.
- 300bps 5y agoI'm honestly sick of seeing these posts. Every single post I've seen on HN, I have had no down time in EC2, S3, Workspaces, FSx for Windows, Directory Service, Console, etc. Over 200 services, 86 availability zones and 26 regions. We might as well post every 5 minutes if we post every time something on AWS goes down. And yes AWS is more dependent on us-east-1 but one of the outrages posts was for a minor outage in us-west-1.
- ineedasername 5y agoWell, there really shouldn't be be a service-- even across all regions and products-- going down every five minutes. Especially because it would mean service that needed to be in multiple regions around the world would almost always be having some type of issue.
- ineedasername 5y agoHmm, am I wrong in this? I admittedly don't work on things of worldwide or nationwide of that scale so if my statement isn't accurate I like to learn from it. Thanks for any additional insight & explanation.
- gregjor 5y agoYou aren’t necessarily wrong, but there’s a difference between a service at AWS scale experiencing frequent problems (inevitable if you think about it) and what actual impact those problems have. Overall the vast majority of AWS users don’t experience frequent problems. By analogy cars crash every few minutes in my city (Las Vegas), but I can’t conclude that indicates some inherent (or easily fixable) problem with the road infrastructure or the design of cars, or the skill of drivers. The frequent crashes rarely affect me.
- ineedasername 5y agoThanks, that makes sense. I always like to know why I'm wrong, but getting that explanation can be a fine line to avoid looking like you're complaining about downvotes. I could care less about the votes, but I do want to learn. Much appreciated.
- bool3max 5y ago
- ineedasername 5y agoNo, but headlines are supposed to be neutral and not flame/click bait. Even actual headlines that go in that direction are frequently modified. Maybe a fix from @dang?
- nefitty 5y agoJesus, let the man rest. It's Christmas Eve.
- ineedasername 5y agoWhy does Christmas Eve play into this? It's a time of year where we try to be a bit more charitable, so why should I not expect more civility here? And if the GP or original poster of this was seeming more rest on Christmas Eve then they could simply have avoided posting today. Edit: I just realized you probably meant Dang? That's fair.
- yjftsjthsd-h 5y agoIt's not about feelings; the site guidelines say, > Please don't do things to make titles stand out, like using uppercase or exclamation points, or saying how great an article is. It's implicit in submitting something that you think it's important. > [several other points, which don't apply here] > Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize. And while I am sympathetic to adding meaningful information ("4th time this month" is technically against the guidelines but I wouldn't object), that's not adding anything more other than telling us how freakynit feels about it, which bluntly isn't useful.
- ksec 5y agoI think it would have been fine if it had Tell HN in front of it. Without it the title would be against the rules and guidelines. One could argue the word Goddamn seems to be provocative. Which could be appropriate if it was really down. But it is not. I think @dang need to edit the title.
- SubiculumCode 5y agoIt certainly could be editorialized. Alternatively, AWS may be in fact damned by god.
- walrus01 5y agoIt may be editorialized but it's also exactly the words I would expect to come out of the mouths of some of my most experienced, senior, and clueful colleagues when asked to describe the present reliability of AWS US-EAST-1
- codeduck 5y agoGood old chaos-east-1
- t8e56vd4ih 5y agonice try, Jeff ...
- bgroat 5y agoThere's a video that's being going around HN lately about civilization collapsing and the loss of the ability to maintain robust systems. Now that I'm looking for it I see it everywhere.
- BackBlast 5y agoI would like more information about this. This is the first that I have seen this video mentioned.
- skyzyx 5y agoPro-Tip: Get the hell out of US-East-1.
- deleted 5y ago[deleted]
- altdataseller 5y agoIt’s Christmas Eve, I aint answering any PagerDuty notification today or tomorrow.
- voidfunc 5y ago
- Macha 5y agoYes, they'll understand if the impact is worth paying someone(s) for 24/7/365 support, and so far they've decided the answer is no. In my company's industry, it probably is correct - our customers don't work christmas eve/day either. At my previous company they did - and so oncall and its compensation was specified in the contract and there was a formal rota.
- geofft 5y agoAlso, quite literally, people (both management and other engineers) often don't fully think through the consequences of systems that have neither enough automatic failover nor enough staffing by non-distracted humans, and build systems assuming that it will be fine. It's probably better to suffer an unexpected outage now than in the future when your company is bigger.
- aaomidi 5y agoIt's funny, SREs and SDEs generally get paid around the same in most places. I know some companies compensate oncall time differently, but a lot don't. SREs work a loooot more hours than SDEs, and I feel like that's partly on SREs for just accepting that workload. In my current position, if I get woken up at night even once, I take that as 8 hours worked and take a whole day off. The other side of offering "unlimited time off" :). This creates a real cost on technical debt, and forces actually addressing them rather than letting human capital deal w/ them.
- lumost 5y agoCuriously this made my transition from SRE to SDE somewhat interesting. After I transitioned I kept working the same hours I would as an SRE and would be surprised at the lack of output from my coworkers. It wasn't until later that I realized it was simply the difference in hours worked. As an SRE, your only option for climbing out of the mess is to work harder/smarter/extra, and you are always lowest on the totem pole when messes happen. You either learn to keep up with the crazy schedules, or get the hell out of dodge.
- mushufasa 5y agowhat the hell this is unacceptable. is this some sort of sabotage? AWS may have to live this down for years in RFPs
- voidfunc 5y agoWonder how much of this is caused by engineers rushing to fix log4j issues at AWS so they can enjoy the holidays? Their stack is heavily Java-based isnt it?
- joebiden2024 5y agoUnclear on this one, but wasn’t the last one caused by power failure in an AZ?
- wlonkly 5y agoLog4j on the transfer switch, then!
- gjsman-1000 5y agoAmazon’s SLAs are quickly becoming only useful for kindling. At least your fireplace will give some warmth for your relatives. Merry Christmas!
- gowthamgts12 5y agolooks like it's a connectivity outage across all services - https://www.thousandeyes.com/outages/ https://www.thousandeyes.com/outages/
- darkwater 5y agoNice link, didn't know it. Thanks!
- theossuary 5y agoComplete speculation; but I wonder if how Amazon works their employee's so hard is the root cause of this. Layer on COVID as an additional stressor and give it a few years and this could be the result. Curious to hear from those at Amazon, what's it been like during COVID?
- valleyjo 5y agoAttrition has been super high during Covid and they raised the pay a lot for new offers to try and attract talent. (source: I got an offer for S3 mid-Nov. this is what the hiring manager told me).
- dv_dt 5y agoThe rule of thumb of efficiency is that it can make things worse when it breaks. How that applies to organizations is if you have an incredible org with a higher plates to plate spinner ratio than ever, then if you have a catastrophe, the org may not have enough people to actually get the plates spinning again as fast as a less efficient, larger crew with more reserve.
- loeg 5y agoThey're losing engineers left and right, especially in H2 2021. It's probably all related. Their recruiters' emails are increasingly desperate. If I was considering a job at Amazon now, I would be very interested in a good answer for how they plan to address their attrition problem (and paying new hires well isn't a viable answer alone).
- tomcooks 5y agoHappens when you delegate, don't give them money.
- forty 5y agoI got worried when I saw this then checked https://status.aws.amazon.com/ https://status.aws.amazon.com/ and all seems good, I'm relieved ;)
- gjsman-1000 5y agoI hope you are kidding because Amazon has been very late with updating that page in the past despite problems being obvious.
- rosndo 5y agoFor once the AWS status page is actually telling the truth, there is no outage.
- inaprovaline 5y agoIt takes years to build trust with your customers, and very little to disrupt it. I think these recent events will have people rethink their cloud strategy. This could be a good opportunity for Google to take on Amazon.
- moonchrome 5y agoI'm not usually the person to complain about UI performance but Google cloud admin is the slowest trash I had to use in a while, even Azure is better. So here's me hoping it doesn't pick up steam.
- monkeybutton 5y agoThe GCP admin console is painfully slow and yet is still somehow faster than the new Google Ads UI.
- delfinom 5y agoThe GCP admin console once created an impossible to delete cloud function and I spent 2 months convincing support that I knew how to click a button.
- motoboi 5y agoAzure CLI is absolutely insanely bad. Google CLI is great. Aws is trying to get better. GUI on the three sites is awful.
- pensatoio 5y agoI used to complain about the AWS UI, then I switched to GCP...
- remote-dev 5y agoIt has been awhile since I used the GCP admin, but I am intimately familiar with how awful the Azure UI is... is it really better than GCP? I have never liked how Azure seems to be designed as if you're in a virtual machine or something. The horizontal scrolling is a mess.
- tyingq 5y agoI would be curious to hear what any current or former AWS employees think the internal consequences become for this. From the outside, it feels like they are going to have to do something different to get some customer confidence back. Some sort of "mea culpa" with an explanation of what they are going to change.
- manojlds 5y agoJeff Bezos comes back /s
- oonis 5y agoIt’ll be a lot of finger pointing and teams covering their own ass
- tyingq 5y agoThat seems to be the status quo. I'm curious if the current trend is enough to break that tradition.
- sgt 5y agoOr maybe breaking stuff is celebrated? For every type of downtime there will be lessons learned.
- cconstantine 5y agoEvery place I'v worked has celebrated fixing things that have broken, and forces engineers to fight for prevention. It's not exactly celebrating things breaking, but it sure looks a lot like it.
- Arainach 5y agoThat's only true if lessons are learned. If an outage happens and you do nothing to fix it (or worse, do something entirely too specific without fixing the real problem and thinking you've done it) the exact same problem will happen again and you'll get another opportunity to not learn the lesson.
- 5y ago
- deleted 5y ago[deleted]
- k__ 5y agoThat's because they're running on-premise.
- dainiusse 5y agoThat's an old joke already
- mr-karan 5y agoAS6453 is having a mega outage according to https://www.thousandeyes.com/outages/ https://www.thousandeyes.com/outages/. Lot of services for users in India were inaccessible for ~1h or so.
- markus_zhang 5y agoSome cloudfare seem to be down too?
- cowsandmilk 5y agoSo not AWS, but internet outage in general in India?
- JCM9 5y ago…or there are Internet issues that are causing issues for people. Do we actually know AWS is having issues or people are speculating? Not seeing any issues here, but am seeing people reporting broader internet issues at the moment. Post title seems a bit quick on the trigger to point blame.
- organic_popcorn 5y agoI have been wondering if Charlie Bell leaving AWS has anything to do with these recent outages. His ops meetings were fucking brutal. I’m curious how they are since he left.
- greatpostman 5y agoA lot of senior AWS employees have left. What the root causes of these outages are is a well guarded secret. I work at Amazon and no one I know knows anything in detail about what went down and why
- ferdowsi 5y agoWhy isn't there any investigative journalism looking into wtf is happening at AWS? What the point of all these tech journalists writing all those puff pieces for access if they're not going to use it to pierce the NDA shield at times like this?
- hotpotamus 5y agoI wonder how many are hosted on AWS and if that has any influence on whether they'd be willing to investigate.
- albert_e 5y agoBeing merely hosted on AWS should not be an influence at all.
- hotpotamus 5y agoI agree that it should not, but would you want to bite the hand that hosts you?
- joconde 5y ago“Breaking: AWS censors website that criticized them” As if they didn’t already get enough negative attention.
- johnday 5y agoIf there is sufficiently competent investigative journalism going on, you wouldn't know it either way until a report was released.
- chrisseaton 5y agoYou can't figure these kind of outages out from the outside, and Amazon aren't going to explain it you except for their official explanations. Plus - 'access' means doing what the subject tells you, certainly not doing anything investigative.
- 5y ago
- throwaway82931 5y agoI've always heard about technical debt at AWS, with thousand line functions everyone is terrified to touch, demotivated employees doing whatever it takes to close tickets without dealing with the root causes, and so on. I presume that this rash of downtime is just the rickety structure inevitably creaking and breaking. It's probably too late now to fix things — dealing with legacy code requires patience, discipline, and understanding which the management of Amazon doesn't have. They'll just yell at people louder, and hold people "accountable" by punishing anyone where anything breaks.
- mogul100 5y ago
- deleted 5y ago[deleted]
- goldenkey 5y agoI worked there and can vouch for what you said. AWS is built like a house of cards. They just painted it orange and white to make it look cohesive. What's funny though is that the CSS isn't even sourced from one place. They tell front-end engineers to use a dropper tool and to just copy and paste style from other places. DRY is not in Amazon's vocabulary. Gotta love their leadership principles: > Frugality (we'll give you a shitty laptop, shitty chair, and a shitty desk.) > Be right a lot (are you too stupid to forecast the future?!)"
- sergiotapia 5y agoI never ever want to hear people disparage DigitalOcean or Linode ever again. The "cloud" has now been objectively proven to be just as flakey.
- deleted 5y ago[deleted]
- rosndo 5y agoThere is no AWS outage, just routing issues with shit asian ISPs affecting connectivity to many providers. Amazon has nothing whatsoever to do with this. Imagine Comcast or Verizon fucking up their configs, you might not be able to access AWS stuff but that doesn’t mean that AWS is down. But hey, the real story wouldn’t make the front page, so lets just stick to the fabricated narrative.
- qbasic_forever 5y agoAnyone deploying to production Christmas eve, or even the week of Christmas, should have to get direct approval from the CTO of the company to do so. Just a terrible, terrible idea.
- oleg_antonyan 5y agoperfect idea if you're small b2b company b/c the trafic is super low
- kw-maller 5y agoI don't know about "4th time this month alone" which seems a bit editorialised, but our company is off AWS for now, and most likely for good. We're not in the cloud or tech business per se, and as such our customers are not really understanding of technical issues which unfortunately means they are blaming us, and our own reputation is on the line because of AWS' shortcomings. We did consider Google but for now OVH is the only major provider which is both reliable and secure (w.r.t. court-and-gag letters from government and intelligence agencies) as far as we're concerned. We still use Hetzner and Scaleway for some older stuff and also because of Scaleway's low prices, but it's likely we're moving everything to OVH in the future.
- blibble 5y agomeanwhile my raspberry pi cluster at home on residential FTTP has 100% uptime over the last 2 years (not entirely serious)
- authed 5y agoBut you know it wouldn't if it had to rely on your home connection AND Amazon. It's like all those major website that rely on 10 domains to serve one webpage...
- mogul100 5y ago
- tillinghast 5y agoThis looks to be an ISP issue and not AWS.
- sathyabhat 5y ago> 10:41 AM PST Between 8:59 AM and 9:32 AM PST and between 9:40 AM and 10:16 AM PST we observed Internet connectivity issues with a network provider outside of our network in the AP-SOUTH-1 Region. This impacted Internet connectivity from some customer networks to the AP-SOUTH-1 Region. Connectivity between EC2 instances and other AWS services within the Region was not impacted by this event. The issue has been resolved and we continue to work with the external provider to ensure it does not reoccur. https://status.aws.amazon.com/#AP_block https://status.aws.amazon.com/#AP_block
- yosefjaved1 5y agoAccording to ThousandEyes, it looks like it was AS16509 that was affected. The outage lasted for 28 min; however, it didn't seem to affect many applications for too long. I think it's because there was a backup server that kicked in for AS16509; however, it also went down but only for 8 min. https://www.thousandeyes.com/outages/ https://www.thousandeyes.com/outages/
- yosefjaved1 5y agoAccording to ThousandEyes, the bigger outage though is TATA Communications server AS6453. That was down or at least having issues of some sort for 1 hr 11 min.
- markus_zhang 5y ago212 cases doesn't seem to be a lot. Does anyone know if it's legit? If they are burning left and right the 9s are going to leave soon.
- gregjor 5y agoThese threads always remind me of Umberto Eco's essay Sports Chatter. He describes the degress of participation in sport: 1. Playing the game yourself. 2. Watching other people play. 3. Talking about people playing the game. 4. Listening to someone else talk about people playing the game (i.e. sports commentators). 5. Talking about what sports commentators have to say. In sports and in threads like this we're mostly at 3, 4, or 5. Even people who work at AWS posting here may only be "watching," i.e. working with second-hand information. While it's entertaining to nerds to speculate about root causes and possible solutions, and the dire predicted fallout from problems (especially when they happen to companies they dislike), it's just chatter and doesn't actually address any real issue, or inform any decisions that mean anything.
- trembonator 5y ago
- ryrymcfly 5y agoIt's almost as though it were a bad idea to put all the eggs in one basket