13 ms·
AWS's us-east-1 region is experiencing issues
- mtrunkat 5y agoIn our case (Apify.com) there was a complete outage of SQS (15mins+), most likely DNS problems + EC2 instances got restarted probably as a result of an SQS outage. EDIT: Also AWS Lambda seems to be down and AWS EC2 APIs having a very high error rate and machines slow startup times.
- BigGreenTurtle 5y agoYep, I saw empty responses for sqs.us-east-1.amazonaws.com for a while. Seems okay now though.
- lyjackal 5y agonoticed issues with SQS for a couple minutes. Errors from java sdk, `com.amazonaws.SdkClientException: Unable to execute HTTP request: sqs.us-east-1.amazonaws.com`
- PeterBarrett 5y agoSQS went down for us in us-east-1 and we lost health checks on instances there. Fully recovered now.
- asah 5y agous-east-1 again!
- deleted 5y ago[deleted]
- xilni 5y agoThis is why you are strongly urged not to rely on one region or AZ.
- pid-1 5y agoGiven the total amount of money I've lost due a single AZ being down, it was totally worth it to NOT go multi az or multi region so far. Multi AZ isn't that hard, but generally requires extra costs (one nat gw per az, etc...) But multi region in AWS is a royal pain in the ass. Many services (like SSO) do not play well with multi region setups, making things really complicated even if you IaCed your whole stack.
- evrydayhustling 5y agoThose costs are the actual reason you are encouraged to go multi-AZ! (I actually love that we have strategies and infrastructure for multi-region... it just tends to come up at scales and for applications where it is not justified.)
- systemvoltage 5y agoSeems like it would be conflict of interest to increase robustness of single AZ (so it never goes down or has its own redundancy) vs. increased revenues from multi AZ deployment. What's the point of cloud if we have to manage robustness of their own infrastructure. I can understand if that's due to natural disasters and earthquakes, but the idea should be that a single AZ should never go down barring extraordinary circumstances. AWS should be auto-balancing, handling downtimes of a single AZ without the customer ever noticing it. It might not be a good analogy, but if a single Cloudflare edge datacenter goes down, it will automatically route traffic through others. Transparent and painless to the customer. I understand AWS is huge, and different services have different redundancy mechanisms, but just conceptually it feels like they're in a conflict of interest to increase robustness of their data centers - "We told you to have multi-AZ deployment, not our fault". Another way to put this is make sure as an AWS customer, to 3x multiply all costs + management of multi-AZ deployment into your total costs.
- thedougd 5y agoThey would simply charge for the privilege. An EC2 'always on' or whatever option that enabled your instance to live migrate between availability zones would be a nice and expensive option.
- csdvrx 5y agoAs usual?
- operator1 5y agoWhat’s up with all of the multi-platform outages lately? Seems abnormal looking at historical data. Are there issues affecting the internet backbone or something? Or just a coincidence?
- 9wzYQbTYsAIc 5y agoElevated risk of cyberattacks due to foreign meddling. https://www.cisa.gov/shields-up https://www.cisa.gov/shields-up
- btgeekboy 5y agoThis is a pretty big claim to make. Do you have any sources that back it up?
- 9wzYQbTYsAIc 5y agoIndeed I do. It is public information within America that we are to be at “Shields Up” readiness.
- super_linear 5y agoAbsolutely no way to prove this but maybe Q1 deadlines coming up and people trying to launch things and make changes?
- 300bps 5y agoImportant to keep in mind that AWS has 250 services in 84 Availability Zones in 26 regions. This outage is reportedly impacting 5 services in 1 region. For those impacted, pretty terrible. But as a heavy user of AWS, I’ve seen these notices posted multiple times on HN and haven’t been impacted by one yet.
- quxbar 5y agoFor businesses with uptime guarantees and lots of boxes to spin up in failover scenario, this has been a very eventful 12 months. At least that's what I'm experiencing.
- temptemptemp111 5y ago
- saltypal 5y agoBased on our telemetry, this started as NXDOMAINs for sqs.us-east-1.amazonaws.com beginning in modest volumes at 20:43 UTC and becoming a total outage at 20:48 UTC. Naturally, it was completely resolved by 20:57, 5 minutes before anything was posted in the "Personal Health Dashboard" in the AWS console. It takes a while to find a Vice President, I guess.
- mcqueenjordan 5y agoOr perhaps triaging, root-causing, and fixing the issue is the highest-order bit?
- nostrebored 5y agoIt definitely is. For an issue like this, you will see relevant teams and delegates looped in very quickly. Getting approved wording about an outage requires some very senior people though. Often they have to be paged in as well. Having worked at a few other large tech companies now -- Amazon's incident response process is honestly great. It's one of the things I miss about working there.
- saltypal 5y agoThis. We have a 4-person team and posted our own incident about this 7 minutes before Amazon did. Surely they can aim a little higher.
- ElevenLathe 5y agoIME, this actually becomes more challenging as a company gets larger, not less (but that doesn't mean it can't be done).
- smachiz 5y agosure, but if those people are updating the status pages to say something isn't right and we're looking into it, we're doomed.
- 5y ago
- halestock 5y agoI can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing? Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.
- zwirbl 5y agoJust like the tech priests in Warhammer 40k, keeping occult old engineering, thatno one could build anymore, running
- viraptor 5y agoNot familiar with 40k. Was it a similar idea to nuclear-power-as-religion from Foundation?
- aaronax 5y agoOr how about Anathem, with the Ita class doing computer things and nuclear materials cared for by a select group?
- 9wzYQbTYsAIc 5y agoThat was probably one of Stephenson’s most insightful moments of science fiction. Scientists only get to talk to the public every 100 years or something, wasn’t it? IT was never allowed to talk to scientists. Seemed like a modernist idea, even at the time of publishing.
- atty 5y agoNot far off. The “golden age” of humanity was shattered long ago, with the mortal wounding of the god emperor, and knowledge of most of the greatest technology was lost.Millennia later, a cult has grown up that both worships and maintains technology as having machine spirits, which are somehow linked to the machine god itself. That god may or may not be the same or related to the god emperor of mankind, depending on the interpretation. Honestly the lore of w40k is quite fun to read, if you’re into dystopian and fantasy sci-fi.
- amar0c 5y agoMy Aruba Instant ON Ap's are "offline" (orange) even tho they work and I am online. My first tought is that some Cloud went nirvana state
- etaioinshrdlu 5y agoDoes AWS have a plan to improve this region? Do they acknowledge the problem? It's been a joke for years how bad us-east-1 is.
- consumer451 5y agoNuke the entire site from orbit It's the only way to be sure
- consumer451 5y agoJust in case PT's Stasi as a Service company ($PLTR) has a hard time parsing this, I want to make clear that this is a joke based on a quote from Aliens the movie. I am not endorsing anything violent. It is a joke. I would delete my parent comment if I could.
- easton 5y agoFrom temuze last time: "If you're having SLA problems I feel bad for you son I got two 9 problems cuz of us-east-1"
- didip 5y agoThis is why us-east-1 is perfect for chaos-testing, non-prod, environment.
- TameAntelope 5y agoYeah, if you're still running only in us-east-1 at this point, you kind of asked for it...
- fotta 5y agoSomehow AWS managed to make their new status page more opaque than the old one. It's like they want you to scroll through their gigantic list so they can fix the issue before you find the right line.
- 0xCAP 5y agoIs us-east cursed or what?!
- adenner 5y agoIt is just the one that everyone uses...
- hughrr 5y agoI am no longer surprised and this is worrying.
- jedberg 5y agoProtip to anyone building new infrastructure in AWS: If you're gonna only use one region in the US, make it us-east-2 or us-west-2. us-east-1 is their oldest and biggest region but also the least stable in the US (ok technically us-west-1 is worse but you can't get that one anymore).
- jasoneckert 5y agoMaybe the reason AWS keeps going down is because they run all their stuff on-prem...
- merb 5y agoI'm not sure if gcloud or azure would help. I run two servers on hetzner which is way cheaper than azure/gcloud they would be better off there.
- 9wzYQbTYsAIc 5y agoThey might benefit from migrating to the Azure cloud. I’ve heard that some of the Windows servers actually run faster than some of the Linux servers on Azure.
- 0xbadcafebee 5y agoMy schadenfreude is strong whenever us-east-1 goes down
- stjohnswarts 5y agoSeems fine he
- extant_lifeform 5y agoThe URA target needs to be bumped up to 25%. Churn and burn.
- karmakaze 5y agoIt's a meme by this point that us-east-1 is not 'the cloud'--it's a snowflake, a pet, etc.