8 ms·
Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow
by indoordin0saur 1y ago
Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.
- wavemode 1y agoSEV-0 for my company this morning. We can't connect to RDS anymore.
- assholesRppl2 1y agoYep, confirmed worse - DynamoDB now returning "ServiceUnavailableException"
- perching_aix 1y agoIn addition to those, Sagemaker also fails for me with an internal auth error specifically in Virginia. Fun times. Hope they recover by tomorrow.
- autophagian 1y agoYeah. We had a brief window where everything resolved and worked and now we're running into really mysterious flakey networking issues where pods in our EKS clusters timeout talking to the k8s API.
- cj00 1y agoYeah, networking issues cleared up for a few hours but now seem to be as bad as before.
- JCM9 1y agoAgree… still seeing major issues. Briefly looked like it was getting better but things falling apart again.
- ljdtt 1y agofirst time i see "fubar", is that a common expression on the industry? jsut curious (english is not my native language)
- parliament32 1y agoYes, although it's military in origin.
- vishnugupta 1y agoIt used to be quite common but has fallen out of usage.
- strictnein 1y agoFUBAR: Fucked Up Beyond All Recognition Somewhat common. Comes from the US military in WW2.
- sorentwo 1y agoIt is an old US military term that means “F*ked Up Beyond All Recognition”
- dingnuts 1y agoFUBAR being a bit worse than SNAFU: "situation normal: all fucked up" which is the usual state of us-east-1
- D-Coder 1y agoMy favorite is JANFU: Joint Army-Navy Fuck-Up.
- joeyphoen 1y agoBut you probably have seen the standard example variable names "foo" and "bar" which (together at least) come from `fubar`
- worik 1y ago
- whaleofatw2022 1y agoDangerous curiosity ask, is whether the number of folks off for Diwali is a factor or not? I.e. lots of folks that weren't expected to work today and/or trying to round them up to work the problem.
- junon 1y agoSeeing as how this is us-east-1, probably not a lot.
- redeux 1y agoI believe the implication is that a lot of critical AWS engineers are of Indian descent and are off celebrating today.
- herewulf 1y agojunon's implication may be that AWS engineers of Indian descent would tend to be located on the West Coast.
- jmedefind 1y agoNorth Virginia has a very large Indian community. All the schools in the area have days off for Indian Holidays since so many would be out of school otherwise.
- hinkley 1y agoThis broke in the middle of the day IST did it not? Why would you start waking up people in VA if it’s 3 in the morning there if you don’t have to?
- AutoDunkGPT 1y agoI bet you haven't gotten an email back from AWS support during twilight hours before. There are 153k Amazon employees based in India according to LinkedIn.
- Forricide 1y agoDefinitely seems to be getting worse, outside of AWS itself, more websites seem to be having sporadic or serious issues. Concerning considering how long the outage has been going.
- busymom0 1y agoThat's probably why Reddit has been down too
- PeterCorless 1y agoDowndetector had 5,755 reports of AWS problems at 12:52 AM Pacific (3:53 AM Eastern). That number had dropped to 1,190 by 4:22 AM Pacific (7:22 AM Eastern). However, that number is back up with a vengeance. 9,230 reports as of 9:32 AM Pacific (12:32 Eastern). Part of that could be explained by more people making reports as the U.S. west coast awoke. But I also have a feeling that they aren't yet on top of the problem.
- rogerrogerr 1y agoWhere do they source those reports from? Always wondered if it was just analysis of how many people are looking at the page, or if humans somewhere are actually submitting reports.
- jedberg 1y agoIt's both. They count a hit from google as a report of that site being down. They also count that actual reports people make.
- hunter2_ 1y agoSo if my browser auto-completes their domain name and I accept that (causing me to navigate directly to their site and then I click AWS) it's not a report; but if my browser doesn't or I don't accept it (because I appended "AWS" after their site name) causing me to perform a Google search and then follow the result to the AWS page on their site, it's a report? That seems too arbitrary... they should just count the fact that I went to their AWS page regardless of how I got to it.
- jedberg 1y agoI don't know the exact details, but I know that hits to their website do count as reports, even if you don't click "report". I assume they weight it differently based on how you got there (direct might actually be more heavily weighted, at least it would be if I was in charge).
- 1y ago
- outworlder 1y agoI'm wondering why your and other companies haven't just evicted themselves from us-east-1. It's the worst region for outages and it's not even close. Our company decided years ago to use any region other than us-east-1. Of course, that doesn't help with services that are 'global', which usually means us-east-1.
- lordnacho 1y agoIs there some reason why "global" services aren't replicated across regions? I would think a lot of clients would want that.
- bcrosby95 1y agoGlobal replication is hard and if they weren't designed with that in mind its probably a whole lot of work.
- ineedasername 1y agoI thought part of the point of using AWS was that such things were pretty much turnkey?\
- DevelopingElk 1y agoMostly AWS relies on each region being its own isolated copy of each service. It gets tricky when you have globalized services like IAM. AWS tries to keep those to a minimum.
- rhplus 1y agoData residency laws may be a factor in some global/regional architectures.
- lordnacho 1y agoSo provide a way to check/uncheck which zones you want replication to. Most people aren't going to need more than a couple of alternatives, and they'll know which ones will work for them legally.
- mjrpes 1y agoDown detector agrees: https://downdetector.com/status/amazon/ https://downdetector.com/status/amazon/ Amazon says service is now just "degraded" and recovering, but searching for products on Amazon.com still does not work for me. https://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status
- ilamont 1y agoSearch, Seller Central, Amazon Advertising not working properly for me. Attempting to access from New York. When this is fixed, I am very interested in seeing recorded spend for Sunday and Monday.
- ayhanfuat 1y agoAmazon Ads is down indeed https://status.ads.amazon.com/ https://status.ads.amazon.com/
- jmuguy 1y agoYeah we were fine until about 1030 eastern and have been completely down since then, Heroku customer.
- markus_zhang 1y agoIt has been quite a while, wondering how many 9s are dropped. 365 day * 24 * 0.0001 is roughly 8 hours, so it already lost the 99.99% status.
- codeduck 1y agoI'm sure they'll find some way to weasel out of this.
- alex_young 1y agoIt's not down time, it's degradation. No outage, just degradation of a fraction[0] of the resources. [0] Fraction is ~ 1
- indoordin0saur 1y agoThis 100% seems to be what they're saying. I have not been able to get a single Airflow task to run since 7 hours ago. Being able to query Redshift only recently came back online. Despite this all their messaging is that the downtime was limited to some brief period early this morning and things have been "coming back online". Total lie, it's been completely down for the entire business day here on the east coast.
- randomname11 1y agoWe continue to see early signs of progress!
- 8organicbits 1y agoCheck the terms of your contract. The public terms often only offer partial service credit refunds, if you ask for it, via a support request.
- Keyframe 1y agoIt doesn't count. It's not downtime, it's unscheduled maintenance event.
- mvdtnz 1y agoThe problems now seem mostly related to starting new instances. Our capacity is slowly decaying as existing services spin down and new EC2 workloads fail to start.
- steveBK123 1y agoAgreed, every time the impacted services list internally gets shorter, the next update it starts growing again. A lot of these are second order dependencies like Astronomer, Atlassian, Confluent, Snowflake, Datadog, etc... the joys of using hosted solutions to everything.
- hinkley 1y agoBefore my old company spun off, we didn’t know the old ops team had put on-prem production and our Atlassian instances in the same NAS. When the NAS shit the bed, we lost half of production and all our run books. And we didn’t have autoscaling yet. Wouldn’t for another 2 years. Our group is a bunch of people that has no problem getting angry and raising voices. The whole team was so volcanically angry that it got real quiet for several days. Like everyone knew if anyone unclenched that there would be assault charges.
- throwaway-aws9 1y agoYou have to remember that health status dashboards at most (all?) cloud providers require VP approval to switch status. This stuff is not your startup's automated status dashboard. It's politics, contracts, money.
- hinkley 1y agoWhich makes them a flat out lie since it ceases to be a dashboard if it’s not live. It’s just a status page.
- davedx 1y agoAndy Jassy is the Tim Cook of Amazon Rest and vest CEOs
- hinkley 1y agoDon’t insult Tim Cook like that. He got a lot of impossible shit done as COO. They do need a more product minded person though. If Jobs was still around we’d have smart jewelry by now. And the Apple Watch would be thin af.
- napolux 1y agoworst of all: ring alarm unstoppable siren because the app is down and the keyboard was removed by my parents and put "somewhere in the basement".
- bartread 1y agoIs it hard wired? If so, and if the alarm module doesn’t have an internal battery, can you go to the breaker box and turn off the circuit it’s on? You should be able to switch off each breaker in turn until it stops if you don’t know which circuit it’s on. If it doesn’t stop, that means it has a battery backup. But you can still make life more bearable. Switch off all your breakers (you probably have a master breaker for this), then open up the alarm box and either pull the battery or - if it’s non-removable - take the box off the wall, put it in a sealed container, and put the sealed container somewhere… else. Somewhere you can’t hear it or can barely hear it until the battery runs down. Meanwhile you can turn the power back on but make sure you’ve taped the bare ends of the alarm power cable, or otherwise electrically insulated them, until you’re able to reinstall it.
- napolux 1y agoI'll keep it in mind, thx. I was lucky to find the keypad in the "this is the place where we put electronic shit" in the basement.
- bartread 1y agoNice. Well, whatever, I’m glad you managed to stop it from driving you up the wall.
- Liftyee 1y agoI have a Ring alarm. It has a battery backup and is powered by AC adaptor, so no need to turn off entire circuits (but no easy silence). All the sensors I have are wireless (not sure if they offer wired). I would honestly do your box option. Stuff it in there with some pillows and leave it in the shed for a while.
- belter 1y agoThis looks like one their worst outage in 15 years and us-east-1 still shows as degraded but I had no outages, as dont use us-east-1. Are you seeing issues on other regions? https://health.aws.amazon.com/health/status?path=open-issues https://health.aws.amazon.com/health/status?path=open-issues The closest to their identification of a root cause seems to be this one: "Oct 20 8:43 AM PDT We have narrowed down the source of the network connectivity issues that impacted AWS Services. The root cause is an underlying internal subsystem responsible for monitoring the health of our network load balancers. We are throttling requests for new EC2 instance launches to aid recovery and actively working on mitigations."
- hinkley 1y agoI wonder how many people discovered their autoscaling settings went batshit when services went offline, either scaling way down or way up, or went metastable and started fishtailing.
- tlogan 1y agoI noticed the same thing and it seems to have gotten much worse around 8:55 a.m. Pacific Time. By the way, Twilio is also down, so all those login SMS verification codes aren’t being delivered right now.
- baubino 1y agoBasic services at my worksite have been offline for almost 8 hours now (things were just glitchy for about 4 hours before that). This is nuts.
- indoordin0saur 1y agoHave not gotten a data pipeline to run to success since 9AM this morning when there was a brief window of functioning systems. Been incredibly frustrating seeing AWS tell the press that things are "effectively back to normal". They absolutely are not! It's still a full outage as far as we are concerned.
- jread 1y agoLambda create-function control plane operations are still failing with InternalError for us - other services have recovered (Lambda, SNS, SQS, EFS, EBS, and CloudFront). Cloud availability is the subject of my CS grad research, I wrote a quick post summarizing the event timeline and blast radius as I've observed it from testing in multiple AWS test accounts: https://www.linkedin.com/pulse/analyzing-aws-us-east-1-outage-jason-read-a1pfc/ https://www.linkedin.com/pulse/analyzing-aws-us-east-1-outag...
- jonplackett 1y agoThe problem is now that, what’s anyone going to do? Leave? I remember a meme years ago about Nestle. It was something like: GO ON, BOYCOT US - I BET YOU CAN’T - WE MAKE EVERYTHING. Same meme would work for Aws today.
- MaKey 1y ago> Same meme would work for Aws today. Not really, there are enough alternatives.
- jonplackett 1y agoHow any just run on AWS underneath though? And it’s not lie there aren’t other brands of chocolate either…
- hinkley 1y agoIt’s amazing how much you can avoid them by eating food that still looks like what it started as though. They own a lot of processed food.