79 ms·
AWS us-east-1 outage
- booleanbetrayal 5y agoWe're seeing load balancer failure in us-east-1 AZ's, so we are not exactly sure why this is being characterized as Console outage ...
- muttantt 5y agoRunning anything on us-east-1 is asking for trouble...
- kchoudhu 5y agoAt this point I have no idea why anyone would put anything in us-east-1. Also isolation is not as good as they would have you believe: I am unable to login to AWS Quicksight in us-west-2...
- crad 5y agoBeen in us-east-1 for a long time. Things like Direct Connect and other integrations aren't easy or cheap to move and when you have other, bigger priorities, moving regions is not an easy decision to prioritize.
- bradhe 5y agoMan, some conclusions are being _jumped_ to by this reply.
- InTheArena 5y agoThere is a very long history of US-east-1 being horrible. Just bad. We've told every client we can to get out of there. It's one of the oldest amazon regions, and I think too much old legacy and weird stuff happens there. Use US-west-2.
- jedberg 5y agoOr US-East-2.
- blahyawnblah 5y agoIsn't us-east-1 where they deploy everything first? And the only region that has 100% of all available services?
- nurgasemetey 5y agoIt seems that IMDB and Goodreads are also affected
- kingcharles 5y agoYeah, and Audible and Amazon.com search and the Amazon retail stores. Basically Amazon fucked all their own products too.
- crescentfresh 5y agoThe majority of our errors stem from: - writing to Firehose (S3-backed) - publishing to eventbridge - terraform commands to ECS' API are stuck/hanging Other spurious errors involving kinesis but nothing alarming. us-east-1
- duckworth 5y agoAfter over 45 minutes https://status.aws.amazon.com/ https://status.aws.amazon.com/ now shows "AWS Management Console - Increased Error Rates" I guess 100% is technically an increase.
- Slartie 5y ago"Fixed a bug that could cause [adverse behavior affecting 100% of the user base] for some users"
- sophacles 5y ago"some" as in "not all". I'm sure there are some tiny tiny sites that were unaffected because no one went to them during the outage.
- Slartie 5y ago"some" technically includes "all", doesn't it? It excludes "none", I suppose, but why should it exclude "all" (except if "all" equals "none")?
- brasetvik 5y agoI can't remember seeing problems be more strongly worded than "Increased Error Rates" or "high error rates with S3 in us-east-1" during the infamous S3 outage of 2017 - and that was after they struggled to even update their own status page because of S3 being down. :)
- schleck8 5y agoDuring the Facebook outage FB wrote something along the lines of "We noticed that some users are experiencing issues with our apps" eventhough nothing worked anymore
- remram 5y agoTheir entire infrastructure was unroutable for a while if I remember correctly. That is a euphemism if I ever saw one.
- SubiculumCode 5y agoMust be why I can't seem to access my amazon account. I thought my account had gotten compromised.
- picodguyo 5y agoFunny, I just asked Alexa to set a timer and she said there was a problem doing that. Apparently timers require functioning us-east-1 now.
- minig33 5y agoI can’t turn on my lights… the future is weird
- balls187 5y agoYeah, my chrismas tree was fugazi.
- lsaferite 5y agoAnd that is why my lighting automation has a baseline req that it works 100% without the internet and preferably without a central controller.
- rodgerd 5y agoHomeKit compatibility is a useful proxy for local API, since it's a hard requirement for HomeKit certification.
- organsnyder 5y agoI love my Home Assistant setup for this reason. I can even get light bulbs pre-flashed with ESPHome now (my wife was bemused when I was updating the firmware on the lightbulbs).
- SavantIdiot 5y agoSo basically it's just a toggle switch?
- lsaferite 5y agoMy lighting automation uses Insteon currently. My primary req is that they are smart and connected without needing a central controller o a connection to the internet. My switches all understand lighting scenes and can manage those in a P2P manner, without a central controller. The central controller is primarily used when I want to add actual automations vs. scenes. Even the central controller aspect works 100% disconnected from the internet though. I can easily layer on top any automations I like. For instance, I have my exterior lights driven my the angle of the sun. Then, on top of that I can add internet based triggers for automation as needed. This is where I add in voice assistant triggering of automation and scenes. Edit: Just to add, very simple binary automations are even possible without a central controller. Like, I have Insteon motion sensors that trigger a lighting scene when they detect motion. These are super simplistic though.
- wrren 5y agoLooks like their health check logic also sucks, just like mine.
- sayed2020 5y agoNeed google voice unlimited,
- _of 5y ago...and imdb
- soco 5y agoAnd Netflix.
- MR4D 5y agoAnd Venmo, and McDonald's, and.... This one is pretty epic (pun intended). Bad enough that Down Detector [0] shows "Reports indicate there may be a widespread outage at Amazon Web Services, which may be impacting your service." in a red alert bar at the top. [0] - https://downdetector.com/ https://downdetector.com/
- mrguyorama 5y agoIt occurs to me that it's very nice that Netflix, Youtube, and other streaming services tend to be on separate infrastructure so they don't all go down at once
- ec109685 5y agoI'm surprised Netflix is down. They are multi-region: https://netflixtechblog.com/active-active-for-multi-regional-resiliency-c47719f6685b https://netflixtechblog.com/active-active-for-multi-regional...
- stephenr 5y agoSo, we're getting failures (for customers) trying to use amazon pay from our site. AFAIK there is no "status page" for Amazon Pay, but the rest of Amazon's services seem to be a giant Rube Goldberg machine so it's hard to imagine this isn't too.
- itisit 5y agohttp://status.mws.amazon.com/ http://status.mws.amazon.com/
- stephenr 5y agoThanks.. seems to be about as accurate as the regular AWS status board is..
- itisit 5y agoThey just added a banner. My guess is they don't know enough yet to update the respective service statuses.
- stephenr 5y agoI have basically zero faith in Amazon at this point. We first noticed failures because a tester happened to be testing in an env that uses the Amazon Pay sandbox. I checked the prod site, and it wouldn't even ask me to login. When I tried to login to SellerCentral to file a ticket - it told me my password (from a pw manager) was wrong. When I tried to reset, the OTP was ridiculously slow. Clicking "resend OTP" gives a "the OTP is incorrect" error message. When I finally got an OTP and put it in, the resulting page was a generic Amazon "404 page not found". A while later, my original SellerCentral password, still un-changed because I never got another OTP to reset it, worked. What the fuck kind of failure mode is that "services are down, so password must be wrong".
- itisit 5y agoSorry to hear. If multi-cloud is the answer, I wouldn't be surprised to see folks go back to owning and operating their own gear.
- blueside 5y agoI was currently in the process of buying some tickets on Ticketmaster and the entire presale event had to be postponed for at least 4 hours due to this AWS outage. I'm not complaining, I enjoyed the nostaliga - sometimes the web still feels like the late 90s
- kingcharles 5y agoI'm complaining. I could not get my Happy Meal at McDonald's this morning. Bad start to the day.
- ec109685 5y agoSurprised Netflix is down. I thought they were hot/hot multi-region: https://netflixtechblog.com/active-active-for-multi-regional-resiliency-c47719f6685b https://netflixtechblog.com/active-active-for-multi-regional...
- ManuelKiessling 5y agoJust watched two episodes of Better Call Saul on Netflix Germany without issues (while not being able to run my Terraform plan against my eu-central-1 infrastructure…).
- skapadia 5y agoI'm absolutely terrified by our reliance on cloud providers (and modern technology, networks, satellites, electric grids, etc. in general), but the cloud hasn't been around that long. It is advancing extremely fast, and every problem makes it more resilient.
- SkyPuncher 5y agoWould you prefer to have more of something good that you can't occasionally have OR less of something good that you can always have? ---- The answer likely depends on the specific thing, but I'd argue most people would take the better version of something at the risk of it not working 1 or 2 days per year.
- skapadia 5y agoIMO, the cloud is not the overall problem. Our insatiable desire to want things right HERE, right NOW is the core issue. The cloud is just a solution to try to meet that demand.
- filip5114 5y agoCan confrim us-east-1
- romanhotsiy 5y agoCan confirm lambda is down in us-east-1. Other services seems to work for us.
- imnoscar 5y agoSTS or console login not working either.
- judge2020 5y agoGot alerted to 503 errors for SES, so it's not just the management console.
- adamtester 5y agoeu-west-1 is down for us
- PeterBarrett 5y agoI hope you stay being the only person who has said that, 1 region being gone is enough for me!
- adamtester 5y agoI should have said, only the Console and CLI was down for us, our services remained up!
- ComputerGuru 5y agoConsole has lots of us-east-1 dependencies.
- larrik 5y agoWe are having a number of rolling issues, but the site is sort of up? I worry it'll get worse before it gets better. Nothing on their status page. But the Console is not working.
- commandlinefan 5y agoWe're seeing some stuff is up, and some stuff is down and some of the stuff that was up a little while is down now. It's getting worse as of 9:53 AM CST.
- chazu 5y agoECR borked for us in east-1
- the-rc 5y agoYou can get new tokens. Image pulling times out after ~30s, which tells me that maybe ECR is actually up, but it can't verify the caller's credentials or access image metadata from some other internal service. It's probably something low level that crashed, taking down anything built above it.
- the-rc 5y agoActually, images that do not exist will return the appropriate error within a few seconds, so it's really timing out when talking to the storage layer or similar.
- albatross13 5y agoWelp, this is awkward (._. )
- sakopov 5y agoAWS Management console is dead/dying. Numerous errors across major services like S3 and EC2 in us-east-1. This looks pretty bad.
- zedpm 5y agoYep, PHD isn't loading, Cloudwatch is reporting SQS errors, metrics aren't loading, can't pull logs. This is in US-East-1.
- tonyhb 5y agoOur services that are in us-east-2 are up, but I'm wondering how long that will hold true.
- saggy4 5y agoIt seems that only console is having the problem CLI works fine
- albatross13 5y agoNot entirely true- we federate through ADFS and `saml2aws login` is currently failing with: error logging into aws role using saml assertion: error retrieving STS credentials using SAML: ServiceUnavailable: status code: 503
- zedpm 5y agoI'm having CLI issues as well, they're using the same APIs under the hood. For example, I'm getting 503 errors for cloudwatch DescribeLogGroups.
- saggy4 5y agoTried a few cli commands seems to be working fine for me. Maybe it is not for everyone or maybe It is just the start of something very worse. :(
- technics256 5y agotry aws ecr describe-registry and you will get an error
- saggy4 5y agoYes Indeed, getting failures in CLI as well
- rhines 5y agoCLI for EC2 works for me, but not ELB.
- MatthewCampbell 5y agoCloudFormation changesets are reporting "InternalFailure" for us in us-east-1.
- zegl 5y agoI love that every time this happens, 100% of the services on https://status.aws.amazon.com https://status.aws.amazon.com are green.
- strictfp 5y ago"some customers may experience a slight elevation in error rates" --> everything is on fire, absolutely nothing works
- davecap1 5y agoOn top of that, the "Personalized Health Dashboard" doesn't work because I can't seem to log in to the console.
- qudat 5y agoStatus pages are hard
- jtdev 5y agoWhy? Twitter and HN can tell me that AWS is having an outage, why can’t AWS?
- AH4oFVbPT4f8 5y agoThey sent their CEO into space, I am sure they have the resources to figure it out.
- 5y ago
- deleted 5y ago[deleted]
- davikawasaki 5y agoEKS works for us
- saggy4 5y agoI think the only the console is down. CLI is working fine for me in ap-southeast-1
- romanhotsiy 5y agoIt's funny that the first place I go to learn about the outage is Hacker News and not https://status.aws.amazon.com/ https://status.aws.amazon.com/ (it's still reports everything to be "operating normally"...)
- albatross13 5y agoYeah, I tend to go off of https://downdetector.com/status/aws-amazon-web-services/ https://downdetector.com/status/aws-amazon-web-services/
- murph-almighty 5y agoI always got the impression that downdetector worked by logging the number of times they get a hit for a particular service and using that as a heuristic to determine if something is down. If so, that's brilliant.
- albatross13 5y agoI think it's a bit simpler for AWS- there's a big red "I have a problem with AWS" button on that page. You click it, tell it what your problem is, and it logs a report. Unless that's what you were driving at and I missed it, it's early. Too early for AWS to be down :( Some 3600 people have hit that button in the last ~15 minutes.
- cmg 5y agoIt's brilliant until the information is bad. When Facebook's properties all went down in October, people were saying that AT&T and other cell phone carriers were also down - because they couldn't connect to FB/Insta/etc. There were even some media reports that cited Downdetector, seeming without understanding that they are basically crowdsourced and sometimes the crowd is wrong.
- authed 5y agoI usually go on Twitter first for outages.
- 5y ago
- etimberg 5y agoSeeing issues in ca-central-1 and us-east
- taf2 5y agoWe're not down and we're in us-east-1... maybe there is more to this issue?
- bradhe 5y agoI think could just be the console?
- abarringer 5y agoOur services in AWS East are down.
- fsagx 5y agoEverything fine with S3 in us-east-1 for me. Also just not able to access the console.
- taf2 5y agoFound that it seems Lambda is impacted
- technics256 5y agoour EKS/EC2 instances are OK
- AH4oFVbPT4f8 5y agoUnable to log into the console for us-east-1 for me too
- biohax2015 5y agoGetting 502 in Parameter Store. Cloudformation isn't returning either -- and that's how we deploy code :(
- iso1210 5y agoMy raspberry pi is still working just fine
- marginalia_nu 5y agoYeah, strange, my self-hosted server isn't affected either.
- iso1210 5y agoSeems "the cloud" had a major outage less than a month ago, my laptop has a higher uptime. $ 16:04 up 46 days, 7:02, 9 users, load averages: 3.68 3.56 3.18 US East 1 was down just over a year ago https://www.theregister.com/2020/11/25/aws_down/ https://www.theregister.com/2020/11/25/aws_down/ Meanwhile I moved one of my two internal DNS servers to a second site on 11 Nov 2020, and it's been up since then. One of my monitoring machines has been filling, rotating and deleting logs for 1,712 days with a load average in the c. 40 range for that whole time, just works. If only there was a way to run stuff with an uptime of 364 days a year without using the cloud /s
- nfriedly 5y agoI think the point of the could isn't increased uptime - the point is that when it's down, bring it back up is someone else's problem. (Also, OpEx vs CapEx financial shenanigans...) All the same, I don't disagree with your point.
- iso1210 5y ago> the point is that when it's down, bring it back up is someone else's problem. When it's down, it's my problem, and I can't do anything about it other than explain why I have no idea the system is broken and can't do anything about it. "Why is my dohicky down? When will it be back?" "Because it's raining, no idea" May be accurate, it's also of no use. But yes, Opex vs Capex, of course that's why you can lease your servers. It's far easier to spend company money with another $500 a month on AWS than spend $500 a year for a new machine.
- endisneigh 5y agoAzure, Google Cloud, AWS and others need to have a “Status alliance” where they determine the status of each of their services by a quorum using all cloud providers. Status pages are virtually useless these days
- smt88 5y agoThey can do this without an alliance. They very intentionally choose not to do it. Every major company has moved away from having accurate status pages.
- dentemple 5y agoIt's because none of these companies are held responsible for missing their actual SLAs, as opposed to their self-reported SLA compliance. So unless regulation gets implemented that says otherwise, there's zero incentive for any company to maintain an accurate status page.
- iso1210 5y agoThe only uptimes I'm concerned with are my own services that my own monitoring keeps on top of, this varies - if the monitoring page goes down for 10 seconds I'm not worried, if one leg of a smpte-2022-7 is down for a second that's fine, if it keeps going down for a second that's a concern, etc. If something I'm responsible goes down to the point that my stakeholders are complaining (which is something seriously wrong), they are not going to be happy with "oh the cloud was down, not my fault" If AWS is down or not it meaningless to me, if my service running on AWS is down or not is the key metric. If a service is down and I can't get into it, then chatter on things like outages mailing list, or HN, will let me know if it's yet another cloud failure, or if it's something that's affecting my machine only.
- xtracto 5y ago>It's because none of these companies are held responsible for missing their actual SLAs, as opposed to their self-reported SLA compliance. Right, there should be an "alliance" of customers from different large providers (something like a Union but instead of workers, it would be customers). They are the ones that should measure SLAs and hold the provider accountable.
- eoinboylan 5y agoCan't SSM into ec2 in us-east-1
- kp195_ 5y agoThe AWS console seems kind of broken for us in us-west-1 (Northern California), but it seems like the actual services are working
- misthop 5y agoGetting 500 errors across the board on systems trying to hit s3
- abaldwin99 5y agoDitto
- artembugara 5y agoI think there should be some third-party status checker alliance. It's a joke. Each time AWS/Azure/GCP is down their status page says all is fine.
- cphoover 5y agoWant to build a startup?
- artembugara 5y agoalready running one.
- synergy20 5y agoNo wonder I could not read books from amazon all of a sudden, what about their cloud-based redundancy design?
- tgtweak 5y agoThe book preview webservice or actual ebooks (kindle, etc)?
- doesnotexist 5y agoFor me, it's been that I am unable to download books to the kindle app on my computer
- doesnotexist 5y agoCan also confirm that the kindle app is failing for me and has been for the past few hours.
- lordnacho 5y agoThis got me thinking, are there any major chat services that would go down if a particular AWS/GCP/etc data centre went down? You don't want your service to go down, plus your team's comms at the same time.
- DoctorOW 5y agoSlack is pretty much full AWS, I've been switched over to Teams so I can't check.
- School-Cotton 5y agoMy company's Slack instance is currently fine.
- rodiger 5y agoSlack is working fine for me
- dahak27 5y agoEspecially if enough Amazon internal tools rely on it - would be funny if there were a repeat of the FB debacle where Amazon employees somehow couldn't communicate/get back into their offices because of the problem they were trying to fix
- School-Cotton 5y agoLast I knew, Amazon used all Microsoft stuff for business communication.
- itsyaboi 5y agoSlack, as of last year. https://slack.com/blog/news/slack-aws-drive-development-agility https://slack.com/blog/news/slack-aws-drive-development-agil...
- nostrebored 5y ago
- shaftoe444 5y agoCompany wide can't log in to console. Many, many SNS and SQS errors in us-east-1.
- _wldu 5y agoSame here and I use us-east-2.
- 7six 5y agoeu-central-1 as well
- tmarice 5y agoAWS OpenSearch is also returning 500 in us-east-1.
- WesolyKubeczek 5y agoOur instances are up (when I poke with SSH, say), but the console itself is under the weather.
- PaulHoule 5y agoMakes me glad I am in us-east-2.
- muttantt 5y agous-east-2 is truly a hidden gem
- kohanz 5y agoExcept that it had a significant outage just a couple of weeks ago. Source: most of our stuff is on us-east-2.
- heyitsguay 5y agoI'm having issues with us-east-2 now. Console is down, then when I try to sign into a particular service I just get "please try again later".
- muttantt 5y agoIt's related to east-1, looks like some single point of failure not letting you access east-2 console URLs
- newhouseb 5y agoWe're flapping pretty hard in us-east-2 (looks API Gateway related, which is probably because it's an edge deployment which has a bunch of us-east-1 dependencies).
- crad 5y agoWe make heavy usage of Kinesis Firehose in us-east-1. Issues started ~1:24am ET and resolved around 7:31am ET. Then really kicked in at a much larger scale at 10:32am ET. We're now seeing failures with connections to RDS Postgres and other services. Console is completely unavailable to me.
- m3nu 5y agoRoute53 is not updating new records. Console is also out.
- grumple 5y agoKinesis was the cause last Thanksgiving too iirc. It's the backbone of many services.
- dylan604 5y ago>Issues started ~1:24am ET and resolved around 7:31am ET. First engineer found a clever hack using bubble gum >Then really kicked in at a much larger scale at 10:32am ET. Bubble gum dried out, and the connector lost connection again. Now, connector also fouled by the gum making a full replacement required.
- mabbo 5y agoAre the actual services down, or is it just the console and/or login page? For example, the sign-up page appears to be working: https://portal.aws.amazon.com/billing/signup#/start https://portal.aws.amazon.com/billing/signup#/start Are websites that run on AWS us-east up? Are the AWS CLIs working?
- nowahe 5y agoI can't access anything related to Cloudfront, either through the CLI or console : $ aws cloudfront list-distributions An error occurred (HttpTimeoutException) when calling the ListDistributions operation: Could not resolve DNS within remaining TTL of 4999 ms However I can still access the distribution fine
- 2bitlobster 5y agoInteractive Video Service (IVS) is down too
- Guest19023892 5y agoOne of my sites went offline an hour ago because the web server stopped responding. I can't SSH into it or get any type of response. The database server in the same region and zone is continuing to run fine though.
- bijoo 5y agoInteresting, is the site on a particular type of EC2 instance, e.g. bare metal? I see c4.xlarge is doing fine in us-east-1.
- Guest19023892 5y agoIt's just a t3a.nano instance since it's a project under development. However, I have a high number of t3a.nano instances in the same region operating as expected. This particular server has been running for years, so although it could be a coincidence it just went offline within minutes of the outage starting, it seems unlikely. Hopefully no hardware failures or corruption, and it'll just need a reboot once I can get access to AWS again.
- deleted 5y ago[deleted]
- p2t2p 5y agoI have alerts going off because of that…
- nickysielicki 5y agoThe Amazon.com storefront was giving me issues loading search results — this is the worst possible time of year for Amazon to have issues. It’s horrifying and awesome to imagine hundreds of thousands (if not millions) of dollars of lost orders an hour — just from sluggish load times. Hugops to those dealing with this.
- throwanem 5y agoThird worst time. It's not BFCM and it's not the week before Christmas; from prior high-volume ecommerce experience I suspect their purchase rate is elevated at this time but nowhere near those two peaks.
- rickreynoldssf 5y agoEC2 at least seems fine but the console is definitely busted as of 16:00UTC
- Bedon292 5y agoI am still getting 'Invalid region parameter' for resources in us-east-1, the others are fine.
- valeness 5y agoThis is more than just east. I am seeing the same error on us-west-2 resources.
- singlow 5y agoI am seeing some issues but only with services that have global aspects such as s3. I can't create an s3 bucket even though I am in us-west-2 because I think the names are globally unique and creating them depends on us-east-1.
- herodoturtle 5y agoCan't access Lightsail console even though our instances are in a totally different Region.
- deleted 5y ago[deleted]
- l72 5y agoMost of our services in us-east-1 are still responding although we cannot log into the console. However, it looks like dynamodb is timing out most requests for us.
- techthumb 5y agohttps://status.aws.amazon.com https://status.aws.amazon.com hasn't been updated to reflect outage yest
- deleted 5y ago[deleted]
- soheil 5y agoCan confirm I can once again login to the Console and everything seems to be back to normal in us-east-2.
- deleted 5y ago[deleted]
- temuze 5y agoFriends tell friends to pick us-east-2. Virginia is for lovers, Ohio is for availability.
- tgtweak 5y agoIf you're not multi-cloud in 2021 and are expecting 5-9's, I feel bad for you.
- ryuta 5y agoHow do you become multi-cloud if your root domain is in Route53? Have Backup domains on the client side?
- tgtweak 5y agodns records should be synced to secondary provider, and that provider added to your secondary/tertiary domain dns. Multi-provider dns is a solved problem.
- temuze 5y agoIf you're having SLA problems I feel bad for you son I got two 9 problems cuz of us-east-1
- ShroudedNight 5y ago> ~~I got two 9 problems cuz of us-east-1~~ I left my two nines problems in us-east-1
- unethical_ban 5y agoIf you're not multi-region, I feel bad for you. If your company is shoehorning you into using multiple clouds and learning a dozen products, IAM and CICD dialects simultaneously because "being cloud dependent is bad", I feel bad for you. Doing one cloud correctly from a current DevSecOps perspective is a multi-year ask. I estimate it takes about 25 people working full time on managing and securing infrastructure per cloud, minimum. This does not include certain matrixed people from legacy network/IAM teams. If you have the people, go for it.
- PragmaticPulp 5y agoI worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his department at Amazon, being at fault for an outage was one of the worst things that could happen to you. He wanted to avoid that mark any way possible. YMMV, of course. Amazon is a big company and I've had other friends work there in different departments who said this wasn't common at all. I will always remember the look of sheer panic he had when we insisted that he update the status page to accurately reflect an outage, though.
- broknbottle 5y agoThis gets posted every time there's an AWS outage. It mind as well be a copy pasta at this point.
- Spivak 5y agoI mean it's true at every company I've ever worked at too. If you can lawyer incidents into not being an outage you avoid like 15 meetings with the business stakeholders about all the things we "have to do" to prevent things like this in the future that get canceled the moment they realize that how much dev/infra time it will take to implement.
- JoelMcCracken 5y agowell, this is the first time I've seen it, so I am glad it was posted this time.
- jjoonathan 5y agoFirst time I've seen it too. Definitely not my first "AWS us-east-1 is down but the status board is green" thread, either.
- rconti 5y ago
- deleted 5y ago[deleted]
- htrp 5y agoLooks like all aws internal APIs are down....
- taf2 5y agostatus page looks very green even 17 minutes later...
- echlipse 5y agoimdb.com is down right now. But https://www.imdb.com/chart/top https://www.imdb.com/chart/top is reachable right now. Strange. https://imgur.com/a/apBT86o https://imgur.com/a/apBT86o
- tgtweak 5y agoI'm going with "BGP error" which is likely config-related, likely human error. Seems to be the trend with the last 5-6 big cloud outages.
- RubberShoes 5y ago"AWS Management Console Home page is currently unavailable. You can monitor status on the AWS Service Health Dashboard." And then Health Dashboard is 100% green. What a joke.
- sbr464 5y agobooker.com/mindbody random areas are affected
- markus_zhang 5y agoJust curious does it still make sense to claim that up time is X numbers of 9? (e.g. 99.999%)
- throwanem 5y agoYep. In this case, zero nines.
- deleted 5y ago[deleted]
- avsteele 5y agoGetting strange errors trying to manage my amazon account right now, could this be related? 494 ERROR and "We're sorry Something went wrong with our website, please try again later."
- sirfz 5y agoAWS ec2 API requests are failing and ECR also seems to be down (error 500 on pull)
- anovikov 5y agoHaha my developer called me in panic telling that he crashed Amazon - was doing some load tests with Lambda
- imstil3earning 5y agothats cute xD
- xtracto 5y agoHow can you own a developer? is it expensive to buy one?
- DataGata 5y agoDon't get nitty about saying "my X". People say "my plumber" or "my hairstylist" or whatever all the time.
- rossdavidh 5y agoIf he actually knows how to crash Amazon, you have a new business opportunity, albeit not a very nice one...
- Justsignedup 5y agothank you for that big hearty laugh! :)
- politelemon 5y agoIt'd be hilarious if you kept that impression going for the duration of the outage.
- tgtweak 5y agoPostmortem: unbounded auto-scaling of lambda combined with oversight on internal rate limits caused unforseen internal ddos.
- throwthere 5y agoJust wait for the medium article “How I ran up a $400 million AWS bill.”
- whalesalad 5y agoI can still hit EC2 boxes and networking is okay. DynamoDB is 100% down for the count, every request is an Internal Server Error.
- jacobkg 5y agoAWS Connect is down, so our customer support phone system is down with it
- nostrebored 5y agoHighly recommend talking to your account team to recommend regional failovers and DR for Amazon Connect! With enough feedback from customers, stuff like this can get prioritized.
- jacobkg 5y agoThanks, will definitely do that!
- anonu 5y agoI think its just the console - as my EC2 in us-east-1 are still reachable.
- jrs235 5y agoI think it's affecting more than just AWS. Try searching amazon.com. That's broken for me.
- Tea418 5y agoIf you have trouble logging in to AWS Console, you can use a regional console endpoint such as https://eu-central-1.console.aws.amazon.com/ https://eu-central-1.console.aws.amazon.com/
- bennyp101 5y agoYea, Amazon Music has gone down for me in the UK now :( Looks like it might be getting worse
- taormina 5y agoHave folks considered a class-action lawsuit against these blatantly fraudulent SLAs to recoup costs?
- itsdrewmiller 5y agoIn my experience, despite whatever is published, companies will private acknowledge and pay their SLA terms. (Which still only gets you, like, one day's worth of reimbursement if you're lucky.)
- adrr 5y agoRetail SLAs are a small risk compared to the enterprise SLAs where an outage like this could cost Amazon tens of millions. I assume these contracts have discount tiers based on availability and anything below 99% would be a 100% discount for that bill cycle.
- jaywalk 5y agoBut those enterprise SLAs are the ones they'll be paying out. Retail SLAs are the ones that you'll have to fight for.
- swasheck 5y agothey're a bit later than normal with their large annual post-thanksgiving us-east-1 outage
- deleted 5y ago[deleted]
- zackbloom 5y agoI'm now getting failures searching for products on Amazon.com itself. This is somewhat surprising, as the narrative always was that Amazon didn't do a great job of dogfooding their own cloud platform.
- blahyawnblah 5y agoDoesn't amazon.com run on us-east-1?
- di4na 5y agoThey did more of it starting a few years back. It has been interesting to see how some services evolved far faster when retail started to use them. Seems that some customer are far more centric than others if you catch my drift...
- zackbloom 5y agoUpdate: I'm also getting Internal Errors trying to log into the Amazon.com site now as well.
- whoknowswhat11 5y agoMy Amazon order history showed no orders, but now is showing my orders again - so stuff seems to be getting either fixed or intermittent outages.
- m12k 5y agoA former colleague told me years ago that us-east-1 is basically the guinea pig where changes get tested before being rolled out to the other regions, and as a result is less stable than the others. Does anyone know if there's any truth to this?
- arpinum 5y agoThis is not true. Lambda updates us-east-1 last.
- wizwit999 5y agofalse, it's often 4th iirc, SFO (us-west-1) is actually usually first.
- treesknees 5y agoI can't see why they'd use the most common/popular region as a guinea pig.
- Kye 5y agoProblem: you've gone as far as you can go testing internally or with test groups. You know there are edge cases you'll only identify by having enough people test it. Solution: push it to production on the zone with the most users and see what breaks.
- sharpy 5y agoThe guideline has been to deploy to it last. If the team follows pipeline best practices, they are supposed to deploy to a single small region first, wait 24 hours, and then deploy to more, wait more, and deploy to more, until finally deploying to us-east-1.
- shepherdjerred 5y agoAt my org it was deployed in the middle, around the fourth wave iirc
- mystcb 5y agoNot sure it helps, but got this update from someone inside AWS a few moments ago. "We have identified the root cause of the issues in the US-EAST-1 Region, which is a network issue with some network devices in that Region which is affecting multiple services, including the console but also services like S3. We are actively working towards recovery."
- rozenmd 5y agoThey finally updated the status page: https://status.aws.amazon.com/ https://status.aws.amazon.com/
- mystcb 5y agoAhh good spot, it does seem that the AWS person I am speaking too has a few more bits other than what is shown on the page, they just messaged me the same message there, but added: "All teams are engaged and continuing to work towards mitigation. We have confirmed the issue is due to multiple impaired network devices in the US-EAST-1 Region." Doesn't sound like they are having a good day there!
- alpha_squared 5y agoThat's a copy-paste, we got the same thing from our AWS contact. It's just enough info to confirm there's an issue, but not enough to give any indication on the scope or timeline to resolution.
- jdc0589 5y agoI love how they are sharing this stuff out to some clients, but its technically under NDA.
- alfalfasprout 5y agoYeah, we got updates via NDA too lol. Such a joke that a status page update is considered privileged lol.
- alexatalktome 5y ago
- singlow 5y agoI am able to access the console for us-west-2 by going to a region specific URL: https://us-west-2.console.aws.amazon.com/ https://us-west-2.console.aws.amazon.com/ It does tack me to a non-region specific login page which is up, and then redirects back to us-west-2 which works. If I go to my bookmarked login page it breaks because, presumably, it hits something that is broken in us-east-1.
- imstil3earning 5y agocant scale our Kubernetes cluster due to 500s from ECR :(
- mohanmcgeek 5y agoI don't think it's a console outage. Goodreads has been down for a while
- andrew_ 5y agoThis is also effecting Elastic Beanstalk and Elastic Container Registry. We're getting 500s via API for both services.
- mohanmcgeek 5y agoSame happening right now with Goodreads
- ipmb 5y agoLooks like they've acknowledged it on the status page now. https://status.aws.amazon.com/ https://status.aws.amazon.com/ > 8:22 AM PST We are investigating increased error rates for the AWS Management Console. > 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US-EAST-1. Customers may be able to access region-specific consoles going to https://console.aws.amazon.com/ https://console.aws.amazon.com/. So, to access the US-WEST-2 console, try https://us-west-2.console.aws.amazon.com/ https://us-west-2.console.aws.amazon.com/
- bobviolier 5y agohttps://status.aws.amazon.com/ https://status.aws.amazon.com/ still shows all green for me
- banana_giraffe 5y agoIt's acting odd for me. Shows all green in Firefox, but shows the error in Chrome even after some refreshes. Not sure what's caching where to cause that.
- junyoon 5y agofirefox has more aggressive caching than other browsers I think
- jeremyjh 5y agoThey are still lying about it, the issues are not only affecting the console but also AWS operations such as S3 puts. S3 still shows green.
- lsaferite 5y agoIt's certainly affecting a wider range of stuff from what I've seen. I'm personally having issues with API Gateway, CloudFormation, S3, and SQS
- joshstrange 5y agoThis seems to be affecting Audible as well. I can't buy a book which sucks since I just finished the previous one in the series and I'm stuck in bed sick.
- 2bitlobster 5y agoI wonder what the cost is on the US economy
- stoneham_guy 5y agoAWS Management Console Home page is currently unavailable. That's the error I am getting when logging to aws console
- stoneham_guy 5y agoAWS Management Console Home page is currently unavailable.
- dimitar 5y agohttps://status.hashicorp.com/incidents/3qc302y4whqr https://status.hashicorp.com/incidents/3qc302y4whqr - seems to have affected Hashicorp Cloud too
- joelbondurant 5y agoThey should put everything in the cloud so hardware issues can't happen.
- johnsimer 5y agoEverything seems to be functioning normally for me now
- jrs235 5y agoSearch on Amazon.com seems to be broken too. This doesn't appear to just be affecting their AWS revenue.
- authed 5y agoI've had issues login-in at amazon.com too... and IMDB is also down
- jrochkind1 5y agoThis is effecting heroku. While my heroku apps are currently up, I am unable to push new versions. Logging in to heroku dashboard (which does work), there is a message pointing to this heroku status incident for "Availability issues with upstream provider in the US region": https://status.heroku.com/incidents/2390 https://status.heroku.com/incidents/2390 How can there be an outage severe enough to be effecting middleman customers like heroku, but the AWS status page is still all green?!?! If whoever runs the AWS status page isn't embaressed, they really ought to be.
- VWWHFSfQ 5y agoAWS management APIs in the us-east-1 region is what is affected. I'm guessing Heroku uses at least the S3 APIs when deploying new versions, and those are failing (intermittently?). I advise not touching your Heroku setup right now. Even something like trying to restart a dyno might mean it doesn't come back since the slug is probably stored on S3 and that will fail.
- zahil 5y agoIs it just me or is it becoming more and more frequent, outages involving large websites?
- pixelmonkey 5y agoLooks like Kinesis Firehose is either the root cause, or severely impacted: https://twitter.com/amontalenti/status/1468265799458639877 https://twitter.com/amontalenti/status/1468265799458639877 Segment is publicly reporting issues delivering to Firehose, and one of my company's real-time monitors also triggered for Kinesis Firehose an hour ago. Update: By my sniff of it, some “core” APIs are down for S3 and EC2 (e.g. GET/PUT on S3 and node create/delete on EC2). Systems like Kinesis Firehose and DynamoDB rely on these APIs under the hood (“serverless” is just “a server in someone else’s data center”). Further update: There is a workaround available for the AWS Console login issue. You can use https://us-west-2.console.aws.amazon.com/ https://us-west-2.console.aws.amazon.com/ to get in -- it's just the landing page that is down (because the landing page is in the affected region).
- deleted 5y ago[deleted]
- soheil 5y agoCan't even login this is the error I'm getting: Internal Error Please try again later
- deleted 5y ago[deleted]
- john37386 5y agoIt seems a bit long to fix! They probably paint themselves in a corner just like facebook few weeks ago. This make me think; Could it be that one day the internet will have a total global outage and it will take few days to recover?
- rossdavidh 5y agoIf we have a total global outage, Stack Overflow will be unavailable, and the internet will never be fixed. :) Mostly joking, I hope...
- jaywalk 5y agoSome brave soul at Stack Overflow will have to physically go into the datacenter, roll up a cart with a keyboard, monitor and printer and start printing off a bunch of Networking answers.
- lesam 5y agoThe StackOverflow datacenter is famously tiny - like, 10 Windows servers. So even if the rest of the internet goes down hopefully they stay up. They might have to rebuild the internet out from their server room though.
- orhmeh09 5y agoLarge portions of stackoverflow are archived and available for download with the Kiwix app (typically used for offline Wikipedia). Same with the arch wiki. Might keep it around for a rainy day.
- iso1210 5y agoI'm not sure how you get a total global outage in a distributed system. Lets say a major transit provider (Century link for example) advertises traffic as a "go via me", but then drops the traffic, lets also assume it drops the costs of routes to pretty much zero. That would certainly have a major effect, until their customers/peers stop their peers. That might be tricky if they are remote and not on the same AS as their router access points and have no completely out of band access, but you're still talking hours at most.
- mbordenet 5y agoI suspect the ex-Amazonian PragmaticPulp cites was let go from Amazon for a reason. The COE process works, provided the culture is healthy and genuinely interested in fixing systemic problems. Engineers who seek to deflect blame are toxic and unhelpful. Don't hire them!
- jonnycomputer 5y agoAh, this might explain why my AWS requests were so slow, or timing out, this afternoon.
- whoisjuan 5y agous-east-1 is so unreliable that it probably should be nuked. It's the region with the worst track record. I guess it doesn't help that is one of the oldest.
- zrail 5y agoUS-EAST-1 is the oldest, largest, and most heterogeneous region. There are many data centers and availability zones within the region. I believe it's where AWS rolls out changes first, but I'm not confident on that.
- JamesAdir 5y agonoob question: Aren't companies using several regions for availability and redundancy?
- blahyawnblah 5y agoIdeally, yes. In practice, most are hosted in a single region but with multiple availability zones (this is called high availability). What you're talking about is fault tolerance (across multiple regions). That's harder to implement and costs more.
- yunwal 5y agoI'm seeing outages across several regions for certain services (SNS), so cross-region failover doesn't necessarily help here. Additionally, for complex apps, automatic cross-region disaster recovery can take tens or even hundreds of dev years, something most small to midsized companies can't afford.
- alephnan 5y agoVanguard has been slow all day. I'm going to guess Vanguard has a depencency on US-east-1
- AzzieElbab 5y agoBetting on dynamo again
- numberwhun 5y agoAmazon is having an outage in us-east-1, and it is bleeding over elsewhere, like eu: https://status.aws.amazon.com/ https://status.aws.amazon.com/
- alex_young 5y agoEDIT: As pointed out below, I missed that this was for the Amazon Connect service, and not an update for all of US-EAST-1. Preserved for consistency, but obviously just a comprehension issue on my side. At least the updates are amusing: "9:18 AM PST We can confirm degraded Contact handling by agents in the US-EAST-1 Region. Agents may experience issues logging in or being connected with end-customers." WTF is "contact handling", an "agent" or an "end-customer"? How about something like "We are confirming that some users are not able to connect to AWS services in us-east-1. We're looking into it."
- hiyer 5y agoPayments in Amazon India is down - likely because of this.
- _moof 5y agoThis is your regularly scheduled reminder: https://www.whoownsmyavailability.com/ https://www.whoownsmyavailability.com/
- andreyazimov 5y agoMy payment processor Paddle also stopped working.
- whoknowswhat11 5y agoAnyone understand why these services go down for so long? That's the part I find interesting.
- woshea901 5y agoN. Virginia consistently has more problems than other zones. Is it possible this zone is also hosting government computers/could it be a more frequent target for this reason?
- longhairedhippy 5y agoThe real reason is us-east-1 was the first and by far the biggest region, the same reason that new services always launch there but other regions are are not necessarily required (some services have to launch in every region). The us-east-1 region is consistently pushing the limits of scale for the AWS services, thus is has way more problems than other regions.
- saisundar 5y agoYikes, ring, the security system is also down. Wonder if crime rates might eventually spike up if aws goes down, in an utopian world where Amazon gets everyone to use ring.
- kingcharles 5y agoI'm now imagining a team of criminals sitting around in face masks and hoodies refreshing AWS status page all day... "AWS is down! Christmas came early boys! Roll out..."
- errcorrectcode 5y agoAlexa (jokes and flash briefing) is currently partially down for me. Skills and routines are working. Amazon customer service can't help me handle an order. It us-east-1 is out, then half of Amazon is out too.
- bkirkby 5y agofwiw, we are seeing errors when trying to publish to SNS although the aws status pages say nothing about SNS.
- kuya11 5y agoThe blatant status page lies are getting absolutely ridiculous. How many hours does a service need to be totally down until it gets properly labelled as a "disruption"?
- bearjaws 5y agoYeah, we are seeing SQS, API Gateway (both socket and non websocket) and S3 all completely unavailable. Status page shows nothing despite having gotten several updates.
- amir734jj 5y agoMy job (although 50% of time) at Azure is unit testing/monitoring services under different scenarios and flows to detect small failures that will be overlooked in public status page. Our tests run multiple times daily and we have people constantly monitoring logs. It concerns me when I see all AWS services are 100% green when I know there is an outage.
- jamesfmilne 5y agoOh, sweet summer child. The reason you care about your status page being 100% accurate is that your stock price is not directly linked to your status page.
- joshocar 5y agoI don't know how accurate this information is, but I'm hearing that the monitor can't be updated because the service is in the region that is down.
- Rapzid 5y agoKinda hard to believe after they were blasted for that very situation during/after the S3 outage way back. If that's the case, it's 100% a feature. They want as little public proof of an outage after it's over and to put the burden on customers completely to prove they violated SLAs.
- jgworks 5y agoSomeone posted this on our company slack: https://stop.lying.cloud/ https://stop.lying.cloud/
- exabrial 5y agoJust a reminder that Colocation is always an option :)
- birdyrooster 5y agoGives me a flashback to the December 24th, 2012 outage. I guess not much changes in 9 years time.
- strictfp 5y ago"some customers may experience a slight elevation in error rates" --> everything is on fire
- soco 5y agoI'm also experiencing a slight elevation in billing rates - got alarms for 10x consumption and I can't check on them... Edit: also API access is failing, terraform can't take anything down because "connection was forcibly closed"
- gchamonlive 5y agoImagine triggering a big instance for machine learning or a huge EMR cluster that would otherwise be short lived and not being able to scale it down. I am quite sure the AWS support will be getting many refund requests over the course of the week.
- hvgk 5y agoECR and API are fucked so it’s impossible to scale anything to the point fire can come out :)
- kello 5y agoah corporate speak at it's finest
- Xenoamorphous 5y agoMaybe when the error rate hits 100% they’ll say “error rate now stable”.
- retbull 5y ago"Only direction is up"
- kingcharles 5y agoDoes McDonalds use AWS for the backend to their app? If I find out this is why I couldn't get my Happy Meal this morning I'm going to be really, really grumpy. EDIT: I'm REALLY grumpy now: https://aws.amazon.com/blogs/industries/aws-is-how-mcdonalds-transforms-the-drive-thru-experience/ https://aws.amazon.com/blogs/industries/aws-is-how-mcdonalds...
- hoofhearted 5y agoThe McDonalds App was showing on the frontpage of Down Detector at the same time as all the Amazon dependent services last I checked.
- Crespyl 5y agoApparently Taco Bell too, not being able to place an order and then also not being able to fall back to McDonalds was how I realized there was a larger outage :p What am I supposed to do for lunch now? Go to the drive through and order like a normal person? /s Grumble grumble
- n0cturne 5y agoI just walked into the Amazon Books store at our local mall. They are letting everyone know at the entrance that “some items aren’t available for purchase right now because our systems are down.” So at least Amazon retail is feeling some of the pain from this outage!
- CoachRufus87 5y agoDoes there exist a resource that tracks outages on a per-region basis over time?
- dalrympm 5y agoContrary to what the status page says, CodePipeline is not working. Hitting the CLI I can start pipelines but they never complete and I get a lot of: Connection was closed before we received a valid response from endpoint URL: "https://codepipeline.us-east-1.amazonaws.com/ https://codepipeline.us-east-1.amazonaws.com/".
- alexatalktome 5y agoRumor is that our internal pipelines are the root cause. The CICD pipelines (not tests, the literal pipeline infrastructure) failed to block certain commits and pushed them to production when not ready. We’ve been told to manually disable them to ensure integrity of our services when it recovers
- deleted 5y ago[deleted]
- ComputerGuru 5y agoThe AWS status page no longer loads for me. /facepalm
- all_usernames 5y ago25 Regions, 85 Availability Zones in this global cloud service and I can't login because of a failure in a single region (their oldest). Can't login to AWS console at signin.aws.amazon.com: Unable to execute HTTP request: sts.us-east-1.amazonaws.com. Please try again.
- mancerayder 5y agoThe Personal Health Dashboard is unhealthy. It says unknown error, failure or exception. They need a monitor for the monitoring.
- jonnylangefeld 5y agoDoes anyone know why Google is showing the same spike on down detector as everything else? How does Google depend on AWS? https://downdetector.com/status/google/ https://downdetector.com/status/google/
- NobodyNada 5y agoIt’s because Down Detector works off of user reports rather than automatically detecting outages somehow. So, every time a major service goes down (whether infrastructure like AWS or Cloudflare, or user-facing like YouTube or Facebook), some users will blame Google, ISPs, cellular providers, or some other unrelated service.
- gmm1990 5y agoSome google sheets functions aren't updating in a timely manner for me. Maybe people google as a backup for aws and they have to throttle certain service from a higher load
- cbtacy 5y agoIt's funny but when I saw "AWS Outage" breaking, my first thought was "I bet it's US-east-1 again." I know it's cheap but seriously... not worth it. Many of us have the scars to prove this.
- nemothekid 5y agoSome sage advice I learned a while ago: "Avoid us-east-1 as much as possible".
- soco 5y agoBut if you use CloudFront, there you go.
- dr-detroit 5y agoIf you need to be up all the time don't you use more than 1 region or do you need the ultra low ping for running 9-11 operator phone systems that calculate real time firing solutions to snipe ICBMs out of low orbit?
- hvgk 5y agoWell I got to bugger off home early so good job Amazon. Edit: to be clear this is because I’m utterly helplessly unable to do anything at the moment.
- AnIdiotOnTheNet 5y agoYep, that's a consideration of going with cloud tech: if something goes wrong you're often powerless. At least with on-prem you know who to wake up in the middle of the night and you'll get straight-forward answers about what's going on.
- hvgk 5y agoDepends which provider you host your crap with. I’ve had real trouble trying to get a top tier incident even acknowledged by one of the pre cloud providers. To be fair when it’s AWS when something goes snap it’s not my problem which I’m happy about (until some wise ass at AWS hires me) :)
- AnIdiotOnTheNet 5y ago> Depends which provider you host your crap with. That's what I'm saying: you host it yourself in facilities owned by your company if you're not willing to have everyone twiddle their thumbs during this sort of event. Your DR environment can be co-located or hosted elsewhere.
- ZebusJesus 5y agoMe thinks Venmo uses AWS because they are down as well. Status gator has AWS as on off on off on off. I can access my servers hosted in the west coast but I cannot access the AWS console, this is making for an interesting morning.
- kello 5y agoDon't envy the engineers working on this right now. Good luck!
- joelhaasnoot 5y agoAmazon.com is also throwing errors left and right
- lowwave 5y agothis is exact kinda over centralisation issues I was talking about. I'm one of first developers using AWS EC2, sure back when scaling is hard for small dev shops. In now day and age any one who is technically inclined, can figure out using the new technologies. Why even use AWS. Get something like Hetzner, Linodes please!
- pojzon 5y agoSome manager: But it does not web scale!
- dr-detroit 5y agothe average HN poster: I run things on a box under my desk and you cannot teach me why thats bad!!!
- binaryblitz 5y agoHow are those better than using AWS/Azure/GCP/etc? I'd say the correct way to handle situations is to have things in multiple regions, and potentially multiple clouds if possible. Obviously, things like databases would be harder to keep in sync on multi cloud, but not impossible.
- ricardobayes 5y agoThis was funny at first but now I can't even play Elder Scrolls Online :(
- john37386 5y agoimdb seems down too and returning 503. Is it related? Here is the output. Kind of funny. D'oh! Error 503 We're sorry, something went wrong. Please try again...wait...wait...yep, try reload/refresh now. But if you are seeing this again, please report it here. Please explain which page you were at and where on it that you clicked Thank you!
- takeda 5y agoIMDB belongs to Amazon, so likely on AWS too. This also confirms it: https://downdetector.com/status/imdb/ https://downdetector.com/status/imdb/
- binaryblitz 5y agoTIL Amazon owns IMDB
- hulahoop 5y agoI heard from my cousin who works at an Amazon warehouse that the conveyor belts stopped working and items were messed up and getting randomly removed off the belts.
- bilalq 5y agoSome advice that may help: * Visit the console directly from another region's URL (e.g., https://us-east-2.console.aws.amazon.com/console/home?region=us-east-2 https://us-east-2.console.aws.amazon.com/console/home?region...). You can try this after you've successfully signed in but see the console failing to load as well. * If your AWS SSO app is hosted in a region other than us-east-1, you're probably fine to continue signing in with other accounts/roles. Of course, if all your stuff is in us-east-1, you're out of luck. EDIT: Removed incorrect advice about running AWS SSO in multiple regions.
- binaryblitz 5y agoI don't think you can run SSO in multiple regions on the same AWS account.
- bilalq 5y agoThanks, corrected.
- staticassertion 5y ago> Might also be a good idea to run AWS SSO in multiple regions if you're not already doing so. Is this possible? > AWS Organizations only supports one AWS SSO Region at a time. If you want to make AWS SSO available in a different Region, you must first delete your current AWS SSO configuration. Switching to a different Region also changes the URL for the user portal. [0] This seems to indicate you can only have one region. [0] https://docs.aws.amazon.com/singlesignon/latest/userguide/regions.html https://docs.aws.amazon.com/singlesignon/latest/userguide/re...
- bilalq 5y agoGood call. I just assumed you could for some reason. I guess the fallback is to devise your own SSO implementation using STS in another region if needed.
- TameAntelope 5y agoWe run as much as we can out of us-east-2 because it has more uptime than us-east-1, and I don't think I've ever regretted that decision.
- adwww 5y agoLeft a big terraform plan running while I put the kids to bed, checked back now and Amazon is on fire.... was it me?!
- dylan604 5y agoThe fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS. If you can't connect to the cloud, fail, move on and load the app so that local things are allowed to work. I'm guessing other IoT things suffer from this same short sitedness as well.
- s_dev 5y ago>The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. Whats a graceful fallback? Switching to another hosting service when AWS goes down? Wouldn't that present another set of complications for a very small edge case at huge cost?
- rehevkor5 5y agoUsually this refers to falling back to a different region in AWS. It's typical for systems to be deployed in multiple regions due to latency concerns, but it's also important for resiliency. What you call "a very small edge case" is occurring as we speak, and if you're vulnerable to it you could be losing millions of dollars.
- simmanian 5y agoAWS itself has a huge single point of failure on us-east-1 region. Usually, if us-east-1 goes down, others soon follow. At that point, it doesn't matter how many regions you're deploying to.
- scoopertrooper 5y agoMy workloads on Sydney and London are unaffected. I can't speak for anywhere else.
- griffinkelly 5y agoHosting and processing all the photos for the California International Marathon on EC2, this doesnt make easier dealing with impatient customers any easier
- dwighttk 5y agoIs this why goodreads hasn’t been working today?
- megakid 5y agoI live in London and I can't launch my Roomba vacuum to clean up after dinner because of this. Hurry up AWS, fix it!
- ChrisArchitect 5y agoWe have always been at war with us-east-1.
- alde 5y agoS3 bucket creation is failing across all regions for us, so this isn't an us-east-1 only issue.
- macintux 5y agoGiven the global uniqueness requirement, it’s quite possible that S3 is always going to fail for bucket creation if a region is (mostly) offline.
- boldman 5y agoI think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevented from working and tens of million dollars of profit are simply vanishing since the start of the outage this morning. And now institutions like government and banks are throwing their entire capability into the cloud with no recourse or recovery plan. It seems bad now but I wonder how much worse it might be when no one actually has access to money because all financial traffic is going through AWS and it goes down. We are incredibly blind to just trust just 3 cloud providers with the operational success of basically everything we do. Why hasn't the industry come up with an alternative?
- bowmessage 5y agoBecause the majority of consumers don't know better / don't care and still buy products from companies with no backup plan. Because, really, how can any of us know better until we're burned many times over?
- tw04 5y ago>It seems bad now but I wonder how much worse it might be when no one actually has access to money because all financial traffic is going through AWS and it goes down. Most financial institutions are implementing their own clouds, I can't think of any major one that is reliant on public cloud to the extent transactions would stop. >Why hasn't the industry come up with an alternative? You mean like building datacenters and hosting your own gear?
- baoyu 5y ago> Most financial institutions are implementing their own clouds https://www.nasdaq.com/Nasdaq-AWS-cloud-announcement https://www.nasdaq.com/Nasdaq-AWS-cloud-announcement
- 5y ago
- thefourthchime 5y agoIf someone needs to get to the console, you can make a url like this: https://us-west-1.console.aws.amazon.com/ https://us-west-1.console.aws.amazon.com/
- yabones 5y agoWorks for a lot of things, but not Route53... Which is great because that's the only thing I need to do in AWS today :)
- 999900000999 5y agoWhat if it never comes back up ?
- lgylym 5y agoSo reinvent is over. Time to deploy.
- keyle 5y agoWhy does that webpage renders like a dog?... I get that it's under load but the rendering itself is chugging something rare. Edit: wow that webpage is humongous... never heard of paging?
- torbTurret 5y agoAny leads on the cause? Some services on us-east-1 seem to be working just fine. Others not.
- fnord77 5y agocan't wait for the postmortem on this
- bamboozled 5y agoI don't think AWS knows what's going on judging by their updates, yes DynamoDB might be having issues, but so is IAM it seems, we're getting issues terminating resources for example.
- Rapzid 5y agoThey build all their services on top of other services. Hence the 2017 S3 outage that broke the internet.
- giantg2 5y agoWhat a fun time today was
- at_a_remove 5y agoI wonder if this has anything to do with their help pages being ... gone?
- fnord77 5y agonow amazon is giving more detailed updates of the problem to bigger customers *under NDA*. WTF
- john37386 5y agoFirst fastly, then facebook, now AWS us east. Who's next? Global Google outage?
- deleted 5y ago[deleted]
- sizzle 5y agoAnyone else having trouble with their Alexa devices? Mine are acting really wonky, couldn’t listen to NPR in the morning really messed up my routine.
- Ecco 5y agoMy Alexa-controlled lights didn’t turn on this evening.
- sizzle 5y agoSame for my smart plugs, this is infuriating to say the least. I hate having to re-enable skills in the Alexa app when these smart devices randomly decide to unlink.
- tyingq 5y agoSetting a timer seemed to be dependent on us-east-1, like "alexa set a timer for 10 minutes".
- Goety 5y agoDamn AWS is still getting hammered
- gregw2 5y agoFYI, there was a Redshift query outage this morning, and Redshift COPY operations from S3 to Redshift have been failing for quite a while now. Status page shows all green but AWS confirms differently when on a customer phone call.
- gregw2 5y agoRedshift Concurrency Scaling seems like it went down and UNLOAD to S3 operations were also failing much of the day. Seems like all has been working fine the last 3-4 hours.
- jcun4128 5y agoSide effect I notice is duplicated AWS SMS
- hooande 5y agoSo is the big lesson here...don't use us-east-1?
- itsdrewmiller 5y agoLatest update: "[4:35 PM PST] With the network device issues resolved, we are not working towards recovery of any impaired services. We will provide additional updates for impaired services within the appropriate entry in the Service Health Dashboard." I guess they gave up.
- ryeights 5y agoHahaha, corrected to “now working” now.
- fasdghvhjf 5y agoIt was a typo. "Not" has changed to "now".
- booleanbetrayal 5y agoI'm pretty well versed in Kinesis / Firehose errors these days. I just got a new Firehose error I've never seen before: > Firehose error: Slow down. I wonder if they had to introduce a new exception / rate-limit to mediate this issue ...
- taf2 5y agoI noticed some additional rate limiting as we're processing a backlog of data from earlier today... It's possible they just have increased the normal thresholds while everyone is in a similar recovery mode and they too are bring capacity back to normal levels.
- JDK11-0-12 5y agoNetflix was working though
- belter 5y agoI am going to take a public bet this was an issue caused by a code deployment to the AWS service Amazon Event Bridge...
- alblue 5y agoThe most annoying aspect of this for us was that the AWS Organizations API only has a single endpoint, in us-east-1. https://docs.aws.amazon.com/general/latest/gr/ao.html https://docs.aws.amazon.com/general/latest/gr/ao.html As a result, even if you weren’t using that region but you were using the API you were hosed for 6+ hours, And the status page never acknowledged that it was out of action. We had the following Terraform in our production pipeline: data “aws_organizations_organization” “current” {} As a result, all of our deployments to our EU regions were borked. Of course, we couldn’t raise a support case because the support system was also down, and despite escalating to our TAM weren’t able to get the status page to reflect reality. My concern is that the Organizations API specifically will be brushed under the carpet and we will still have a single point of failure in a region which we never intend to use.
- advpetc 5y agoI cannot open the status page from aws. Is it just me or everyone?
- jarechiga 5y agoSo what monitors the monitoring services?
- daxfohl 5y agoTangent, still no postmortem on the github outage two weeks ago. Just "problem resolved".