21 ms·
AWS us-east-1 down
The status page says everything is fine though.
- last_responder 3y agoNo issues on our Linodes .
- totaldude87 3y agomuch reliable https://downdetector.com/status/aws-amazon-web-services/ https://downdetector.com/status/aws-amazon-web-services/
- BluePen7 3y agoThey've just acknowledged degradation with lambda
- ciguy 3y agoIt appears to be an outage in IAM which is trickling down to every service which relies on IAM auth.
- pliuchkin 3y agoBut IAM is supposed to be Global, not us-east-1
- Androider 3y agoAll the regions are equal, but some regions are more equal than others.
- paulddraper 3y agoIn that it globally depends on us-east-1.
- otterley 3y agoAs I’ve said elsewhere, this is not accurate for existing IAM resources.
- dijit 3y agoI wonder if this is a coincidence or if us-east-1 is simply down enough that I'm just experiencing selection bias; but I posted a poll on twitter earlier today: https://twitter.com/dijit/status/1668678588713824257 https://twitter.com/dijit/status/1668678588713824257 Contents: > Has anyone ever actually had customers accept an outage because AWS was down; or is this just cloud evangelicalism copium? > [ ] Yeah, outages free pass > [ ] No, they say to use AZ's
- robrtsql 3y ago> No, they say to use AZ's Using 3 AZs in us-east-1 won't save you. I guess a demanding customer would have said 'you should have implemented disaster recovery so you could failover to us-east-2' but that's easier said than done. The more regional AWS services you adopt, the bigger the impact is. How does one recover from a regional outage if their pipeline is in that region?
- fnordpiglet 3y agoI’ve worked for several systemically important megacorps where certain things had to not only run cross region but also cross provider. It’s absurdly difficult, and only should be done if you need five or more 9’s of availability. Almost nothing actually does.
- fbellag 3y agoWhat I did once I was in the position of _having_ to provide that level of support, was to run the pipeline in a third region, different from the "prod" ones. That way, worst case you can't do deployments during the outage... Another alternative studied was to use a thirdparty ci/cd service, outside of our network. It was discarded bc you never know where that would actually run
- robrtsql 3y ago> It was discarded bc you never know where that would actually run Yep, I considered that switching to GitHub Actions would _theoretically_ eliminate the need for disaster recovery for CI/CD (since the handling of disasters is out of your hands) but in practice their SLA is far worse than just running CodePipeline in a single region.
- monero-xmr 3y agoYou can't even access the tools (web or CLI) in order to put your own system into maintenance mode...
- rotten 3y agoLooks like it is a problem with lambdas.
- trnsfrmrsr 3y agodef broken for us
- trnsfrmrsr 3y agoissues with STS and IAM galore #thisIsFine
- gerenuk 3y agoNetlify is down as well. https://www.netlifystatus.com https://www.netlifystatus.com
- deleted 3y ago[deleted]
- smnirven 3y agoyep can't even get into console to diagnose / troubleshoot / fix
- bovermyer 3y agoCLI works OK still.
- dveeden2 3y agohttps://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status reports: Increased Error Rates and Latencies Jun 13 12:08 PM PDT We are investigating increased error rates and latencies in the US-EAST-1 Region. They list Lambda as the only affected service
- ciguy 3y agoThey've added a dozen or so more as potentially down now. Anything that uses IAM, which I suspect is the core of the issue.
- notatoad 3y agostatus page won't load for me. are they still hosting their status page on their own infrastructure?
- j-sizz 3y agoPerhaps they should host it on GCP
- xenospn 3y agoOuch
- uniformlyrandom 3y agoIt is a perfectly cromulent practice to host a replica of your status page at your competitor.
- uberism 3y agos3 is down for us
- arixzajicek 3y agothis happened less than an hour after I altered our prod scheme, thought I brought down production, what a relief
- taylodl 3y agoNo, you brought down us-east-1 instead! Thanks!
- spicybright 3y agoThis is why you don't give prod credentials to the new guy!
- thefourthchime 3y agoI had a similar reaction. Oh no WTF, how did i break that?! Then my buddy texted me about us-east-1 being down. Then i thought "Oh thank god, this shitshow is someone else's fault."
- arixzajicek 3y agoHad our login and other features go down less than an hour after I altered our prod scheme and thought I did something wrong, what a relief it was to see this
- samwillis 3y agoOther submission linking to AWS status page: https://news.ycombinator.com/item?id=36315441 https://news.ycombinator.com/item?id=36315441 And what a surprise it's US-EAST-1 again...
- rbosinger 3y agoIt's always during the demos to the stakeholders, isn't it?
- deleted 3y ago[deleted]
- mardifoufs 3y agoI'm so glad my demo today was specifically about local inference on... Windows. I guess working I finally found an upside to doing ML outside Linux ; we don't have Windows VMs on AWS!! :)
- forgetfulness 3y agoTried to log on to OkCupid and it was down, I guess I should thank Jeff for taking me off the Skinner Box for a while.
- jarym 3y agous-east-1 seems to be very 'special' compared to the other regions - I wonder if they will ever align it with the rest of them.
- thefourthchime 3y agoI think a lot of companies just do everything there and pinky promise one day they'll go multi-region.
- rotten 3y agoI worked with a devops person who moved everything we had set up in other regions _to_ US-East-1 because that is where you are supposed to run stuff. According to him, the other regions were just for DR stuff.
- jarym 3y agoSurely not an AWS certified devops person? I don't think they teach mythology!
- goodells 3y agoYep, it has issues so frequently. I wonder how many companies/teams start using AWS and blindly choose us-east-1 without realizing what they're getting into. <rant> It's also quite annoying sometimes that some things _need_ to be in us-east-1, and if e.g. you are using Terraform and specify a different default region, AWS will happily let you create useless resources in regions that aren't us-east-1 that then mysteriously break stuff because they aren't in this one blessed region. AWS Certificate Manager (ACM) certificates are like this, I believe. </rant>
- mschuster91 3y agoACM certificates themselves can be had in any region (and you can use them for stuff like ELBs), but since the Cloudfront control plane is in us-east-1, if you want Cloudfront (and IIRC, also if you want custom domain names for an S3 bucket, but don't quote me on that) you'll have to create an additional certificate in us-east-1. Sigh.
- compumike 3y agoThe second law of thermodynamics guarantees that there will always / eventually be downtime… and that’s okay. Design for downtime. Shameless self-plug for https://heiioncall.com/ https://heiioncall.com/ for free website / HTTP endpoint monitoring and cron job monitoring if you want to know about your own app’s downtime. EDIT: thanks to those of you who have signed up in the past few minutes! Let us know if you have any feedback.
- CodesInChaos 3y ago"eventually" is doing a lot of work in that sentence. Our sun will last another 5 billion years, and the heat death is something like 100 trillion years away.
- compumike 3y agoSure :) Though a lot of practical thermal-related causes of electronics failure seem to operate on timescales of years to decades, like electromigration https://en.m.wikipedia.org/wiki/Electromigration https://en.m.wikipedia.org/wiki/Electromigration or even just cooling fan bearing failure. And I don’t think it would be a huge stretch to point to electromigration as a case of diffusion, a natural entropy increasing process, re-randomizing the arrangement of atoms within a transistor (and therefore making it fail eventually).
- intsunny 3y agoFor those wondering: Currently PDT is 7 hours behind UTC. AWS can do so many things, reporting critical outage updates in UTC is not one of those things.
- messe 3y ago> AWS can do so many things, reporting critical outage updates in UTC is not one of those things. Thank you for reminding me about one of my biggest mildest annoyances from working at AWS.
- takeda 3y agoI thought it uses your browser time zone, is it not?
- deleted 3y ago[deleted]
- meepmorp 3y agoSays the same here and I'm on the other coast.
- paulddraper 3y agoNo. It's all PDT.
- rurp 3y agoThe inconsistency with timezones across different services in the AWS console has always baffled and annoyed me. Some places have a time without a timezone and I can never tell right away if it's utc, local time, or region time.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- 3y ago
- ltcoleman 3y agoWe are seeing console, codebuild, etc. access issues. Possibly all using Lambdas, foundationally?
- waz0wski 3y agothe last big us-east-1 outage was ... DNS - and it's usually DNS or software-defined core networking causing these cascading failures Loss of DNS causes inter-service api calls to fail, then IAM and all other services fail. Anything not built to handle those situations with backoff causes a 'stampeding herd' of failure/retry and exacerbate the outage Review the AWS statements about outages here - https://aws.amazon.com/premiumsupport/technology/pes/ https://aws.amazon.com/premiumsupport/technology/pes/
- bastard_op 3y agoMaybe azure was hosting some services there, getting ddos'd and all.
- whoisjuan 3y agoWhy is it always us-east-1 though? I have always stayed away from that region because it seems significantly less reliable than other regions.
- deleted 3y ago[deleted]
- JoshTriplett 3y agous-east-1 is AWS's oldest region, and has the most legacy infrastructure, in ways that many other regions do not.
- nicoffeine 3y agoI thought I read that this is where they deploy new changes first. Can anyone confirm?
- mparnisari 3y agoWhen i worked at aws, IIRC, us-east-1 was one of the last regions we deployed to. So this is very confusing to me
- throwaway019254 3y agoI don't believe it's true. I was working on one of the biggest AWS services and we always deployed to small regions first. @dijit is right: https://news.ycombinator.com/item?id=36315736 https://news.ycombinator.com/item?id=36315736
- foobiekr 3y agoI have a suspicion that AWS uses some regions as canaries. Because we control both ends of things, I have personally noted that certain AWS functions clearly break in Australia first.
- temp_praneshp 3y agoFrom observing my wife's teams over the years, they deploy new _products_ early to that region, but deploying code changes starts in smaller regions.
- hatmanstack 3y agoI'm sure aws re:Inforce starting has nothing to do with it... https://reinforce.awsevents.com/livestreams/ https://reinforce.awsevents.com/livestreams/
- deleted 3y ago[deleted]
- 89vision 3y agoWe're on use1 and havent seen any degradation
- nathants 3y agofinally an opportunity to test a full deploy from scratch, and restore from backup, in a new region. i wonder if it will work first try? the true test of devops culture.
- danryan 3y agoGood luck! My own attempt failed because SSO is down.
- neodypsis 3y agoIs there an estimate for the time they will take to solve this?
- JoBrad 3y ago2.5 hours, from start of incident.
- impulser_ 3y agoWhy does everyone keep deploying their products to this one region when it always seems like the one that fails? We don't use big cloud were I work, so maybe I'm missing something. Does East-1 offer something other don't?
- andrewmcwatters 3y agoThere's a lot of software, iirc even Amazon's own dashboards, that simply defaults to us-east-1.
- akira2501 3y agoThat's my favorite part of Amazon's console. That miniature heart attack you have when you ask "WHERE ARE ALL MY LAMBDAS AND DYNAMO INSTANCES?" Then you realize that they just switched you back to us-east-1 for some reason and a wave of familiar relief washes over you.
- redwood 3y agoOut of curiosity have you been building with the serverless framework? I'm curious what drives people to use dynamodb in particular
- akira2501 3y agoNo, I really don't like any of the serverless or amplify type of frameworks that AWS produces. What I needed was a distributed store for managing a large number of small configuration "files" and other state "links" across the infrastructure I built. In particular, I needed the ability for a write conflicts to be detected and managed immediately and for consistent reads to be available in some cases. Looking at the size of data and number of transactions, DynamoDB on a per-request model was going to be significantly cheaper and easier than standing up a bunch of DB instances or other "fixed" infrastructure. The ultimate model was to actually implement something like a POSIX filesystem on top of the "nosql" layer. In practice, I've been really happy with that decision, and it makes working on the code that interacts with this system very familiar and easy to understand. It's even got symlinks, acls, and automatically expiring advisory locks.
- super_linear 3y agoSeems like it's time for end of the quarter AWS demos and someone got a little too eager for launch.
- thedigitalone 3y agoToast POS is down 100%, don't go out to lunch.
- jjice 3y agoAs a side note, I wonder if businesses won't even accept cash if they can't go through their POS system. If not, it's a shame that these modern internet connected POSs lock out stuff like that.
- inconceivable 3y agosome restaurants have their owner or manager run square or stripe on their cell phone.
- xyst 3y agoI don’t know about Stripe but Square used to offer the CC swipe dongle you can connect to your phone. Then process payments through their app
- JoBrad 3y agoI was at a swim meet last week, and one of the food trucks was using Apple Tap to Pay because Toast didn't have a solution that worked for them, on site. After they finish up at an area, they then enter a single transaction for all of the day's business into Toast.
- p_l 3y agoDepending on country and exact POS setup, they might not be able to take cash if POS is down. For example, in Poland, your typical restaurant or shop needs to generate tax receipt (as well as properly calculate the tax), and uses either a separate receipt POS device, or POS with appropriate receipt printer (the devices are certified and for example do simultaneous two prints - one for client one for seller - or use digitally-signed storage for seller copy). If the POS isn't designed properly to operate in case of network failure... welp, can't take cash either, at least not legally.
- 3y ago
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- dangoodmanUT 3y agoYes
- dangoodmanUT 3y agoWhat's interesting is that I can still access my EKS cluster, but none of the deployments are "ready" that have LBs attached to them. Pods can create fine though!
- eis 3y agoSeems like it took IMDB with it. Surprised that Amazon is not able to keep their own property up when one of their zones goes down. Not a great example.
- andrewinardeer 3y agoIs this possibly why my Ring doorbell omitted a phantom chime about two hours ago at 4am?
- grumple 3y agoIt's fun watching each service fail sequentially while the aws service dashboard just updates them to "Informational" status, whatever that means. Even management console is down, and their suggested region specific workaround does not work, at least for us-east-1. I can see some processes via api but I don't have code prepared for monitoring every service from my local.
- grumple 3y agoAnd now the service health page is down.
- adubashi 3y agoIt doesn't matter if your infra is in another region, because there will almost always be transitive dependencies on us-east-1. IAM is deployed in us-east-1 and there will always be a transitive dependency on us-east-1
- mooreds 3y agoControl plane will almost always be impacted, I agree. Our data plane was fine (for example, ec2 instances and s3 buckets in other regions were fine).
- luhn 3y agoI have never had a production issue in other regions due to a us-east-1 outage. The worst that ever happened was I had to wait to update a Cloudfront distribution because the control plane (based in us-east-1) was down, but the existing configuration continued working fine throughout. I don't know what the architecture of IAM looks like, but somehow it's never suffered a global outage. AWS is really, really good at regional isolation.
- jcims 3y ago>I don't know what the architecture of IAM looks like, but somehow it's never suffered a global outage. Authentication possibly, but the control plane has gone down preventing changes.
- TheSoftwareGuy 3y agoI think the data plane is regional
- tedmiston 3y agoI have. Not being able to update your existing resources is still an outage from a DevOps perspective. It might be an API level outage vs an end-user level outage from your customer's perspective, but if the functionality is down, it's an outage.
- jedberg 3y agoUsually it only prevents changes, but the runtime isn't affected.
- hollowturtle 3y agoeu-west-1 having some issue as well for me
- lonnyk 3y agocan you elaborate?
- theshrike79 3y agoGuesses, which one is it this time: * DNS * Misconfigured switch It's always one of those two.
- mdaniel 3y ago* Cert expiry
- kyleee 3y agoThe AWS status page is now down as well
- WalterSobchak 3y agohttps://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status seems to be working fine for me (been refreshing it every couple of minutes).
- Max-Ganz-II 3y agoI kicked off a Redshift cluster in every region, they've all run and completed, except for `us-east-1`, which is stuck creating the cluster. Been about an hour now.
- rafamcc 3y agoanyone getting Gateway Time-out?
- rafamcc 3y agogetting Gateway Time-out
- JohnMakin 3y agoI'm not sure it's the case here, but the issue with these cloud providers is they use their own services to maintain their infrastructure - that's why when something like lambda gets degraded, which would not shock me if they're using everywhere, you start to see random crap like console and IAM go down as well.
- johnnyApplePRNG 3y agoI can't even update my billing information right now :/
- deleted 3y ago[deleted]
- gawshinde 3y agohttps://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status now it is showing. Lot of services are impacted.
- issafram 3y agoI love how their status says that services are just degraded
- throaway87c10f0 3y agoMysterious lack of "AWS is bad for the internet because it is so centralized" dialog up in here. edit: for those that would downvote: HN _just_ yesterday: https://news.ycombinator.com/item?id=36295352 https://news.ycombinator.com/item?id=36295352 https://news.ycombinator.com/item?id=36295305 https://news.ycombinator.com/item?id=36295305
- deleted 3y ago[deleted]
- dijit 3y agoits just tired at this point. everyone knows, nobody seems to care. Another comment of mine in this thread asks the question if you can excuse downtime of your service due to AWS outages. Consensus seems to be: yes which is a pretty huge deal, well worth the insane cost increase of AWS by itself. No other hosting provider would grant you such an excuse. I would weep for the centralised future of the internet, but its already here, so theres no point.
- klysm 3y agoEven if people _do_ care, there isn't much to do about it.
- hosteur 3y agoIf people do care they could use other hosting providers such as Hetzner or OVH, no?
- klysm 3y agoThe economics of the situation make it so that at your job that's not very realistic.
- YetAnotherNick 3y agoWhy did you come to the conclusion that Hetzner or OVH is more reliable. At least their SLA credits doesn't say that.
- vvoyer 3y agoWorth knowing: today and tomorrow is AWS re:Inforce 2023 https://reinforce.awsevents.com/ https://reinforce.awsevents.com/.
- FullyFunctional 3y agoI "love" it when my vacuum stops working because an online book sellers servers went down. #modernlife This is a good reminder to avoid cloud-centric products, but they are getting harder and harder to avoid.
- MrBruh 3y agoDid this actually happen? The vacuum part
- FullyFunctional 3y agoYes. This morning I found my Roomba (i9 something IIRC) sitting idle in the middle of the kitchen. When I launched the iOS app, it appologised it couldn't connect due to AWS being down (I regret not taking a screenshot).
- assimpleaspossi 3y agoMy son delivers part-time for Amazon and all the drivers at his warehouse were sent home. So if your delivery is late or non-existent today....
- prfssnl 3y agoAmazon Flex?
- assimpleaspossi 3y agoYep
- prfssnl 3y agoI heard they are paying Flex drivers for that day. Well, most of them.
- assimpleaspossi 3y agoThey did. Say what one wants about Amazon or how they treat delivery drivers. My son told me of some experiences he's had--including having his car totaled on an icy river bridge at four in the morning--and Amazon was on top of it, empathetic, and did more than what was expected.
- deleted 3y ago[deleted]
- JTbane 3y agob-but muh five nines of reliability...
- noradbase 3y agoI guess I'll use the downtime to see what's new on Reddi... oh... yeah.
- uberism 3y agoIt appears to have been fixed and resolved for us.
- chaosmachine 3y agoIf you're having problems accessing the console, the workaround is just to use a different region, eg: https://ca-central-1.console.aws.amazon.com/console/home https://ca-central-1.console.aws.amazon.com/console/home This assumes you don't actually need anything from us-east-1, though :)
- jmacjmac 3y agoCan we find stats somewhere, something like number of outages by regions?
- jdlyga 3y agoMost stuff is back up, but AWS MediaConvert still seems to be down.
- jdlyga 3y agoI posted a comment about AWS Media Convert down below, but it's back working for me.
- mlhpdx 3y agoHow many folks actually use multi-region deployments with automatic failover (e.g. latency based routing in route 53)?
- klysm 3y agoThe difficulty of making this work in practice is pretty high. It also isn't cheap, so I would guess not many.
- mlhpdx 3y agoSome parts are expensive (EC2, NAT and VPC endpoint proliferation) but others are simple and inexpensive (regional API Gateways, latency based routing, DDB global tables, lambdas, state machines).
- chelobaka 3y agoBoth Vercel and Netlify went down with it. I wonder what % of the internet went down because of the us-east-1 today.
- throw03172019 3y agoMy Whole Foods grocery pickup order was affected by this outage. They couldn’t check me in. Groceries were packed in the fridge but they told me to come back later. What a waste of time.
- OJFord 3y agoAre you just guessing it's related or is there a reason to be sure? Either way, does seem sjll6 to have no manual fallback 'check-in' (or policy-level ability to bypass the need for it) mechanism.
- doubled112 3y agoMost retail stores simply stop if "the system" is down.
- OJFord 3y agoI've known 'just take your shopping and go' and 'let me make a note of your details', but never 'put your bags down and bugger off but please do comw again'. And why would ordered and paid for shopping be so inaccessible? It's not going to be locked away in a vending machine type thing right, you just need to show up with ID/log-in and collect it. Or so you should.
- doubled112 3y agoI’ve only known “sorry, the till is down” and line ups as far as the eye can see. It's ridiculous that everything stops, but I do understand. The pickup is even more ridiculous. Like you say, it's already through the system. They would just need to mark it as delivered later.
- throw03172019 3y agoThe Whole Foods employee working the pick-up area told me.
- deleted 3y ago[deleted]
- dietr1ch 3y agoYou'd think the largest cloud would be resilient to a zone going down, but I guess not. i find it stupid that clients get to know about regions at all. They should only notice latency hits and batch job queuing latency if something bad happens underneath, but no services should go down.
- deleted 3y ago[deleted]
- nfRfqX5n 3y ago> At that time, we began processing the backlog of asynchronous Lambda invocations that accumulated during the event, including invocations from other AWS services. As of 3:37 PM, the backlog was fully processed pretty cool that stuff was stored in a backlog and eventually processed!
- capableweb 3y agoI don't want to diminish anything, but in terms of dealing with queues, backlogs are pretty much a basic requirement from the beginning.
- great_psy 3y agoI am guessing they mean SQS queue … so basically the SQS aid doing what is supposed to: - try to process event - send to DLQ if it fails - there’s a redrive button in the DLQ to … well restive the events after the lambda is fixed
- merek 3y agoDoes anyone know if the entire region suffered the outage, or one/some AZs?
- idontwantthis 3y agoHas anyone checked if Washington, DC is still there?
- KingMachiavelli 3y agoWeird, I didn't notice the actual outage at all except high ping to a non-AWS server/IP on the west coast. Normally the latency is ~85ms, today it has been >170ms. SSH is basically unusable at that latency and even bandwidth is very low (not sure how that could be).
- fierro 3y agoanyone have a link to the AWS summary / postmortem? Saw it on their status page the other day, but can not find it anywhere now.