7 ms·
Ask HN: Anybody else EC2 issues on us-east-1?
Originally just us-east-1d was not reacting well, but now most EC2 instances don't do much anymore.
- swvjeff 9y agoYes: https://status.aws.amazon.com/ https://status.aws.amazon.com/ us-east-1b seems to be affected for me. 4:36 PM PDT We are investigating network connectivity issues for some instances in a single Availability Zone in the US-EAST-1 Region.
- gregwtmtno 9y agoI'm having problems in us-east-1c
- misframer 9y agoThe AZ names are different for each account.
- swvjeff 9y agoWow, how did I not know this? Thanks for the info.
- vacri 9y agoThey randomise the AZ letters, because humans tend to shove things in 'a' before anywhere else - it's a psychological load-balancer. If you want to figure out which of your AZs corresponds to another account's AZs, you can compare spot-prices, which are individual per AZ. Also, for some reason, my account doesn't have a 'b', just a-c-d-e. Weird.
- misframer 9y agoThey also introduced us-east-1f, and it's the same for everyone because it's new :).
- misframer 9y agoNot sure why I was downvoted. This was mentioned in an issue of Last Week in AWS [0]. [0] https://snarkive.lastweekinaws.com/issue_14-_US-East-1_Drops_An_F-Bomb.html https://snarkive.lastweekinaws.com/issue_14-_US-East-1_Drops...
- nodesocket 9y agoInteresting that the idea is to distribute load across the zones (because people default to using A), but Google Cloud zones are uniform across all accounts and it seems to work for Google.
- deleted 9y ago[deleted]
- JonRB 9y agoInteresting! Do you know anywhere I can read more about this?
- misframer 9y agoYep, it's in the documentation [0]. [0] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-availability-zones http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-reg...
- gregwtmtno 9y agoThanks! did not know that.
- rickpmg 9y ago1d here I heard that aws actually scrambled availability zones... so your 1a might be my 1c, etc.. Haven't confirmed it, but the comments seem to bear that out.
- misframer 9y agoIt's in the documentation [0]. [0] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-availability-zones http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-reg...
- thenayr 9y agoYep, phone just exploded with alerts
- chadr 9y agoYes
- rickpmg 9y agoyes
- hyperanthony 9y agoYes, seeing issues with the autoscaling API for a single zone in us-east-1.
- deleted 9y ago[deleted]
- dijit 9y agous-east-1 is the most common region for failures. It's literally a joke between me and my friend, when either of us are unavailable for social time.
- amichal 9y agocheck https://phd.aws.amazon.com/phd/home?region=us-east-1#/event-log https://phd.aws.amazon.com/phd/home?region=us-east-1#/event-... when this happens... As of now it says for me "04:36 PM PDT We are investigating network connectivity issues for some instances in a single Availability Zone in the US-EAST-1 Region. 04:58 PM PDT We can confirm that some instances are unreachable and some EBS volumes are experiencing degraded performance in a single Availability Zone in the US-EAST-1 Region. Engineers are engaged and we are working to resolve the issue. 05:05 PM PDT We have identified the root cause and are beginning to see recovery for instances and EBS volumes in the affected Availability Zone in the US-EAST-1 Region. We continue to work toward full resolution. "
- leesalminen 9y agoHad a similar issue earlier today with Rackspace's version of EBS.
- deleted 9y ago[deleted]
- misterbowfinger 9y agoyup
- cstejerean 9y agous-east-1 is the region that has most issues. Every time you hear about an AWS outage it's typically just us-east-1, and it brings down half the internet. It's really puzzling why everyone keeps hosting their projects in us-east-1. The advantage of that region is that it tends to get new instance types and services first. And if you need to be on the bleeding edge, then you have to agree to deal with some risks. But for everyone else that doesn't need the bleeding edge, why not just run your instances in us-west-2 or us-east-2 for example? I've run services in us-west-2 for years and I've never had to deal with fallout from an AWS outage.
- koolba 9y agoBecause we want to be colo with everybody else that's on us-east-1. It makes for lower latency, higher transfers, and when shit hits the fan none of your customers notice that your SaaS product is down because they can't access their entire site!
- debaserab2 9y agoI think you answered your own question. In the case of AWS, sometimes being "bleeding edge" means you're allowing amazon to scaffold infrastructure with a new service that you'd otherwise have to deal with yourself. For a small company, an AWS service being bleeding edge is still a lot more resilient than doing it yourself.
- deleted 9y ago[deleted]
- cstejerean 9y agoI doubt that the vast majority of customers in us-east-1 are using bleeding edge services. By the time you figure out how to integrate some new AWS offering into your infrastructure it has probably been rolled out to the other regions and the kinks worked out. Unless you need latest and greatest GPU offerings for DL, in which case maybe.
- paulddraper 9y agoOnly two regions have good availability for AWS services. For example, email (SES). Only Virgina and Oregon. And Oregon goes down too. If all you needed was VPS, you'd probably find better offerings elsewhere.
- tuna 9y agoalways.
- tj-teej 9y agolol
- whatthesmack 9y agoIs this related to the L3 fiber cut and related Comcast outages in the northeast?
- encoderer 9y agoCronitor saw our customers first impacted at 4:29, largely recovered by 5:04. Another interesting thing is that AZ identifiers are randomized from customer-to-customer so when people report their failures, like here in this thread, it can sometimes seem like a problem is region-wide when in fact it's isolated to an AZ.
- dmourati 9y agoI don't follow your point about randomized AZs. I agree they are randomized and have observed and correlated region letters across multiple customers/accounts. How does that make it seem like a problem is region wide though?
- jimwalsh 9y agoAmazon are starting to push 'new' features and services to Ohio pretty quickly if you still need a non-Virginia AWS East Coast location.
- coreyw 9y agoThis is exactly why you should design your services to run in multiple availability zones to mitigate issues like this. We run our most critical services in at least 3 availability zones and we are moving the rest of our services soon as well. While these problems are unfortunate, it is part of relying on Amazon to manage resources. Always plan for service degradation and look for ways to mitigate against issues like this.
- tuna-piano 9y agoSome services, like AWS Redshift do not allow multi-AZ deployments. Not very helpful when Redshift didn't have a single note in their status page for 30+ minutes after it went down. Any bets on whether the status page will show up as Red for EC2/Redshift tomorrow? I'll take 100 to 1 odds for $1 that it won't be red.
- jwaldrip 9y agoHighly recommended switching to google cloud.