6 ms·
Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2
by jedberg 4y ago
Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2."
Now I guess we have to move to us-west-2. :)
Update: looks like it's only one zone anyway, so my statement still stands!
- acwan93 4y agoIn all seriousness, we've been deploying everything on us-west-2, and it seems to have dodged most of the outages recently. Is there something special about that data center?
- arecurrence 4y agoClassically, us-east-1 received most of the hate given its immense size (it used to be several times larger than any other) and status as the first large aws data center. It also seemed to launch new aws features first but that may have been my imagination. If true, I'm sure always running the latest builds was not great for stability. us-west-2 has had outages as well but it is less common, even rare. I've been pushing companies to make their initial deployments onto us-west-2 for over ten years now. I occasionally get kudos messages in my inbox :)
- teknopaul 4y ago93.99999
- doubled112 4y agoThere are six 9s in there. Pretty solid!
- rjh29 4y agoMaybe Amazon should make us-east-1's actual datacenter change depend on the customer, as they do with the AZs :P
- creeble 4y agoDoesn't AWS IoT run only in us-east-1? And I think Alexa skills, if anybody cares about those.
- oxfordmale 4y agoI believe us-east-1 runs some of the control plane and an us-east outage can effectively take a service in a different region offline as it can break IAM authentication
- jedberg 4y agoIt's never been a default datacenter. For a long time the default when you first logged into the console was us-east-1 so a lot of companies set up there (that's where all of reddit was run for a long time and Netflix too). At some point they switched the default to us-east-2. So anyone who is in us-west-2 is there intentionally, which makes me assume there is a smaller footprint there (but I have no idea).
- tinym 4y agoRather the opposite - us-west-2 is big but not the biggest region, or the smallest, or the oldest or newest, it's not partitioned off like the China or GovCloud regions. Because us-west-2 is fairly typical it tends to be one of the last regions to get software updates, after they've been tested in prod elsewhere
- alex_young 4y agoLooks like this particular issue was due to power loss, and for power us-west-2 has one clear advantage: It's power is directly from the Columbia river and highly unlikely to have demand based outages.
- mnordhoff 4y agoMaybe not the entire region. Amazon was reportedly building a data center complex next to the natural gas Hermiston Generating Plant some distance from the river.
- nodesocket 4y agoJinxed for sure. I refuse to deploy resource into us-east-1 unless required by the service.
- deathanatos 4y agoThe AWS status board, posted elsewhere in the comments, seems to think this is an AZ outage, not a regional one. Edit: although, one of our vendors that uses AWS has said that they think ELB registration is impacted (but I don't recall if that's regional?) and R53 is impacted (which is supposed to be global, IIRC). Dunno how much truth there is to it as we don't use AWS directly.
- water-your-self 4y agoAWS is notorious did underreporting and failing to report. They do not have asafe culture and its bad for your career if there is a major outage
- fishnchips 4y agoStay in us-east-1, they provide Chaos Monkey for free. It's a feature.
- sergiotapia 4y agoI'm moving to us-weast-1
- apengwin 4y agowhat kind of compass are ya reading lad?
- tomxor 4y agoI presume they are trying to express an extra cardinal dimension perpendicular to the plane. Deep underground in Bezos new evil lair perhaps?
- sergiotapia 4y agohttps://www.youtube.com/watch?v=RuJNUXT2a9U https://www.youtube.com/watch?v=RuJNUXT2a9U
- _ea1k 4y agoBetter to use us-nouth-3
- jedberg 4y agoChaos Kong is the one that takes out whole regions. ;)
- iasay 4y agoI think it's Chaos Cthulhu the way it's going recently...
- dudeinjapan 4y ago
- highwaylights 4y agoIf us-west-2 goes down in the next few days we’ll expect an explanation.
- yodsanklai 4y agoNaive question: don't people who care about resiliency have their services in more than one datacenter? or datacenter failure is considered such a rare event that's it's not worth the cost/trouble of using more?
- vhgyu75e6u 4y agoAWS works with multiple availability zones (AZ) per region, some products by default deploy in several ones at the same time, while others leave it up to you.
- vineyardmike 4y agoSome people care about it but not enough to justify the added downsides - multi-data center is expensive (you pay per data center) and it’s complex (data sharding/duplication/sync). If you’re Amazon where every second is millions of $ in transactions you care more than StartUp that has 1 request per minute. Even if you accept the risk, you still care when your DC goes down. Also, a large chunk of AWS is managed from a single data center so if that one goes down you may still have issues with your service in another data center.
- jedberg 4y agoAWS makes it pretty easy to operate in multiple AZs within a region (each AZ is considered a separate datacenter but in real life each AZ is multiple datacenters that are really close to each other). That being said, there is still an added cost and complexity to operate in multiple AZs, because you have to synchronize data across the AZs. Also you have to have enough reserved instances to move into when you lose an AZ, because if you're running lean and each zone is serving 33% of your traffic, suddenly the two that are left need to serve 50% each. The bigger companies with overhead reservations will get all the instances before you can launch any on demand during an AZ failure.
- Tehnix 4y ago> each AZ is considered a separate datacenter but in real life each AZ is multiple datacenters that are really close to each other For AWS specifically, I’m fairly certain they maintain a minimum distance and are much more strict on requirements to be on different grids etc than other Cloud providers. A few years ago they were calling out Azure and Google Cloud on exactly what you describe (having data centers essentially on the same street almost).
- papito 4y agoPlease don't move to us-west. We are probably going to have an 11-point earthquake the next day. Thanks!