8 ms·
I'm wondering why your and other companies haven't just evicted themselves from us-east-1. It's the worst region for outages and it's not even close. Our compa
by outworlder 1y ago
I'm wondering why your and other companies haven't just evicted themselves from us-east-1. It's the worst region for outages and it's not even close.
Our company decided years ago to use any region other than us-east-1.
Of course, that doesn't help with services that are 'global', which usually means us-east-1.
- lordnacho 1y agoIs there some reason why "global" services aren't replicated across regions? I would think a lot of clients would want that.
- bcrosby95 1y agoGlobal replication is hard and if they weren't designed with that in mind its probably a whole lot of work.
- ineedasername 1y agoI thought part of the point of using AWS was that such things were pretty much turnkey?\
- DevelopingElk 1y agoMostly AWS relies on each region being its own isolated copy of each service. It gets tricky when you have globalized services like IAM. AWS tries to keep those to a minimum.
- rhplus 1y agoData residency laws may be a factor in some global/regional architectures.
- lordnacho 1y agoSo provide a way to check/uncheck which zones you want replication to. Most people aren't going to need more than a couple of alternatives, and they'll know which ones will work for them legally.
- JoshTriplett 1y ago> Is there some reason why "global" services aren't replicated across regions? On AWS's side, I think us-east-1 is legacy infrastructure because it was the first region, and things have to be made replicable. For others on AWS who aren't AWS themselves: because AWS outbound data transfer is exorbitantly expensive. I'm building on AWS, and AWS's outbound data transfer costs are a primary design consideration for potential distribution/replication of services.
- me551ah 1y agoIt is absolutely crazy how much AWS charges for data. Internet access in general has become much cheaper and Hetzner gives unlimited AWS. I don't recall AWS ever decreasing prices for outbound data transfer
- Sanzig 1y agoI think there's two reasons: one, it makes them gobs of money. Two, it discourages customers from building architectures which integrate non-AWS services, because you have to pay the data transfer tax. This locks everyone in. And yes, AWS' rates are highway robbery. If you assume $1500/mo for a 10 Gbps port from a transit provider, you're looking at $0.0005/GB with a saturated link. At a 25% utilization factor, still only $0.002/GB. AWS is almost 50 times that. And I guarantee AWS gets a far better rate for transit than list price, so their profit margin must be through the roof.
- JoshTriplett 1y ago> I think there's two reasons: one, it makes them gobs of money. Two, it discourages customers from building architectures which integrate non-AWS services, because you have to pay the data transfer tax. This locks everyone in. Which makes sense, but even their rates for traffic between AWS regions are still exorbitant. $0.10/GB for transfer to the rest of the Internet somewhat discourages integration of non-Amazon services (though you can still easily integrate with any service where most of your bandwidth is inbound to AWS), but their rates for bandwidth between regions are still in the $0.01-0.02/GB range, which discourages replication and cross-region services. If their inter-region bandwidth pricing was substantially lower, it'd be much easier to build replicated, highly available services atop AWS. As it is, the current pricing encourages keeping everything within a region, which works for some kinds of services but not others.
- DevelopingElk 1y agoMy guess is that for IAM it has to do with consistency and security. You don't want regions disagreeing on what operations are authorized. I'm sure the data store could be distributed, but there might be some bad latency tradeoffs. The other concerns could have to do with the impact of failover to the backup regions.
- belter 1y agoRegions disagree on what operations are authorized. :-) IAM uses eventual consistency. As it should... "Changes that I make are not always immediately visible": - "...As a service that is accessed through computers in data centers around the world, IAM uses a distributed computing model called eventual consistency. Any changes that you make in IAM (or other AWS services), including attribute-based access control (ABAC) tags, take time to become visible from all possible endpoints. Some delay results from the time it takes to send data from server to server, replication zone to replication zone, and Region to Region. IAM also uses caching to improve performance, but in some cases this can add time. The change might not be visible until the previously cached data times out... ...You must design your global applications to account for these potential delays. Ensure that they work as expected, even when a change made in one location is not instantly visible at another. Such changes include creating or updating users, groups, roles, or policies. We recommend that you do not include such IAM changes in the critical, high availability code paths of your application. Instead, make IAM changes in a separate initialization or setup routine that you run less frequently. Also, be sure to verify that the changes have been propagated before production workflows depend on them..." https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot.html#troubleshoot_general_eventual-consistency https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoo...
- zikduruqe 1y ago"Is there some reason why "global" services aren't replicated across regions?" us-east-1 is so the government to slurp up all the data. /tin-foil hat
- jedberg 1y agoSome AWS services are only available in us-east-1. Also a lot of people have not built their infra to be portable and the occasional outage isn't worth the cost and effort of moving out.
- twistedpair 1y agoServices like SES Inbound are only available in 2x US regions. AWS isn't great about making all services available in all regions :/
- kondro 1y agoOne of those still isn’t us-east-1 though and email isn’t latency-bound.
- zamalek 1y agoWe're on Azure and they are worse in every aspect, bad deployment of services, and status pages that are more about PR than engineering. At this point, is there any cloud provider that doesn't have these problems? (GCP is a non-starter because a false-positive YouTube TOS violation get you locked out of GCP[1]). [1]: https://9to5google.com/2021/02/26/stadia-port-of-terraria-cancelled-after-co-creator-is-locked-out-of-his-google-accounts/ https://9to5google.com/2021/02/26/stadia-port-of-terraria-ca...
- ecshafer 1y agoGlobal auth is and has been a terrible idea.
- sleepybrett 1y agoSo did a previous company i worked at, all our stuff was in west-2.. then east-1 went down and some global backend services that aws depended on also went down and effected west-2. I'm not sure a lot of companies are really looking at the costs of multi-region resiliency and hot failovers vs being down for 6 hours every year or so and writing that check.
- DrBenCarson 1y agoYep. Many, many companies are fine saying “we’re going to be no more available than AWS is.”
- frankchn 1y agoCustomers are generally a lot more understanding if half the internet goes down at the same time as you.
- chrisweekly 1y agoYes, and that's a major reason so many just use us-east-1.
- perching_aix 1y agocheapest + has the most capacity
- andrewl-hn 1y agoSeveral reasons, really: 1. The main one: it's the cheapest region, so when people select where to run their services they pick it because "why pay more?" 2. It's the default. Many tutorials and articles online show it in the examples, many deployment and other devops tools use it as a default value. 3. Related to n.2. AI models generate cloud configs and code examples with it unless asked otherwise. 4. It's location make it Europe-friendly, too. If you have a small service and you'd like to capture European and North American audience from a single location us-east-1 is a very good choice. 5. Many Amazon features are available in that region first and then spread out to other locations. 6. It's also a region where other cloud providers and hosting companies offer their services. Often there's space available in a data center not far from AWS-running racks. In hybrid cloud scenarios where you want to connect bits of your infrastructure running on AWS and on some physical hardware by a set of dedicated fiber optic lines us-east-1 is the place to do it. 7. Yes, for AWS deployments it's an experimental location that has higher risks of downtime compared to other regions, but in practice when a sizable part of us-east-1 is down other AWS services across the world tend to go down, too (along with half of the internet). So, is it really that risky to run over there, relatively speaking? It's the world's default hosting location, and today's outages show it.
- derefr 1y ago> it's the cheapest region In every SKU I've ever looked at / priced out, all of the AWS NA regions have ~equal pricing. What's cheaper specifically in us-east-1? > Europe-friendly Why not us-east-2? > Many Amazon features are available in that region first and then spread out to other locations. Well, yeah, that's why it breaks. Using not-us-east-1 is like using an LTS OS release: you don't get the newest hotness, but it's much more stable as a "build it and leave it alone" target. > It's also a region where other cloud providers and hosting companies offer their services. Often there's space available in a data center not far from AWS-running racks. This is a better argument, but in practice, it's very niche — 2-5ms of speed-of-light delay doesn't matter to anyone but HFT folks; anyone else can be in a DC one state away with a pre-arranged tier1-bypassing direct interconnect, and do fine. (This is why OVH is listed on https://www.cloudinfrastructuremap.com/ https://www.cloudinfrastructuremap.com/ despite being a smaller provider: their DCs have such interconnects.) For that matter, if you want "low-latency to North America and Europe, and high-throughput lowish-latency peering to many other providers" — why not Montreal [ca-central-1]? Quebec might sound "too far north", but from the fiber-path perspective of anywhere else in NA or Europe, it's essentially interchangeable with Virginia.
- indoordin0saur 1y agoWe have discussions coming up to evict ourselves from AWS entirely. Didn't seem like there was much of an appetite for it before this but now things might have changed. We're still small enough of a company to where the task isn't as daunting as it might otherwise be.
- nijave 1y agoFor us, we had some minor impacts but most stuff was stable. Our bigger issue was 3rd party SaaS also hosted on us-east-1 (Snowflake and CircleCI) which broke CI and our data pipeline
- Eridrus 1y agoThis was a major issue, but it wasn't a total failure of the region. Our stuff is all in us-east-1, ops was a total shitshow today (mostly because many 3rd party services besides aws were down/slow), but our prod service was largely "ok", a total of <5% of customers were significantly impacted because existing instances got to keep running. I think we got a bit lucky, but no actual SLAs were violated. I tagged the postmortem as Low impact despite the stress this caused internally. We definitely learnt something here about both our software and our 3rd party dependencies.
- oofbey 1y agoOne advantage to being in the biggest region: when it goes down the headlines all blame AWS, not you. Sure you’re down too, but absolutely everybody knows why and few think it’s your fault.