4 ms·

Some AWS services are only available in us-east-1. Also a lot of people have not built their infra to be portable and the occasional outage isn't worth the cos

Some AWS services are only available in us-east-1. Also a lot of people have not built their infra to be portable and the occasional outage isn't worth the cost and effort of moving out.

twistedpair 1y ago
Services like SES Inbound are only available in 2x US regions. AWS isn't great about making all services available in all regions :/
- kondro 1y ago
  One of those still isn’t us-east-1 though and email isn’t latency-bound.
  - hvb2 1y ago
    Except for OTP codes when doing 2fa in auth
    - kondro 1y ago
      100ms isn’t going to make a difference to email-based OTP. Also, who’s using email-based OTP?
- zamalek 1y ago
  We're on Azure and they are worse in every aspect, bad deployment of services, and status pages that are more about PR than engineering. At this point, is there any cloud provider that doesn't have these problems? (GCP is a non-starter because a false-positive YouTube TOS violation get you locked out of GCP[1]). [1]: https://9to5google.com/2021/02/26/stadia-port-of-terraria-cancelled-after-co-creator-is-locked-out-of-his-google-accounts/ https://9to5google.com/2021/02/26/stadia-port-of-terraria-ca...
  - ecshafer 1y ago
    Global auth is and has been a terrible idea.
  - fragmede 1y ago
    If you can't figure out how to use a different Google account for YouTube from the GCP billing account, I don't know what to say. Google's in the wrong here, but spanner's good shit! (If you can afford it. and you actually need it. you probably don't.)
    - zamalek 1y ago
      The problem isn't specifically getting locked out of GCP (though it is likely to happen for those out of the loop on what happened). It is that Google themselves can't figure out that a social media ban shouldn't affect your business continuity (and access to email or what-have-you). It is an extremely fundamental level of incompetence at Google. One should "figure out" the viability of placing all of one's eggs in the basket of such an incompetent partner. They screwed the authentication issue up and, this is no slippery slope argument, that means they could be screwing other things up (such as being able to contact a human for support, which is what the Terraria developer also had issues with).
  - tapoxi 1y ago
    Don't worry there was a global GCP outage a few months ago
  - derefr 1y ago
    [flagged]
    - hshdhdhj4444 1y ago
      That’s an incredibly long comment that does nothing to explain why a YouTube ToS violation should lead to someone’s GCP services being cut off. Also, Steve Jobs already wrote your comment better. You should have just stolen it. “You’re holding it wrong”.
shermantanktop 1y ago
Same calculation everyone makes but that doesn’t stop them from whining about AWS being less than perfect.
bartread 1y ago
> the occasional outage isn't worth the cost and effort of moving out. And looked at from the perspective of an individual company, as a customer of AWS, the occasional outage is usually an acceptable part of doing business. However, today we’ve seen a failure that has wiped out a huge number of companies used by hundreds of millions - maybe billions - of people, and obviously a huge number of companies globally all at the same time. AWS has something like 30% of the infra market so you can imagine, and most people reading this will to some extent have experienced, the scale of disruption. And the reality is that whilst bigger companies, like Zoom, are getting a lot of the attention here, we have no idea what other critical and/or life and death services might have been impacted. As an example that many of us would be familiar with, how many houses have been successfully burgled today because Ring has been down for around 8 out of the last 15 hours (at least as I measure it)? I don’t think that’s OK, and I question the wisdom of companies choosing AWS as their default infra and hosting provider. It simply doesn’t seem to be very responsible to be in the same pond as so many others. Were I a legislator I would now be casting a somewhat baleful eye at AWS as a potentially dangerous monopoly, and see what I might be able to do to force organisations to choose from amongst a much larger pool of potential infra providers and platforms, and I would be doing that because these kinds of incidents will only become more serious as time goes on.
- jedberg 1y ago
  You're suffering from survivorship bias. You know that old adage about the bullet holes in the planes, and someone pointed out that you should reinforce that parts without bullet holes, because these are the planes that came back. It's the same thing here. Do you think other providers are better? If people moved to other providers, things would still go down, more likely than not it would be more downtime in aggregate, just spread out so you wouldn't notice as much. At least this way, everyone knows why it's down, our industry has developed best practices for dealing with these kinds of outages, and AWS can apply their expertise to keeping all their customers running as long as possible.
  - bartread 1y ago
    That’s a pretty bold claim. Where’s your data to back it up? More importantly you appear to have misunderstood the scenario I’m trying to avoid, which is the precise situation we’ve seen in the past 24 hours where a very large proportion of internet services go down all at the same time precisely because they’re all using the same provider. And then finally the usual outcome of increased competition is to improve the quality of products and services. I am very aware of the WWII bomber story, because it’s very heavily cited in corporate circles nowadays, but I don’t see that it has anything to do with what I was talking about. AWS is chosen because it’s an acceptable default that’s unlikely to be heavily challenged either by corporate leadership or by those on the production side because it’s good CV fodder. It’s the “nobody gets fired for buying IBM” of the early mid-21st century. That doesn’t make it the best choice though: just the easiest. And viewed at a level above the individual organisation - or, perhaps from the view of users who were faced with failures across multiple or many products and services from diverse companies and organisations - as with today (yesterday!) we can see it’s not the best choice.

Some AWS services are only available in us-east-1. Also a lot of people have not built their infra to be portable and the occasional outage isn't worth the cost and effort of moving out.

twistedpair 1y ago

Services like SES Inbound are only available in 2x US regions. AWS isn't great about making all services available in all regions :/

kondro 1y ago
One of those still isn’t us-east-1 though and email isn’t latency-bound.
- hvb2 1y ago
  Except for OTP codes when doing 2fa in auth
  - kondro 1y ago
    100ms isn’t going to make a difference to email-based OTP. Also, who’s using email-based OTP?
zamalek 1y ago
We're on Azure and they are worse in every aspect, bad deployment of services, and status pages that are more about PR than engineering. At this point, is there any cloud provider that doesn't have these problems? (GCP is a non-starter because a false-positive YouTube TOS violation get you locked out of GCP[1]). [1]: https://9to5google.com/2021/02/26/stadia-port-of-terraria-cancelled-after-co-creator-is-locked-out-of-his-google-accounts/ https://9to5google.com/2021/02/26/stadia-port-of-terraria-ca...
- ecshafer 1y ago
  Global auth is and has been a terrible idea.
- fragmede 1y ago
  If you can't figure out how to use a different Google account for YouTube from the GCP billing account, I don't know what to say. Google's in the wrong here, but spanner's good shit! (If you can afford it. and you actually need it. you probably don't.)
  - zamalek 1y ago
    The problem isn't specifically getting locked out of GCP (though it is likely to happen for those out of the loop on what happened). It is that Google themselves can't figure out that a social media ban shouldn't affect your business continuity (and access to email or what-have-you). It is an extremely fundamental level of incompetence at Google. One should "figure out" the viability of placing all of one's eggs in the basket of such an incompetent partner. They screwed the authentication issue up and, this is no slippery slope argument, that means they could be screwing other things up (such as being able to contact a human for support, which is what the Terraria developer also had issues with).
- tapoxi 1y ago
  Don't worry there was a global GCP outage a few months ago
- derefr 1y ago
  [flagged]
  - hshdhdhj4444 1y ago
    That’s an incredibly long comment that does nothing to explain why a YouTube ToS violation should lead to someone’s GCP services being cut off. Also, Steve Jobs already wrote your comment better. You should have just stolen it. “You’re holding it wrong”.

shermantanktop 1y ago

Same calculation everyone makes but that doesn’t stop them from whining about AWS being less than perfect.

bartread 1y ago

> the occasional outage isn't worth the cost and effort of moving out. And looked at from the perspective of an individual company, as a customer of AWS, the occasional outage is usually an acceptable part of doing business. However, today we’ve seen a failure that has wiped out a huge number of companies used by hundreds of millions - maybe billions - of people, and obviously a huge number of companies globally all at the same time. AWS has something like 30% of the infra market so you can imagine, and most people reading this will to some extent have experienced, the scale of disruption. And the reality is that whilst bigger companies, like Zoom, are getting a lot of the attention here, we have no idea what other critical and/or life and death services might have been impacted. As an example that many of us would be familiar with, how many houses have been successfully burgled today because Ring has been down for around 8 out of the last 15 hours (at least as I measure it)? I don’t think that’s OK, and I question the wisdom of companies choosing AWS as their default infra and hosting provider. It simply doesn’t seem to be very responsible to be in the same pond as so many others. Were I a legislator I would now be casting a somewhat baleful eye at AWS as a potentially dangerous monopoly, and see what I might be able to do to force organisations to choose from amongst a much larger pool of potential infra providers and platforms, and I would be doing that because these kinds of incidents will only become more serious as time goes on.

jedberg 1y ago
You're suffering from survivorship bias. You know that old adage about the bullet holes in the planes, and someone pointed out that you should reinforce that parts without bullet holes, because these are the planes that came back. It's the same thing here. Do you think other providers are better? If people moved to other providers, things would still go down, more likely than not it would be more downtime in aggregate, just spread out so you wouldn't notice as much. At least this way, everyone knows why it's down, our industry has developed best practices for dealing with these kinds of outages, and AWS can apply their expertise to keeping all their customers running as long as possible.
- bartread 1y ago
  That’s a pretty bold claim. Where’s your data to back it up? More importantly you appear to have misunderstood the scenario I’m trying to avoid, which is the precise situation we’ve seen in the past 24 hours where a very large proportion of internet services go down all at the same time precisely because they’re all using the same provider. And then finally the usual outcome of increased competition is to improve the quality of products and services. I am very aware of the WWII bomber story, because it’s very heavily cited in corporate circles nowadays, but I don’t see that it has anything to do with what I was talking about. AWS is chosen because it’s an acceptable default that’s unlikely to be heavily challenged either by corporate leadership or by those on the production side because it’s good CV fodder. It’s the “nobody gets fired for buying IBM” of the early mid-21st century. That doesn’t make it the best choice though: just the easiest. And viewed at a level above the individual organisation - or, perhaps from the view of users who were faced with failures across multiple or many products and services from diverse companies and organisations - as with today (yesterday!) we can see it’s not the best choice.