5 ms·

Is it possible for AWS to have a multi-region outage - as in is there anything connecting them that could bring them all (or several) down at once? (Apart from

by smegel 9y ago

Is it possible for AWS to have a multi-region outage - as in is there anything connecting them that could bring them all (or several) down at once? (Apart from the result of a botched patching or update to the core software stack that was done worldwide at the same time and hopefully never happens).

HatchedLake721 9y ago
No. Hence them rolling out new features region by region.
dmurray 9y ago
A major solar flare and coronal mass ejection? It wouldn't just be Amazon that was affected, though.
dijit 9y ago
Unless you work at amazon, you can't know that. It looks very separated on the outside, but I've worked in so many companies that have appeared incredibly competent externally but have "snowflake" servers which keep things ticking over- Given Bezos treatment of workers I have absolutely no confidence that everything is as cleanly engineered as they claim.
blazespin 9y ago
Yes, there are ways to bring down all of their arch at once, but you'd have to get through a lot of barriers to do it.
mrep 9y ago
That would go against a core principle at aws which is to have every region completely isolated. Also, deployments are designed to be exponential and no region should ever have a cross region dependency.
askvictor 9y ago
A cascading electrical grid failure? I don't know if there are any interconnects between the regions with the DC's, but if there were that might be a concern. Though at that stage, presumably most of the US is without power, hence not so much need for AWS.
- smegel 9y ago
  Well I guess a nationwide power outage will have bigger implications than Netflix going down...
  - yjftsjthsd-h 9y ago
    Yeah, it might take down Netflix and YouTube! Then what are we supposed to do while we wait for the electricity to come back on?!
- plandis 9y ago
  I think each DC has at least two power sources and probably a backup generator. I think that's why cloud providers have been so reluctant to open in Africa, diversified power is apparently a problem.
Rjevski 9y ago
Well given that you can manage services across all regions from a single web interface, I assume someone compromising this web interface would be able to control and bring stuff down across all regions.
- jrimbault 9y ago
  Do you think there's a global world admin web console ?
  - Rjevski 9y ago
    Well as a customer how do you interact with it? Do you have one admin interface per region with its own URL or do you just go on an AWS homepage, login and control everything from there?
    - TrickyRick 9y ago
      > Do you have one admin interface per region This ^^^ When I go to login, it redirects me to: https://us-east-1.signin.aws.amazon.com https://us-east-1.signin.aws.amazon.com And for example the S3 console is on the URL https://console.aws.amazon.com/s3/home?region=us-east-1 https://console.aws.amazon.com/s3/home?region=us-east-1 Of course this doesn't mean I can't easily switch to a different region, or for that matter control all my regions from a single interface, but there is some distinction.
    - plandis 9y ago
      Separate end points. Also, AWS recommends that you might want multiple different accounts for more isolation [1] [1] https://aws.amazon.com/answers/account-management/aws-multi-account-security-strategy/ https://aws.amazon.com/answers/account-management/aws-multi-...
    - jrimbault 9y ago
      I meant like truly admin, amazon intranet admin.
dantiberian 9y ago
My memory may be wrong, but I thought several regions were affected by the recent S3 outage? Also, I suspect that if us-east-1 went down completely that that would have a debilitating effect on the others.
TrickyRick 9y ago
Just bringing down us-east was enough to cause quite a bit of trouble recently: https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/

by smegel 9y ago

HatchedLake721 9y ago

No. Hence them rolling out new features region by region.

dmurray 9y ago

A major solar flare and coronal mass ejection? It wouldn't just be Amazon that was affected, though.

dijit 9y ago

Unless you work at amazon, you can't know that. It looks very separated on the outside, but I've worked in so many companies that have appeared incredibly competent externally but have "snowflake" servers which keep things ticking over- Given Bezos treatment of workers I have absolutely no confidence that everything is as cleanly engineered as they claim.

blazespin 9y ago

Yes, there are ways to bring down all of their arch at once, but you'd have to get through a lot of barriers to do it.

mrep 9y ago

That would go against a core principle at aws which is to have every region completely isolated. Also, deployments are designed to be exponential and no region should ever have a cross region dependency.

askvictor 9y ago

A cascading electrical grid failure? I don't know if there are any interconnects between the regions with the DC's, but if there were that might be a concern. Though at that stage, presumably most of the US is without power, hence not so much need for AWS.

smegel 9y ago
Well I guess a nationwide power outage will have bigger implications than Netflix going down...
- yjftsjthsd-h 9y ago
  Yeah, it might take down Netflix and YouTube! Then what are we supposed to do while we wait for the electricity to come back on?!
plandis 9y ago
I think each DC has at least two power sources and probably a backup generator. I think that's why cloud providers have been so reluctant to open in Africa, diversified power is apparently a problem.

Rjevski 9y ago

Well given that you can manage services across all regions from a single web interface, I assume someone compromising this web interface would be able to control and bring stuff down across all regions.

jrimbault 9y ago
Do you think there's a global world admin web console ?
- Rjevski 9y ago
  Well as a customer how do you interact with it? Do you have one admin interface per region with its own URL or do you just go on an AWS homepage, login and control everything from there?
  - TrickyRick 9y ago
    > Do you have one admin interface per region This ^^^ When I go to login, it redirects me to: https://us-east-1.signin.aws.amazon.com https://us-east-1.signin.aws.amazon.com And for example the S3 console is on the URL https://console.aws.amazon.com/s3/home?region=us-east-1 https://console.aws.amazon.com/s3/home?region=us-east-1 Of course this doesn't mean I can't easily switch to a different region, or for that matter control all my regions from a single interface, but there is some distinction.
  - plandis 9y ago
    Separate end points. Also, AWS recommends that you might want multiple different accounts for more isolation [1] [1] https://aws.amazon.com/answers/account-management/aws-multi-account-security-strategy/ https://aws.amazon.com/answers/account-management/aws-multi-...
  - jrimbault 9y ago
    I meant like truly admin, amazon intranet admin.

dantiberian 9y ago

My memory may be wrong, but I thought several regions were affected by the recent S3 outage? Also, I suspect that if us-east-1 went down completely that that would have a debilitating effect on the others.

TrickyRick 9y ago

Just bringing down us-east was enough to cause quite a bit of trouble recently: https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/