11 ms·

Oracle also had issues around the same time: https://ocistatus.oraclecloud.com/#/incidents/ocid1.oraclecloudincident.oc1.phx.amaaaaaavwew44aa7zoskanlspjh4ll6wxh

by dgl 4y ago

Oracle also had issues around the same time: https://ocistatus.oraclecloud.com/#/incidents/ocid1.oraclecloudincident.oc1.phx.amaaaaaavwew44aa7zoskanlspjh4ll6wxhwxrbkbed4d4cnupxexzqzvlyq https://ocistatus.oraclecloud.com/#/incidents/ocid1.oraclecl... Obviously the weather is easy to blame, but I wonder if the underlying cause is the same datacenter. It's kind of annoying when clouds use availability zones as such an opaque thing, it's not possible to map a zone from one cloud to another and potentially your failure domains overlap. Shameless self promotion: I have a map of all the cloud regions (but not going into as much detail as availability zones): https://cloud-regions.bodge.cloud https://cloud-regions.bodge.cloud -- the clouds just don't publish the data.

threeseed 4y ago
It is possible to infer the locations if you had a lot of time. AWS at least in the past use to co-locate with other third party servers in the data centre. And so you if you had such a server you could ping AWS endpoints to triangulate where physically those servers might be.
- jhugo 4y ago
  I'm pretty sure a majority of AWS AZs globally are still not in AWS-owned-and-operated facilities.
  - sneak 4y ago
    At this point I would be quite surprised if they were not majority AWS owned and operated.
    - jhugo 4y ago
      Of the ones that I know details about in Europe and Asia, not a single one of them are AWS owned and operated. I'm sure the situation is different in the US, and wouldn't be surprised if they own and operate all of their US DCs.
tambre 4y ago
Why does your map have AWS's eu-north-1 in Estonia? Officially it's said to be in Stockholm and I can find no references nor know of it existing there.
- dgl 4y ago
  Strange, I'm not sure where I got that from, will fix.
  - sneak 4y ago
    You have some locations simply pinned to the central squares of cities when that is definitely not where the facilities are. Maybe include a +/- error range on the provided data?
- cjg_ 4y ago
  The eu-north-1 AZ’s are in Västerås, Eskilstuna and Katrineholm, all ~100 km from Stockholm and ~40-80 km from each other.
jeffbee 4y ago
I also think it would be nice if Google prominently stated which cloud regions are hosted in their real datacenters and which are in third party facilities. It's not just that you run the risk of some off-brand piece of junk overheating, but also because they so loudly trumpet their PUE and carbon neutrality, but then resell cloud capacity in other people's facilities where you have no idea what the carbon impact is.
- lupire 4y ago
  That's not how carbon impact works. You don't buy "clean" products. You buy products from vendors who have an overall average mix of production that meats a "clean" threshold, and/or buys credits from someone else who exceeds threshold.
  - sofixa 4y ago
    In Google's (and for instance Scaleway too), the point is that some of their DCs use renewable energy only, thus are "clean". If however you use the "wrong" Google region which is hosted in Equinix's DCs, which happen to be powered by coal, it's not even remotely close. However Google make the distinction and you can easily check which region is "clean" and which isn't.
- dgl 4y ago
  They do tell you the carbon impact right on the region selector: https://cloud.google.com/blog/topics/sustainability/pick-the-google-cloud-region-with-the-lowest-co2 https://cloud.google.com/blog/topics/sustainability/pick-the... and also https://cloud.withgoogle.com/region-picker/ https://cloud.withgoogle.com/region-picker/ is quite nice.
- mike_d 4y ago
  Google doesn't really have a concept of datacenters internally. In simplified terms everything runs on a borg cell, which usually but not always exists in a single building. Within a region one of your GCP services could be running on one cell, with another service running on another in a different building. If a failure happens or maintenance needs to be done, a cell could be drained to another one in the same region.
deleted 4y ago
[deleted]
angch 4y ago
You have some of the empty lat, lon data (e.g. AWS GovCloud (US)) interpreted as at (lat,lon) = (0,0), putting them incorrectly on Null Island.
- dgl 4y ago
  Thanks, hidden.
sofixa 4y ago
For the AZs that's slightly complicated by the fact that at least for AWS, they are randomised (your eu-west-1a isn't necessarily the same as mine), so it might be pretty much impossible to actually check for actual overlap even if you know one DC is used by both Google and Oracle. Spreading out regions seems more appropriate to increase redundancy.
- BillinghamJ 4y ago
  Your own mappings are available in the console and on the API. So definitely still possible to tell
- deleted 4y ago
  [deleted]
- dgl 4y ago
  Amazon does provide a way to map to the real availability zone: https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html https://docs.aws.amazon.com/ram/latest/userguide/working-wit... It's also not appropriate in all cases to spread out, for example the nearest zones to London are around ~7ms away in mainland Europe.
  - klohto 4y ago
    The provided info is only relative to AWS. If you wanna map geographical destination across clouds you’re out of luck unless you have reports with precise location available. Generally, AWS already provides certain assurances that AZs are spread out.
- formerly_proven 4y ago
  I didn't know this, but it makes perfect sense and randomizing the a/b/c is a tried and true solution - e.g. in the electrical grid, L1/2/3 are rotated for each customer, because people tend to connect more stuff to L1 on average.
  - thunderrabbit 4y ago
TheOtherHobbes 4y ago
UK infra - physical and electronic - is typically rated to around 30C. So higher temps are going to cause failures in multiple locations. If yesterday's temps lasted for a week instead of a day or two a lot of essential stuff would stop working.
- nsteel 4y ago
  Carrier-grade (i.e. critical) network equipment should be rated for 40C ambient air and then up to 55C for some short length of time (3 days??) to allow for air conditioning failure. This is what we design and test for. Cheap stuff, including Google's software solutions running on 'commodity hardware', won't handle that. You get what you pay for.
  - ben_w 4y ago
    Does the UK even design with the assumption of air conditioning being necessary and present, let alone designing for it failing?
    - FartyMcFarter 4y ago
      Datacenters always have A/C I thought?
      - ben_w 4y ago
        Thanks :)
    - Relestio 4y ago
      Google and other companies do make risk assessments including temperature scenarios.
    - nsteel 4y ago
      Yes. You must be able to control the temperature of a building densely packed full of hot radiators. You might be able to avoid active AC within the Arctic circle but you'd m still want to filter that incoming (very) cold air, at which point maybe you might as well basically have an AC system. And yes, you must ensure you can survive a failure because these systems do fail (normally when you need them most) and then otherwise everything inside cooks.
exikyut 4y ago
Wow, this map is really cool. I'm (very) idly curious how accurate the reporting for the Sydney region is, because I just looked up the lat/lon, and found myself in the middle of an urban mall environment that I've walked past many times when I've been in the city. Being able to look up at the buildings there and know there are indeed 5 different clouds somewhere up above my head in that specific location would be really cool. Being able to point at a specific building would be even cooler. I do of course (sadly) appreciate the flip side of this coin which is one of (many of) the reasons precise data is not published. So I guess I'm just wondering out loud, probably rhetorically :), how I might find out one day. "<-- That building" is enough resolution for me :) EDIT: A quick Google found baxtel.com (among other websites) has address-level locations for most providers. The buildings are all so boring, haha! (Understandably so though.)
- larrybud 4y ago
  It’s pretty far off, at least for azure. It shows a number of azure regions as being located in downtown urban areas, which is typically not the case

by dgl 4y ago

threeseed 4y ago

It is possible to infer the locations if you had a lot of time. AWS at least in the past use to co-locate with other third party servers in the data centre. And so you if you had such a server you could ping AWS endpoints to triangulate where physically those servers might be.

jhugo 4y ago
I'm pretty sure a majority of AWS AZs globally are still not in AWS-owned-and-operated facilities.
- sneak 4y ago
  At this point I would be quite surprised if they were not majority AWS owned and operated.
  - jhugo 4y ago
    Of the ones that I know details about in Europe and Asia, not a single one of them are AWS owned and operated. I'm sure the situation is different in the US, and wouldn't be surprised if they own and operate all of their US DCs.

tambre 4y ago

Why does your map have AWS's eu-north-1 in Estonia? Officially it's said to be in Stockholm and I can find no references nor know of it existing there.

dgl 4y ago
Strange, I'm not sure where I got that from, will fix.
- sneak 4y ago
  You have some locations simply pinned to the central squares of cities when that is definitely not where the facilities are. Maybe include a +/- error range on the provided data?
cjg_ 4y ago
The eu-north-1 AZ’s are in Västerås, Eskilstuna and Katrineholm, all ~100 km from Stockholm and ~40-80 km from each other.

jeffbee 4y ago

I also think it would be nice if Google prominently stated which cloud regions are hosted in their real datacenters and which are in third party facilities. It's not just that you run the risk of some off-brand piece of junk overheating, but also because they so loudly trumpet their PUE and carbon neutrality, but then resell cloud capacity in other people's facilities where you have no idea what the carbon impact is.

lupire 4y ago
That's not how carbon impact works. You don't buy "clean" products. You buy products from vendors who have an overall average mix of production that meats a "clean" threshold, and/or buys credits from someone else who exceeds threshold.
- sofixa 4y ago
  In Google's (and for instance Scaleway too), the point is that some of their DCs use renewable energy only, thus are "clean". If however you use the "wrong" Google region which is hosted in Equinix's DCs, which happen to be powered by coal, it's not even remotely close. However Google make the distinction and you can easily check which region is "clean" and which isn't.
dgl 4y ago
They do tell you the carbon impact right on the region selector: https://cloud.google.com/blog/topics/sustainability/pick-the-google-cloud-region-with-the-lowest-co2 https://cloud.google.com/blog/topics/sustainability/pick-the... and also https://cloud.withgoogle.com/region-picker/ https://cloud.withgoogle.com/region-picker/ is quite nice.
mike_d 4y ago
Google doesn't really have a concept of datacenters internally. In simplified terms everything runs on a borg cell, which usually but not always exists in a single building. Within a region one of your GCP services could be running on one cell, with another service running on another in a different building. If a failure happens or maintenance needs to be done, a cell could be drained to another one in the same region.

deleted 4y ago

[deleted]

angch 4y ago

You have some of the empty lat, lon data (e.g. AWS GovCloud (US)) interpreted as at (lat,lon) = (0,0), putting them incorrectly on Null Island.

dgl 4y ago
Thanks, hidden.

sofixa 4y ago

For the AZs that's slightly complicated by the fact that at least for AWS, they are randomised (your eu-west-1a isn't necessarily the same as mine), so it might be pretty much impossible to actually check for actual overlap even if you know one DC is used by both Google and Oracle. Spreading out regions seems more appropriate to increase redundancy.

BillinghamJ 4y ago
Your own mappings are available in the console and on the API. So definitely still possible to tell
deleted 4y ago
[deleted]
dgl 4y ago
Amazon does provide a way to map to the real availability zone: https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html https://docs.aws.amazon.com/ram/latest/userguide/working-wit... It's also not appropriate in all cases to spread out, for example the nearest zones to London are around ~7ms away in mainland Europe.
- klohto 4y ago
  The provided info is only relative to AWS. If you wanna map geographical destination across clouds you’re out of luck unless you have reports with precise location available. Generally, AWS already provides certain assurances that AZs are spread out.
formerly_proven 4y ago
I didn't know this, but it makes perfect sense and randomizing the a/b/c is a tried and true solution - e.g. in the electrical grid, L1/2/3 are rotated for each customer, because people tend to connect more stuff to L1 on average.
- thunderrabbit 4y ago

TheOtherHobbes 4y ago

UK infra - physical and electronic - is typically rated to around 30C. So higher temps are going to cause failures in multiple locations. If yesterday's temps lasted for a week instead of a day or two a lot of essential stuff would stop working.

nsteel 4y ago
Carrier-grade (i.e. critical) network equipment should be rated for 40C ambient air and then up to 55C for some short length of time (3 days??) to allow for air conditioning failure. This is what we design and test for. Cheap stuff, including Google's software solutions running on 'commodity hardware', won't handle that. You get what you pay for.
- ben_w 4y ago
  Does the UK even design with the assumption of air conditioning being necessary and present, let alone designing for it failing?
  - FartyMcFarter 4y ago
    Datacenters always have A/C I thought?
    - ben_w 4y ago
      Thanks :)
  - Relestio 4y ago
    Google and other companies do make risk assessments including temperature scenarios.
  - nsteel 4y ago
    Yes. You must be able to control the temperature of a building densely packed full of hot radiators. You might be able to avoid active AC within the Arctic circle but you'd m still want to filter that incoming (very) cold air, at which point maybe you might as well basically have an AC system. And yes, you must ensure you can survive a failure because these systems do fail (normally when you need them most) and then otherwise everything inside cooks.

exikyut 4y ago

Wow, this map is really cool. I'm (very) idly curious how accurate the reporting for the Sydney region is, because I just looked up the lat/lon, and found myself in the middle of an urban mall environment that I've walked past many times when I've been in the city. Being able to look up at the buildings there and know there are indeed 5 different clouds somewhere up above my head in that specific location would be really cool. Being able to point at a specific building would be even cooler. I do of course (sadly) appreciate the flip side of this coin which is one of (many of) the reasons precise data is not published. So I guess I'm just wondering out loud, probably rhetorically :), how I might find out one day. "<-- That building" is enough resolution for me :) EDIT: A quick Google found baxtel.com (among other websites) has address-level locations for most providers. The buildings are all so boring, haha! (Understandably so though.)

larrybud 4y ago
It’s pretty far off, at least for azure. It shows a number of azure regions as being located in downtown urban areas, which is typically not the case