4 ms·

I can't ignore the feeling that Google Cloud is sub par compared to AWS. How did this again cause a multi zone failure. Why haven't they fixed those dependencie

I can't ignore the feeling that Google Cloud is sub par compared to AWS. How did this again cause a multi zone failure. Why haven't they fixed those dependencies the last few times they had a full region failure.

dekhn 4y ago
zones and regions have different definition in google cloud than AWS. Multiple zones are physically co-located and are not truly availability zones because the physical proximity causes shared fates even when they have independent systems (network, power) that should allow one to fail while another doesn't. Even two datacenters in the same city are prey to the same meteor.
- styren 4y ago
  In the same building is quite a bit different than 10km apart, even if a huge meteor would lead to the same conclusion for both scenarios.
- ericpauley 4y ago
  Wow, this is a big issue! Is there any way to guarantee AWS-level physical redundancy in GCP without paying the latency/inter-region data transfer (higher than inter-zone outside US: https://cloud.google.com/vpc/network-pricing#egress-within-gcp https://cloud.google.com/vpc/network-pricing#egress-within-g...) pricing? A nice thing about EC2 is that you're getting a pretty dumb, predictable service. There have been multi-zone or global control plane issues but the physical metal has bona fide redundancy between zones/regions.
- lamontcg 4y ago
  I don't know what it is like these days but us-east AZs used to be in different datacenters that were on different flood plains and power companies. They were just on a very high capacity (for the day) fiber ring. You still had a couple miles of light-delay latency in between them. A sufficiently big enough meteor, or a powerful enough massive hurricane, could probably take out multiple ones at the same time.
  - londons_explore 4y ago
    I believe the Google design is one big pool of machines, perhaps spread across a few buildings, but they hope that any failure only affects a few racks. They will arrange/move workloads such that any one customer will only see an outage in one 'zone'. Clearly that didn't work here,
    - local_crmdgeon 4y ago
      That's .... that's not a great strategy. I get that it's cheap, but incidents like this will cost a LOT more with lost business.
- secondcoming 4y ago
  Then why do they charge extra for cross-zonal traffic?
  - kccqzy 4y ago
    Presumably for inter-zonal traffic, it uses up bandwidth on their B4 network, but intra-zonal traffic does not and has basically unlimited bandwidth. B4: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/41761.pdf https://storage.googleapis.com/pub-tools-public-publication-...
    - dilyevsky 4y ago
      This is not going through b4 but inter-zonal links are still choke points because they are separate networks
- deathanatos 4y ago
  > A cluster represents a distinct physical infrastructure that is housed in a data center. > Google designs zones to minimize the risk of correlated failures caused by physical infrastructure outages And they have stated that the flood "caused a multi-cluster failure". > Zones should be considered a single failure domain within a region. (—GCP's documentation.)
  - sitkack 4y ago
    "should be considered" -> we would like you to think Zones are an artificial construct.
  - illumin8 4y ago
    I guess you just discovered the difference between a GCP Zone and an AWS Availability Zone. By definition, AWS availability zones do not share fault domains, other than a geographic region up to hundreds of miles wide. Even for services used by multiple AZs, such as transit to the Internet and other regions, there are two transit centers operating in separate fault domains. In contrast, many GCP Zones share the same physical datacenter. What they are actually providing you are simply different racks, rows of racks, or rooms in a single physical facility. Caveat emptor.
- deleted 4y ago
  [deleted]
ec109685 4y ago
It was not only a region outage, but a global outage! "GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating-to-zonal https://cloud.google.com/compute/docs/internal-dns#migrating... " Embarrassing.
influx 4y ago
AWS has had datacenters that were flooded, but they failed over and physically moved racks via trucks. Customers never knew.
- local_crmdgeon 4y ago
  This has happened much more than people know.
  - Aeolun 4y ago
    If it happens without me knowing then they’re doing their job very well.
nonethewiser 4y ago
> I can't ignore the feeling that Google Cloud is sub par compared to AWS. That goes without saying at this point. More importantly, it’s proven worse than Azure.
- outworlder 4y ago
  Depending on the metric. I have not seen any incidents where the isolation between customers was breach. Azure had several. Their compute offerings are better. We could go on. On the other hand, Azure was(and still is) upfront about not having AZs - now that they have rolled out, hopefully those are not in the same building.
1letterunixname 4y ago
Well, GCP did fire most of their senior customer techs 18 months ago. Alphabet is most ruled by arbitrary fiat of empire-builders. Killing products and footgun changes are standard.

dekhn 4y ago

zones and regions have different definition in google cloud than AWS. Multiple zones are physically co-located and are not truly availability zones because the physical proximity causes shared fates even when they have independent systems (network, power) that should allow one to fail while another doesn't. Even two datacenters in the same city are prey to the same meteor.

styren 4y ago
In the same building is quite a bit different than 10km apart, even if a huge meteor would lead to the same conclusion for both scenarios.
ericpauley 4y ago
Wow, this is a big issue! Is there any way to guarantee AWS-level physical redundancy in GCP without paying the latency/inter-region data transfer (higher than inter-zone outside US: https://cloud.google.com/vpc/network-pricing#egress-within-gcp https://cloud.google.com/vpc/network-pricing#egress-within-g...) pricing? A nice thing about EC2 is that you're getting a pretty dumb, predictable service. There have been multi-zone or global control plane issues but the physical metal has bona fide redundancy between zones/regions.
lamontcg 4y ago
I don't know what it is like these days but us-east AZs used to be in different datacenters that were on different flood plains and power companies. They were just on a very high capacity (for the day) fiber ring. You still had a couple miles of light-delay latency in between them. A sufficiently big enough meteor, or a powerful enough massive hurricane, could probably take out multiple ones at the same time.
- londons_explore 4y ago
  I believe the Google design is one big pool of machines, perhaps spread across a few buildings, but they hope that any failure only affects a few racks. They will arrange/move workloads such that any one customer will only see an outage in one 'zone'. Clearly that didn't work here,
  - local_crmdgeon 4y ago
    That's .... that's not a great strategy. I get that it's cheap, but incidents like this will cost a LOT more with lost business.
secondcoming 4y ago
Then why do they charge extra for cross-zonal traffic?
- kccqzy 4y ago
  Presumably for inter-zonal traffic, it uses up bandwidth on their B4 network, but intra-zonal traffic does not and has basically unlimited bandwidth. B4: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/41761.pdf https://storage.googleapis.com/pub-tools-public-publication-...
  - dilyevsky 4y ago
    This is not going through b4 but inter-zonal links are still choke points because they are separate networks
deathanatos 4y ago
> A cluster represents a distinct physical infrastructure that is housed in a data center. > Google designs zones to minimize the risk of correlated failures caused by physical infrastructure outages And they have stated that the flood "caused a multi-cluster failure". > Zones should be considered a single failure domain within a region. (—GCP's documentation.)
- sitkack 4y ago
  "should be considered" -> we would like you to think Zones are an artificial construct.
- illumin8 4y ago
  I guess you just discovered the difference between a GCP Zone and an AWS Availability Zone. By definition, AWS availability zones do not share fault domains, other than a geographic region up to hundreds of miles wide. Even for services used by multiple AZs, such as transit to the Internet and other regions, there are two transit centers operating in separate fault domains. In contrast, many GCP Zones share the same physical datacenter. What they are actually providing you are simply different racks, rows of racks, or rooms in a single physical facility. Caveat emptor.
deleted 4y ago
[deleted]

ec109685 4y ago

It was not only a region outage, but a global outage! "GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating-to-zonal https://cloud.google.com/compute/docs/internal-dns#migrating... " Embarrassing.

influx 4y ago

AWS has had datacenters that were flooded, but they failed over and physically moved racks via trucks. Customers never knew.

local_crmdgeon 4y ago
This has happened much more than people know.
- Aeolun 4y ago
  If it happens without me knowing then they’re doing their job very well.

nonethewiser 4y ago

> I can't ignore the feeling that Google Cloud is sub par compared to AWS. That goes without saying at this point. More importantly, it’s proven worse than Azure.

outworlder 4y ago
Depending on the metric. I have not seen any incidents where the isolation between customers was breach. Azure had several. Their compute offerings are better. We could go on. On the other hand, Azure was(and still is) upfront about not having AZs - now that they have rolled out, hopefully those are not in the same building.

1letterunixname 4y ago

Well, GCP did fire most of their senior customer techs 18 months ago. Alphabet is most ruled by arbitrary fiat of empire-builders. Killing products and footgun changes are standard.