4 ms·

Awesome Google... Now learn what an availability zone is and stop creating them with firewalls across the same data center. Oh and make your data centers small

by belter 2y ago

Awesome Google... Now learn what an availability zone is and stop creating them with firewalls across the same data center. Oh and make your data centers smaller. Not so big they can be seen in Google Maps. Because otherwise, you will be unable to move those whale sized workloads to an alternative. https://youtu.be/mDNHK-SzXEM?t=564 https://youtu.be/mDNHK-SzXEM?t=564 https://news.ycombinator.com/item?id=35713001 https://news.ycombinator.com/item?id=35713001 "Unmasking Google Cloud: How to Determine if a Region Supports Physical Zone Separation" - https://cagataygurturk.medium.com/unmasking-google-cloud-how-to-determine-if-a-region-supports-physical-zone-separation-1a7bfbb3d280 https://cagataygurturk.medium.com/unmasking-google-cloud-how...

tecleandor 2y ago
Making a datacenter not visible from Google Maps, at least on most big cities where Google zones are deployed, would mean making them smaller than a car. Or even smaller than a dishwasher. If I check London (where europe-west2 is kinda located) on Google Maps right now, I can easily discern manhole covers or people. If I check Jakarta (Asia-southeast2) things smaller than a car get confusing, but you can definitely see them.
- belter 2y ago
  Your comment does not address the essence of the point I was trying to make. If you have a monstrous data-center, instead of many smaller, in relative size, you are putting too many eggs on a giant basket.
  - joshuamorton 2y ago
    What if you have dozens of big data centers?
    - jiggawatts 2y ago
      To reinforce your point: The scale of cloud data centres reflects the scale of their customer base, not the size of the basket for each individual customer. Larger data centres actually improve availability through several mechanisms: more power components such as generators means the failure of any one is just a few percent instead of a total blackout. You can also partition core infrastructure like routers and power rails into more fault domains and update domains. Some large clouds have two update domains and five fault domains on top of three zones that are more than 10km apart. You can’t beat ~30 individual partitions with your data centres at a reasonable cost!
      - belter 2y ago
        I provided three different references. Despite the massive downvotes on my comment I guess by Google engineers, as a troll...:-)I take comfort on the fact nobody was able to advance a reference to prove me wrong.
        joshuamorton 2y ago
        You haven't actually made an argument. It is true that the nomenclature "AWS Availability Zone" has a different meaning than "GCP Zone" when discussing the physical separation between zones within the same region. It's unclear why this is inherently a bad thing, as long as them same overall level of reliability is achieved.
        belter 2y ago
        The phrase "as long as the same overall level of reliability is achieved" is logically flawed when discussing physically co-located vs. geographically separated infrastructure.
        joshuamorton 2y ago
        Justify that claim. In my experience, the set of issues that would affect 2 buildings close to each other, but not two buildings a mile apart, is vanishingly small, usually just last mile fiber cuts or power issues (which are rare and mitigated by having multiple independent providers), as well as issues like building fires (which are exceedingly rare, we know of, perhaps two of notable impact in more than a decade across the big three cloud providers). Everything else is done at the zone level no matter what (onsite repair work, rollouts, upgrades, control plane changes, etc.) or can impact an entire region (non-last mile fiber or power cuts, inclement weather, regional power starvation, etc.) There is a potential gain from physical zone isolation, but it protects against a relatively small set of issues. Is it really better to invest in that, or to invest the resources in other safety improvements?
        2y ago
traceroute66 2y ago
[flagged]
- anewplace 2y ago
  This isn't even close to true. You can just go on Google Maps and visually see the literally *hundreds* of wholly-owned and custom built data centers from AWS, MS, and Google. Edge locations (like Cloud CDN) are often in colos, but the main regions compute/storage are not. Most of them are even labeled on Google Maps. Here's a couple search terms you can just type into Google Maps and see a small fraction of what I mean: - "Google Data Center Berkeley County" - "Microsoft Data Center Boydton" - "GXO council bluffs" (two locations will appear, both are GCP data centers) - "Google Data Center - Henderson" - "Microsoft - DB5 Datacentre" (this one is in Dublin, and is huuuuuge) - "Meta Datacenter Clonee" - "Google Data Center (New Albany)" (just to the east of this one is a massive Meta data center campus, and to the immediate east of it is a Microsoft data center campus under construction) And that's just a small sample. There are hundreds of these sites across the US. You're somewhat right that a lot of international locations are colocated in places like Equinix data centers, but even then it's not all of them and varies by country (for example in Dublin they mostly all have their own buildings, not colo). If you know where to look and what the buildings look like, the custom-build and self-owned data centers from the big cloud providers are easy to spot since they all have their own custom design.
  - traceroute66 2y ago
    [flagged]
    - anewplace 2y ago
      Yea, you're not the only "insider" here. And you're 100% wrong. Just because you completely misunderstand what those Amazon/MS employees are doing in those buildings doesn't mean that you know what you're talking about. The big cloud players have the vast majority of their compute and storage hosted out of their own custom built and self-owned data centers. The stuff you see in colos is just the edge locations like Cloudfront and Cloud CDN, or the new-ish offerings like AWS Local Zones (which are a mix between self-owned and colo, depending on how large the local zone is). Most of this is publicly available by just reading sites like datacenterdynamics.com regularly, btw. No insider knowledge needed.
nl 2y ago
To address the availability point of your comment, Google's terminology is slightly different to AWS. On GCP it sounds like you want to have a multi region architecture, not multi-zone (if you want firewalls outside the same data center). > Resources that live in a zone, such as virtual machine instances or zonal persistent disks, are referred to as zonal resources. Other resources, like static external IP addresses, are regional. Regional resources can be used by any resource in that region, regardless of zone, while zonal resources can only be used by other resources in the same zone. https://cloud.google.com/compute/docs/regions-zones https://cloud.google.com/compute/docs/regions-zones (No affiliation with Google, just had a similar confusion at one point)
- erik_seaberg 2y ago
  You also need to go multi-region with AWS. I liked their AZ story but in practice it hasn't avoided multi-zone outages (maybe deploys?)

by belter 2y ago

tecleandor 2y ago

Making a datacenter not visible from Google Maps, at least on most big cities where Google zones are deployed, would mean making them smaller than a car. Or even smaller than a dishwasher. If I check London (where europe-west2 is kinda located) on Google Maps right now, I can easily discern manhole covers or people. If I check Jakarta (Asia-southeast2) things smaller than a car get confusing, but you can definitely see them.

belter 2y ago
Your comment does not address the essence of the point I was trying to make. If you have a monstrous data-center, instead of many smaller, in relative size, you are putting too many eggs on a giant basket.
- joshuamorton 2y ago
  What if you have dozens of big data centers?
  - jiggawatts 2y ago
    To reinforce your point: The scale of cloud data centres reflects the scale of their customer base, not the size of the basket for each individual customer. Larger data centres actually improve availability through several mechanisms: more power components such as generators means the failure of any one is just a few percent instead of a total blackout. You can also partition core infrastructure like routers and power rails into more fault domains and update domains. Some large clouds have two update domains and five fault domains on top of three zones that are more than 10km apart. You can’t beat ~30 individual partitions with your data centres at a reasonable cost!
    - belter 2y ago
      I provided three different references. Despite the massive downvotes on my comment I guess by Google engineers, as a troll...:-)I take comfort on the fact nobody was able to advance a reference to prove me wrong.
      - joshuamorton 2y ago
        You haven't actually made an argument. It is true that the nomenclature "AWS Availability Zone" has a different meaning than "GCP Zone" when discussing the physical separation between zones within the same region. It's unclear why this is inherently a bad thing, as long as them same overall level of reliability is achieved.
        belter 2y ago
        The phrase "as long as the same overall level of reliability is achieved" is logically flawed when discussing physically co-located vs. geographically separated infrastructure.
        joshuamorton 2y ago
        Justify that claim. In my experience, the set of issues that would affect 2 buildings close to each other, but not two buildings a mile apart, is vanishingly small, usually just last mile fiber cuts or power issues (which are rare and mitigated by having multiple independent providers), as well as issues like building fires (which are exceedingly rare, we know of, perhaps two of notable impact in more than a decade across the big three cloud providers). Everything else is done at the zone level no matter what (onsite repair work, rollouts, upgrades, control plane changes, etc.) or can impact an entire region (non-last mile fiber or power cuts, inclement weather, regional power starvation, etc.) There is a potential gain from physical zone isolation, but it protects against a relatively small set of issues. Is it really better to invest in that, or to invest the resources in other safety improvements?
        2y ago

traceroute66 2y ago

[flagged]

anewplace 2y ago
This isn't even close to true. You can just go on Google Maps and visually see the literally *hundreds* of wholly-owned and custom built data centers from AWS, MS, and Google. Edge locations (like Cloud CDN) are often in colos, but the main regions compute/storage are not. Most of them are even labeled on Google Maps. Here's a couple search terms you can just type into Google Maps and see a small fraction of what I mean: - "Google Data Center Berkeley County" - "Microsoft Data Center Boydton" - "GXO council bluffs" (two locations will appear, both are GCP data centers) - "Google Data Center - Henderson" - "Microsoft - DB5 Datacentre" (this one is in Dublin, and is huuuuuge) - "Meta Datacenter Clonee" - "Google Data Center (New Albany)" (just to the east of this one is a massive Meta data center campus, and to the immediate east of it is a Microsoft data center campus under construction) And that's just a small sample. There are hundreds of these sites across the US. You're somewhat right that a lot of international locations are colocated in places like Equinix data centers, but even then it's not all of them and varies by country (for example in Dublin they mostly all have their own buildings, not colo). If you know where to look and what the buildings look like, the custom-build and self-owned data centers from the big cloud providers are easy to spot since they all have their own custom design.
- traceroute66 2y ago
  [flagged]
  - anewplace 2y ago
    Yea, you're not the only "insider" here. And you're 100% wrong. Just because you completely misunderstand what those Amazon/MS employees are doing in those buildings doesn't mean that you know what you're talking about. The big cloud players have the vast majority of their compute and storage hosted out of their own custom built and self-owned data centers. The stuff you see in colos is just the edge locations like Cloudfront and Cloud CDN, or the new-ish offerings like AWS Local Zones (which are a mix between self-owned and colo, depending on how large the local zone is). Most of this is publicly available by just reading sites like datacenterdynamics.com regularly, btw. No insider knowledge needed.

nl 2y ago

To address the availability point of your comment, Google's terminology is slightly different to AWS. On GCP it sounds like you want to have a multi region architecture, not multi-zone (if you want firewalls outside the same data center). > Resources that live in a zone, such as virtual machine instances or zonal persistent disks, are referred to as zonal resources. Other resources, like static external IP addresses, are regional. Regional resources can be used by any resource in that region, regardless of zone, while zonal resources can only be used by other resources in the same zone. https://cloud.google.com/compute/docs/regions-zones https://cloud.google.com/compute/docs/regions-zones (No affiliation with Google, just had a similar confusion at one point)

erik_seaberg 2y ago
You also need to go multi-region with AWS. I liked their AZ story but in practice it hasn't avoided multi-zone outages (maybe deploys?)