3 ms·

https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... Multiple Google Cloud services i

https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... Multiple Google Cloud services in the europe-west9 region are impacted. 'Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. We expect some unavailability in the europe-west9 region. There is no current ETA for full recovery of operations in the europe-west9 region at this time. We expected to see an extended outage for some services. Customers are advised to failover to other regions if they are impacted.' can you please explain to me how water intrusion in a data center cause a region failure? there is only one reason. they lied. most likely overhyping the region resiliency. how after seing that can you trust them to host critical apps is beyond belief. and if you look the resolution link I shared they kept hiding reality behind fuzzy language. 'After Regional Spanner, IAM and GCE/PD came back online for the region, almost all of the services in the region fully recovered soon after that. The following services either took longer to recover or had additional impact during the outage. Google Cloud Storage, Google BigQuery, Cloud Bigtable, Cloud SQL, Cloud Pub/Sub, Google Cloud Dataflow Please log a case with our Support team if you would like to receive additional information regarding these services.' please look the last sentence and I can tell you it was way more than a day for those services. try to get additional info on how long bigquery/bigtable were offline.

toast0 3y ago
> can you please explain to me how water intrusion in a data center cause a region failure? there is only one reason. they lied. most likely overhyping the region resiliency. how after seing that can you trust them to host critical apps is beyond belief. Clearly, they fucked up, and didn't distribute critical infrastructure adequately across the region. There's lots of ways this kind of thing happens. I've had 'redundant fiber' that was running in the same bundle and got severed by the same backhoe, for example. The easiest way to determine if there's a dependency on a particular building is to forcibly de-energize all the equipment in it. In this particular case, it seems like there was a mistake in allocation of instances for the regional Spanner database, resulting in that database becoming unavailable when the building was de-energized; and then lots of services depend on regional Spanner, because that's its intended use. It sounds like they did an audit of regional Spanners after this incident, but who knows what other hidden dependencies there are there, or at any cloud provider. Having your critical apps available in multiple distributed datacenters is best practice; having that all in one city isn't really. In an ideal world, you're not dependent on any single entity you don't control; but egress charges make that difficult. Re: Bigquery, there's a separate post for that [1] which seems to indicate "The issue with Google BigQuery has been resolved for all affected users as of Wednesday, 2023-04-26 17:05 US/Pacific." The incident began around "26 Apr 2023 06:25 PDT", so resolution was next-day (but that's about 35 hours). If you've got other data on BigQuery, that's fine; I'm not saying your experience is wrong, I'm saying it doesn't match the public record. [1] https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiq...
- finikytou 3y ago
  its not only spanner multiple services got hit. the console wasnt even accessible. to me it looks like they have one datacenter and in this datacenter multiple "availability zone"
  - toast0 3y ago
    Yeah, because the console and IAM auth rely, and pretty much everything at GCP relies on regional spanner; that's a key point of GCP design for better or worse. It's like if they only had BGP in the one datacenter. If that datacenter turns off, nothing works in the other two that are nearby.

toast0 3y ago

> can you please explain to me how water intrusion in a data center cause a region failure? there is only one reason. they lied. most likely overhyping the region resiliency. how after seing that can you trust them to host critical apps is beyond belief. Clearly, they fucked up, and didn't distribute critical infrastructure adequately across the region. There's lots of ways this kind of thing happens. I've had 'redundant fiber' that was running in the same bundle and got severed by the same backhoe, for example. The easiest way to determine if there's a dependency on a particular building is to forcibly de-energize all the equipment in it. In this particular case, it seems like there was a mistake in allocation of instances for the regional Spanner database, resulting in that database becoming unavailable when the building was de-energized; and then lots of services depend on regional Spanner, because that's its intended use. It sounds like they did an audit of regional Spanners after this incident, but who knows what other hidden dependencies there are there, or at any cloud provider. Having your critical apps available in multiple distributed datacenters is best practice; having that all in one city isn't really. In an ideal world, you're not dependent on any single entity you don't control; but egress charges make that difficult. Re: Bigquery, there's a separate post for that [1] which seems to indicate "The issue with Google BigQuery has been resolved for all affected users as of Wednesday, 2023-04-26 17:05 US/Pacific." The incident began around "26 Apr 2023 06:25 PDT", so resolution was next-day (but that's about 35 hours). If you've got other data on BigQuery, that's fine; I'm not saying your experience is wrong, I'm saying it doesn't match the public record. [1] https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiq...

finikytou 3y ago
its not only spanner multiple services got hit. the console wasnt even accessible. to me it looks like they have one datacenter and in this datacenter multiple "availability zone"
- toast0 3y ago
  Yeah, because the console and IAM auth rely, and pretty much everything at GCP relies on regional spanner; that's a key point of GCP design for better or worse. It's like if they only had BGP in the one datacenter. If that datacenter turns off, nothing works in the other two that are nearby.