6 ms·

wasn’t that OVH?

toast0 3y ago
I think they're referring to this incident?[1] although the timeline in the preliminary report seems to indicate that service disruption in europe-west9 but outside the directly affected zone had ended the day after the incident began. In the directly affected zone, there was water and smoke damage, and it seems like 58% of europe-west9-a was decomissioned as a result. Losing roughly half of the capacity in one of three availability zones for a location would probably make instance availability a lot tighter, and might have impacted the poster in ways that aren't reflected in the report? I work for a gcp customer, but we don't have anything deployed to europe-west9, so I don't have direct exposure. As I recall, we did see some difference in geodns routing on that day, but it was unclear if it was related or not. [1] https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf...
- finikytou 3y ago
  definitly not one day but weeks as I posted links just above. and again they lied about what is a region. don't know why it didnt reach WW news.
  - toast0 3y ago
    The register link says europe-west9-a was still down after weeks, and that -c had a five hour outage the weekend before publishing. The other article was published near the time of the incident and doesn't say anything about the weeks after. europe-west9-a was clearly messed up, and for a long time; which I think is fairly reasonable if you have to toss out at least 58% of the hardware because of water and smoke damage. c'est la vie. That -b and -c were unavailable during the incident is not how availability zones are supposed to work, and it's bad; but you seem to be claiming that -b and -c were down for weeks afterward and I don't see that reflected in the links you shared. If you demand a 3 availability zone system, and one availability zone had a disaster; the right thing to do is to move to another region which has three working azs (and hope that one is properly setup; or request an audit if you've got leverage). Cloudyness is supposed to make it easier to move infrastructure. But if likeminded customers are all moving infrastructure, there's going to be a capacity crunch. OTOH, GCP has several regions in europe-west, and there's a lot of interconnections in Europe in general, so unless you had a requirement to domicile data in France, lots of options.
    - finikytou 3y ago
      https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... Multiple Google Cloud services in the europe-west9 region are impacted. 'Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. We expect some unavailability in the europe-west9 region. There is no current ETA for full recovery of operations in the europe-west9 region at this time. We expected to see an extended outage for some services. Customers are advised to failover to other regions if they are impacted.' can you please explain to me how water intrusion in a data center cause a region failure? there is only one reason. they lied. most likely overhyping the region resiliency. how after seing that can you trust them to host critical apps is beyond belief. and if you look the resolution link I shared they kept hiding reality behind fuzzy language. 'After Regional Spanner, IAM and GCE/PD came back online for the region, almost all of the services in the region fully recovered soon after that. The following services either took longer to recover or had additional impact during the outage. Google Cloud Storage, Google BigQuery, Cloud Bigtable, Cloud SQL, Cloud Pub/Sub, Google Cloud Dataflow Please log a case with our Support team if you would like to receive additional information regarding these services.' please look the last sentence and I can tell you it was way more than a day for those services. try to get additional info on how long bigquery/bigtable were offline.
      - toast0 3y ago
        > can you please explain to me how water intrusion in a data center cause a region failure? there is only one reason. they lied. most likely overhyping the region resiliency. how after seing that can you trust them to host critical apps is beyond belief. Clearly, they fucked up, and didn't distribute critical infrastructure adequately across the region. There's lots of ways this kind of thing happens. I've had 'redundant fiber' that was running in the same bundle and got severed by the same backhoe, for example. The easiest way to determine if there's a dependency on a particular building is to forcibly de-energize all the equipment in it. In this particular case, it seems like there was a mistake in allocation of instances for the regional Spanner database, resulting in that database becoming unavailable when the building was de-energized; and then lots of services depend on regional Spanner, because that's its intended use. It sounds like they did an audit of regional Spanners after this incident, but who knows what other hidden dependencies there are there, or at any cloud provider. Having your critical apps available in multiple distributed datacenters is best practice; having that all in one city isn't really. In an ideal world, you're not dependent on any single entity you don't control; but egress charges make that difficult. Re: Bigquery, there's a separate post for that [1] which seems to indicate "The issue with Google BigQuery has been resolved for all affected users as of Wednesday, 2023-04-26 17:05 US/Pacific." The incident began around "26 Apr 2023 06:25 PDT", so resolution was next-day (but that's about 35 hours). If you've got other data on BigQuery, that's fine; I'm not saying your experience is wrong, I'm saying it doesn't match the public record. [1] https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiq...
finikytou 3y ago
https://www.theregister.com/2023/05/10/google_cloud_paris_outage_persists/ https://www.theregister.com/2023/05/10/google_cloud_paris_ou... https://thenewstack.io/paris-is-drowning-gcps-region-failure-in-age-of-operational-resilience/ https://thenewstack.io/paris-is-drowning-gcps-region-failure...

wasn’t that OVH?

I think they're referring to this incident?[1] although the timeline in the preliminary report seems to indicate that service disruption in europe-west9 but outside the directly affected zone had ended the day after the incident began. In the directly affected zone, there was water and smoke damage, and it seems like 58% of europe-west9-a was decomissioned as a result. Losing roughly half of the capacity in one of three availability zones for a location would probably make instance availability a lot tighter, and might have impacted the poster in ways that aren't reflected in the report? I work for a gcp customer, but we don't have anything deployed to europe-west9, so I don't have direct exposure. As I recall, we did see some difference in geodns routing on that day, but it was unclear if it was related or not. [1] https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf...

finikytou 3y ago
definitly not one day but weeks as I posted links just above. and again they lied about what is a region. don't know why it didnt reach WW news.
- toast0 3y ago
  The register link says europe-west9-a was still down after weeks, and that -c had a five hour outage the weekend before publishing. The other article was published near the time of the incident and doesn't say anything about the weeks after. europe-west9-a was clearly messed up, and for a long time; which I think is fairly reasonable if you have to toss out at least 58% of the hardware because of water and smoke damage. c'est la vie. That -b and -c were unavailable during the incident is not how availability zones are supposed to work, and it's bad; but you seem to be claiming that -b and -c were down for weeks afterward and I don't see that reflected in the links you shared. If you demand a 3 availability zone system, and one availability zone had a disaster; the right thing to do is to move to another region which has three working azs (and hope that one is properly setup; or request an audit if you've got leverage). Cloudyness is supposed to make it easier to move infrastructure. But if likeminded customers are all moving infrastructure, there's going to be a capacity crunch. OTOH, GCP has several regions in europe-west, and there's a lot of interconnections in Europe in general, so unless you had a requirement to domicile data in France, lots of options.
  - finikytou 3y ago
    https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... Multiple Google Cloud services in the europe-west9 region are impacted. 'Water intrusion in a data center in europe-west9 caused a multi-cluster failure that led to a shutdown of multiple zones. We expect some unavailability in the europe-west9 region. There is no current ETA for full recovery of operations in the europe-west9 region at this time. We expected to see an extended outage for some services. Customers are advised to failover to other regions if they are impacted.' can you please explain to me how water intrusion in a data center cause a region failure? there is only one reason. they lied. most likely overhyping the region resiliency. how after seing that can you trust them to host critical apps is beyond belief. and if you look the resolution link I shared they kept hiding reality behind fuzzy language. 'After Regional Spanner, IAM and GCE/PD came back online for the region, almost all of the services in the region fully recovered soon after that. The following services either took longer to recover or had additional impact during the outage. Google Cloud Storage, Google BigQuery, Cloud Bigtable, Cloud SQL, Cloud Pub/Sub, Google Cloud Dataflow Please log a case with our Support team if you would like to receive additional information regarding these services.' please look the last sentence and I can tell you it was way more than a day for those services. try to get additional info on how long bigquery/bigtable were offline.
    - toast0 3y ago
      > can you please explain to me how water intrusion in a data center cause a region failure? there is only one reason. they lied. most likely overhyping the region resiliency. how after seing that can you trust them to host critical apps is beyond belief. Clearly, they fucked up, and didn't distribute critical infrastructure adequately across the region. There's lots of ways this kind of thing happens. I've had 'redundant fiber' that was running in the same bundle and got severed by the same backhoe, for example. The easiest way to determine if there's a dependency on a particular building is to forcibly de-energize all the equipment in it. In this particular case, it seems like there was a mistake in allocation of instances for the regional Spanner database, resulting in that database becoming unavailable when the building was de-energized; and then lots of services depend on regional Spanner, because that's its intended use. It sounds like they did an audit of regional Spanners after this incident, but who knows what other hidden dependencies there are there, or at any cloud provider. Having your critical apps available in multiple distributed datacenters is best practice; having that all in one city isn't really. In an ideal world, you're not dependent on any single entity you don't control; but egress charges make that difficult. Re: Bigquery, there's a separate post for that [1] which seems to indicate "The issue with Google BigQuery has been resolved for all affected users as of Wednesday, 2023-04-26 17:05 US/Pacific." The incident began around "26 Apr 2023 06:25 PDT", so resolution was next-day (but that's about 35 hours). If you've got other data on BigQuery, that's fine; I'm not saying your experience is wrong, I'm saying it doesn't match the public record. [1] https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiqaq#scTMecZFsPpiygYrQ9sG https://status.cloud.google.com/incidents/TbcwMSkKy8MTmeeEiq...

finikytou 3y ago

https://www.theregister.com/2023/05/10/google_cloud_paris_outage_persists/ https://www.theregister.com/2023/05/10/google_cloud_paris_ou... https://thenewstack.io/paris-is-drowning-gcps-region-failure-in-age-of-operational-resilience/ https://thenewstack.io/paris-is-drowning-gcps-region-failure...