4 ms·

https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident https://cloud.google.com/blog/products/infrastructure/detail... Goo

https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident https://cloud.google.com/blog/products/infrastructure/detail... Google Cloud accidentally wiped an Australian super[annuation] (pension) fund's entire cloud deployment earlier this year. I think that if you really want durable backups, they have to be reducible to object storage and put in someone else's cloud.

milesward 2y ago
... not quite. I worked directly with the folks involved on getting more RCA details public. This customer used a single product on GCP, a specific type of VMware hosting, and the "subscription" to that product failed, which turned those resources off. It's more like turning off all their VM's, rather than deleting their entire account, identities, access structures, etc.
- buran77 2y ago
  The reporting on that was a bid muddy with Google and Unisuper officially saying different things in different places. Regardless, calling it "more like turning off all their VM's" sounds like heavily downplaying the reality. The downtime alone confirms it was way more than that. From their joint statement [0]: > when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies. > an extensive recovery of our Private Cloud which includes hundreds of virtual machines, databases and applications. > UniSuper had backups in place with an additional service provider. These backups have minimised data loss Strangely enough on this last point a Google blog post [1] says: > This incident did not impact: The customer’s data backups stored in Google Cloud Storage (GCS) in the same region. [0] https://www.unisuper.com.au/about-us/media-centre/2024/a-joint-statement-from-unisuper-and-google-cloud https://www.unisuper.com.au/about-us/media-centre/2024/a-joi... [1] https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident https://cloud.google.com/blog/products/infrastructure/detail...
throwaway2037 2y ago
Thank you to share that blog post. That blog post specifically mentioned that no data was lost. I am confused by your comment about durable backups. Deeper question: Do people think on-prem backups are more reliable than cloud? I would say for 95% of orgs: no.
- endgame 2y ago
  No data was lost because APRA rules require funds to back up across multiple clouds.

milesward 2y ago

... not quite. I worked directly with the folks involved on getting more RCA details public. This customer used a single product on GCP, a specific type of VMware hosting, and the "subscription" to that product failed, which turned those resources off. It's more like turning off all their VM's, rather than deleting their entire account, identities, access structures, etc.

buran77 2y ago
The reporting on that was a bid muddy with Google and Unisuper officially saying different things in different places. Regardless, calling it "more like turning off all their VM's" sounds like heavily downplaying the reality. The downtime alone confirms it was way more than that. From their joint statement [0]: > when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies. > an extensive recovery of our Private Cloud which includes hundreds of virtual machines, databases and applications. > UniSuper had backups in place with an additional service provider. These backups have minimised data loss Strangely enough on this last point a Google blog post [1] says: > This incident did not impact: The customer’s data backups stored in Google Cloud Storage (GCS) in the same region. [0] https://www.unisuper.com.au/about-us/media-centre/2024/a-joint-statement-from-unisuper-and-google-cloud https://www.unisuper.com.au/about-us/media-centre/2024/a-joi... [1] https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident https://cloud.google.com/blog/products/infrastructure/detail...

throwaway2037 2y ago

Thank you to share that blog post. That blog post specifically mentioned that no data was lost. I am confused by your comment about durable backups. Deeper question: Do people think on-prem backups are more reliable than cloud? I would say for 95% of orgs: no.

endgame 2y ago
No data was lost because APRA rules require funds to back up across multiple clouds.