5 ms·

Given the level of impact that this incident caused, I am surprised that the remediations did not go deeper. They ensured that the same problem could not happen

Given the level of impact that this incident caused, I am surprised that the remediations did not go deeper. They ensured that the same problem could not happen again in the same way, but that's all. So some equivalent glitch somewhere down the road could lead to a similar result (or worse; not all customers might have the same "robust and resilient architectural approach to managing risk of outage or failure"). Examples of things they could have done to systematically guard against inappropriate service termination / deletion in the future: 1. When terminating a service, temporarily place it in a state where the service is unavailable but all data is retained and can be restored at the push of a button. Discard the data after a few days. This provides an opportunity for the customer to report the problem. 2. Audit all deletion workflows for all services (they only mention having reviewed GCVE). Ensure that customers are notified in advance whenever any service is terminated, even if "the deletion was triggered as a result of a parameter being left blank by Google operators using the internal tool". 3. Add manual review for any termination of a service that is in active use, above a certain size. Absent these broader measures, I don't find this postmortem to be in the slightest bit reassuring. Given the are-you-f*ing-kidding-me nature of the incident, I would have expected any sensible provider who takes the slightest pride in their service, or even is merely interested in protecting their reputation, to visibly go over the top in ensuring nothing like this could happen again. Instead, they've done the bare minimum. That says something bad about the culture at Google Cloud.

2OEH8eoCRo0 2y ago
That sounds reasonable. Perhaps they felt that a larger change to process would be riskier overall.
- TheCleric 2y ago
  No it would probably be even worse from Google’s perspective: more expensive.
rezonant 2y ago
Hard agree. They clearly were more interested in making clear that there's not a systemic problem in how GCP's operators manage the platform, which read strongly and alarmingly that there is a systemic problem in how GCP's operators manage the platform. The lack of the common sense measures you outline in their postmortem just tells me that they aren't doing anything to fix it.
- ok_dad 2y ago
  “There’s no systemic problem.” Meanwhile, the operators were allowed to leave a parameter blank and the default was to set a deletion time bomb. Not systemic my butt! That’s a process failure, and every process failure like this is a systemic problem because the system shouldn’t allow a stupid error like this.
  - joshuamorton 2y ago
    If you're arguing that that was the systemic problem, then it's been fully fixed, as the manual operation was removed and so validation can no longer be bypassed.
    - rezonant 2y ago
      I think you glossed over the importance of the term process failure. The idea is that this one particular form missing the appropriate care is indicative of a wider lack of discipline amongst the engineers building it. Definitionally, you cannot solve a process problem by fixing a specific bug.
      - joshuamorton 2y ago
        "we removed the system that can enable a process failure" fixes the process failure. I didn't misunderstand anything.
        cowboylowrez 2y ago
        joshua nails it. companies do not do root cause anymore. its just the great enshitification, google edition.
phito 2y ago
It's a joke that they're not doing these things. How can you be an giant cloud provider and not think of putting safe guards around data deletion. I guess that realistically they thought of it many times but never implemented it because our costs money.
- pm90 2y ago
  It’s probably because implementing such safeguards wouldn't help anyones promo packet. I really dislike that most of our major cloud infrastructure is provided by big tech rather than eg infrastructure vendors. I trust equinix a lot more than Google because thats all they do.
  - metadat 2y ago
    Understandable, however public clouds are a huge mix of both hardware and software, and it takes deep proficiency at both to pull it off. Equinix are definitely in the hardware and routing business.. may be tough to work upstream. Hardware always get commoditized to the max (sad but true).
  - Thorrez 2y ago
    I work in GCP and have seen a lot of OKRs about improving reliability. So implementing something like this would help someone's promo packet.
    - cbarrick 2y ago
      This is exactly the kind of work that would get SREs promoted.
    - passion__desire 2y ago
      It is funny Google has internal memegen but not ideagen. Ideate away your problems, guys.
  - lima 2y ago
    As a customer of Equinix Cloud... No thank you. Infrastructure vendors are terrible software engineers.
Ocha 2y ago
I wouldn’t be surprised if VMware support is getting deprecated in GCP so they just don’t care - waiting for all customers to move off of it
- snewman 2y ago
  My point is that if they had this problem in their VMware support, they might have a similar problem in one of their other services. But they didn't check (or at least they didn't claim credit for having checked, which likely means they didn't check).
sangnoir 2y ago
> When terminating a service, temporarily place it in a state where the service is unavailable but all data is retained and can be restored at the push of a button. Discard the data after a few days. This provides an opportunity for the customer to report the problem Replacing actual deletion with deletion flags may lead to lead to other fun bugs like "Google Cloud fails to delete customer data, running afoul of EU rules". I suspect Google would err on the side of accidental deletions rather than accidental non-deletions: at least in the EU.
- pm90 2y ago
  I highly doubt this was the reason. Google has similar deletion protection for other resources eg GCP projects are soft deleted for 30 days before being nuked.
- boesboes 2y ago
  Not really how it works, GDPR protects individuals and allow them to request deletion with the data owner. They need to then, within 60(?) days, respond to any request. Google has nothing to do with that beyond having to make sure their infra is secure. There even are provisions for dealing with personal data in backups. EU law has nothing to do with this.
- mcherm 2y ago
  > I suspect Google would err on the side of accidental deletions rather than accidental non-deletions: at least in the EU. I certainly hope not, because that would be incredibly stupid. Customers understand the significance of different kinds of risk. This story got an incredible amount of attention among the community of people who choose between different cloud services. A story about how Google had failed to delete data on time would not have gotten nearly as much attention. But let us suppose for a moment that Google has no concern for their reputation, only for their legal liability. Under EU privacy rules, there might be some liability for failing to delete data on schedule -- although I strongly suspect that the kind of "this was an unavoidable one-off mistake" justifications that we see in this article would convince a court to reduce that liability. But what liability would they face for the deletion? This was a hedge fund managing billions of dollars. Fortunately, they had off-site backups to restore their data. If they hadn't, and it had been impossible to restore the data, how much liability could Google have faced? Surely, even the lawyers in charge of minimizing liability would agree: it is better to fail by keeping customers accounts then to fail by deleting them.
steveBK123 2y ago
>> 1. When terminating a service, temporarily place it in a state where the service is unavailable but all data is retained and can be restored at the push of a button. Discard the data after a few days. This provides an opportunity for the customer to report the problem. This is so obviously "enterprise software 101" that it is telling Google is operating in 2024 without it. Since my new hire grad days, the idea of immediately deleting data that is no longer needed was out of the question. Soft deletes in databases with a column you mark delete. Move/rename data on disk until super duper sure you need to delete it (and maybe still let the backup remain). Etc..
- nikanj 2y ago
  There are many voices in the industry arguing against soft deletes. Mostly coming from a very Chesterton's Fence perspective. For some examples https://www.metabase.com/learn/analytics/data-model-mistakes#3-soft-deletes https://www.metabase.com/learn/analytics/data-model-mistakes... https://www.cultured.systems/2024/04/24/Soft-delete/ https://www.cultured.systems/2024/04/24/Soft-delete/ https://brandur.org/soft-deletion https://brandur.org/soft-deletion Many more can easily be found.
  - snewman 2y ago
    For the use case we're discussing here, of terminating an entire service, the soft delete would typically be needed only at some high level, such as on the access list for the service. The impact on performance, etc. should be minimal.
    - steveBK123 2y ago
      Precisely, before you delete a customer account, you disable its access to the system. This is a scream test. Once you've gone through some time and due diligence you can contemplate actually deleting the customer data and account.
      - Nathanba 2y ago
        I think the reason why someone wouldn't want to do this is because it will cost Google money to keep it active on any level.
ajross 2y ago
FWIW, you're solving the bug by fiat, and that doesn't work. Surely analogs to all those protections are already in place. But a firm and obvious requirement of a software system that is capable of deleting data the ability to delete data. And if it can do it you can write a bug that short-circuits any architectural protection you put in place. Which is the definition of a bug. Basically I don't see this as helpful. This is just a form of the "I would never have written this bug" postmortem response. And yeah, you would. We all would. And do.
mehulashah 2y ago
I’m completely baffled by Google’s “postmortem” myself. Not only is it obviously insufficient to anyone that has operated online services as you point out, but the conclusions are full of hubris. I.e. this was a one time incident, it won’t happen again, we’re very sorry, but we’re awesome and continue to be awesome. This doesn’t seem to help Google Cloud’s face-in-palm moment.
- playingalong 2y ago
  It looks like they could read the SRE book by Google. BTW available for free at https://sre.google/sre-book/table-of-contents/ https://sre.google/sre-book/table-of-contents/ A bit chaotic (a mix of short essays) and simplistic (assuming one kind of approach or design), but definitely still worth a read. No exaggeration to state it was category defining.
markhahn 2y ago
most of this complaint is explicitly answered in the article. must have been TL...
PcChip 2y ago
Could it have been a VMware expiration setting somewhere, and thus VMware itself deleted the customer’s tenant? If so then Google wouldn’t have a way to prove it won’t happen again except by always setting the expiration flag to “never” instead of leaving it blank
yalok 2y ago
I would add one more - 4. Add an option to auto-backup all the data from the account to the outside backup service of users choice. This would help not just with these kind of accidents, but also any kind of data corruption/availability issues. I would pay for this even for my personal gmail account.
belter 2y ago
Can you imagine if there was no backup? Google would be in for to cover the +/- 200 billion in losses? This is why the smart people at Berkshire Hathaway don't offer Cyber Insurance: https://youtu.be/INztpkzUaDw?t=5418 https://youtu.be/INztpkzUaDw?t=5418
- ratorx 2y ago
  I’d be very surprised if there wasn’t legalese in the contract/ToS about liability limitations etc. Would maybe expect it to be more than infrastructure costs for a big company custom contract, but probably not unlimited/as high as that, because it seems like such a blatant legal risk… Disclaimer: Am Googler who knows nothing real about this. This is rampant speculation on my part.

2OEH8eoCRo0 2y ago

That sounds reasonable. Perhaps they felt that a larger change to process would be riskier overall.

TheCleric 2y ago
No it would probably be even worse from Google’s perspective: more expensive.

rezonant 2y ago

Hard agree. They clearly were more interested in making clear that there's not a systemic problem in how GCP's operators manage the platform, which read strongly and alarmingly that there is a systemic problem in how GCP's operators manage the platform. The lack of the common sense measures you outline in their postmortem just tells me that they aren't doing anything to fix it.

ok_dad 2y ago
“There’s no systemic problem.” Meanwhile, the operators were allowed to leave a parameter blank and the default was to set a deletion time bomb. Not systemic my butt! That’s a process failure, and every process failure like this is a systemic problem because the system shouldn’t allow a stupid error like this.
- joshuamorton 2y ago
  If you're arguing that that was the systemic problem, then it's been fully fixed, as the manual operation was removed and so validation can no longer be bypassed.
  - rezonant 2y ago
    I think you glossed over the importance of the term process failure. The idea is that this one particular form missing the appropriate care is indicative of a wider lack of discipline amongst the engineers building it. Definitionally, you cannot solve a process problem by fixing a specific bug.
    - joshuamorton 2y ago
      "we removed the system that can enable a process failure" fixes the process failure. I didn't misunderstand anything.
      - cowboylowrez 2y ago
        joshua nails it. companies do not do root cause anymore. its just the great enshitification, google edition.

phito 2y ago

It's a joke that they're not doing these things. How can you be an giant cloud provider and not think of putting safe guards around data deletion. I guess that realistically they thought of it many times but never implemented it because our costs money.

pm90 2y ago
It’s probably because implementing such safeguards wouldn't help anyones promo packet. I really dislike that most of our major cloud infrastructure is provided by big tech rather than eg infrastructure vendors. I trust equinix a lot more than Google because thats all they do.
- metadat 2y ago
  Understandable, however public clouds are a huge mix of both hardware and software, and it takes deep proficiency at both to pull it off. Equinix are definitely in the hardware and routing business.. may be tough to work upstream. Hardware always get commoditized to the max (sad but true).
- Thorrez 2y ago
  I work in GCP and have seen a lot of OKRs about improving reliability. So implementing something like this would help someone's promo packet.
  - cbarrick 2y ago
    This is exactly the kind of work that would get SREs promoted.
  - passion__desire 2y ago
    It is funny Google has internal memegen but not ideagen. Ideate away your problems, guys.
- lima 2y ago
  As a customer of Equinix Cloud... No thank you. Infrastructure vendors are terrible software engineers.

Ocha 2y ago

I wouldn’t be surprised if VMware support is getting deprecated in GCP so they just don’t care - waiting for all customers to move off of it

snewman 2y ago
My point is that if they had this problem in their VMware support, they might have a similar problem in one of their other services. But they didn't check (or at least they didn't claim credit for having checked, which likely means they didn't check).

sangnoir 2y ago

> When terminating a service, temporarily place it in a state where the service is unavailable but all data is retained and can be restored at the push of a button. Discard the data after a few days. This provides an opportunity for the customer to report the problem Replacing actual deletion with deletion flags may lead to lead to other fun bugs like "Google Cloud fails to delete customer data, running afoul of EU rules". I suspect Google would err on the side of accidental deletions rather than accidental non-deletions: at least in the EU.

pm90 2y ago
I highly doubt this was the reason. Google has similar deletion protection for other resources eg GCP projects are soft deleted for 30 days before being nuked.
boesboes 2y ago
Not really how it works, GDPR protects individuals and allow them to request deletion with the data owner. They need to then, within 60(?) days, respond to any request. Google has nothing to do with that beyond having to make sure their infra is secure. There even are provisions for dealing with personal data in backups. EU law has nothing to do with this.
mcherm 2y ago
> I suspect Google would err on the side of accidental deletions rather than accidental non-deletions: at least in the EU. I certainly hope not, because that would be incredibly stupid. Customers understand the significance of different kinds of risk. This story got an incredible amount of attention among the community of people who choose between different cloud services. A story about how Google had failed to delete data on time would not have gotten nearly as much attention. But let us suppose for a moment that Google has no concern for their reputation, only for their legal liability. Under EU privacy rules, there might be some liability for failing to delete data on schedule -- although I strongly suspect that the kind of "this was an unavoidable one-off mistake" justifications that we see in this article would convince a court to reduce that liability. But what liability would they face for the deletion? This was a hedge fund managing billions of dollars. Fortunately, they had off-site backups to restore their data. If they hadn't, and it had been impossible to restore the data, how much liability could Google have faced? Surely, even the lawyers in charge of minimizing liability would agree: it is better to fail by keeping customers accounts then to fail by deleting them.

steveBK123 2y ago

>> 1. When terminating a service, temporarily place it in a state where the service is unavailable but all data is retained and can be restored at the push of a button. Discard the data after a few days. This provides an opportunity for the customer to report the problem. This is so obviously "enterprise software 101" that it is telling Google is operating in 2024 without it. Since my new hire grad days, the idea of immediately deleting data that is no longer needed was out of the question. Soft deletes in databases with a column you mark delete. Move/rename data on disk until super duper sure you need to delete it (and maybe still let the backup remain). Etc..

nikanj 2y ago
There are many voices in the industry arguing against soft deletes. Mostly coming from a very Chesterton's Fence perspective. For some examples https://www.metabase.com/learn/analytics/data-model-mistakes#3-soft-deletes https://www.metabase.com/learn/analytics/data-model-mistakes... https://www.cultured.systems/2024/04/24/Soft-delete/ https://www.cultured.systems/2024/04/24/Soft-delete/ https://brandur.org/soft-deletion https://brandur.org/soft-deletion Many more can easily be found.
- snewman 2y ago
  For the use case we're discussing here, of terminating an entire service, the soft delete would typically be needed only at some high level, such as on the access list for the service. The impact on performance, etc. should be minimal.
  - steveBK123 2y ago
    Precisely, before you delete a customer account, you disable its access to the system. This is a scream test. Once you've gone through some time and due diligence you can contemplate actually deleting the customer data and account.
    - Nathanba 2y ago
      I think the reason why someone wouldn't want to do this is because it will cost Google money to keep it active on any level.

ajross 2y ago

FWIW, you're solving the bug by fiat, and that doesn't work. Surely analogs to all those protections are already in place. But a firm and obvious requirement of a software system that is capable of deleting data the ability to delete data. And if it can do it you can write a bug that short-circuits any architectural protection you put in place. Which is the definition of a bug. Basically I don't see this as helpful. This is just a form of the "I would never have written this bug" postmortem response. And yeah, you would. We all would. And do.

mehulashah 2y ago

I’m completely baffled by Google’s “postmortem” myself. Not only is it obviously insufficient to anyone that has operated online services as you point out, but the conclusions are full of hubris. I.e. this was a one time incident, it won’t happen again, we’re very sorry, but we’re awesome and continue to be awesome. This doesn’t seem to help Google Cloud’s face-in-palm moment.

playingalong 2y ago
It looks like they could read the SRE book by Google. BTW available for free at https://sre.google/sre-book/table-of-contents/ https://sre.google/sre-book/table-of-contents/ A bit chaotic (a mix of short essays) and simplistic (assuming one kind of approach or design), but definitely still worth a read. No exaggeration to state it was category defining.

markhahn 2y ago

most of this complaint is explicitly answered in the article. must have been TL...

PcChip 2y ago

Could it have been a VMware expiration setting somewhere, and thus VMware itself deleted the customer’s tenant? If so then Google wouldn’t have a way to prove it won’t happen again except by always setting the expiration flag to “never” instead of leaving it blank

yalok 2y ago

I would add one more - 4. Add an option to auto-backup all the data from the account to the outside backup service of users choice. This would help not just with these kind of accidents, but also any kind of data corruption/availability issues. I would pay for this even for my personal gmail account.

belter 2y ago

Can you imagine if there was no backup? Google would be in for to cover the +/- 200 billion in losses? This is why the smart people at Berkshire Hathaway don't offer Cyber Insurance: https://youtu.be/INztpkzUaDw?t=5418 https://youtu.be/INztpkzUaDw?t=5418

ratorx 2y ago
I’d be very surprised if there wasn’t legalese in the contract/ToS about liability limitations etc. Would maybe expect it to be more than infrastructure costs for a big company custom contract, but probably not unlimited/as high as that, because it seems like such a blatant legal risk… Disclaimer: Am Googler who knows nothing real about this. This is rampant speculation on my part.