4 ms·

Sounds like a pretty thorough review in that they didn't stop at just an investigation of the specific tool / process, but also examined the rest for any auto d

Sounds like a pretty thorough review in that they didn't stop at just an investigation of the specific tool / process, but also examined the rest for any auto deletion problems and also confirmed soft delete behavior. They could have gone one step further by reviewing all cases of default behavior for anything that might be surprising. That said, it can be difficult to assess what is "surprising", as it's often the people who know the least about a tool/API who also utilize its defaults.

x0x0 2y ago
Sounds more like some pants browning because incidents like this are a great reason to just use aws. Like come on: > After the end of the system-assigned 1 year period, the customer’s GCVE Private Cloud was deleted. No customer notification was sent because the deletion was triggered as a result of a parameter being left blank by Google operators using the internal tool, and not due a customer deletion request. Any customer-initiated deletion would have been preceded by a notification to the customer. ... Tada! We're so incompetent we let giant deletes happen with no human review. Thank god this customer didn't trust us and kept off-gcp backups or they'd be completely screwed. > There has not been an incident of this nature within Google Cloud prior to this instance. It is not a systemic issue. Translated to English: oh god, every aws and Azure salesperson has sent 3 emails to all their prospects citing our utter fuckup.
- markfive 2y ago
  > Thank god this customer didn't trust us and kept off-gcp backups or they'd be completely screwed. Except that, from the article, the customer's backups that were used to recover were in GCP, and in the same region.
  - ceejayoz 2y ago
    I'm curious about that bit. https://www.unisuper.com.au/contact-us/outage-update https://www.unisuper.com.au/contact-us/outage-update says "UniSuper had backups in place with an additional service provider. These backups have minimised data loss, and significantly improved the ability of UniSuper and Google Cloud to complete the restoration."
    - politelemon 2y ago
      That's the bit that's sticking out to be as contradictory. I'm inclined to not believe what GCP have said here as an account deletion is an account deletion, why would some objects be left behind. No doubt this little bit must be causing some annoyance among UniSuper's tech teams.
      - flaminHotSpeedo 2y ago
        I'm inclined to not believe GCP because they edited their status updates retroactively and lied in their postmortem about the Clichy fire in Paris not affecting multiple "zones"
    - graemep 2y ago
      The had another provider because the regulator requires it. I suspect a lot of businesses in less regulated industries do not.
  - skywhopper 2y ago
    I think you misread. Here’s the relevant statement from the article: “Data backups that were stored in Google Cloud Storage in the same region were not impacted by the deletion, and, along with third party backup software, were instrumental in aiding the rapid restoration.”
rezonant 2y ago
> and also confirmed soft delete behavior. Where exactly do they mention they have confirmed soft delete behavior systemically? All they said was they have ensured that this specific automatic deletion scenario can no longer happen, and it seems the main reason is because "these deployments are now automated". They were automated before, now they are even more automated. That does zero to assure me that their deletion mechanisms are consistently safe, only that there's no operator at the wheel any more.

x0x0 2y ago

Sounds more like some pants browning because incidents like this are a great reason to just use aws. Like come on: > After the end of the system-assigned 1 year period, the customer’s GCVE Private Cloud was deleted. No customer notification was sent because the deletion was triggered as a result of a parameter being left blank by Google operators using the internal tool, and not due a customer deletion request. Any customer-initiated deletion would have been preceded by a notification to the customer. ... Tada! We're so incompetent we let giant deletes happen with no human review. Thank god this customer didn't trust us and kept off-gcp backups or they'd be completely screwed. > There has not been an incident of this nature within Google Cloud prior to this instance. It is not a systemic issue. Translated to English: oh god, every aws and Azure salesperson has sent 3 emails to all their prospects citing our utter fuckup.

markfive 2y ago
> Thank god this customer didn't trust us and kept off-gcp backups or they'd be completely screwed. Except that, from the article, the customer's backups that were used to recover were in GCP, and in the same region.
- ceejayoz 2y ago
  I'm curious about that bit. https://www.unisuper.com.au/contact-us/outage-update https://www.unisuper.com.au/contact-us/outage-update says "UniSuper had backups in place with an additional service provider. These backups have minimised data loss, and significantly improved the ability of UniSuper and Google Cloud to complete the restoration."
  - politelemon 2y ago
    That's the bit that's sticking out to be as contradictory. I'm inclined to not believe what GCP have said here as an account deletion is an account deletion, why would some objects be left behind. No doubt this little bit must be causing some annoyance among UniSuper's tech teams.
    - flaminHotSpeedo 2y ago
      I'm inclined to not believe GCP because they edited their status updates retroactively and lied in their postmortem about the Clichy fire in Paris not affecting multiple "zones"
  - graemep 2y ago
    The had another provider because the regulator requires it. I suspect a lot of businesses in less regulated industries do not.
- skywhopper 2y ago
  I think you misread. Here’s the relevant statement from the article: “Data backups that were stored in Google Cloud Storage in the same region were not impacted by the deletion, and, along with third party backup software, were instrumental in aiding the rapid restoration.”

rezonant 2y ago

> and also confirmed soft delete behavior. Where exactly do they mention they have confirmed soft delete behavior systemically? All they said was they have ensured that this specific automatic deletion scenario can no longer happen, and it seems the main reason is because "these deployments are now automated". They were automated before, now they are even more automated. That does zero to assure me that their deletion mechanisms are consistently safe, only that there's no operator at the wheel any more.