5 ms·

If we should only read things written by organizations that make no mistakes, then we will never read anything.

randmeerkat 2y ago
> If we should only read things written by organizations that make no mistakes, then we will never read anything. That was a “mistake” that should not have even been possible. If the pension fund had not used a multi cloud strategy the entire business would have been lost. A mistake is not configuring Kafka correctly and losing some data, deleting an entire account should not be given a pass.
- joshuamorton 2y ago
  The recent postmortem says they were able to recover from backups on gcp, so I don't think this is true.
  - randmeerkat 2y ago
    > The recent postmortem says they were able to recover from backups on gcp, so I don't think this is true. “UniSuper, an Australian pension fund that manages $135 billion worth of funds and has 647,000 members, had its entire account wiped out at Google Cloud, including all its backups that were stored on the service. UniSuper thankfully had some backups with a different provider and was able to recover its data, but according to UniSuper's incident log, downtime started May 2, and a full restoration of services didn't happen until May 15.” Google didn’t recover the data, the customer recovered their data from a different cloud provider.
    - joshuamorton 2y ago
      From https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident https://cloud.google.com/blog/products/infrastructure/detail... > This incident did not impact: > Any other Google Cloud service. > Any other customer using GCVE or any other Google Cloud service. > The customer’s other GCVE Private Clouds, Google Account, Orgs, Folders, or Projects. > The customer’s data backups stored in Google Cloud Storage (GCS) in the same region. ... > Data backups that were stored in Google Cloud Storage in the same region were not impacted by the deletion, and ... were instrumental in aiding the rapid restoration. Emphasis mine. You're quoting, as far as I can tell, an ArsTechnica article that makes unsourced claims about backups being deleted, neither UniSuper's nor Google's previous statements ever mentioned anything about backups being deleted.
      - randmeerkat 2y ago
        > were instrumental in aiding the rapid restoration. I don’t call 13 days a rapid restoration. I also don’t trust Google’s post-mortem documentation more than an independent news organization to be honest about what really happened. Especially while Google is actively gaslighting their users about the errors in its AI search [1]. 1. https://www.theverge.com/2024/5/24/24164119/google-ai-overview-mistakes-search-race-openai https://www.theverge.com/2024/5/24/24164119/google-ai-overvi...
        joshuamorton 2y ago
        It is, we'll go with, weird, to presume a random news article making baseless claims is correct over the, like, actual people who addressed the problem. I'll reiterate, no one involved in the restoration (Unisuper or Google) ever said anything about Google's backups being deleted, in fact basically everything Google and Unisuper have said specific that it was only the VM config that was removed. Ars made up the thing about backups being deleted, which makes an exciting headline, but it doesn't appear at all reliable or based in reporting, just conjecture.
        randmeerkat 2y ago
        > Ars made up the thing about backups being deleted, which makes an exciting headline, but it doesn't appear at all reliable or based in reporting, just conjecture. So you just label a reputable news outlet as fake news and then move on..?
        joshuamorton 2y ago
        I'm saying Ars is jumping to conclusions a bit too quickly. You can call that fake news if you want, I didn't.
        randmeerkat 2y ago
        “This is an isolated, ‘one-of-a-kind occurrence’ that has never before occurred with any of Google Cloud’s clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again.” “UniSuper had backups in place with an additional service provider. These backups have minimised data loss, and significantly improved the ability of UniSuper and Google Cloud to complete the restoration.” https://www.unisuper.com.au/about-us/media-centre/2024/a-joint-statement-from-unisuper-and-google-cloud https://www.unisuper.com.au/about-us/media-centre/2024/a-joi... Those quotes were pulled directly from UniSuper’s website. Google deleted an account, lost the data, and then took 13 days to recover the pension fund from data stored on another data provider. Maybe you should consider that your employment at Google is damaging your objectivity.

> If we should only read things written by organizations that make no mistakes, then we will never read anything. That was a “mistake” that should not have even been possible. If the pension fund had not used a multi cloud strategy the entire business would have been lost. A mistake is not configuring Kafka correctly and losing some data, deleting an entire account should not be given a pass.

joshuamorton 2y ago
The recent postmortem says they were able to recover from backups on gcp, so I don't think this is true.
- randmeerkat 2y ago
  > The recent postmortem says they were able to recover from backups on gcp, so I don't think this is true. “UniSuper, an Australian pension fund that manages $135 billion worth of funds and has 647,000 members, had its entire account wiped out at Google Cloud, including all its backups that were stored on the service. UniSuper thankfully had some backups with a different provider and was able to recover its data, but according to UniSuper's incident log, downtime started May 2, and a full restoration of services didn't happen until May 15.” Google didn’t recover the data, the customer recovered their data from a different cloud provider.
  - joshuamorton 2y ago
    From https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident https://cloud.google.com/blog/products/infrastructure/detail... > This incident did not impact: > Any other Google Cloud service. > Any other customer using GCVE or any other Google Cloud service. > The customer’s other GCVE Private Clouds, Google Account, Orgs, Folders, or Projects. > The customer’s data backups stored in Google Cloud Storage (GCS) in the same region. ... > Data backups that were stored in Google Cloud Storage in the same region were not impacted by the deletion, and ... were instrumental in aiding the rapid restoration. Emphasis mine. You're quoting, as far as I can tell, an ArsTechnica article that makes unsourced claims about backups being deleted, neither UniSuper's nor Google's previous statements ever mentioned anything about backups being deleted.
    - randmeerkat 2y ago
      > were instrumental in aiding the rapid restoration. I don’t call 13 days a rapid restoration. I also don’t trust Google’s post-mortem documentation more than an independent news organization to be honest about what really happened. Especially while Google is actively gaslighting their users about the errors in its AI search [1]. 1. https://www.theverge.com/2024/5/24/24164119/google-ai-overview-mistakes-search-race-openai https://www.theverge.com/2024/5/24/24164119/google-ai-overvi...
      - joshuamorton 2y ago
        It is, we'll go with, weird, to presume a random news article making baseless claims is correct over the, like, actual people who addressed the problem. I'll reiterate, no one involved in the restoration (Unisuper or Google) ever said anything about Google's backups being deleted, in fact basically everything Google and Unisuper have said specific that it was only the VM config that was removed. Ars made up the thing about backups being deleted, which makes an exciting headline, but it doesn't appear at all reliable or based in reporting, just conjecture.
        randmeerkat 2y ago
        > Ars made up the thing about backups being deleted, which makes an exciting headline, but it doesn't appear at all reliable or based in reporting, just conjecture. So you just label a reputable news outlet as fake news and then move on..?
        joshuamorton 2y ago
        I'm saying Ars is jumping to conclusions a bit too quickly. You can call that fake news if you want, I didn't.
        randmeerkat 2y ago
        “This is an isolated, ‘one-of-a-kind occurrence’ that has never before occurred with any of Google Cloud’s clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again.” “UniSuper had backups in place with an additional service provider. These backups have minimised data loss, and significantly improved the ability of UniSuper and Google Cloud to complete the restoration.” https://www.unisuper.com.au/about-us/media-centre/2024/a-joint-statement-from-unisuper-and-google-cloud https://www.unisuper.com.au/about-us/media-centre/2024/a-joi... Those quotes were pulled directly from UniSuper’s website. Google deleted an account, lost the data, and then took 13 days to recover the pension fund from data stored on another data provider. Maybe you should consider that your employment at Google is damaging your objectivity.