4 ms·

Google’s “best practices” lead them to deleting an entire customer’s $135 billion pension account [1]. I’m surprised anyone is still reading anything Google wri

Google’s “best practices” lead them to deleting an entire customer’s $135 billion pension account [1]. I’m surprised anyone is still reading anything Google writes. 1. https://arstechnica.com/gadgets/2024/05/google-cloud-accidentally-nukes-customer-account-causes-two-weeks-of-downtime/ https://arstechnica.com/gadgets/2024/05/google-cloud-acciden...

klabb3 2y ago
You’re assuming that those systems were all implemented to the letter of that guide. That’s never the case. Often these type of guidelines are written to address recurring problems found in an organization.
thirteenfingers 2y ago
That was seven years later. Maybe the problem is that Google stopped reading what Google wrote.
- passion__desire 2y ago
  Google : We will breach the rules we preach.
  - wiseowise 2y ago
    Right, because every line of code written by tens of thousands of Google engineers is being validated against every guidebook.
- randmeerkat 2y ago
  > That was seven years later. Maybe the problem is that Google stopped reading what Google wrote. The problem is that it was never that good. Anyone who has used K8s at scale will tell you at length how it doesn’t scale. People should stop focusing on tech companies like celebrities and focus instead on domain problems related to their business.
  - lima 2y ago
    The funny thing with k8s is that Google doesn't use it (except GKE, and there's a reason it's one cluster per customer). Their internal tooling scales just fine, but all it shares with k8s is some of the underlying concepts. Unlike, say, Bazel, gVisor or Gerrit, which are the real thing (minus some secret sauce tied to internal infra). k8s is good software, and best-in-class when it comes to open source options, but the idea that it is "open source Borg" is silly.
    - randmeerkat 2y ago
      > k8s is good software, and best-in-class when it comes to open source options No it isn’t, it’s a solution in search of a problem that is needlessly complex, wastes engineering cycles on what could have been product development, and has violated every principal of orthogonal design.
dieortin 2y ago
If we should only read things written by organizations that make no mistakes, then we will never read anything.
- randmeerkat 2y ago
  > If we should only read things written by organizations that make no mistakes, then we will never read anything. That was a “mistake” that should not have even been possible. If the pension fund had not used a multi cloud strategy the entire business would have been lost. A mistake is not configuring Kafka correctly and losing some data, deleting an entire account should not be given a pass.
  - joshuamorton 2y ago
    The recent postmortem says they were able to recover from backups on gcp, so I don't think this is true.
    - randmeerkat 2y ago
      > The recent postmortem says they were able to recover from backups on gcp, so I don't think this is true. “UniSuper, an Australian pension fund that manages $135 billion worth of funds and has 647,000 members, had its entire account wiped out at Google Cloud, including all its backups that were stored on the service. UniSuper thankfully had some backups with a different provider and was able to recover its data, but according to UniSuper's incident log, downtime started May 2, and a full restoration of services didn't happen until May 15.” Google didn’t recover the data, the customer recovered their data from a different cloud provider.
      - joshuamorton 2y ago
        From https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident https://cloud.google.com/blog/products/infrastructure/detail... > This incident did not impact: > Any other Google Cloud service. > Any other customer using GCVE or any other Google Cloud service. > The customer’s other GCVE Private Clouds, Google Account, Orgs, Folders, or Projects. > The customer’s data backups stored in Google Cloud Storage (GCS) in the same region. ... > Data backups that were stored in Google Cloud Storage in the same region were not impacted by the deletion, and ... were instrumental in aiding the rapid restoration. Emphasis mine. You're quoting, as far as I can tell, an ArsTechnica article that makes unsourced claims about backups being deleted, neither UniSuper's nor Google's previous statements ever mentioned anything about backups being deleted.
gtirloni 2y ago
Oh, completely ignoring anything anyone from Google ever writes again? This is akin to the cancel culture which we all know is how society should work. /s
- randmeerkat 2y ago
  > Oh, completely ignoring anything anyone from Google ever writes again? This is akin to the cancel culture which we all know is how society should work. /s Maybe if Google focused on doing actual work instead of writing feel good engineering pieces, they wouldn’t have the Google graveyard and an unstable cloud offering that may spontaneously delete multi-billion dollar accounts.
  - wiseowise 2y ago
    Alphabet has 2T market cap, get your head out of your sitting place, lol.
    - randmeerkat 2y ago
      > Alphabet has 2T market cap, get your head out of your sitting place, lol. That same sort of thinking is what led to the downfall of yahoo.
      - wiseowise 2y ago
        Maybe. Times are different now.
        randmeerkat 2y ago
        > Maybe. Times are different now. Are they?
        gtirloni 2y ago
        Are they not?

klabb3 2y ago

You’re assuming that those systems were all implemented to the letter of that guide. That’s never the case. Often these type of guidelines are written to address recurring problems found in an organization.

thirteenfingers 2y ago

That was seven years later. Maybe the problem is that Google stopped reading what Google wrote.

passion__desire 2y ago
Google : We will breach the rules we preach.
- wiseowise 2y ago
  Right, because every line of code written by tens of thousands of Google engineers is being validated against every guidebook.
randmeerkat 2y ago
> That was seven years later. Maybe the problem is that Google stopped reading what Google wrote. The problem is that it was never that good. Anyone who has used K8s at scale will tell you at length how it doesn’t scale. People should stop focusing on tech companies like celebrities and focus instead on domain problems related to their business.
- lima 2y ago
  The funny thing with k8s is that Google doesn't use it (except GKE, and there's a reason it's one cluster per customer). Their internal tooling scales just fine, but all it shares with k8s is some of the underlying concepts. Unlike, say, Bazel, gVisor or Gerrit, which are the real thing (minus some secret sauce tied to internal infra). k8s is good software, and best-in-class when it comes to open source options, but the idea that it is "open source Borg" is silly.
  - randmeerkat 2y ago
    > k8s is good software, and best-in-class when it comes to open source options No it isn’t, it’s a solution in search of a problem that is needlessly complex, wastes engineering cycles on what could have been product development, and has violated every principal of orthogonal design.

dieortin 2y ago

If we should only read things written by organizations that make no mistakes, then we will never read anything.

randmeerkat 2y ago
> If we should only read things written by organizations that make no mistakes, then we will never read anything. That was a “mistake” that should not have even been possible. If the pension fund had not used a multi cloud strategy the entire business would have been lost. A mistake is not configuring Kafka correctly and losing some data, deleting an entire account should not be given a pass.
- joshuamorton 2y ago
  The recent postmortem says they were able to recover from backups on gcp, so I don't think this is true.
  - randmeerkat 2y ago
    > The recent postmortem says they were able to recover from backups on gcp, so I don't think this is true. “UniSuper, an Australian pension fund that manages $135 billion worth of funds and has 647,000 members, had its entire account wiped out at Google Cloud, including all its backups that were stored on the service. UniSuper thankfully had some backups with a different provider and was able to recover its data, but according to UniSuper's incident log, downtime started May 2, and a full restoration of services didn't happen until May 15.” Google didn’t recover the data, the customer recovered their data from a different cloud provider.
    - joshuamorton 2y ago
      From https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident https://cloud.google.com/blog/products/infrastructure/detail... > This incident did not impact: > Any other Google Cloud service. > Any other customer using GCVE or any other Google Cloud service. > The customer’s other GCVE Private Clouds, Google Account, Orgs, Folders, or Projects. > The customer’s data backups stored in Google Cloud Storage (GCS) in the same region. ... > Data backups that were stored in Google Cloud Storage in the same region were not impacted by the deletion, and ... were instrumental in aiding the rapid restoration. Emphasis mine. You're quoting, as far as I can tell, an ArsTechnica article that makes unsourced claims about backups being deleted, neither UniSuper's nor Google's previous statements ever mentioned anything about backups being deleted.

gtirloni 2y ago

Oh, completely ignoring anything anyone from Google ever writes again? This is akin to the cancel culture which we all know is how society should work. /s

randmeerkat 2y ago
> Oh, completely ignoring anything anyone from Google ever writes again? This is akin to the cancel culture which we all know is how society should work. /s Maybe if Google focused on doing actual work instead of writing feel good engineering pieces, they wouldn’t have the Google graveyard and an unstable cloud offering that may spontaneously delete multi-billion dollar accounts.
- wiseowise 2y ago
  Alphabet has 2T market cap, get your head out of your sitting place, lol.
  - randmeerkat 2y ago
    > Alphabet has 2T market cap, get your head out of your sitting place, lol. That same sort of thinking is what led to the downfall of yahoo.
    - wiseowise 2y ago
      Maybe. Times are different now.
      - randmeerkat 2y ago
        > Maybe. Times are different now. Are they?
        gtirloni 2y ago
        Are they not?