4 ms·

Please tell me more about The Great Oops

bigbuppo 8mo ago
It's inevitable that one of the major cloud providers will irrecoverably delete all customer data with one single fat-fingered command. Though in google's case I'll also consider the prophecy to be fulfilled if they delete their own data. It will forever be known as The Great Oops.
- tokyobreakfast 8mo ago
  That seems unlikely. Is Google run by one Homer Simpson?
  - bigbuppo 8mo ago
    Yes.
- Arainach 8mo ago
  It's not inevitable, it's essentially impossible. There are a few things that can cause tremendously widespread outages, essentially all of them network configuration changes. Actually deleting customer data is dramatically more difficult to the point of impossible - there are so many different services in so many different locations with so many layers of access control. There is no "one command" that can do such a thing - at the scale of a worldwide network of data centers there is no "rm -rf /".
  - ocdtrekkie 8mo ago
    Ah, but you fail to account for Google's incredible knack for building tools designed to do things at scale. Or put AI in things that don't need it. The possibility Google will either manage to unleash a malicious AI on their infrastructure and/or develop a way to destroy a lot of data at scale quite efficiently or some combination of the two is far from zero. Bear in mind, this "Little Oops" should also have been impossible: https://www.techspot.com/news/103207-google-reveals-how-blank-parameter-mistake-erased-135.html https://www.techspot.com/news/103207-google-reveals-how-blan...
    - Arainach 8mo ago
      .....no? "We deployed this private cloud with a missing parameter and it wasn't caught" is as different from "we wiped out all customer data" as hello world is from Kubernetes. No one promised this "should be impossible". Did you confuse "we'll take steps to ensure this never happens again"?
      - ocdtrekkie 8mo ago
        It's pretty much half the puzzle actually. You contend there's no global rm rf for a global cloud provider, but clearly a missing parameter can rm rf a customer in an irrecoverable manner. The only half you're missing is... how every major cloud outage happens today... a bad configuration update. These companies have hundreds of thousands of servers, but they also use orchestration tools to distribute sets of changes to all of them. You only need a command to rm rf one box, if you are distributing that command to every box. Now sure, there are tons of security precautions and checks and such to prevent this! But pretending it's impossible is delusional. People do stupid stuff, at scale, every day. The most likely scenario is a zero day in an environment necessitating an extremely rapid global rollout, combined with a plain, simple error.
        Arainach 8mo ago
        The release process, monitoring checks, etc. for a customer's private cloud is generally significantly different from the release process for a global product. I'm not going to get any more specific for all the standard NDA reasons, but having worked for Google and Microsoft among others....no, the risk you describe doesn't translate from one to the other.
        ocdtrekkie 8mo ago
        I understand you believe the checks cannot fail that catastrophically, and I agree that the likelihood they do is quite low. But it can happen, and it only has to happen once. (Also FYI, telling me your work history just tells me you've drunk the koolaid, ain't proof you know more.)
  - rossjudson 8mo ago
    Delete a decryption key. Good luck! I'll see you at the end of time. Break your control plane, and you can't stop the propagation of poison. Propagate the wrong trust bundle... everywhere. Also, it's not about the delete command. It's about the automatic cleanup following behind it that shreds everything, or repurposes the storage.
    - bigbuppo 8mo ago
      Children of the kubernetic line.
      - esseph 8mo ago
        Cyclic infrastructure dependencies suck :(
  - GeekyBear 8mo ago
    Google accidentally deleted customer location history data from customer devices (after intentionally deleting it from Google servers) just last year. If didn't back it up yourself, it is gone forever.
- JyB 8mo ago
  I don’t know if you’re being serious but that’s laughable
  - SchemaLoad 8mo ago
    The idea that all customer data will be deleted is far fetched, but I feel like there have been some massive incidents. Crowdstrike comes to mind, but I feel its entirely possible that Apple/Google/etc could push out some kind of config update which bricks phones in a way they are unable to download another update to fix them. Though I'm sure the major players are all over this risk which is why it hasn't happened.
    - aragilar 8mo ago
      Google wiped all of UniSuper not too long ago by mistake, I don't see why such a occurrence couldn't happen more widely.

It's inevitable that one of the major cloud providers will irrecoverably delete all customer data with one single fat-fingered command. Though in google's case I'll also consider the prophecy to be fulfilled if they delete their own data. It will forever be known as The Great Oops.

tokyobreakfast 8mo ago
That seems unlikely. Is Google run by one Homer Simpson?
- bigbuppo 8mo ago
  Yes.
Arainach 8mo ago
It's not inevitable, it's essentially impossible. There are a few things that can cause tremendously widespread outages, essentially all of them network configuration changes. Actually deleting customer data is dramatically more difficult to the point of impossible - there are so many different services in so many different locations with so many layers of access control. There is no "one command" that can do such a thing - at the scale of a worldwide network of data centers there is no "rm -rf /".
- ocdtrekkie 8mo ago
  Ah, but you fail to account for Google's incredible knack for building tools designed to do things at scale. Or put AI in things that don't need it. The possibility Google will either manage to unleash a malicious AI on their infrastructure and/or develop a way to destroy a lot of data at scale quite efficiently or some combination of the two is far from zero. Bear in mind, this "Little Oops" should also have been impossible: https://www.techspot.com/news/103207-google-reveals-how-blank-parameter-mistake-erased-135.html https://www.techspot.com/news/103207-google-reveals-how-blan...
  - Arainach 8mo ago
    .....no? "We deployed this private cloud with a missing parameter and it wasn't caught" is as different from "we wiped out all customer data" as hello world is from Kubernetes. No one promised this "should be impossible". Did you confuse "we'll take steps to ensure this never happens again"?
    - ocdtrekkie 8mo ago
      It's pretty much half the puzzle actually. You contend there's no global rm rf for a global cloud provider, but clearly a missing parameter can rm rf a customer in an irrecoverable manner. The only half you're missing is... how every major cloud outage happens today... a bad configuration update. These companies have hundreds of thousands of servers, but they also use orchestration tools to distribute sets of changes to all of them. You only need a command to rm rf one box, if you are distributing that command to every box. Now sure, there are tons of security precautions and checks and such to prevent this! But pretending it's impossible is delusional. People do stupid stuff, at scale, every day. The most likely scenario is a zero day in an environment necessitating an extremely rapid global rollout, combined with a plain, simple error.
      - Arainach 8mo ago
        The release process, monitoring checks, etc. for a customer's private cloud is generally significantly different from the release process for a global product. I'm not going to get any more specific for all the standard NDA reasons, but having worked for Google and Microsoft among others....no, the risk you describe doesn't translate from one to the other.
        ocdtrekkie 8mo ago
        I understand you believe the checks cannot fail that catastrophically, and I agree that the likelihood they do is quite low. But it can happen, and it only has to happen once. (Also FYI, telling me your work history just tells me you've drunk the koolaid, ain't proof you know more.)
- rossjudson 8mo ago
  Delete a decryption key. Good luck! I'll see you at the end of time. Break your control plane, and you can't stop the propagation of poison. Propagate the wrong trust bundle... everywhere. Also, it's not about the delete command. It's about the automatic cleanup following behind it that shreds everything, or repurposes the storage.
  - bigbuppo 8mo ago
    Children of the kubernetic line.
    - esseph 8mo ago
      Cyclic infrastructure dependencies suck :(
- GeekyBear 8mo ago
  Google accidentally deleted customer location history data from customer devices (after intentionally deleting it from Google servers) just last year. If didn't back it up yourself, it is gone forever.
JyB 8mo ago
I don’t know if you’re being serious but that’s laughable
- SchemaLoad 8mo ago
  The idea that all customer data will be deleted is far fetched, but I feel like there have been some massive incidents. Crowdstrike comes to mind, but I feel its entirely possible that Apple/Google/etc could push out some kind of config update which bricks phones in a way they are unable to download another update to fix them. Though I'm sure the major players are all over this risk which is why it hasn't happened.
  - aragilar 8mo ago
    Google wiped all of UniSuper not too long ago by mistake, I don't see why such a occurrence couldn't happen more widely.