4 ms·

My vote: Bad configuration got pushed.

• SteveNuts 8y ago
  I thought they had all sorts of crazy canary deployment processes that will automatically stop/rollback the deployment if the failure metrics start increasing?
  • corodra 8y ago
    Ran out of canaries? Well, could be an unforeseeable-wack-crazy-bug. Can't plan for every possible failure.
    • SteveNuts 8y ago
      That's very true, there's always a bug somewhere lurking, even if you're Google.
  • p0rkbelly 8y ago
    There was a GCP Global network outage maybe a year or two ago, where their canary of a canary (Yo dawg, I heard you like bugs, so I I put a but inside of your bug) caused a cascading network failure. It was a bad config push -- brownouts fighting with brown outs. Consistency issues. I bet this is happening.