6 ms·

The people tasked with fixing this aren't the ones providing the updates.

marcinzm 8y ago
Fair point but still seems odd that the people providing updates took the weekend off during a large scale customer impacting issue. I'm sure all the people spending the weekend trying to mitigate the impact of this on their infrastructure would love to have timely updates.
- izacus 8y ago
  It's weekend, why wouldn't you take it off? It's just silly software.
  - jschwartzi 8y ago
    More to the point, why would you depend on Google for any critical infrastructure after this?
    - user5994461 8y ago
      Because you tried to run your own infrastructure and it was so much worse.
    - Draiken 8y ago
      Then in a few months from now, people will be saying the same things after an AWS or Azure outage. Am I the only one that finds this slightly humorous?
  - ramphastidae 8y ago
    A lot of people's businesses, reputations and livelihoods depend on "silly software," not to mention that they are paying customers themselves.
    - Twirrim 8y ago
      I know sarcasm doesn't translate well on the internet, but how the heck did you manage to miss it in OPs post?!
      - ramphastidae 8y ago
        Ah, I guess I missed it. My fault.
  - DoreenMichele 8y ago
    A quick search suggests that silly software is used by dozens of companies classified as hospitals or healthcare. Let's hope you don't have a life threatening medical emergency that can't wait near an affected healthcare facility while that silly software is down. https://idatalabs.com/tech/products/kubernetes https://idatalabs.com/tech/products/kubernetes
    - steelframe 8y ago
      > Let's hope you don't have a life threatening medical emergency that can't wait near an affected healthcare facility while that silly software is down. If your ability to operate an ER is dependent on a remote data center, you have no business being a public health provider.
- tweenagedream 8y ago
  This is a problem in the web ui creating additional node pools, there is a very simple workaround to use gcloud. What major impact are you referring to? It's likely the fix is checked in and will start roll out on Monday. Disclaimer: I work on Google Cloud and while I believe we could use more words here, this doesn't seem like a huge problem. It's embarrassing the the issue with the ui was shipped, and I'm sure this will be addressed in the post mortem as well as whether it could have been mitigated quicker than a roll forward.
  - marcinzm 8y ago
    >This is a problem in the web ui creating additional node pools, there is a very simple workaround to use gcloud. What major impact are you referring to? Based on comments in this thread even gcloud is failing and so are other non-kubernetes services. Which may be inaccurate but there's a lot of people saying the same thing so maybe it is. You're right however that the linked issue is only about the UI. So Google isn't even tracking the service distribution issue in it's issue tracker much less updating people on. I personally think that's even worse...
  - kenan_warren 8y ago
    It's not just the UI though. Resources have been exhausted, or that's the error I'm getting, in a lot of regions for both my work and personal accounts (GKE and GCE). Someone on the GCP slack also said they were getting similar issues from Dataflow so it seems to be widespread across products.
trhway 8y ago
They have whole book describing who fixes, provides updates, etc. Fun meditative reading while waiting for the outage to get fixed. https://landing.google.com/sre/sre-book/chapters/managing-incidents/ https://landing.google.com/sre/sre-book/chapters/managing-in... Looks like this time Mary took the whole week off without telling Josephine :)
INTPenis 8y ago
Understandable but in my experience the incident manager assigned is still supposed to keep track of progress during weekends when you have a major incident. And this is likely a major incident with significant customer impact. The way google is handling all this gives a pretty poor impression. Seems like this kubernetes is just a PoC.
- joshuamorton 8y ago
  Incident manager isn't public comms person. The person updating that status dashboard may or may not be an engineer, the IM certainly is.
  - rorykoehler 8y ago
    They should take a few mins out of their weekend to update the dashboard regardless. If my small 25 employee company can do it Google can do it.
    - joshuamorton 8y ago
      See my other comment, to say what, exactly? That's yes, it's still being investigated?
      - tmd83 8y ago
        Yeah why not? A billion dollar cloud provider can't have one person communicating to customers that are facing multi day outage. That not updating is an option is absurd in that time range.
      - rorykoehler 8y ago
        Yes precisely. At least with a new timestamp we will then know the status dashboard isn't also broken.
  - michaelt 8y ago
    At a smaller, less corporate company the engineer/public comms person dividing line would not be so ossified the divide couldn't be bridged when the situation called for it.

Fair point but still seems odd that the people providing updates took the weekend off during a large scale customer impacting issue. I'm sure all the people spending the weekend trying to mitigate the impact of this on their infrastructure would love to have timely updates.

izacus 8y ago
It's weekend, why wouldn't you take it off? It's just silly software.
- jschwartzi 8y ago
  More to the point, why would you depend on Google for any critical infrastructure after this?
  - user5994461 8y ago
    Because you tried to run your own infrastructure and it was so much worse.
  - Draiken 8y ago
    Then in a few months from now, people will be saying the same things after an AWS or Azure outage. Am I the only one that finds this slightly humorous?
- ramphastidae 8y ago
  A lot of people's businesses, reputations and livelihoods depend on "silly software," not to mention that they are paying customers themselves.
  - Twirrim 8y ago
    I know sarcasm doesn't translate well on the internet, but how the heck did you manage to miss it in OPs post?!
    - ramphastidae 8y ago
      Ah, I guess I missed it. My fault.
- DoreenMichele 8y ago
  A quick search suggests that silly software is used by dozens of companies classified as hospitals or healthcare. Let's hope you don't have a life threatening medical emergency that can't wait near an affected healthcare facility while that silly software is down. https://idatalabs.com/tech/products/kubernetes https://idatalabs.com/tech/products/kubernetes
  - steelframe 8y ago
    > Let's hope you don't have a life threatening medical emergency that can't wait near an affected healthcare facility while that silly software is down. If your ability to operate an ER is dependent on a remote data center, you have no business being a public health provider.
tweenagedream 8y ago
This is a problem in the web ui creating additional node pools, there is a very simple workaround to use gcloud. What major impact are you referring to? It's likely the fix is checked in and will start roll out on Monday. Disclaimer: I work on Google Cloud and while I believe we could use more words here, this doesn't seem like a huge problem. It's embarrassing the the issue with the ui was shipped, and I'm sure this will be addressed in the post mortem as well as whether it could have been mitigated quicker than a roll forward.
- marcinzm 8y ago
  >This is a problem in the web ui creating additional node pools, there is a very simple workaround to use gcloud. What major impact are you referring to? Based on comments in this thread even gcloud is failing and so are other non-kubernetes services. Which may be inaccurate but there's a lot of people saying the same thing so maybe it is. You're right however that the linked issue is only about the UI. So Google isn't even tracking the service distribution issue in it's issue tracker much less updating people on. I personally think that's even worse...
- kenan_warren 8y ago
  It's not just the UI though. Resources have been exhausted, or that's the error I'm getting, in a lot of regions for both my work and personal accounts (GKE and GCE). Someone on the GCP slack also said they were getting similar issues from Dataflow so it seems to be widespread across products.

trhway 8y ago

They have whole book describing who fixes, provides updates, etc. Fun meditative reading while waiting for the outage to get fixed. https://landing.google.com/sre/sre-book/chapters/managing-incidents/ https://landing.google.com/sre/sre-book/chapters/managing-in... Looks like this time Mary took the whole week off without telling Josephine :)

INTPenis 8y ago

Understandable but in my experience the incident manager assigned is still supposed to keep track of progress during weekends when you have a major incident. And this is likely a major incident with significant customer impact. The way google is handling all this gives a pretty poor impression. Seems like this kubernetes is just a PoC.

joshuamorton 8y ago
Incident manager isn't public comms person. The person updating that status dashboard may or may not be an engineer, the IM certainly is.
- rorykoehler 8y ago
  They should take a few mins out of their weekend to update the dashboard regardless. If my small 25 employee company can do it Google can do it.
  - joshuamorton 8y ago
    See my other comment, to say what, exactly? That's yes, it's still being investigated?
    - tmd83 8y ago
      Yeah why not? A billion dollar cloud provider can't have one person communicating to customers that are facing multi day outage. That not updating is an option is absurd in that time range.
    - rorykoehler 8y ago
      Yes precisely. At least with a new timestamp we will then know the status dashboard isn't also broken.
- michaelt 8y ago
  At a smaller, less corporate company the engineer/public comms person dividing line would not be so ossified the divide couldn't be bridged when the situation called for it.