6 ms·

Fair point but still seems odd that the people providing updates took the weekend off during a large scale customer impacting issue. I'm sure all the people spe

Fair point but still seems odd that the people providing updates took the weekend off during a large scale customer impacting issue. I'm sure all the people spending the weekend trying to mitigate the impact of this on their infrastructure would love to have timely updates.

izacus 8y ago
It's weekend, why wouldn't you take it off? It's just silly software.
- jschwartzi 8y ago
  More to the point, why would you depend on Google for any critical infrastructure after this?
  - user5994461 8y ago
    Because you tried to run your own infrastructure and it was so much worse.
  - Draiken 8y ago
    Then in a few months from now, people will be saying the same things after an AWS or Azure outage. Am I the only one that finds this slightly humorous?
- ramphastidae 8y ago
  A lot of people's businesses, reputations and livelihoods depend on "silly software," not to mention that they are paying customers themselves.
  - Twirrim 8y ago
    I know sarcasm doesn't translate well on the internet, but how the heck did you manage to miss it in OPs post?!
    - ramphastidae 8y ago
      Ah, I guess I missed it. My fault.
- DoreenMichele 8y ago
  A quick search suggests that silly software is used by dozens of companies classified as hospitals or healthcare. Let's hope you don't have a life threatening medical emergency that can't wait near an affected healthcare facility while that silly software is down. https://idatalabs.com/tech/products/kubernetes https://idatalabs.com/tech/products/kubernetes
  - steelframe 8y ago
    > Let's hope you don't have a life threatening medical emergency that can't wait near an affected healthcare facility while that silly software is down. If your ability to operate an ER is dependent on a remote data center, you have no business being a public health provider.
tweenagedream 8y ago
This is a problem in the web ui creating additional node pools, there is a very simple workaround to use gcloud. What major impact are you referring to? It's likely the fix is checked in and will start roll out on Monday. Disclaimer: I work on Google Cloud and while I believe we could use more words here, this doesn't seem like a huge problem. It's embarrassing the the issue with the ui was shipped, and I'm sure this will be addressed in the post mortem as well as whether it could have been mitigated quicker than a roll forward.
- marcinzm 8y ago
  >This is a problem in the web ui creating additional node pools, there is a very simple workaround to use gcloud. What major impact are you referring to? Based on comments in this thread even gcloud is failing and so are other non-kubernetes services. Which may be inaccurate but there's a lot of people saying the same thing so maybe it is. You're right however that the linked issue is only about the UI. So Google isn't even tracking the service distribution issue in it's issue tracker much less updating people on. I personally think that's even worse...
- kenan_warren 8y ago
  It's not just the UI though. Resources have been exhausted, or that's the error I'm getting, in a lot of regions for both my work and personal accounts (GKE and GCE). Someone on the GCP slack also said they were getting similar issues from Dataflow so it seems to be widespread across products.

izacus 8y ago

It's weekend, why wouldn't you take it off? It's just silly software.

jschwartzi 8y ago
More to the point, why would you depend on Google for any critical infrastructure after this?
- user5994461 8y ago
  Because you tried to run your own infrastructure and it was so much worse.
- Draiken 8y ago
  Then in a few months from now, people will be saying the same things after an AWS or Azure outage. Am I the only one that finds this slightly humorous?
ramphastidae 8y ago
A lot of people's businesses, reputations and livelihoods depend on "silly software," not to mention that they are paying customers themselves.
- Twirrim 8y ago
  I know sarcasm doesn't translate well on the internet, but how the heck did you manage to miss it in OPs post?!
  - ramphastidae 8y ago
    Ah, I guess I missed it. My fault.
DoreenMichele 8y ago
A quick search suggests that silly software is used by dozens of companies classified as hospitals or healthcare. Let's hope you don't have a life threatening medical emergency that can't wait near an affected healthcare facility while that silly software is down. https://idatalabs.com/tech/products/kubernetes https://idatalabs.com/tech/products/kubernetes
- steelframe 8y ago
  > Let's hope you don't have a life threatening medical emergency that can't wait near an affected healthcare facility while that silly software is down. If your ability to operate an ER is dependent on a remote data center, you have no business being a public health provider.

tweenagedream 8y ago

This is a problem in the web ui creating additional node pools, there is a very simple workaround to use gcloud. What major impact are you referring to? It's likely the fix is checked in and will start roll out on Monday. Disclaimer: I work on Google Cloud and while I believe we could use more words here, this doesn't seem like a huge problem. It's embarrassing the the issue with the ui was shipped, and I'm sure this will be addressed in the post mortem as well as whether it could have been mitigated quicker than a roll forward.

marcinzm 8y ago
>This is a problem in the web ui creating additional node pools, there is a very simple workaround to use gcloud. What major impact are you referring to? Based on comments in this thread even gcloud is failing and so are other non-kubernetes services. Which may be inaccurate but there's a lot of people saying the same thing so maybe it is. You're right however that the linked issue is only about the UI. So Google isn't even tracking the service distribution issue in it's issue tracker much less updating people on. I personally think that's even worse...
kenan_warren 8y ago
It's not just the UI though. Resources have been exhausted, or that's the error I'm getting, in a lot of regions for both my work and personal accounts (GKE and GCE). Someone on the GCP slack also said they were getting similar issues from Dataflow so it seems to be widespread across products.