9 ms·

Terrance here from Google Cloud Support. There are only 3 things I can say about this situation. 1) These issues are currently unrelated. 2) We learn a lot fro

Terrance here from Google Cloud Support. There are only 3 things I can say about this situation. 1) These issues are currently unrelated. 2) We learn a lot from these situations. 3) A lot of these types of issues can be mitigated by running in more then 1 region. I really cant promise that today's situations will never happen again. There are a lot of moving pieces in our system and sometimes there are things outside of Google's control.

tus87 7y ago
No, you are too busy trying to stop Trump being elected in 2020 to run a decent service for your paying customers.
gizmo385 7y ago
> There are a lot of moving pieces in our system and sometimes there are things outside of Google's control. Are you implying that the cause of this outage is not Google's fault? If so, can you go into more details about that?
- opportune 7y ago
  Not him but oftentimes cloud outages can be due to issues with the network connections to the datacenter, or power outages. Datacenters also sometimes have other single points of failure such as DNS, but those are within the company's control. https://www.networkworld.com/article/3373646/network-problems-responsible-for-more-data-center-outages.html https://www.networkworld.com/article/3373646/network-problem... https://www.datacenterknowledge.com/uptime/equinix-power-outage-one-reason-behind-aws-cloud-disruption https://www.datacenterknowledge.com/uptime/equinix-power-out...
  - stingraycharles 7y ago
    But data centers are typically designed with network and power failures in mind, not? Isn’t this why these kind of ring based network topologies exist, so that whenever a single network connection fails, it can still easily be routed around?
    - jjeaff 7y ago
      Almost always, yes, but the problem is that everyone has to start routing around the problem and it creates congestion. Those redundant pipes don't sit idle. They are sharing the traffic. As mentioned in another thread, in this case, Google has rerouted google.com traffic out of the region to try to mitigate the congestion.
    - dodobirdlord 7y ago
      On a smaller scale, to link up a few datacenters that are a few miles apart? Sure. On a grand scale though, no. Nobody's running an extra undersea cable from Japan to Singapore so that they can have a ring topology. Or trenching a second PBps of cables across the Appalachian Mountains. When something like that gets busted you go and reroute your least important traffic and send out the repair crew.
- dodobirdlord 7y ago
  > The disruptions with Google Cloud Networking and Load Balancing have been root caused to physical damage to multiple concurrent fiber bundles serving network paths in us-east1, and we expect a full resolution within the next 24 hours. From the dashboard. Looks like this can be blamed on an Act of Backhoe.
mathattack 7y ago
“You should be using more than 1 region” could also be “you should be using more than one provider”, no?
- BurritoAlPastor 7y ago
  Well, sure, if you hate your devops team and you want to make sure they can’t use any of the proprietary functionality of either provider. At which point, if you want to be managing a fleet of vanilla Linux boxes yourself, why use a cloud provider at all?
  - stingraycharles 7y ago
    This seems overly negative. There are lots of ways to do hybrid clouds, especially if you’re doing it for only the more critical parts of your application.
  - mathattack 7y ago
    Staying on current versions, and the ability to scale usage up and down?
  - fkdo 7y ago
    Why would you want to lock into a cloud provider? You're losing a lot of operational flexibility for less devops and sysaadmin work. You are really limiting your tech stack by using standardized things like Jenkins, Docker, K8, mqtt, kafka.
    - StreamBright 7y ago
      For the same reason you want "to lock in" (meaning use) any solution. You do not want to build or operate it yourself. Why don't you take this further? Why to use a water utility if you can just drill your own wells? Most businesses are better of on cloud because their core business is not to build and operate datacenters but provide services to their customers (on the top of datacenters running their apps).
    - bradstewart 7y ago
Decabytes 7y ago
Thanks for the reply Terrance. But isn't it more expensive to run in more than one region?
- tscanausa 7y ago
  Absolutely. For some customer it is the right thing for other customers it may not be the right thing. Every provider will have failures. So the question mostly boils down to does paying for more then 1 region cost more or less then paying for the the lost productivity or revenue of an outage like this. From some places the most costly things they spend money on is employees. If your whole company comes to a stop for even 1 hour. It may cost more then the engineering effort for multi zone, multi region or multi cloud for your critical environments.
iamaelephant 7y ago
Cool man let me know how I can run my Calendar in multiple regions.
deleted 7y ago
[deleted]
marme 7y ago
how do you use multiple regions when Google only supports certain things in limited regions like Dataflow Shuffle only being available in a single region in north america https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#cloud-dataflow-shuffle https://cloud.google.com/dataflow/docs/guides/deploying-a-pi...
deleted 7y ago
[deleted]

tus87 7y ago

No, you are too busy trying to stop Trump being elected in 2020 to run a decent service for your paying customers.

gizmo385 7y ago

> There are a lot of moving pieces in our system and sometimes there are things outside of Google's control. Are you implying that the cause of this outage is not Google's fault? If so, can you go into more details about that?

opportune 7y ago
Not him but oftentimes cloud outages can be due to issues with the network connections to the datacenter, or power outages. Datacenters also sometimes have other single points of failure such as DNS, but those are within the company's control. https://www.networkworld.com/article/3373646/network-problems-responsible-for-more-data-center-outages.html https://www.networkworld.com/article/3373646/network-problem... https://www.datacenterknowledge.com/uptime/equinix-power-outage-one-reason-behind-aws-cloud-disruption https://www.datacenterknowledge.com/uptime/equinix-power-out...
- stingraycharles 7y ago
  But data centers are typically designed with network and power failures in mind, not? Isn’t this why these kind of ring based network topologies exist, so that whenever a single network connection fails, it can still easily be routed around?
  - jjeaff 7y ago
    Almost always, yes, but the problem is that everyone has to start routing around the problem and it creates congestion. Those redundant pipes don't sit idle. They are sharing the traffic. As mentioned in another thread, in this case, Google has rerouted google.com traffic out of the region to try to mitigate the congestion.
  - dodobirdlord 7y ago
    On a smaller scale, to link up a few datacenters that are a few miles apart? Sure. On a grand scale though, no. Nobody's running an extra undersea cable from Japan to Singapore so that they can have a ring topology. Or trenching a second PBps of cables across the Appalachian Mountains. When something like that gets busted you go and reroute your least important traffic and send out the repair crew.
dodobirdlord 7y ago
> The disruptions with Google Cloud Networking and Load Balancing have been root caused to physical damage to multiple concurrent fiber bundles serving network paths in us-east1, and we expect a full resolution within the next 24 hours. From the dashboard. Looks like this can be blamed on an Act of Backhoe.

mathattack 7y ago

“You should be using more than 1 region” could also be “you should be using more than one provider”, no?

BurritoAlPastor 7y ago
Well, sure, if you hate your devops team and you want to make sure they can’t use any of the proprietary functionality of either provider. At which point, if you want to be managing a fleet of vanilla Linux boxes yourself, why use a cloud provider at all?
- stingraycharles 7y ago
  This seems overly negative. There are lots of ways to do hybrid clouds, especially if you’re doing it for only the more critical parts of your application.
- mathattack 7y ago
  Staying on current versions, and the ability to scale usage up and down?
- fkdo 7y ago
  Why would you want to lock into a cloud provider? You're losing a lot of operational flexibility for less devops and sysaadmin work. You are really limiting your tech stack by using standardized things like Jenkins, Docker, K8, mqtt, kafka.
  - StreamBright 7y ago
    For the same reason you want "to lock in" (meaning use) any solution. You do not want to build or operate it yourself. Why don't you take this further? Why to use a water utility if you can just drill your own wells? Most businesses are better of on cloud because their core business is not to build and operate datacenters but provide services to their customers (on the top of datacenters running their apps).
  - bradstewart 7y ago

Decabytes 7y ago

Thanks for the reply Terrance. But isn't it more expensive to run in more than one region?

tscanausa 7y ago
Absolutely. For some customer it is the right thing for other customers it may not be the right thing. Every provider will have failures. So the question mostly boils down to does paying for more then 1 region cost more or less then paying for the the lost productivity or revenue of an outage like this. From some places the most costly things they spend money on is employees. If your whole company comes to a stop for even 1 hour. It may cost more then the engineering effort for multi zone, multi region or multi cloud for your critical environments.

iamaelephant 7y ago

Cool man let me know how I can run my Calendar in multiple regions.

deleted 7y ago

[deleted]

marme 7y ago

how do you use multiple regions when Google only supports certain things in limited regions like Dataflow Shuffle only being available in a single region in north america https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#cloud-dataflow-shuffle https://cloud.google.com/dataflow/docs/guides/deploying-a-pi...

deleted 7y ago

[deleted]