8 ms·

Why so many problems at Google lately? Calendar down two weeks ago[0], and Google Cloud had a larger outage a month ago[1] [0]: https://news.ycombinator.com/it

Why so many problems at Google lately? Calendar down two weeks ago[0], and Google Cloud had a larger outage a month ago[1] [0]: https://news.ycombinator.com/item?id=20213092 https://news.ycombinator.com/item?id=20213092 [1]: https://news.ycombinator.com/item?id=20077421 https://news.ycombinator.com/item?id=20077421

zbowling 7y ago
unrelated. very big company with thousands of products that don't suffer outages. two incidents doesn't make a pattern.
- thsowers 7y ago
  I would argue that two direct Google Cloud outages within a month is pretty concerning for GCP customers, and that it's possible that the calendar outage could also be related in someway since it is likely hosted on GCP, although that is speculation
  - tdhoot 7y ago
    Doubt Calendar is hosted on GCP. Generally Google does not run first-party systems on GCP, instead putting them on Borg (internal cloud).
    - judge2020 7y ago
      Related reading: https://news.ycombinator.com/item?id=9428043 https://news.ycombinator.com/item?id=9428043 https://news.ycombinator.com/item?id=17576720 https://news.ycombinator.com/item?id=17576720
    - samcday 7y ago
      Which, IMO, is actually a big problem. AFAIK Amazon are running a lot of actual production loads on AWS. Dogfooding can be extremely valuable, especially if a massive portion of your staff have the same profession as your target market. I've been using Google Cloud in a new role I started recently. There's definitely some parts of GCP I like, but whenever I use the Web Console I get the distinct impression nobody at Google actually uses it. If they did, I'm fairly sure all the annoying little warts I encounter would not exist.
      - acomjean 7y ago
        yeah. With google maps I ran into this little bug when trying to get a new key/update my payments info like they forced all google map users to do. The UI was maddeningly obtuse. This is from the second time I tried.. They did fix it eventually. Very complex system for distributing new keys taking payments. https://issuetracker.google.com/issues/124188941 https://issuetracker.google.com/issues/124188941
      - keepper 7y ago
        It took amazon over over 6 years to do so though. EC2 was released in 2006. Amazon.com last non ec2 server was 2012. But a lot of features of amazon.com still don't run on the main AWS offerings. GCP has not been out for that long. Also, quite easier to run an e-commerce site than to run the web's largest search engine as well as the largest email provider, as well as the largest maps provider. Each of these has an order of magnitude more traffic than amazon.com I'm sure they'll get there though, just not the same scale. Not even close.
        WaxProlix 7y ago
        Large parts of AWS don't run on AWS either, due to issues with circular dependencies and similar problems. Similarly, if all of AWS onboards to use your AWS service, suddenly that's the business. Your 'real' customers and their traffic are dwarfed by the rest of AWS, making it hard to keep those real customers at the forefront. There's also an issue with those deps of cascading failures; having two separate fabrics/strata for internal and external offerings is similar to having a multi-regional offering in that it's more robust to random failures and such.
      - tdhoot 7y ago
        I agree, it's a huge problem. It also leads to divergence between internal variants of systems and external where in many cases the internal variants are leagues better than even what competitors have. But unfortunately since they're not on GCP they don't drive cloud adoption for Google.
      - alteria 7y ago
        I'm curious - what are some examples of the warts you encounter?
- deleted 7y ago
  [deleted]
- mbrumlow 7y ago
  So you are making a case for smaller companies run by different people in different ways? So that we don't have huge outages with common systems shared across entire platforms misbehave? When you really care about high availability and security you really don't want all your systems run with the same software, hardware, and coded by the same teams. What does google (or amazon/msft) do to ensure a software echo chambers are not made within their infrastructure that potentially could cause mass scale outages by way of the same bug or bugs propagating through their systems? GCP, AWS, and Azure is the grate decentralization of the internet.
  - svachalek 7y ago
    Afaik, these are all as homogenous as they can make them, but there are limits to that. It's hard to move big, old things forward which creates some diversity, but that's probably worse than consistently running the latest stuff everywhere. If you want heterogeneous environments you have to cobble it together yourself by using multiple services.
- deleted 7y ago
  [deleted]
foobiekr 7y ago
Regression to the mean.
reading-at-work 7y ago
Don't forget the Google Fi outage from a short while ago: https://www.theverge.com/2019/6/3/18650851/google-fi-service-outage-cant-make-calls https://www.theverge.com/2019/6/3/18650851/google-fi-service...
tscanausa 7y ago
Terrance here from Google Cloud Support. There are only 3 things I can say about this situation. 1) These issues are currently unrelated. 2) We learn a lot from these situations. 3) A lot of these types of issues can be mitigated by running in more then 1 region. I really cant promise that today's situations will never happen again. There are a lot of moving pieces in our system and sometimes there are things outside of Google's control.
- tus87 7y ago
  No, you are too busy trying to stop Trump being elected in 2020 to run a decent service for your paying customers.
- gizmo385 7y ago
  > There are a lot of moving pieces in our system and sometimes there are things outside of Google's control. Are you implying that the cause of this outage is not Google's fault? If so, can you go into more details about that?
  - opportune 7y ago
    Not him but oftentimes cloud outages can be due to issues with the network connections to the datacenter, or power outages. Datacenters also sometimes have other single points of failure such as DNS, but those are within the company's control. https://www.networkworld.com/article/3373646/network-problems-responsible-for-more-data-center-outages.html https://www.networkworld.com/article/3373646/network-problem... https://www.datacenterknowledge.com/uptime/equinix-power-outage-one-reason-behind-aws-cloud-disruption https://www.datacenterknowledge.com/uptime/equinix-power-out...
    - stingraycharles 7y ago
      But data centers are typically designed with network and power failures in mind, not? Isn’t this why these kind of ring based network topologies exist, so that whenever a single network connection fails, it can still easily be routed around?
avocado4 7y ago
> Why so many problems at Google lately? I recently left Google to start a startup and now everything is falling apart.

zbowling 7y ago

unrelated. very big company with thousands of products that don't suffer outages. two incidents doesn't make a pattern.

thsowers 7y ago
I would argue that two direct Google Cloud outages within a month is pretty concerning for GCP customers, and that it's possible that the calendar outage could also be related in someway since it is likely hosted on GCP, although that is speculation
- tdhoot 7y ago
  Doubt Calendar is hosted on GCP. Generally Google does not run first-party systems on GCP, instead putting them on Borg (internal cloud).
  - judge2020 7y ago
    Related reading: https://news.ycombinator.com/item?id=9428043 https://news.ycombinator.com/item?id=9428043 https://news.ycombinator.com/item?id=17576720 https://news.ycombinator.com/item?id=17576720
  - samcday 7y ago
    Which, IMO, is actually a big problem. AFAIK Amazon are running a lot of actual production loads on AWS. Dogfooding can be extremely valuable, especially if a massive portion of your staff have the same profession as your target market. I've been using Google Cloud in a new role I started recently. There's definitely some parts of GCP I like, but whenever I use the Web Console I get the distinct impression nobody at Google actually uses it. If they did, I'm fairly sure all the annoying little warts I encounter would not exist.
    - acomjean 7y ago
      yeah. With google maps I ran into this little bug when trying to get a new key/update my payments info like they forced all google map users to do. The UI was maddeningly obtuse. This is from the second time I tried.. They did fix it eventually. Very complex system for distributing new keys taking payments. https://issuetracker.google.com/issues/124188941 https://issuetracker.google.com/issues/124188941
    - keepper 7y ago
      It took amazon over over 6 years to do so though. EC2 was released in 2006. Amazon.com last non ec2 server was 2012. But a lot of features of amazon.com still don't run on the main AWS offerings. GCP has not been out for that long. Also, quite easier to run an e-commerce site than to run the web's largest search engine as well as the largest email provider, as well as the largest maps provider. Each of these has an order of magnitude more traffic than amazon.com I'm sure they'll get there though, just not the same scale. Not even close.
      - WaxProlix 7y ago
        Large parts of AWS don't run on AWS either, due to issues with circular dependencies and similar problems. Similarly, if all of AWS onboards to use your AWS service, suddenly that's the business. Your 'real' customers and their traffic are dwarfed by the rest of AWS, making it hard to keep those real customers at the forefront. There's also an issue with those deps of cascading failures; having two separate fabrics/strata for internal and external offerings is similar to having a multi-regional offering in that it's more robust to random failures and such.
    - tdhoot 7y ago
      I agree, it's a huge problem. It also leads to divergence between internal variants of systems and external where in many cases the internal variants are leagues better than even what competitors have. But unfortunately since they're not on GCP they don't drive cloud adoption for Google.
    - alteria 7y ago
      I'm curious - what are some examples of the warts you encounter?
deleted 7y ago
[deleted]
mbrumlow 7y ago
So you are making a case for smaller companies run by different people in different ways? So that we don't have huge outages with common systems shared across entire platforms misbehave? When you really care about high availability and security you really don't want all your systems run with the same software, hardware, and coded by the same teams. What does google (or amazon/msft) do to ensure a software echo chambers are not made within their infrastructure that potentially could cause mass scale outages by way of the same bug or bugs propagating through their systems? GCP, AWS, and Azure is the grate decentralization of the internet.
- svachalek 7y ago
  Afaik, these are all as homogenous as they can make them, but there are limits to that. It's hard to move big, old things forward which creates some diversity, but that's probably worse than consistently running the latest stuff everywhere. If you want heterogeneous environments you have to cobble it together yourself by using multiple services.
deleted 7y ago
[deleted]

foobiekr 7y ago

Regression to the mean.

reading-at-work 7y ago

Don't forget the Google Fi outage from a short while ago: https://www.theverge.com/2019/6/3/18650851/google-fi-service-outage-cant-make-calls https://www.theverge.com/2019/6/3/18650851/google-fi-service...

tscanausa 7y ago

Terrance here from Google Cloud Support. There are only 3 things I can say about this situation. 1) These issues are currently unrelated. 2) We learn a lot from these situations. 3) A lot of these types of issues can be mitigated by running in more then 1 region. I really cant promise that today's situations will never happen again. There are a lot of moving pieces in our system and sometimes there are things outside of Google's control.

tus87 7y ago
No, you are too busy trying to stop Trump being elected in 2020 to run a decent service for your paying customers.
gizmo385 7y ago
> There are a lot of moving pieces in our system and sometimes there are things outside of Google's control. Are you implying that the cause of this outage is not Google's fault? If so, can you go into more details about that?
- opportune 7y ago
  Not him but oftentimes cloud outages can be due to issues with the network connections to the datacenter, or power outages. Datacenters also sometimes have other single points of failure such as DNS, but those are within the company's control. https://www.networkworld.com/article/3373646/network-problems-responsible-for-more-data-center-outages.html https://www.networkworld.com/article/3373646/network-problem... https://www.datacenterknowledge.com/uptime/equinix-power-outage-one-reason-behind-aws-cloud-disruption https://www.datacenterknowledge.com/uptime/equinix-power-out...
  - stingraycharles 7y ago
    But data centers are typically designed with network and power failures in mind, not? Isn’t this why these kind of ring based network topologies exist, so that whenever a single network connection fails, it can still easily be routed around?

avocado4 7y ago

> Why so many problems at Google lately? I recently left Google to start a startup and now everything is falling apart.