6 ms·

Etsy is 404 too: https://www.etsy.com https://www.etsy.com Seems to be a bigger issue. edit: Nest is down too: http://nest.com http://nest.com Fitbit.com is

Etsy is 404 too: https://www.etsy.com https://www.etsy.com Seems to be a bigger issue. edit: Nest is down too: http://nest.com http://nest.com Fitbit.com is 404 too: https://www.fitbit.com https://www.fitbit.com Big GCP issue? edit2: Downdetector.com shows multiple website and services as down, including Pokemon GO or Rocket League. GCP status page is still green all over the board: https://status.cloud.google.com https://status.cloud.google.com 19:10 CET update: Some websites are coming back, including spotify.com, but their app still does not work for me. information about outage just added to GCP status page, direct link: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R... Description: We are experiencing an issue with Cloud Networking beginning at Tuesday, 2021-11-16 09:53 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2021-11-16 10:40 US/Pacific with current details. We apologize to all who are affected by the disruption. 19:20 CET update: Description: We believe the issue with Cloud Networking is partially resolved. Customers will be unable to apply changes to their load balancers until the issue is fully resolved. We do not have an ETA for full resolution at this point. We will provide an update by Tuesday, 2021-11-16 11:28 US/Pacific with current details. Spotify desktop app still not working for me. 19:45 CET: Spotify app is back online for me.

Jugurtha 5y ago
I have things running on GCP that are OK. I think it's related to nameservers.
- terom 5y ago
  I don't think so, https://dnsviz.net/d/spotify.com/YHUfXQ/dnssec/ https://dnsviz.net/d/spotify.com/YHUfXQ/dnssec/ shows that spotify.com resolved to the same 35.186.224.25 as it does now.
- profmonocle 5y ago
  It is (was?) definitely a load balancer issue. We had multiple load balancers on completely different DNS providers fail.
panzagl 5y ago
homedepot.com is 404, various pages on lowes.com are 404
whymauri 5y ago
Status board got updated: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...
iso1631 5y ago
Another cloud issue? Wow, nearly a week's uptime. https://news.ycombinator.com/item?id=29197832 https://news.ycombinator.com/item?id=29197832
bobo_legos 5y ago
Homedepot.com appears down as well.
parthdesai 5y ago
discord seems to be down as well
ly 5y ago
When Spotify went down I switched to Youtube Music. Funny how that's still running but Spotify isn't, if it is in fact a Google issue.
- progbits 5y ago
  Internal Google infra and GCP are different beasts. Sure some failure modes take both down but more likely than not they are independent.
  - rmbyrro 5y ago
    That's why many choose AWS, they eat their own dog food.
cscheid 5y ago
https://deno.land https://deno.land down too. I guess a lot of CI actions will be failing just like ours...
- sholladay 5y ago
  deno.land is working for me, while lots of other sites are not. Did they failover to another provider?
  - profmonocle 5y ago
    It seems things are gradually coming back. All of our load balancers started working again.
  - cscheid 5y ago
    Working now, yes. Though deno.land's current IP still nslookup's to something inside googleusercontent.com so maybe google has fixed their Side. But it was definitely offline earlier today (cf. our github CI failures and downforeveryoneorjustme)
nuccy 5y ago
Parts of google are also down it seems, if one clicks three lines on the top right on google.com in mobile browser, then the https://www.google.com/mobile/?newwindow=1 https://www.google.com/mobile/?newwindow=1 opens, which also appears 404 not found.
rodolphoarruda 5y ago
Evernote too.
Nextgrid 5y ago
I’m assuming all these websites point to Google web frontend servers and for some reason it’s no longer able to map the Host header to the proper backend to proxy to.
- notreallyserio 5y ago
  I was seeing regular levels of health checks throughout the entire outage, so they still had configs in place. That seems plausible.
- jeffbee 5y ago
  This is a sort of repetitive outage for Google. They've wiped out the GSLB configs before. A year ago there was also that big outage where they blanked out the whole Gmail delivery configs and started rejecting all mails (even for gmail.com). Config safety is not their strong suit.
- profmonocle 5y ago
  For what it's worth, not only HTTP(S) load balancers went down. We have a couple TCP proxy load balancers that went down too.
karimf 5y ago
Gojek and Tokopedia too.
deleted 5y ago
[deleted]
XCSme 5y ago
All work for me in the EU apart from Etsy.
hk1337 5y ago
Egnyte had a pretty major outage today too. https://status.egnyte.com https://status.egnyte.com

Jugurtha 5y ago

I have things running on GCP that are OK. I think it's related to nameservers.

terom 5y ago
I don't think so, https://dnsviz.net/d/spotify.com/YHUfXQ/dnssec/ https://dnsviz.net/d/spotify.com/YHUfXQ/dnssec/ shows that spotify.com resolved to the same 35.186.224.25 as it does now.
profmonocle 5y ago
It is (was?) definitely a load balancer issue. We had multiple load balancers on completely different DNS providers fail.

panzagl 5y ago

homedepot.com is 404, various pages on lowes.com are 404

whymauri 5y ago

Status board got updated: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...

iso1631 5y ago

Another cloud issue? Wow, nearly a week's uptime. https://news.ycombinator.com/item?id=29197832 https://news.ycombinator.com/item?id=29197832

bobo_legos 5y ago

Homedepot.com appears down as well.

parthdesai 5y ago

discord seems to be down as well

ly 5y ago

When Spotify went down I switched to Youtube Music. Funny how that's still running but Spotify isn't, if it is in fact a Google issue.

progbits 5y ago
Internal Google infra and GCP are different beasts. Sure some failure modes take both down but more likely than not they are independent.
- rmbyrro 5y ago
  That's why many choose AWS, they eat their own dog food.

cscheid 5y ago

https://deno.land https://deno.land down too. I guess a lot of CI actions will be failing just like ours...

sholladay 5y ago
deno.land is working for me, while lots of other sites are not. Did they failover to another provider?
- profmonocle 5y ago
  It seems things are gradually coming back. All of our load balancers started working again.
- cscheid 5y ago
  Working now, yes. Though deno.land's current IP still nslookup's to something inside googleusercontent.com so maybe google has fixed their Side. But it was definitely offline earlier today (cf. our github CI failures and downforeveryoneorjustme)

nuccy 5y ago

Parts of google are also down it seems, if one clicks three lines on the top right on google.com in mobile browser, then the https://www.google.com/mobile/?newwindow=1 https://www.google.com/mobile/?newwindow=1 opens, which also appears 404 not found.

rodolphoarruda 5y ago

Evernote too.

Nextgrid 5y ago

I’m assuming all these websites point to Google web frontend servers and for some reason it’s no longer able to map the Host header to the proper backend to proxy to.

notreallyserio 5y ago
I was seeing regular levels of health checks throughout the entire outage, so they still had configs in place. That seems plausible.
jeffbee 5y ago
This is a sort of repetitive outage for Google. They've wiped out the GSLB configs before. A year ago there was also that big outage where they blanked out the whole Gmail delivery configs and started rejecting all mails (even for gmail.com). Config safety is not their strong suit.
profmonocle 5y ago
For what it's worth, not only HTTP(S) load balancers went down. We have a couple TCP proxy load balancers that went down too.

karimf 5y ago

Gojek and Tokopedia too.

deleted 5y ago

[deleted]

XCSme 5y ago

All work for me in the EU apart from Etsy.

hk1337 5y ago

Egnyte had a pretty major outage today too. https://status.egnyte.com https://status.egnyte.com