34 ms·
Error 404 (Not Found)
- gassius 5y agoFunny enough, datadog, which I was using to investigate on of my vercel sites, is down too Yeah, Vercel is running some GCP services it seems
- Borrible 5y agoYes, but we're still on DEFCON 5.
- al_james 5y agoNetlify is also failing for us, and reporting bad TLS certs. Not sure if they use GCP https://www.netlifystatus.com/ https://www.netlifystatus.com/
- al_james 5y agoYes confirmed its because netlify's origin servers use Google Load Balancers. According to this blog they claim "We can easily move the entire brains of our service between Google, Amazon, and Rackspace in around 10 minutes with no service interruptions.". Lets see if thats true. https://www.netlify.com/blog/2018/05/14/how-netlify-migrated-to-a-fully-multi-cloud-infrastructure/ https://www.netlify.com/blog/2018/05/14/how-netlify-migrated...
- Wingy 5y agoSeems to be coming back up.
- lenniez 5y agoAlso experience that with all my GCP related infrastructure (Europe)
- mixedbit 5y agoLooks like perhaps an issue with Google Load Balancer. We have a load balancer in front of Google Storage Buckets, and can access resources directly from the buckets, but getting 404 when going through the load balancer.
- neom 5y agoNon-engineer here - Is there an easy way to multi-provider redundancy around this? Can you have LBs on multiple clouds and use dns to move around or something? Or does your LB have to be at the provider the app is at? Sorry if this makes no sense. :o
- sparrc 5y agoyes it's 100% technically possible, the main issue is it would be significantly more expensive.
- te_chris 5y agoYep, it's the GLB. Went down for us at 5:46 GMT. Just responding 404 and logs reporting an internal error.
- htrp 5y agoI can confirm this on my side too
- cinericius 5y agoI wonder if a legal discovery will ever find internal status dashboards that reflect reality rather than fictitious SLA liability-aware status pages.
- hmrr 5y agoYour cynicism is justified and clearly based on the same experience I have :)
- paxys 5y agoYou don't need legal discovery for that. Every "X as a service" contract you sign will explicitly say that SLAs aren't dependent on dashboards/ping tests but rather a mostly subjective measure of "availability".
- contrahax 5y agoSeeing the same - I have projects in us-east1 that went offline first, then us-west1 went offline a few minutes after. Everything green on their status page and nothing in the dashboard - everything returns a 404 so I'm assuming a really high level LB just took a dump.
- contrahax 5y agoSeems to have just resolved itself in us-east1 so I'm hoping us-west1 follows a few minutes after.
- plg 5y agooverleaf.com also
- samasblack 5y agoMy heart goes out to all the math students who were frantically texing their problem sets last minute
- tthebst 5y agojust wanted to start writing my thesis
- jdleesmiller 5y ago(Overleaf co-founder here.) We're back up, so get writing :)
- humanistbot 5y agoThe title of the 404 page on all the down sites has an extra "1" after the exclamation points: "Error 404 (Not Found)!!1"
- MarcScott 5y agoIronically it appears that IsItDownRightNow? is also down, although that could because they're experiencing what is basically going to be the equivalent of a DDOS.
- judge2020 5y agodowndetector.com is the most resistant in my experience.
- Causality1 5y agoI wonder if they have advertising methods in place to turn days like this into a windfall.
- dustinmoris 5y agoThings are back online again!
- pdenton 5y agoI knew Google would one day take control of the web, just thought they'd have a more clever way of doing it.
- authed 5y agoThat's why I like the clouds.
- dustinmoris 5y agoI have a few services running from the same GKE cluster, same ingress controller, same nodes, same GLB, same everything. Some are 404ing at the moment and others work just fine. Feels like a GLB issue. Nothing in my GCP dashboard seems to be aware of the issue however. Only reason I found out is because I use an external service to ping me if a site is down.
- lukeschlather 5y agoAs far as I can tell everything is up it's just that our load balancers aren't routing traffic and just returning 404s.
- jaredeklee13 5y agoGlobal: Experiencing Issue with Cloud networking Incident began at 2021-11-16 10:10 (all times are US/Pacific). https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...
- deleted 5y ago[deleted]
- aalbertson 5y agoLowes search was also down.
- _nickwhite 5y ago1:10PM - either it has resolved itself, or a regional issue, but I'm not seeing anything being down from the East coast USA.
- neelr01 5y agoIts back up now :)
- ishikawa 5y agoit was back in a few minutes. But that wasw pretty weird. App Engine was affected.
- scame-miv 5y agoI was experienced this issue with my spotify app. Initially thought, it was my internet issue lol.
- ruined 5y agonice
- melling 5y agoI noticed Discord went down for me.
- trillic 5y agohttps://www.windy.com https://www.windy.com down, same issue.
- mkl95 5y agoNot Google's best week.
- pcbro141 5y agohttps://downdetector.com/ https://downdetector.com/ yup
- DiFronzo 5y agoOh okay, thought something was on my end.
- andydd 5y agogcp load balancer responding with 404
- terramex 5y agoEtsy is 404 too: https://www.etsy.com https://www.etsy.com Seems to be a bigger issue. edit: Nest is down too: http://nest.com http://nest.com Fitbit.com is 404 too: https://www.fitbit.com https://www.fitbit.com Big GCP issue? edit2: Downdetector.com shows multiple website and services as down, including Pokemon GO or Rocket League. GCP status page is still green all over the board: https://status.cloud.google.com https://status.cloud.google.com 19:10 CET update: Some websites are coming back, including spotify.com, but their app still does not work for me. information about outage just added to GCP status page, direct link: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R... Description: We are experiencing an issue with Cloud Networking beginning at Tuesday, 2021-11-16 09:53 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2021-11-16 10:40 US/Pacific with current details. We apologize to all who are affected by the disruption. 19:20 CET update: Description: We believe the issue with Cloud Networking is partially resolved. Customers will be unable to apply changes to their load balancers until the issue is fully resolved. We do not have an ETA for full resolution at this point. We will provide an update by Tuesday, 2021-11-16 11:28 US/Pacific with current details. Spotify desktop app still not working for me. 19:45 CET: Spotify app is back online for me.
- Jugurtha 5y agoI have things running on GCP that are OK. I think it's related to nameservers.
- terom 5y agoI don't think so, https://dnsviz.net/d/spotify.com/YHUfXQ/dnssec/ https://dnsviz.net/d/spotify.com/YHUfXQ/dnssec/ shows that spotify.com resolved to the same 35.186.224.25 as it does now.
- profmonocle 5y agoIt is (was?) definitely a load balancer issue. We had multiple load balancers on completely different DNS providers fail.
- 5y ago
- cdiddy2 5y agoDiscord down
- deberon 5y agoGCP outage? Status page shows green but a bunch of sites seem down (Rocket League most importantly).
- lillecarl 5y agoMost importantly this game I play for leisure is down!
- mosen 5y agoI think it was a joke.
- NaughtyShiba 5y agoI mean, that’s a fair point… leisure activity, whatever it is, is required for rest and to avoid burndown
- ukd1 5y agoWe're also affected.
- ukd1 5y agoand back up and running!
- dyeje 5y agoDiscord seems to be down.
- Jansin3 5y agoRocket league perhaps epic games even
- timdaub 5y ago- discord doesn't allow me to connect either.
- icecoldfire 5y agoPokémon GO also down
- nicebill8 5y agoGCP - my Cloud Run containers are giving 500's
- dfxt8 5y agoDiscord is down too.
- crackercrews 5y agoCBS.com is down, and some NYT pages as well.
- rasengan 5y ago
- hexfish 5y agoThat's just an inside joke and has been a default error page for Google/GCP for as long as I can remember.
- heywire 5y agoGoogle’s 404 has always been like that
- rasengan 5y agoThat's noteworthy. Do you have an archived url example? :O
- dymk 5y agoIt’s been like that for all the time that I can remember (at least a decade) Not super noteworthy
- rasengan 5y agoYou're right. It's just bad.
- corobo 5y agoFor what it's worth I don't think you should have been flagged or even downvoted just for being wrong. Corrected yes, absolutely. Sinking to the bottom of low contrast lake was a bit much. Flags being gigadownvotes on this site suuuuuucks now. How do people learn if nobody can correct them? Daft. I used to have a vouch option but I guess I used it too much haha
- tfehring 5y agoLooks like it changed to that title in 2011. http://web.archive.org/web/20110408072631/http://www.google.com/404/ http://web.archive.org/web/20110408072631/http://www.google....
- deforciant 5y agohttps://linear.app/ https://linear.app/ is also down
- profmonocle 5y agoI wondered why our alerts started going nuts. Seems like basically every global Google Cloud load balancer went down. Doesn't seem to affect single-region network load balancers. Edit: All of ours are back up. Some other services still seem down though.
- 1cvmask 5y agoIs it regional? Surprised Spotify is not active-active on other platforms like AWS and Azure.
- te_chris 5y agoAffecting us. Busiest time of the year and now down 20 min. It's the Global Load Balancer, so god knows what bit of the global edge has been taken out.
- iso1631 5y agoAt least you don't run your own servers
- sabbakeynejad 5y agoveed.io is down too! Same problem
- oussama-gmd 5y agoSame here, seems to be an issue with load balancers
- dave_aiello 5y agoRight now homedepot.com and the APIs that drive their mobile app are down too.
- fsflover 5y agoTime to switch to https://bandcamp.com https://bandcamp.com.
- casi18 5y agoor https://ipfs.io/ https://ipfs.io/
- dymk 5y agoCool, how do I stream the newest Taylor Swift album on that?
- benbristow 5y agoYou write the domain (ipfs.io) on a blackboard with a chalk pen then take your nails and scratch the board. (jk.)
- BraveryOverload 5y agoDAE THINK POP MUSIC BAD?!
- deleted 5y ago[deleted]
- iso1631 5y agoMy colleague (who loves Taylor Swift) bought the mp3s from somewhere (amazon?) and uploaded them onto her plex server the hour it came out. That server continues to work just fine.
- cheeze 5y agoAs a heavy plex user, I can't imagine using it as my default music player. CX isn't great for music, IMO.
- soheil 5y agoFunny thing is when you google Home Depot or Paramount Plus you get ads served by Google as the first result. When click on it Google then shows you a 404 page. I wonder if they'll get a refund on their Adwords campaign.
- progbits 5y agoWhy? That's not part of the ad contract. They will get refunded for GCP if it goes out of SLO.
- makecheck 5y agoOne of my pet peeves with so many services! Their obnoxious pre-ads can play flawlessly (stealing your time/eyeballs and giving them benefit), and they can still fail to give you the content you exchanged your time/eyeballs to see. Worse, they can repeatedly fail and repeatedly drill the same ads into your brain. There ought to be a law that essentially says if ads are “paying” for content, there must be a flawless link between ads and content such that the system can tell if the content is available (or detect after the fact that something was not delivered properly). And then, based on that, it either is required to ensure the ad never plays (since the content cannot be delivered), or that the user must be compensated in some way (e.g. we see you were forced to see an ad but got nothing so we are crediting $1 to your account).
- artembugara 5y agook, so our API is down. We're on GCP... https://api.newscatcherapi.com/v2/search https://api.newscatcherapi.com/v2/search
- hs86 5y agohttps://toggl.com/ https://toggl.com/ is also affected.
- nagisa 5y agoOne of the websites I've noticed this on is back up.
- mey 5y agoOur infra in us-central1 behind gcp lb is impacted but not us-west1
- humanistbot 5y agoSites that are down according to https://downdetector.com https://downdetector.com include Spotify, Discord, Snapchat, Etsy, Pokemon Go, Epic Games, Target, Paramount+, Evernote
- deleted 5y ago[deleted]
- markbnj 5y agoWe were down. Just came back. Things seem to be resolving.
- soheil 5y agoI also wonder how many companies didn't want to admit they were using Google for their infrastructure. Downdetector shows AWS being affected, it'd be embarrassing if they were caught using Google Cloud Platform.
- grumple 5y agoSeriously doubt that AWS, Facebook are using Google for infra. There's probably some other effect at play, like people using a Google service to connect to these things. Also don't see any effects on those services personally.
- oussama-gmd 5y agoSeems to be an issue with GOOGLE cloud load balancer. our website is down too
- johanam 5y agohttps://overleaf.com/ https://overleaf.com/ is also 404 now
- etimberg 5y agoSeeing this across the board with providers on GCP. Firestore however does not appear to be down
- deleted 5y ago[deleted]
- thegranderson 5y agoSeems like another DNS issue - switching to Cloudflare 1.1.1.1 got me back online...
- Jugurtha 5y agoMy stuff is running OK on GCP, with GKE usage. Maybe it's related to nameservers?
- deforciant 5y agoif you are using regional load balancers or serving traffic directly from nodes then you would be fine :) "only" global LB failed
- jacobkg 5y agoWe bypassed our Google load balancer and pointed DNS directly at the IP addresses of our servers and that seems to have helped
- IceWreck 5y agohttps://www.navidrome.org/ https://www.navidrome.org/ Self hosted Spotify. Compatible with subsonic clients.
- NaughtyShiba 5y agoOn one hand, Spotify is much cheaper, on other, perhaps artistd gets paid more (assuming you acquire music legaly)
- ksajadi 5y agoHalf the internet is down because of a Google Cloud global issue on their load balancers, including Spotify and Etsy and GCP status is all green: https://status.cloud.google.com https://status.cloud.google.com If you ever wondered why GCP is a distant third runner in the enterprise cloud space, here is your answer.
- yupper32 5y agoThe big three cloud platforms all have this issue of delayed status updates. Why do you think it's just GCP?
- mostdataisnice 5y ago...and there's a reason for that. Automations to update the status page are rarely acceptable, since the status page statuses have legal and financial implications. Therefore, the IM usually has to update it (or tell someone to update it). But, realistically, when you get paged, you first need to figure out what exactly is wrong and at least a vague idea of why. Then, you need to tell someone to update the page. Then, it gets updated. The status page will always lag the outage. It's not a conspiracy.
- kazen44 5y agoAlso, in most teams, people who do external communication are different from those doing triage and troubleshooting.
- cheeze 5y agoYeah, but they are still people who are responding to a page, working on wording and getting it approved, and then updating. 20 minutes seems pretty reasonable to me.
- deathanatos 5y agoStatus pages should be driven that way, though. "legal and financial" implications and "It's not a conspiracy" is a poor excuse. Now, I'm on Azure, but it seems like from the comments the situations are similar. So, instead of an automatically updated status page that would help engineers do their jobs, we get a status page that isn't accurate, and customers have pull teeth to get a service credit where/when one is due. And it seems like you can have the cake and eat it too here: while IANAL, a footnote in the SLA or the status page that "this is a machine estimate and not reflective of what goes into the SLA" should do it, no?
- iampliny 5y agoMight be Google Cloud outage: https://news.ycombinator.com/item?id=29243753 https://news.ycombinator.com/item?id=29243753
- arjan_sch 5y agoThe 404's changed into 502's.. I guess that's progress. Fingers crossed it's back up soon
- Thaxll 5y agoIt's back!
- christophclarke 5y agoSnapchat also having issues refreshing
- benbristow 5y agoSnapchat runs on the Google Cloud.
- deleted 5y ago[deleted]
- cglace 5y agoEverything seems to be working on our end as of 1:08 PM EST.
- colewilson 5y agolooks like it's back up?
- cronix 5y agoback up as of now (10:10a PDT) This sure makes it easy to know who is hosted by google by going to downdetector.com.
- crdrost 5y agoYeah, per the status page, there's a temporary mitigation rolled out while the team tries to figure out more... so no more 404s but load balancers will be locked down for a while while investigation continues.
- technologyvault 5y agoBigCommerce is likely one of those. Our BigCommerce store was down for several hours today.
- judge2020 5y agoNot exactly hard to do that in the first place whois $(dig +short spotify.com A) NetName: GOOGLE-CLOUD
- uvdn7 5y agoThat only says spotify.com is using GCP's DNS but not necessarily for hosting? EDIT: I was wrong, as pointed out by shizcakes.
- addcninblue 5y agoIt looks like everything is back now. That was a short outage by recent standards...
- caffeinated_me 5y agoMy company was seeing GCP Airflow environments not responding, but they seem to have recovered in the past few minutes.
- uubk 5y agoWe found extra rules in our GCLB routing config - removing them restored our service.
- jaredeklee13 5y agoGlobal: Experiencing Issue with Cloud networking Incident began at 2021-11-16 10:10 (all times are US/Pacific). https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...
- cronix 5y agoSeems to be fixed.
- mcintyre1994 5y agoLooks like it's made it to the Google status page: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...
- neelr001 5y agoIts back up now :)
- neelr001 5y agoits back up :)
- NaughtyShiba 5y agoSeems to be back already
- saranshk 5y agoOur instances started working again, so seems to be fixed
- algorithm_dk 5y agoThis is clearly the hottest thing on HN right now, and it was bumped from #1 to #6, anyone knows why? Is it some kind of bot protection?
- floatingatoll 5y agoUser flags, because outages are a fact of everyday life.
- mbesto 5y agoWhich is dumb, linking to status pages shouldn't be on HN. A blog that has analysis and explanations of outages or post mortems should. knock-knock dang
- floatingatoll 5y agoDang doesn't see messages like that unless you use the footer Contact link, but I remember a comment from him a while back that I would summarize as "Some site users think it's a good use of HN, and other site users disagree and flag it, and we downweight/dedupe them sometimes and/or if someone emails us with the Contact link". I just didn't want you to wait for a reply that'll never come unless you Contact them.
- mbesto 5y agohehe I know, was just saying more for fun, but I appreciate the comment none the less.
- collinmanderson 5y agoSee also https://news.ycombinator.com/item?id=29243740 https://news.ycombinator.com/item?id=29243740
- milesward 5y agoIt's back up
- davidkuennen 5y agoMy app was down too. Can confirm it is most likely the GCloud LB
- kadomony 5y agoI don’t understand why people post website outages. Do you think the DevOps teams at these billion dollar streaming companies are so clueless that they don’t have monitoring in place? Do you think that people who go to a site when it’s down don’t see the same thing? So whose awareness does this serve?
- blamazon 5y agoIn general, people post things on HN to discuss them. This includes high profile web outages.