11 ms·
Ask HN: Is Google Compute down?
I'm not able to ssh into any of my boxes or access any of the sites, yet my status monitor isn't showing downtime. Spotify is also down for me, which is another GCP customer.
I'm in Los Angeles but the servers are hosted on us-central1
- ntq 8y agoWe are able to connect to some of our services via LTE but not a local wifi network
- ntq 8y agoSeeing the same China routes as mentioned on this thread
- syogi 8y agoI'm in Los Angeles and I can access my GCP Console but I can't access Google services like google.com or Maps or Gmail. EDIT: Some services are intermittently responsive. I had ~5 minutes of no access to anything. Some are slowly coming back.
- johnnyballgame 8y agoYouTube and Spotify unresponsive here.
- dickfickling 8y agoyeah, GCP is having a serious outage. Our site is down, so's Pivotal Tracker Edit: We're also in Los Angeles, connecting to us-central1. Seems to be a pattern?
- hellcow 8y agoBugsnag's app.bugsnag.com is down as well.
- TodayIsTheDay 8y agoDoes anybody else have chinatelecom-gw.transtelecom.net [217.150.59.249] in the traceroute for www.google.com
- kacy 8y agoAlso showing up on a traceroute to spotify.com for me.
- leetbulb 8y agoGitlab.com as well, earlier. 17 195.219.156.146 (195.219.156.146) 152.490 ms 152.423 ms * 18 * * mskn17ra-lo1.transtelecom.net (217.150.55.21) 198.658 ms 19 * * Google-gw.transtelecom.net (217.150.44.9) 192.230 ms 20 * * 108.170.250.111 (108.170.250.111) 172.086 ms
- leesalminen 8y agoWhat is today the day for?
- regnerba 8y agoGoogle IPs seem to be being routed to China for us. We have servers in San Jose that cannot access Google services. Trace route shows everything going to China when leaving the San Jose data center. We can access the same services from Vancouver just fine.
- gred 8y agoRelated? https://news.ycombinator.com/item?id=18385920 https://news.ycombinator.com/item?id=18385920
- lostmsu 8y agoSeems like its time to start or accelerate a working group on secure BGP.
- regnerba 8y agoIt's always great to go to your boss/CEO, explain what's going on, and then have to say "no, there is nothing we can do about it."
- docker_up 8y agoHow many times does this have to happen before China's privileges to do things like this get revoked? At this point, it can't be just a mistake and must be some state-sponsored hacking. Seems like a great way to find out where a particular Spotify user's IP address is.
- QML 8y agoWhat are you going to do? Divide the internet in half? I say that in a joking way but it’s a possibility.
- LinuxBender 8y agoThere is already a wall. This may be a case of something leaking through it.
- 8y ago
- gsibble 8y agoCan confirm many IPs are being sent to China.
- crunchlibrarian 8y agoThis is nuts.
- kenhwang 8y agoMight be related to: https://status.cloud.google.com/incident/cloud-networking/18018 https://status.cloud.google.com/incident/cloud-networking/18... Also in LA, had intermittent issues with google.com and Spotify all morning. edit: linked to wrong issue
- fxdoublecute 8y agounlikely, IMO. this is a routing problem, either a deliberate attack or else a gigantic screwup
- deleted 8y ago[deleted]
- davismwfl 8y agoI am on the East Coast, in Florida and seeing the same thing with traffic heading to China, lots of "chinatelecom-gw.transtelecom.net" in traceroutes I have never seen prior.
- scrollbar 8y agoGetting this as well in SF. transtelecom.net WHOIS says they're Moscow-based
- davismwfl 8y agoYea, just saw that same thing. Definitely something interesting going on, and I am sure no shortage of some frantic research and effort to resolve this all at Google and such right now.
- deleted 8y ago[deleted]
- deleted 8y ago[deleted]
- fxdoublecute 8y agowe manage services deployed in every GCE region, and our monitoring in London is reporting every GCE region having intermittent connectivity. no problems with our services in the other major clouds (we use basically all of them)
- jamalex 8y agoSame thing here in San Diego. Traceroute to spotify.com going through LA, San Jose, NY, London, Amsterdam, Frankfurk, "mskn17ra-lo1.transtelecom.net", then ChinaTelecom-gw.transtelecom.net.
- deleted 8y ago[deleted]
- vamos_davai 8y agoI have trouble accessing YouTube. I live in Sherman Oaks (a town of Los Angeles).
- sethvargo 8y agoHi all - Seth from Google here. Our team is aware and we are working on mitigation. In short, a third party telco provider is advertising on one of our IP blocks. Unfortunately that's all the information I can share at this time.
- fxdoublecute 8y agothanks for the update! FWIW we started noticing the connectivity problems around 2018-11-12 21:17 UTC
- sethvargo 8y agoThanks for the info. We have reports showing it started a bit earlier than that, but every piece of information is helping in managing an incident. I'll make sure the team is aware.
- konschubert 8y agoEDIT: This is a general statement, I am not complaining to google here. This kind of thing should not be possible. Are there any protocol proposals or other kind of upgrades to the routing protocols that would prevent these kind of mistakes/attacks?
- cm2187 8y agoI am surprised how fragile is the internet given how our society is increasingly becoming critically reliant on it.
- QML 8y agoThe internet was not really built with security in mind. Look at DNSSEC or BGPSEC.
- zzzcpan 8y agoOn the other hand it's not that fragile everywhere and for everyone. When ISP markets are not monopolized and the service doesn't rely on a big cloud - much fewer users will get rerouted through random countries and the service itself can failover to properly working datacenters, tolerating all those BGP misdesigns. It's if the internet doesn't like all that centralization with all that market domination. It's naturally resilient only when there is a lot of competition.
- jamalex 8y agoDespite the subdomain, the IP for ChinaTelecom-gw.transtelecom.net (217.150.59.249) seems to be based in Russia, as does the carrier: https://en.wikipedia.org/wiki/TransTelekom https://en.wikipedia.org/wiki/TransTelekom
- sterlind 8y agoSeems likely to be TT's gateway to CT. New theory: TransTelecom brought up a new gateway to ChinaTelecom, which incorrectly gossiped all advertisements from ChinaTelecom. This caused a leak, since CT has bgp highjacking of Google IP ranges for the GFW within China, but ordinarily doesn't leak them outside the country. TransTelecom misconfigured the gateway to broadcast everything advertised by ChinaTelecom, bringing external traffic into the GFW.
- gsibble 8y agoThat's a pretty good theory.
- londons_explore 8y agoI doubt the GFW uses BGP to route traffic to it. It needs to filter traffic to any address, and wouldn't have specific google ranges configured.
- cobookman 8y agoThere's a current BGP prefix hijacking issue currently being mitigated.
- RayHawk 8y agoI'm getting the same thing. Servers are in us-east1 and tracert is ending at chinatelecom-gw.transtelecom.net [217.150.59.249]
- aviv 8y agoFunny, a day after I posted this... https://news.ycombinator.com/item?id=18429099 https://news.ycombinator.com/item?id=18429099 Is our first time actually rolling over the entire stack to AWS - and it worked! GCP outage currently is massive, can't even use other regions. Edit: This also affected AWS Oregon region earlier. I do not know how yet, but they too were unreachable briefly. Seems to be okay now.
- jetforme 8y agoI'm in LA and seeing similar routing through chinatelecom-gw.transtelecom.net
- CydeWeys 8y agoWe urgently need a solution for routing traffic to IP addresses that is better than BGP.
- dasm 8y agoAgreed. This appears to be a repeat of the attack covered here: https://news.ycombinator.com/item?id=18385920 https://news.ycombinator.com/item?id=18385920 I'm not familiar with BGP routing attacks; the article above seems to imply the attacker needs to compromise certs in order to glean useful data from the attack. If that's accurate, is this Google-oriented traffic vulnerable to this type of attack?
- raesene9 8y agofor Google traffic, assuming certificate pinning is in place, I can't see this being that successful. However for more general traffic, well look at the trusted root list in your browser/OS. Realise that every single one of those trusted routes can issue certificates for a given domain...
- dasm 8y agoThanks. Since it appears all this traffic is Google-related, any guesses as to what the attacker could have gained here?
- raesene9 8y agoThis could just be a mistake of course, malicious intent isn't needed :) Of the top of my head, assuming malicious intent, well not all browser (especially older ones) do certificate pinning, so perhaps then Chinese users of Google services using old browsers would find their traffic being intercepted? Past that the leakage would seem fairly minor, a list of source IP addresses and destination hosts.
- raesene9 8y agoor ISPs could implement the proposed BGP security standards... that have been proposed for , well a long time.
- infogulch 8y agoSo... what's the current state of a secure BGP? I feel like this in the top 3 security threats to the whole of the internet.
- xolox 8y agoReading through the comments here I'm recognizing "China Telecom" from an article on a BGP hijack that was published about a week ago, I still had the article open in my browser: https://arstechnica.com/information-technology/2018/11/strange-snafu-misroutes-domestic-us-internet-traffic-through-china-telecom/ https://arstechnica.com/information-technology/2018/11/stran... In another comment in this thread I read: > Seems like its time to start or accelerate a working group on secure BGP. Indeed things can't go on like this for much longer...
- dejaime 8y agoWell, my instance seems to be working
- xstephen95x 8y agoseems like https://arstechnica.com/information-technology/2018/11/strange-snafu-misroutes-domestic-us-internet-traffic-through-china-telecom/ https://arstechnica.com/information-technology/2018/11/stran...
- linuxbuzz 8y agoWorld War III on the Internet front!
- bifrost 8y agoI hate to break it to everyone, but the technology to filter this sorta thing has existed for a very long time, but people often don't use it. Most of the time this sort of thing is accidental (IE: operator error)so a lot of operators kinda ignore it. Check out "IRR Power Tools" if you're interested.