7 ms·
Internet Vulnerability Takes Down Google
- Wiretrip 8y agoOr is this just an advert?
- wongarsu 8y agoIt's a legitimate incident that Google didn't explain all that well beyond "it wasn't us": https://status.cloud.google.com/incident/cloud-networking/18018 https://status.cloud.google.com/incident/cloud-networking/18... OP showing off their tools contributes to the article, so I don't mind
- elorant 8y agoThis could explain why there's so little information coming out from Google as to what caused the outage. China Telekom is state-owned and thus pointing a finger at them could stir the relations with the Chinese government, regardless whether this was a bug or an intended act.
- tumetab1 8y agoOfficial information here https://status.cloud.google.com/incident/cloud-networking/18018 https://status.cloud.google.com/incident/cloud-networking/18...
- phreack 8y agoThe official line[1] is that it was very likely accidental and not malicious. [1]https://arstechnica.com/information-technology/2018/11/major-bgp-mishap-takes-down-google-as-traffic-improperly-travels-to-china/ https://arstechnica.com/information-technology/2018/11/major...
- a012 8y agoExcept it's repeatedly BGP leak accident from China Telecom.
- TheGrumpyBrit 8y agoFrom the article, it wasn't China Telecom who made the initial misconfiguration - it was a small ISP in Nigeria. China Telecom just accepted their BGP update and rebroadcast it.
- raarts 8y agoSo China telecom would have been able to fix it real fast? Hmm
- deleted 8y ago[deleted]
- ahmedalsudani 8y agoIt should be clarified the vulnerability was not on Google's end. It's the sort of thing we're used to by now: misconfigured/malicious BGP routes.
- john37386 8y agoIf google advertised several /24 for these ip networks instead of /19, this would have not happenned.
- tyingq 8y agoIs that a best practice though? Wouldn't the BGP tables get pretty big if everyone did that?
- ahmedalsudani 8y agoThey would, and we would probably see a repeat of this: https://bgpmon.net/what-caused-todays-internet-hiccup/ https://bgpmon.net/what-caused-todays-internet-hiccup/
- apathy 8y agoUgh, old Cisco trash Didn’t Pakistan advertise a /28 at one point that blackholed YouTube? Good times. (/s)
- tyingq 8y agoI believe there are routers that break at around 1M ipv4 entries, and we're at ~700k now. https://www.cidr-report.org/cgi-bin/plota?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp-active%2etxt&descr=Active%20BGP%20entries%20%28FIB%29&ylabel=Active%20BGP%20entries%20%28FIB%29&with=step https://www.cidr-report.org/cgi-bin/plota?file=%2fvar%2fdata...
- baseballMan 8y ago
- cschmidt 8y agoDoes anyone know if there was any relation of this attack and the Facebook outage that happened yesterday as well? Seems weird that both FB and Google have trouble on the same day.
- Allvitende 8y agoNo idea if there is correlation but something is definitely not being discussed.
- luckylittle 8y agoChina is punishing Google for not making the censored search engine?
- slivym 8y agoDespite the routes terminating at China's firewall I don't think it's a good guess to say this was necessarily a Chinese attack. It seems like the interference of the Chinese firewall was simply a side-effect of the new path the network traffic was going. If you were perpetrating this attack it seems more advantageous to re-route the traffic and allow it continue working rather than discoverably interfering with the traffic. Plus if you were going to try to block the route, there's probably a better way of doing it.
- bcaa7f3a8bbc 8y agoChina's firewall has a long history of accidental enforcement of censorship outside its jurisdiction, leaked DNS and BGP poison are the most common problem.
- tyingq 8y ago"However, this also put valuable Google traffic in the hands of ISPs in countries with a long history of Internet surveillance." Curious what Google traffic might still be unencrypted now.
- miemo 8y agowho, America?
- taneq 8y agoHush, we don't talk about that.
- seedie 8y agoMostly Google News, Google Maps and Youtube according to their Transparency Report https://transparencyreport.google.com/https/overview https://transparencyreport.google.com/https/overview
- alex_duf 8y agoIf it was an isolated attack, the damage is probably nil, but if combined with a rogue certificate authority you can do some pretty in-depth spying.
- marknadal 8y agoHmm, 2018 and BGP and DNS aren't cryptographically secure. This is ridiculous, anybody want to help change this? I did a 1.25 hour tech talk on this at a SF meetup https://youtu.be/EHZyaupYjYo https://youtu.be/EHZyaupYjYo explaining from bottom up what we need to replace internet sites, services, and protocols, all end to end secure.
- anc84 8y ago^ cryptocurrency marketing ^
- d33 8y agoJust wanted to say that I find those visualisations pretty pleasant to see.
- vermilingua 8y agoAgreed, this was a very good advertisement for their service.
- whitexn--g28h 8y agoWhy do ISPs peer with China Telekom? BGP is built on trust, if a peer isn’t trustworthy why not drop them?
- rahimnathwani 8y agoChina Telecom is the 2nd largest domestic broadband provider, and the 3rd largest mobile operator, in China. Lots of people live in China.
- chongli 8y agoChina has also made it their goal to bifurcate the internet in order to prop up their authoritarian state. I don't see why the rest of the world should play along with that. It's frustrating in the extreme to see big Western companies repeatedly bow to the Communist party's wishes only to get all of their IP stolen by a domestic clone with the backing of the government.
- rahimnathwani 8y agoWhich 'big Western companies' got 'all of their IP stolen'?
- mtgx 8y agoWhich haven't? Even Google was hacked by the Chinese military.
- MadraRua2000 8y agohttps://www.fireeye.com/content/dam/fireeye-www/services/pdfs/mandiant-apt1-report.pdf https://www.fireeye.com/content/dam/fireeye-www/services/pdf... To answer your question, most large companies have been subject to cyber espionage by the chinese military for several years. This accounts for the speed at which China built its manufacturing base in strategically important industries and its ability to rapidly copy products and the required tooling for their manufacture. The Mandiant report I linked to above focused on tracking the activities of a single but prolific APT group responsible for hacks in 150 major companies worldwide for 7 years since 2006, and that they believe is actually a team of cyber espionage professionals from within the Chinese military. Also just for fun: Anyone here drive a Range Rover Evoke? Heres China's version: https://en.wikipedia.org/wiki/Landwind_X7 https://en.wikipedia.org/wiki/Landwind_X7
- the_duke 8y agoCould this be a not so subtle middle finger from China to the US? (considering the recent political turmoil)
- ccnafr 8y agoUgh... that headline is abysmal. It's just a BGP hijack. Get over it.
- vermilingua 8y agoNo, it’s actually quite accurate. BGP hijacks are a vulnerability in the very core fabric of the internet. Just because we’ve seen a lot of BGP hijacks recently, does not in any way, shape, or form downgrade their severity.
- mcv 8y agoIf it can be done once, I guess it can be done multiple times. I don't know nearly enough about this to understand how BGP hijacks work or why they are possible. Can anyone point me to a simple explanation for a layman?
- Twirrim 8y agoBGP is a gossip protocol. BGP roughly works like this: As BGP nodes come online, they establish connections with "nearby" existing BGP nodes, saying what version and how frequently they'll check in, and what their AS number is (a unique identifier) Once communications have been established, then they can start to report any network routes they know about. "I'm AS 123456, and I am the originator for 3.0.0.0/0" (i.e. they're responsible for it), "Also, I can reach 4.0.0.0/8 with a cost of 6" (A fair number of hops away on the network). Any neighbouring BGP peers update their routing table: "AS 123456 is responsible for 3.0.0.0/8, and I can reach it with a cost of 1, and I can also reach 4.0.0.0/8 with a cost of 7 via AS 123456". If there's a cheaper route to a network address, no changes will happen. Routing changes can propagate quite quickly across the internet. The routing protocol is nice and lightweight, and updates are happening with reasonable frequency, as network connections come and go. The idea is that should damage occur to the network fabric, the network will automatically update and route around it, without need for any intervention. It's entirely built on trust, though. You have to trust that AS 123456 is indeed actually responsible for 3.0.0.0/8. If you get two parties indicating responsibility for a network range, it's possible to end up with routing loops etc, as things get in to a mess. What is legitimate behaviour, though, is for, say, AS 123456 to be the originator for 3.0.0.0/8, and another AS be the originator for 3.0.1.0/24 (i.e just 254 addresses under that space). That's not an unusual situation, and it won't cause routing issues, because more specific is taken as a priority over less specific, rough analogy: "In general mail for General Electric, should be sent here, but if it's for the electronics product division, send it straight to them" There have been different attempts to put filtering in place, provide authentication "Yes, AS 123456 is allowed to be responsible for 3.0.0.0/0" and the like, but nothing has really taken off.
- cauldron 8y agoReminder that China has some big investment in Nigeria, they even built and operates a railway there.
- SlowRobotAhead 8y agoI’ve been around the world to see some amazing/strange investments by China. Whole office buildings in Malaysia, South Africa, Costa Rica, etc that sit empty because a state-backed investor is speculating that this area will someday be more valuable. No where was this more obvious than Malaysia which is their neighbor... but still crazy to see giant brand new office parks that look abandoned. To your point, it doesn’t at all surprise me about Nigeria because I saw it in ZA. China seems to have a world wide plan for the next century.
- yesforwhat 8y ago> China seems to have a world wide plan for the next century. If their government doesn't collapse first.
- mcv 8y agoXi's apparent desire to become president for life is certainly a liability, but other than that, China comes across as being quite stable.
- theandrewbailey 8y agoIt's the belt and road initiative. https://en.wikipedia.org/wiki/Belt_and_Road_Initiative https://en.wikipedia.org/wiki/Belt_and_Road_Initiative If it's not a government project, it could also be capital flight. I think it's one of the main drivers of real estate price expansion around the world. There's concern in China that things could collapse in the next 10 to 20 years. Many people seem to think that losing most of your real estate investment is better than losing ALL of your money. https://www.forbes.com/sites/insideasia/2017/02/22/china-capital-flight-migration/ https://www.forbes.com/sites/insideasia/2017/02/22/china-cap...
- 8y ago
- soonbesleeping 8y agoRelevant: https://youtu.be/VVJldn_MmMY?t=924 https://youtu.be/VVJldn_MmMY?t=924
- deleted 8y ago[deleted]
- TheRealDunkirk 8y ago"... countries with a long history of Internet surveillance." Pot, meet kettle. Seriously, how is this even worth a mention? The US invented wholesale internet surveillance.
- jonathanoliver 8y ago...and the internet.
- social_quotient 8y ago...and wholesale.
- S_A_P 8y agoThis is true, and Im not in support of wholesale surveillance with out reasonable cause and the associated court documents. That said, I don't think I would want to go to Russia or China and say, google "How to overthrow the government". I would expect someone would want to speak with me pretty quickly after I did that...
- vkazanov 8y agoI suspect that in every relatively developed country nowadays executing systematic requests of the kind you mentioned would get you some kind of personal attention. That's why they invented it, right? To make sure the political status quo does not change. That includes political systems in the US, UK, China and Russia.
- okmokmz 8y ago>That said, I don't think I would want to go to Russia or China and say, google "How to overthrow the government". I would expect someone would want to speak with me pretty quickly after I did that... That doesn't mean the US isn't performing the same level of surveillance/tracking, just that they aren't publicly picking up dissenters. In reality, if they were really worried, they'd hit you with a gag order and charge you in the FISA courts so you can't go public https://freedom.press/news/revealed-justice-depts-secret-rules-targeting-journalists-fisa-court-orders/ https://freedom.press/news/revealed-justice-depts-secret-rul...
- bdefore 8y agoGiven its prominence in the article 'BGP' should be defined.
- yesplorer 8y agonitpick but this caught my eye: TransTelecom (AS 20485) in Russia, China Telecom (AS 4809) in China and MainOne (AS 37282), a small ISP in Nigeria. I don't know what qualifies as big ISP but I am certain MainOne with a 14,000 kilometre submarine fibre optic cable may not be one to be classified as a small one. From wikipedia: The Main One Cable is a submarine communications cable stretching from Portugal to South Africa with landings along the route in various west African countries.
- dsfyu404ed 8y agoI get that they need a cool sounding title but they could have just said "BGP hijacking" and saved a click for everyone who knows what that is. We've seen this before and we'll see this again.
- i_cant_speel 8y agoThey don't want to save you a click.
- Quanttek 8y agoCan anyone explain to me what exactly it means that BGP routes "leaked"?
- RKearney 8y agoBy default, Cisco and Juniper routers (and likely others too, those are just the ones I can speak of) will advertise all bgp learned routes to all bgp peers unless you specify a route map/export policy. Leaking a route would imply you’re advertising the route to a peer that you’ve otherwise not intended to, either due to a misconfiguration or by not configuring things at all.
- solotronics 8y agomost likely these routers in Nigera were configured to send traffic destined to Google through China first ( a more specific /24 out of the Google prefix was configured -> China Telecom ASN ), through a route-map mistake this was advertisied out to the public internet instead of staying within the Nigerian ISP ASN.
- Geenirvana 8y agoFrom someone who does full understand BGP, I have a question Somewhere, a BGP route was misconfigured to send data somewhere else. What would happen if a BGP route was terminating at China, and the bad actor who made it happen, decided that they are not going to fix it and just leave it. How would the rest of the BGP network deal with it?
- manelvalls 8y agowell then
- dsr_ 8y agoWhile investigating the alarms, a network engineer at each major network will decide to stop taking routes from the Chinese network making the advertisements, and everything will sort itself out... as far as that network is concerned.
- Hikikomori 8y agoJust to clarify, it was a small Nigerian ISP that caused this, CT just propagated it to others. It's likely that this Nigerian ISP was setting up peering with Google and misconfigured their route policy. If you do this incorrectly you can advertise prefixes you get from one peer to other peers as if you own it. (essentially you replace the full AS path (prefix metadata in BGP) with your own AS number, it makes it looks like you originated the prefix to others) If neither the Nigerian ISP or CT refused to do anything then everyone that is their peer or customer would need to manually filter this "bad" prefix" to stop it. Customers and ISPs that only use CT would still be affected.
- betaby 8y agoSuch thing relatively easy to avoid https://www.youtube.com/watch?v=CSLpWBrHy10 https://www.youtube.com/watch?v=CSLpWBrHy10 Also problems like that are significantly more rare for IPv6.
- rdl 8y agoWhy isn’t there a “premium” commercial Internet, similar to DOD NIPRnet, for b2b stuff and for communications to vetted infrastructure? A random business on the Internet (eyeball, not servers) cares a lot more about routes to AWS and Google than to a bunch of other eyeballs, particularly in foreign countries. You would still want full connectivity, but when things went wrong I could lose connectivity to a Nigerian ISP without critical business risk, but losing access to Google sucks. You could largely accomplish this by prefixing the hell out of everything except the “special” connection, and ensuring DDoS and other security filtering could drop the entire normal network if needed without affecting internal or special.
- basilgohar 8y agoIf the extremely slippery slope of this idea is not immediately obvious to you, then I don't even know what to say. As succinctly as I can put it, this is exactly the opposite sentiment of Net Neutrality.
- kbirkeland 8y agoInternet2 basically provides this service for US research organizations.
- andrewflnr 8y agoWith two completely separate networks, the concept of net neutrality doesn't even apply.
- krallja 8y agoYou can purchase things like AWS Direct Connect.
- rdl 8y agoOnly really at colos. What scares me is businesses with key business partners, satellite offices, etc who just rely on the Internet working normally to communicate between them. If someone like L3 sold a premium network it would be worthwhile.
- furicane 8y agoSooo... paint me stupid but I find the title completely misleading and another, very, VERY annoying thing caught my eye - after reading the text, I caught myself repeating the word "thousandeyes" and it annoyed me to no end. Then I re-read the text and I found that you can kick that word out of every 2nd paragraph and the text would still have merit. Is this yet another marketing ploy where you post something with purposely-misleading title in order to attract traffic? I don't like the fact that word "thousandeyes" got stuck in my head, nor do I like the fact that I got clickbaited. This one is going to my list of "just like every other site since 2015, don't click".
- snek 8y ago> I find the title completely misleading Why? BGP has always been insecure and repeating that fact doesn't make it any less so.
- adminu 8y agoIt's called content marketing. Some do it better than others but it is always trying to attract attention to a company or service by blogging about related content.
- tumetab1 8y agoIt's SEO, they're always trying to produce content with the keywords on each paragraph.
- paulcole 8y agoRepeatedly including your company name, especially if it’s something unique like “thousandeyes” is essentially pointless when it comes to SEO. They own the domain thousandeyes.com. Google is going to recognize the intent for searches for thousandeyes are going to be about that company. Other keywords, yes, those are important for SEO. But repeating your unique company name, not so much.
- rv-de 8y ago> Sooo... paint me stupid why would somebody paint you stupid upon reading your totally spot-on observation? Even in case of not agreeing.
- fewiron9 8y agoIronic that the company reporting this has security problems themselves. I was trying to download their cloud research pdf via google and spammers have gotten hold of them with hundreds of thousands of fake links: https://www.google.com/search?q=site%3Athousandeyes.com+filetype%3Apdf+cloud https://www.google.com/search?q=site%3Athousandeyes.com+file...
- Raphmedia 8y agoHow much private information were foreign entities able to collect from this?
- hlecuanda 8y agoAfter glancing at the headlines, I actually did a double take. I was like "whaa? Taboola creeping into my HN" Given the audience, it wasn't the PR managers finest hour this week. What happens in BGP land is discussed publically on the NÀNOG mailing list, and they are the friendliest crowd ever. I've read them go out of their way to solve issues that just needed Goodwill to do in a couple of minutes, keeping the back channels open even between companies whose rivalry would dictate that they'd talk to each other only via their law firms. 1K@@ will probably be laughed at in the list
- amelius 8y agoWhy not change the title to something like "internet vulnerability blocked access to Google"?
- jayd16 8y agoIs this related to the GCP issues people were posting about this last weekend?
- doener 8y agoAn WSJ article about that: https://www.wsj.com/articles/google-internet-traffic-is-briefly-misdirected-through-russia-china-1542068392 https://www.wsj.com/articles/google-internet-traffic-is-brie...
- carapace 8y agoThis went by on HN a few days ago: "Strange snafu misroutes domestic US Internet traffic through China Telecom" > China Telecom, the large international communications carrier with close ties to the Chinese government, misdirected big chunks of Internet traffic through a roundabout path that threatened the security and integrity of data passing between various providers’ backbones for two and a half years, a security expert said Monday. It remained unclear if the highly circuitous paths were intentional hijackings of the Internet’s Border Gateway Protocol or were caused by accidental mishandling. https://arstechnica.com/information-technology/2018/11/strange-snafu-misroutes-domestic-us-internet-traffic-through-china-telecom/ https://arstechnica.com/information-technology/2018/11/stran... https://news.ycombinator.com/item?id=18403999 https://news.ycombinator.com/item?id=18403999
- i_phish_cats 8y agoNow that's how you do content marketing! Great original research and I definitely know Thousand Eye has great BGP tools :-P