11 ms·
Why Google Went Offline Today and a Bit about How the Internet Works
- hayksaakian 14y agoIt really makes one wonder about the fragility of the internet.
- precisioncoder 14y agoI would say the resilience is what impresses me here. The fact that it's decentralized means that anyone can fix the internet. The fact that this one specific problem was fixed within 26 min by individuals realizing the problem and acting to fix it gives me a warm feeling.
- kami8845 14y agoI think what you mean is that anyone can break the internet (in this case a random ISP from Indonesia) and that in that case only very specific people could fix it (probably at least a senior network engineer at said ISP).
- sp332 14y agoOnly specific routers that you trust (or are trusted by routers you trust) can break your internet. You can fix your internet by un-trusting those routers.
- deleted 14y ago[deleted]
- killermonkeys 14y agoWhy wouldn't PCCW preventing its customers from publishing routes outside its whitelist work? It has been a long time since I worked on BGP but that was common practice from back haul carriers to ISPs even at that point (2003). Given the same back haul provider has allowed this twice, it seems like a reasonable ask.
- jauer 14y agoSome carriers are lazy. There may also be politics involved in making national carriers "ask" for permission to advertise routes.
- stingraycharles 14y agoFor what it's worth, this is quite a vulnerability in the internet's routing system. It's also the reason Youtube went offline after Pakistan was deliberately announcing the wrong routes a few years ago because it didn't agree with some videos being broadcasted by Youtube. http://www.ripe.net/internet-coordination/news/industry-developments/youtube-hijacking-a-ripe-ncc-ris-case-study http://www.ripe.net/internet-coordination/news/industry-deve... http://news.cnet.com/8301-10784_3-9878655-7.html http://news.cnet.com/8301-10784_3-9878655-7.html
- highace 14y agoThis worries me. Am I right in saying a malicious party could actually take down the internet with this?
- sanxiyn 14y agoMalicious party? It already happened by accident. http://www.renesys.com/blog/2009/02/longer-is-not-better.shtml http://www.renesys.com/blog/2009/02/longer-is-not-better.sht...
- davej 14y agoThey would have to be a 'trusted' malicious party. But in theory, yes. I'm sure it would be reverted very quickly though.
- sp332 14y agoYes, but you'd have to con a lot of big players into trusting your BGP routes first. And the effect would only last as long as it took to change some configurations and write you back out of the internet.
- lini 14y agoAnd nothing will change. At least not until someone does this with malicious intent - script kiddie A knocks out big site, or a censoring state decides that it should block a free speech site from the entire Internet.
- forgotusername 14y agoEvil routing has been employed a whole bunch of times going back decades, most visibly a couple of years ago when IIRC Iran (?) started advertising bad routes for a bunch of big sites, including Google
- pilom 14y agoA much more useful thing to do than take out a big site is use BGP to create your own "section of the internet" to do your malicious deeds from, then afterwards remove the BGP routes and your addresses will no longer exist on the internet. So it will be as if you sent packets from a phantom network.
- dsl 14y agoThats why many people have been archiving every advertised route going back to the mid-90's. http://archive.routeviews.org/oix-route-views/ http://archive.routeviews.org/oix-route-views/ http://www.pch.net/resources/data.php?dir=/routing-tables http://www.pch.net/resources/data.php?dir=/routing-tables (link appears to be temporarily broken)
- stephen_g 14y ago
- jhull 14y agoCouldn't a rogue government easily take down the internet this way? Seems like if one guy in Indonesia can take out Google by accident, a government entity could do the same.
- sudhirj 14y agoThe moment people realize that the rogue network was being malicious, they'd stop trusting it - ignoring all announcements it might make. It might take a few hours for order to be restored, though.
- antidoh 14y agoEnough time for a purge or to gain an advantage in an invasion. This might even be on checklists for such actions.
- JohnLBevan 14y agoWould it be possible to claim to own Google's IP, then on receiving the packets intended for Google forward them on to the real IP (without accidentally forwarding them back to yourself)? That way someone could hijack & interrogate these packets without being spotted (at least without causing service outage / only adding slight delay). Alternatively could they route these requests to a clone as an advanced phishing scam?
- ErikD 14y agoThat's what https is for. It should prevent them from doing anything useful with the packets.
- duggan 14y ago“BGP is literally the glue of the Internet” - I think you’ll find BGP is figuratively the glue of the Internet ;)
- virmundi 14y agoIt depends on how one defines virtual glue. But, hey, the whole thing might be moot. http://xkcd.com/1108/ http://xkcd.com/1108/
- prof_hobart 14y agoYou do realise there's more than one meaning for "glue" these days, presumably (including "something that binds together")?
- jganetsk 14y agoThere's also more than meaning for "literally" these days.
- mpyne 14y agoThat only makes it more likely that what was said has a semantically correct meaning though (or is that what you meant?).
- bennyg 14y agoHe meant that there is literally only one definition for literally. Any other use of the word cuts away at its meaning (like what happened in that context).
- prof_hobart 14y agoNot really. It is literally something that binds the internet together (i.e. one of the common definitions of the word glue).
- 14y ago
- flannell 14y agoNot the first time, see here; http://www.theregister.co.uk/2010/04/09/china_bgp_interweb_snafu/ http://www.theregister.co.uk/2010/04/09/china_bgp_interweb_s...
- rlpb 14y ago> When I figured out the problem, I contacted a colleague at Moratel to let him know what was going on. He was able to fix the problem... I wonder how he contacted his colleague. In this case, I presume that routing to other networks were unaffected. But in the general case, with a future of everything over IP, what will network engineers use to communicate about faults?
- jvehent 14y agocellphones ?
- ojii 14y agoThere's a trend to use VoIP on cellphones too (see LTE). So in the future this will not help at all.
- JohnLBevan 14y agoIs there some way to say "ignore DNS results from this provider", such that were you to spot an issue you could block that provider's information (and anyone replicating their version of the truth) and thus find a valid path. If that were possible you wouldn't be reliant on a third party to resolve the issue to get your system working, and once your system worked, you could contact them to resolve the issues for all.
- ef4 14y agoIf someone is giving out bad DNS records, you can just choose to use a different DNS server. But in this case the problem was bad routes. You can certainly force your own routers to use fixed routes instead, but that doesn't help you unless everybody else along the path also does it. So it's not easy. There are tricks one can play -- like advertising your network as a set of smaller, more-specific networks (since routers will usually favor more-specific routes over more general ones).
- Ecio78 14y ago@JohnLBevan this was not a DNS problem, but a BGP problem NB Cannot reply under his post
- ybaumes 14y agoThe author (Tom Paseka) wrote near the conclusion that himself addressed the Google's issue, by contacting a Moratel's engineer. Do you have the same feeling when reading the article? It sounds weird that Google did not triggered a recovery procedure on its own. Maybe I see bad things everywhere and you may call paranoïd, but could it be some sort of ("false") advertising on the side of cloudfare?
- archangel_one 14y agoI'm not a network engineer, but it seems like the kind of thing that might be very hard to detect when you're already inside or near to the google.com domain. Or maybe CloudFlare just got there first. I don't think it's necessary to call BS on Cloudflare without any kind of evidence at all.
- emilw 14y agoStop excusing yourself for not being something, either read more and then comment or trust your gut feeling. (not meant to sound harsh)
- JohnLBevan 14y agoI think it's good to qualify your opinion with your level of expertise. There's no rule that says HN should only be for discussion by experts (hopefully there never will be), and if you don't know something for sure it's best to say so that others don't take your word as gospel. That said, I'm no expert ;).
- amalcon 14y agoThis is basically correct. BGP is weird. The addresses for one of Google's many datacenters were routed incorrectly for packets coming from some subset of IP space. Unless Google is running active ping tests to that subset of IP space, the way they would normally detect it is for someone to call and complain. In this case, the author decided to take a shortcut and call the owner of the "problem peer" directly.
- neya 14y agoBest explanation ever. Wow, seriously, this person can use the right words to help even the non-technical people understand such a complex situation. Thanks for posting this.
- jwr 14y agoI used to manage networks and wondered, while reading the article, why it gets so many points on HN, when it only states obvious things and doesn't really go into detail. And then I realized most people these days have no idea about how packets get from here to there, or even that there are packets at all. Now I understand the appeal, but I guess this means that good introductory material is badly needed.
- vr000m 14y agoThere is an IETF WG called SIDR, which is working on solving this problem of invalid BGP announcements. A good summary is available here http://isoc.org/wp/ietfjournal/?p=2438 http://isoc.org/wp/ietfjournal/?p=2438 and technical details are in the related proposals.
- danyork 14y agoYes, a very good group for people to get involved with if they are interested in this problem.
- yskchu 14y agoHaha, I was in HK today, and one of those hit, using PCCW services also
- trout 14y agoThere are some other ways to fix the problem. Last time with the Youtube problem, they advertised more specific routes. If Pakistan was advertising a /24 network (255 IP addresses) Youtube started advertising two /25 networks (2x 128 addresses). Since they are more specific, they are preferred over the more broad routes. This prevents lack of cooperation, but not malicious behavior. As well, it ends somewhere because many networks will not pass routes smaller than say /24 or /28. Most service providers also do 'inbound route filtering' to filter out any routes that they do not own. This isn't a simple process, which is why PCCW does not do it. Maybe a few more of these incidents and they will. There's also AS Path filtering. This allows networks to be more granular in which paths they trust, by inspecting which AS's a route has gone through. If certain AS or AS path combinations become problematic, the internet at large could blackhole them or do manual route filtering. This would be laborious, but possible. That said if someone can maliciously peer with an active BGP router, the damage to be done is significant. I haven't seen any outage reports from this type of attack, but I'm surprised by that.
- pilom 14y agoMuch more common that malicious outages is malicious creation of ghost networks. Basically a person could say over BGP "W.X.Y.Z is at my office" where that address isn't used by anyone anywhere else on the internet. Then they do their bad deeds from that made up address. Lastly they remove their route via BGP and it is as if their addresses never existed.
- sounds 14y agoThat might work for some unused /24's for a large organization's /8 block, but unused IPv4 addresses are so last year! I suppose the attack will still work for IPv6 for a long time.
- wmf 14y agoThere are a lot of IPv4 addresses that are assigned but not routed on the Internet, so you can easily "borrow" them. This kind of trick does leave a trace, though.
- clebio 14y agoSince I use DuckDuckGo for searches, I probably wouldn't notice this. Not receiving Gmail for a while wouldn't be noteworthy (at least for the first half hour or so). I'm confused about the times the author gives, though. The article is dated today (11/6) and he says this happened 'today' at 6:24pm PST / 02:24 UTC. But unless I'm mistaken, that is a time currently in the future (http://time.gov/timezone.cgi?Pacific/d/-8/java http://time.gov/timezone.cgi?Pacific/d/-8/java). I guess he meant yesterday?
- robk 14y agoYou're counting across the dateline, so for you it was 11/5.
- clebio 14y agoAm I? Not snark: if I'm misunderstanding this, I truly want to know. I'm in central US, CST, and the article gives PST. That conversion has always just been +2 hours.
- Cushman 14y agoAs I read it that was 18:24 yesterday in PST, or 02:24 today in UTC. The use of "today" may just be sloppy dating-- or it may reflect that it was today for most of those affected.
- clebio 14y agoMy guess was that he wrote the article yesterday, but didn't publish it until today. It's not a big deal, was just curious.
- eastdakota 14y agoThat's correct. Tom wrote the article yesterday (11/5) but I didn't review it and hit publish until today (11/6). Sorry for the confusion.
- louischatriot 14y agoVery interesting explanation, thank you.
- runn1ng 14y agoCan I ask a pretty newbie question - how is BGP connected to IP, TCP and DNS protocols? Is it sitting "below" them, "on top" of them, or is it somewhere else?
- ef4 14y agoBGP runs on top of TCP, which runs on top of IP. DNS runs on top of UDP (or sometimes TCP), which runs on top of IP. Edited to elaborate: most computers on the internet don't need to know anything about BGP. It's not directly involved when you establish connections. Think of it as an automatic configuration system running on the various routers.
- pilom 14y agoSaying BGP runs on top of IP is true but doesn't tell you what BGP does or how Global scale IP doesn't work without BGP. BGP is the protocol Internet Routers (i.e. not your home router) use to figure out how to route IP addresses to particular routers. So your home router connects to an internet router at comcast (or your ISP). The Comcast router announces to the rest of the world "Dear world, if you want to connect to any of the IP addresses at X.X.X.X sent those packets to me and I'll deal with them."
- jemfinch 14y agoFirst, TCP and DNS don't come into it: they both piggyback on IP (TCP directly; DNS via UDP in typical use), so IP is all that's really relevant. BGP is how routers communicate with each other. Every major edge router for a network is typically connected to many other edge routers for other networks. Each router announces what amounts to their complete routing table: i.e., for every IPv4/IPv6 address that they know how to route, they announce what networks it traverse on the way to the destination. When a router is deciding which router an IP packet should hop to next, it looks at the packet's destination IP address and consults an in-memory data structure that it has constructed based on the BGP announcements of the routers to which it's connected. Modulo refining nuances (MED/PREF), it looks for two things: 1. It routes the packet according to the most specific network it saw announced. If it sees a packet destined for 1.2.3.4, and one connected router A is announcing a route for 1.2.3.0/24, and another connected router B is announcing a route for 1.2.0.0/16, it will pass along the packet to router A, all other things being equal. 2. As a tiebreaker for announcements with the same network specificity, it looks at the "AS path": the set of networks that the packet will traverse. It picks the router with the shortest path: the least number of traversed networks. So the answer to your direct question is that BGP is "somewhere else": it's what routers use to communicate to each other "How will you route this IP packet?" and then make reasonable decisions about how they should send packets around the network.
- apaprocki 14y agoIf you're interested in peering (couldn't resist the pun) behind the curtain, read the NANOG[1] mailing list. These are the real guys keeping the Internet up and running :) [1]: http://mailman.nanog.org/pipermail/nanog/ http://mailman.nanog.org/pipermail/nanog/
- dsl 14y agoIt is worth noting that the average HN reader should probably subscribe read-only. Unless you have your own AS and enable on routers, you should probably call your ISP with any issues. (though an unfortunate number of people disregard this advice, which results in smaller private splinter mailing lists sigh)
- sneak 14y ago"I'm a network engineer at CloudFlare and I played a small part in helping ensure Google came back online." Uhh, no. Without the "ensure", then maybe.
- xxcode 14y agoTo be accurate, google didn't go down today -- your pathway from your computer to google got 'poisoned'. It wasn't Google's fault.
- dangoldin 14y agoThis is a great write up - thanks for posting. I'm slowly beginning to understand how the internet works day by day due to posts like this.
- henrymazza 14y ago> We use Google Apps for things like email so when we can't reach their servers Very professional way to do so!
- tomjen3 14y agoDoes this mean that in the future we should ignore all routes comming from PCCW (since they rebroadcast all rules without filtering)?
- wmf 14y agoThat's a good way to effectively disconnect yourself from the Internet. A lot of ISPs are not properly filtering.
- sunyc 14y agoalmost all bgp transit provider have prefix filtering,
- jamesinsf 14y agoGreat explanation. Good show and great job! Very smart engineers at Cloudflare!
- jemfinch 14y agoIsn't the title a touch sensationalist? Google did not go "offline": it was briefly unavailable for a relatively small number of networks.
- ColinWright 14y agoYou can't win. If you quote the title given, people complain. If you change it to something more accurate, the mods change it back, and then people complain anyway.
- jemfinch 14y agoYes; to be clear, I was complaining about the author's original title, not the submission title.
- noselasd 14y agoFor the people affected, google was for all purposes "offline". If the estimate of 3–5% of the internet population is correct, that's a lot of people.
- jemfinch 14y agoI think that if 3-5% of the Internet population suddenly stopped querying Google, Google would have noticed before CloudFlare.
- noselasd 14y agoThere weren't any claims that Google were unaware of this. And when things happen at this level, resolving it in 27 minutes can only be done when there's direct contact between people that are able to do anything about it.
- deleted 14y ago[deleted]
- ambiguator 14y agoI don't know much about the processes behind the Internet, but I found this to be a fascinating introduction.
- halayli 14y agoOutage in an ASN != Google went Offline. The title puts the blame on Google which isn't true.
- oldcreek 14y agoThe failure has nothing to do with BGP either.
- ninetax 14y agoWhile this does make sense if I abstract out what a BGP is, I wish I had a deeper knowledge of how the Internet works. Does anyone know of a book that goes from the basics of networking up to how it's all assembled on a large scale? A "big book of internet" if you will.
- ChuckMcM 14y agoAt some point anonymous is going to figure out the bgp 'hack' is actually exploitable, unlike taking the root name servers offline and we see a network routing outage for several days. I wish it wasn't so but sometimes that is the only way these things get fixed
- dsl 14y agoFirst of all to pull off this "hack" you need a router, an AS number, a transit contract with your upstream provider, BGP configured with said upstream, and most importantly your upstream needs to be negligent enough to not apply route filters to your session (which basically means I will only accept routes for IPs owned by company X over company X's session). Secondly, it is pretty easy to track down who is doing it. Assuming a rouge employee used their employers setup (see first point) to announce once of Google's routes and it managed to propagate, smart people at NOCs around the world start emailing and calling each other pretty quickly. Despite CloudFlare trying to take credit here, I'd put money on the fact the network in question received at least a dozen phone calls and emails. There are services like Renesys and BGPmon that "important" companies sign up for that will scream bloody murder and start paging people if someone unauthorized originates your prefixes. Third, as this is a known problem, a solution is already in the works and on its way to being implemented. Basically when you are assigned a block of IP addresses, you also get to publish a cryptographically signed statement of how and where that block should show up in the global routing table. See http://www.nanog.org/meetings/nanog49/presentations/Tuesday/bgp-origin-validation-FINAL.pdf http://www.nanog.org/meetings/nanog49/presentations/Tuesday/...
- guiambros 14y agoWell said, dsl. Almost two decades ago I used to run an ISP in another country, and remember that BGP was already reasonably safe at the time (when v4 started to be implemented), with peers normally rejecting route updates from blocks outside your control. Yes, there's always the risk of a trusted peer mistakenly leaking routes publicly (and a permissive upstream provider not rejecting it outright), but that's a low risk attack vector. I do remember this happening a few times, but were quickly spotted and corrected (true, the internet at the time was a lot smaller; you could probably fit all sysadmins of a country in a room..) I see this article as the CloudFlare guy trying to get credit for an act of civility that many other sysadmins likely have done, silently, in parallel. Of course I'm glad he did, but wouldn't expect anything less. That's just how the internet works. ps: thanks for the link. NANOG is something that I had long ago erased from my brain. Had a chuckle looking at the archives :)
- tcohen 14y agoI wish I understood more of this but still really cool!
- rdl 14y agoI am more curious what caused the 4 minute mid-day outage a few days ago. It wasn't BGP, since google.com was still up, but all personalization was down, and YouTube was down.
- lhnn 14y agoDAE think that the whole "BGP is broken!" argument is a bit overblown? If you're going to have a bunch of autonomous systems/networks operating together, with no central authority, it necessarily comes down to trust and relationships. Shit will occasionally happen. It's important to look at outages, figure out the cause, and work to prevent it. Perhaps, though, this is a best practices issue, and not some fundamental flaw in BGP.
- htf 14y agoYour comment reminds me of this quote from Linus Torvalds: "If you have ever done any security work – and it did not involve the concept of "network of trust" – it wasn't security work, it was – masturbation. I don't know what you were doing. But trust me, it's the only way you can do security, it's the only way you can do development."
- rahasia 14y ago27 minutes, 3-5% traffic, it could means thousand of dollars lost for Google, right? (Does it sueable?)
- jQueryIsAwesome 14y agoUnder which law are they going to sue? USA laws don't apply to Indians ISPs.
- bonchibuji 14y agoIt was an Indonesian ISP, not Indian.
- rurounijones 14y agoSidenote: In the comments I saw a reply about nanog.com being a great plce to meet other networking peeps. http://www.nanog.com/ http://www.nanog.com/ is currently showing a "Welcome to nginx" message
- bonobo 14y agoThat's because the right address is http://www.nanog.org http://www.nanog.org
- rurounijones 14y agoTa muchly.