19 ms·
Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline
- dang 7y agohttps://news.ycombinator.com/item?id=20262214 https://news.ycombinator.com/item?id=20262214 is the earlier thread on this.
- jgrahamc 7y agoThanks. And thanks for the help today updating the title on the original post as the real cause came to light!
- eropple 7y agoLong ago, in the mists of time when I was a wee lad, the internet was a simpler place. There were seven buttons around the world, all pressed down by volunteers. If any four of them were released, the world would end. “The world” was defined as “the internet”, and at the time that meant “the world” was defined as “men with beards and suspenders and real opinions about Star Trek”, and so that wasn’t so bad. Today in 2019, “the world” is defined as “you know, the world“, and there are seven million buttons being held down all over the world. If any four of them are released, the world ends. We have made mistakes, is what I’m saying. (I once had call to explain to nontechnical people how and why the internet is the way it is and why my ops crews tend to be full of people who are a little too calm about things being constantly on fire. This was my best crack at it.)
- deleted 7y ago[deleted]
- ethbro 7y agoYou came up with this analogy when Lost was still airing, no? Although BGP probably makes about the same amount of sense to most people.
- csours 7y agohttps://www.icann.org/news/blog/the-problem-with-the-seven-keys https://www.icann.org/news/blog/the-problem-with-the-seven-k...
- ethbro 7y agoHoisting my pitchfork a bit, but the internet might be better off without hierarchical DNS. I certainly wouldn't call that "the world ending."
- bin0 7y agoSo we run into the age-old problem of "who decides". Also, how do we prevent fragmentation when there is disagreement.
- ethbro 7y agoFreedom isn't free. Web of trust. Inconvenient, but that's a price I'm willing to pay for a network that empowers users rather than commercial interests.
- bin0 7y agoHmm. My issue with that is a majority consensus could decide that, for instance, .gov domains should no longer be relegated to the American government. Same issue as with a crypto 51% takeover.
- vageli 7y ago> Hmm. My issue with that is a majority consensus could decide that, for instance, .gov domains should no longer be relegated to the American government. Same issue as with a crypto 51% takeover. I'm sure other countries would like to use .gov.
- 7y ago
- csours 7y agoThere's no need to fear, for seven men with keys, masters of the universe, well versed universal programming linguistically... and if there is a big blackout.. up to 6 of them can back out! https://youtu.be/Odw9Md9Lm6g?t=164 https://youtu.be/Odw9Md9Lm6g?t=164
- inopinatus 7y agoBack in that oft-forgotten age I was privileged to know, work, and chill with the brave volunteers (and, subsequently, paid RIR staff) holding down the buttons. It’s worth mentioning that even back then there dwelt in the west a large ugly troll whose name was AS701 (AS701 was not alone, either, having two hideous siblings, AS702 and AS703, that lived in other climes). Everyone tiptoed around the beast, because when angered it would flap its routes and there would be a great wailing and severe packet loss. The brave volunteers tried many times to tame the awful creature and I’m very sorry to see that it is still fucking everyone’s announcements even today.
- djhworld 7y agoVery interesting and easy to understand overview, thanks. I'm not familiar with this side of networking, but it sounds to me the "BGP Optimiser" product was left largely to its own devices and automated a configuration change without any explicit approval from a human operator (I could be wrong) With the protocol being prone to problems like leaky routes and sloppy peers accepting them, is it really wise to leave these BGP optimiser products running without some level of supervision? EDIT: of course I guess the human operator might wave the change through too without fully appreciating the problem...
- kazen44 7y agothe issue is that, as stated by the article. Things like this could be prevented by doing proper IRR filtering.
- linsomniac 7y agoI think the reality of it is that these BGP optimizers really can't be human checked. There's just too much it is doing, and for them to be really beneficial they need to respond quickly to network path congestion. I would be surprised if overseeing such a system could be done with fewer than 6 full time people, as a WAG. ... Which is why you should be really sure that these optimized routes never leak! And on top of it, Verizon should never have accepted those announcements.
- kossTKR 7y agoI still get a Cloudflare 1020 error here: http://shadow.tech http://shadow.tech (my location : Scandinavia ) Are these sites waiting for some kind of propagation or cache busting? It's a pretty large gaming service.
- judge2020 7y agoWorks from the ATL DC, what is the airport code that shows up on https://cloudflare-test.judge.sh/#shadow.tech https://cloudflare-test.judge.sh/#shadow.tech ? Might be a local [maybe routing] issue with CF -> shadow's web server.
- AnssiH 7y agoNot working here either (Finland), that page shows HEL for me. shadow.tech shows "Error 1020 Ray ID: 4ec1c24b2a945b25 • 2019-06-24 21:22:45 UTC Access denied What happened? This website is using a security service to protect itself from online attacks."
- judge2020 7y agoOh, 1020 access denied means some firewall rule (any combination of rules [0] and `if`/`or` statements) or access controls (such as blocking IP ranges and countries) blocked access to the website. These are always set up by the site operator, so this isn't caused by any CF issues or the earlier routing issues. I guess best course of action for those who want to access the site is to tweet them https://twitter.com/shadow_official https://twitter.com/shadow_official with your Ray ID. 0: https://developers.cloudflare.com/firewall/cf-firewall-rules/fields-and-expressions/ https://developers.cloudflare.com/firewall/cf-firewall-rules...
- deleted 7y ago[deleted]
- munificent 7y ago> All of the above suggestions are nicely condensed into MANRS (Mutually Agreed Norms for Routing Security) Whoever came up with that name and acronym deserves an award.
- mffnbs 7y agoWow, thanks for pointing that out, I missed it on my once-over.
- jepler 7y agoIs Verizon's engineer on vacation, or did he just call in sick today?
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- jboy55 7y agoHere's a shoutout to all the on-calls who woke up this morning to deal with "someone else's problem". I think everyone who woke up gets to, at least, order a "fancy coffee" and send the bill to Verizon.
- teejmya 7y agoAmen. We need a support group. "Hi, I'm Teejmya, and I was on call last night"
- eastdakota 7y agoYes, apologies and thank you! If you email me (matthewatcloudflaredotcom) your shirt size, preference for men's or women's cut, and your postal address with the subject line: "Verizon BGP Leak On Call Support Group" I'll send you a Cloudflare tshirt. Least we can do.
- nexuist 7y agoThis is so awesome.
- dreamcompiler 7y agoI think somebody should make some "I Fixed the Internet after Verizon Broke It. 20190624" T-shirts.
- IntelMiner 7y ago"Verizon broke the internet and all I got was this lousy T-shirt"
- hn20180220 7y agoI would need a bunch of them
- VectorLock 7y ago
- ziddoap 7y agoThank you for the summary. And, a sincere thank you for not mincing words when it comes to something as important as this. >However, against numerous best practices outlined below, Verizon’s lack of filtering turned this into a major incident that affected many Internet services such as Amazon, Fastly, Linode and Cloudflare. >IRR filtering would not have increased Verizon's costs or limited their service in any way. Again, the only explanation we can conceive of why it wasn't in place is sloppiness or laziness. In an attempt to find any statement given by Verizon, I found that The Register was able to get this amazing statement: "Verizon sent us the following baffling response to today's BGP cockup: "There was an intermittent disruption in internet service for some [Verizon] FiOS customers earlier this morning. Our engineers resolved the issue around 9am ET."" [1] [1]https://www.theregister.co.uk/2019/06/24/verizon_bgp_misconfiguration_cloudflare/ https://www.theregister.co.uk/2019/06/24/verizon_bgp_misconf...
- yingw787 7y agoVerizon's response seems very non-committal and it appears this type of incident may happen again if they don't take any action. Are there ways for companies like Google or Cloudflare to work around ISPs like Verizon without affecting ISP customers, or is this a blocker? Was the 10% of the re-routed traffic from Cloudflare 100% of the traffic from Verizon to Cloudflare?
- eastdakota 7y agoIt's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smaller, less-connected network. The crazy thing about BGP is that any router can announce that it's responsible for a block of IP addresses and, if it's trusted enough, that's what the map of the Internet will reflect. The long term solution is for networks to implement and enforce RPKI. AT&T, for instance, implemented RPKI and we did not see any drop in traffic to their network today. Verizon not only didn't implement RPKI, which would be the best-of-breed approach, but also didn't do even basic route filtering. It's as if a trusted traffic cop (Verizon) overheard from a random passing motorist that the main road was closed and, as a result, directed all traffic off a pier and into the ocean. More about RPKI if you're interested: https://blog.cloudflare.com/rpki/ https://blog.cloudflare.com/rpki/
- unethical_ban 7y agoI am surprised that CF is as aggressive toward Verizon in public as they are. Once you start breaking the Internet for stupid reasons, though, you probably deserve it. I know very little about BGP operations; I did not know that there was PKI and route validation like they described in the article.
- linsomniac 7y ago... Sometimes you just have to call a spade a spade. I know a little about BGP, having run it for a small provider in the distant past, and Verizon should have never been in a position to cause this.
- eastdakota 7y agoMore details on RPKI if you're interested: https://blog.cloudflare.com/rpki/ https://blog.cloudflare.com/rpki/
- fencepost 7y agoThey'd probably be less aggressive if they'd been able to reach anyone there or had received any response (none as of 8 hours after the incident). As noted above in this discussion the CF team thought they had appropriate contact information for all top tier carriers, and I suspect they do have what Verizon would call the appropriate contact info. Not much they can do if Verizon ghosts them, though. I guess they could take steps to null route everything to/from Verizon to see if they could get someone's attention that way.
- icedchai 7y agoAlmost nobody rejects invalid routes. RPKI is basically a research project given that 85% of routes don't even have ROAs. See https://rpki-monitor.antd.nist.gov/ https://rpki-monitor.antd.nist.gov/
- qaq 7y agoOne would think Cloudflare team would have a direct line of communication to all tier 1 Internet providers.
- eastdakota 7y agoWe thought we did. And tried both public and private lines of communication — without reply. Still waiting.
- m_rn 7y agoThank you all at Cloudflare for your attention to detail. I'm currently rethinking my stance of years believing you were crimeflare. I do hope Verizon responds and everyone can learn from this.
- symfoniq 7y agoUnbelievable. Thanks for the excellent write-up.
- dentemple 7y agoSo what's the over/under on Verizon actually picking up?
- avip 7y agoHats off to Verizon for treating everyone equally without discrimination.
- deleted 7y ago[deleted]
- ohithereyou 7y ago"We don't care. We don't have to. We're the phone company." - Verizon, probably It's good to know that Net Neutrality isn't dead.
- hype7 7y agoHaha hats off to you for a string of comments on this thread that have me lol’ing
- dreamcompiler 7y agoIt's clear that Verizon broke the Internet this morning through incompetent BGP management, and they could do it again. Who holds them accountable?
- Uehreka 7y agoglances over at Ajit Pai Nobody.
- driverdan 7y agoIt doesn't need government intervention. It needs other companies to hold them accountable.
- TheOperator 7y agoOh geez looks like the other companies aren't holding them accountable and the government shouldn't hold them accountable so I guess they just aren't accountable for causing broad swathes of economic damage because they were lazily managing their networks. The system works!
- taildrop 7y agoAbsolutely. If you are a Verizon service provider customer, please call them and register your feelings on this matter. Make sure you let them know in no uncertain terms that you are considering switching providers based on their lack of following best practices.
- lostcolony 7y agoI think that may be a little too subtle for the parent, but well done.
- ep103 7y agoAnd will do so, once another provider becomes available in your area.
- 7y ago
- camgunz 7y agoVerizon's lucky it's a blog post that doesn't mince words, rather than a lawsuit.
- clvx 7y agoCan they get a lawsuit?. Has Verizon broken their SLA?. Is there a manual to mitigate all the edge cases? What about being aware internally this had to be improved but it was delayed due bureaucracy.
- dentemple 7y agoAnybody can sue. And it'll still cost Verizon real money to fight those suits, even if they truly don't have liability.
- mjevans 7y agoTorturous influence or something maybe?
- MrStonedOne 7y ago>edge cases? This is not an edge case, allowing downstream networks to broadcast routes for networks they do not own is a very well known security and operational issue with operating an ISP. Massive parts of the internet went down in the 90s to teach us this lesson. Likewise, bureaucracy does not excuse not fixing an issue thats existed since the 90s, and not deploying any 1 of 3 mitigation tricks (let alone all 3). Negligence causing damage from lost sales/traffic is sue-able. The case would basically resolve around rather or not V had an obligation to prevent this from happening, and rather or not they were grossly negligent in that obligation. In my view the answer is yes.
- henryfjordan 7y agoWho has standing though? A Verizon customer might say "my internet was down" but there is 100% a clause in their contract about outages and SLAs. Any company that lost sales likely doesn't have a contract with them, so what are they going to sue for? "Verizon didn't carry my 1s and 0s for free this morning"? Person A on the freeway having an accident and causing Person B to sit in traffic and miss their sales meeting isn't liable for that... Maybe their peers (other telcos) have more standing because they couldn't deliver to their customers as a result but they of course all have a clause in their contracts about outages and SLAs that means ultimately they lost no money so there are no damages. And this is why we need government regulation, either to break up the Telcos or nationalize them
- foota 7y agoAmusing callout to pager duty in the screenshot of the call-log :)
- cabaalis 7y agoWhich is more interesting, philosophically? The internet had a problem due to a single issue, or that the internet's problem was fixed due to a single person calling various people on a cell phone? (device interactions versus human interactions..)
- woliveirajr 7y ago> One of our network engineers made contact with DQE Communications quickly and after a little delay they were able to put us in contact with someone who could fix the problem. DQE worked with us on the phone to stop advertising these “optimized” routes to Allegheny Technologies Inc. We're grateful for their help. Once this was done, the Internet stabilized, and things went back to normal. It's funny how we have to still use phone to help fixing some internet routing problem, even if phone doesn't literally means the old black curly-wire equipment
- azernik 7y agoGotta have a channel that's out-of-band with the internet to fix problems with the internet.
- jedberg 7y agoNowadays with voip “the phone” isn’t as out of band as we’d like.
- heywire 7y agoIs it time to put an HF ham radio rig in each major provider’s office? I shudder thinking of a major outage where even phone communication can’t take place. I’m only half joking. Edit: and maybe we just give Verizon a toy walkie talkie
- kortex 7y agoYou joke, but cascade failure ain't no laughing matter, either. Emergency plans aren't for fair weather, they are for sh!tstorms. I don't want to say "we are too reliant on the internet" because it's cliche and connectivity is just part of growing the modern world. But we sure as heck need several layers of backup plans in case things go sideways. I, for one, hope there is a secret society of HAMs lurking as mild-mannered employees at every telco and ISP, ready to wire things back together when they short out.
- Scoundreller 7y agoI appreciate how sympathetic Cloudflare is to the root-cause party because they answered the phone and undid what they shouldn’t have done. (If my understanding is correct, they shouldn’t have told Verizon about the better routing, while Verizon should have known better)
- laughinghan 7y agoI don't think your understanding is correct. I think they're supposed to be allowed to "optimize" the BGP routes that they advertise to their customer (Allegheny). I'm unclear on whether Allegheny should have relayed that advertisement to Verizon, but it's clear that Verizon should definitely not have then broadcasted that to everyone else.
- saxonww 7y agoI read this as Allegheny's fault, actually. DQE published to Allegheny (DQE's customer), who in turn re-published to Verizon (Allegheny's other provider). While most of the 'prevention' section talks about what Verizon didn't do, it doesn't seem to mention that Allegheny should not have re-published the DQE-published routes up to Verizon. It's startling that Verizon doesn't appear to have any leak mitigations in place, but I feel like Allegheny is getting a pass here because they are small, or something.
- steveklabnik 7y agoAllegheny is a steel company. I don’t think most people expect them to have the same responsibility for internet health as Verizon, even though they are a $4B company. (I’m from Pittsburgh, my grandfather and a bunch of my relatives worked for this company for decades. I’ve been kinda giggling about this intersection of my past and my present all day. I don’t work in the parts of Cloudflare that deal with this kind of thing; I’m glad my co-workers were on top of it.)
- Scoundreller 7y agoI had to do a double take because they seem to be selling themselves as a technology company nowadays. I guess “steel” in your name isn’t good for share prices.
- cesarb 7y ago> For example, our own IPv4 route 104.20.0.0/20 was turned into 104.20.0.0/21 and 104.20.8.0/21. [...] The prefixes Cloudflare announces are signed for a maximum size of 20. RPKI then indicates any more-specific prefix should not be accepted, no matter what the path is. Did RPKI help reduce the scope of this incident, by stopping propagation of these faulty routes earlier than otherwise? Or did it have no effect in this case?
- symfoniq 7y agoAnecdotal, but: The article notes that AT&T has implemented RPKI, and a client mentioned to me that he wasn't having problems accessing Cloudflare-hosted infrastructure via his AT&T phone. The rest of his employees were having major issues though via the municipal fiber service provider.
- eastdakota 7y agoYup. There was no notable impact to AT&T traffic because they rejected the routes because they're filtering based on RPKI. Here's a Tweet from Cloudflare's Head of Network showing the AT&T vs. Verizon graphs: https://twitter.com/Jerome_UZ/status/1143276134907305984 https://twitter.com/Jerome_UZ/status/1143276134907305984
- betaby 7y agoIt didn't. Virtually no major operator rejects invalids. However some do more-or-less strict prefix+ASN filtering.
- icedchai 7y agoI remember the early days of the Internet, when I could log in to an ISP router running BGP, with a blazing fast T1 to an early tier 1 Internet provider. We could literally announce any route we wanted, no filtering. We used to regularly black hole spammers, then turn them back on an hour or two later.
- nikisweeting 7y agoPlease write about your life / these times! Or link me to your blog/writing if you already have. <3
- icedchai 7y agoIt's not terribly interesting. No blog, sorry.
- hyperion2010 7y agoOver a decade ago one of my friends was banned from the Sheffield Uni network for playing around with BGP and knocking the whole campus offline. One kind of has to wonder whether Verizon can suffer the same consequences simply by collective action on the part of other affected parties.
- azernik 7y agoNope - because said collective action would probably involve denying service to tens of millions of Verizon customers.
- MrStonedOne 7y agoThey could do better
- henryfjordan 7y agoCan they? How many choices of ISP do you have? I have 2, Spectrum (Time Warner) or some no-name for the same price with 1/10th the speed. I could not practically switch off Time Warner if I wanted to without moving. I imagine that Verizon customers are in the same boat.
- flatiron 7y agoI work from home in NJ and I knew something was screwy this morning. I wish there was a place I could have checked to see it was this issue. I rebooted pretty much everything in my house.
- BonesJustice 7y agoI’m in NJ and had issues this morning as well. Figured it was a local outage and promptly forgot about it after I left for work. I never expected it was something so serious. Crazy.
- dentemple 7y agoSo how many phone calls do you think Allegheny Technologies or that ISP in Pennsylvania received this morning?
- avip 7y agoCloudFlare continues to raise the bar in terms of communicating technical issues to the public. Thank you for yet another enlightening writeup.
- ggg2 7y agoThey automated a system that relies solely on trust? ...China hackers are surely having a great day.
- VectorLock 7y ago"Why is PagerDuty calling me before I've had my coffee?" Call declined.
- avip 7y agoWell, you just lost yourself a T-shirt Sir. https://news.ycombinator.com/item?id=20269076 https://news.ycombinator.com/item?id=20269076
- himangshuj 7y agohttps://blog.codinghorror.com/coding-for-violent-psychopaths/ https://blog.codinghorror.com/coding-for-violent-psychopaths...
- kevinreedy 7y ago> The RPKI framework that we implemented and deployed globally last year is designed to prevent this type of leak. It enables filtering on origin network and prefix size. The prefixes Cloudflare announces are signed for a maximum size of 20. RPKI then indicates any more-specific prefix should not be accepted, no matter what the path is. Does RPKI prevent Cloudflare from announcing additional /22 routes during an incident like this? Any network with RPKI implemented would reject the /22s, but those who ignore it should pick them up over the leaked /21s.
- ikiris 7y agoAnd you believe the internet optimizer wouldn't have added /23s and /24s.... why?
- eastdakota 7y agoWe could break our prefixes into smaller routes, but 1) the Internet's routers have limited memory; 2) we have a lot of routes; and 3) we want to be good Internet citizens. If every network announced all their routes as /24s — the smallest route generally accepted over the public Internet — the routing table would be a giant mess and would overwhelm many routers' ability to store them. That said, after today we are thinking about ways that, in case of an emergency, we could break the routes down to be more specific than whatever is leaking. Given how broadly peered we are, Cloudflare's network will be as protected as anyone's. However, that's not really a good solution for the Internet generally. Better that we all implement and enforce RPKI.
- lima 7y agoKudos for not deaggregating routes into /24s like many other major ISPs do nowadays.
- hn20180220 7y agoKudos for a CEO that understands in and outs of Internet routing, making me want to join CF's neteng team
- antihero 7y agoWould this have affected stuff in the UK? All sorts of sites like MealPal were inaccessible this morning for a bit
- ethbro 7y agoI wouldn't be surprised if various other transit providers slurped the bogus routes up from Verizon (without filtering), so it's certainly possible.
- qwerty456127 7y agoEvery time I see the BGP abbreviation it's about a huge fuck-up. Either somebody hijacks routes intentionally or something like this happens.
- avip 7y agoAs any other critical underlying infrastructure of our lives, it's taken for granted and ignored until it breaks.
- adambyrtek 7y agoNot trying to defend a legacy protocol, but it's one of these cases where basic infrastructure is not newsworthy when it works just fine, like with plumbing.
- MBCook 7y agoCan someone explain why the optimizer would split one route into two? Wouldn’t it be more optimized to coalesce routes whenever possible?
- teraflop 7y agoIf an ISP has multiple physical connections that it could use to reach Cloudflare's network, it makes sense to distribute the traffic that's addressed to different IPs across different links, instead of using a single route that sends all the traffic over one link and leaves the others idle.
- MBCook 7y agoGood point. Thanks.
- nodesocket 7y agoI saw a few of my static sites that are hosted on Cloudflare and few other auxiliary 3rd party services I use flapping back and forth on PagerDuty, but not all of my Cloudflare sites triggered down. Is this just because the unaffected Cloudflare sites were not within the CIDR range affected?
- tssva 7y agoAS701 was the UUNET/Worldcom AS for the US and eventually the US/Canada network. Verizon bought Worldcom in 2006 and they became Verizon Business. From the late 90s to early 2000s I worked for UUNET/Worldcom as an engineer in the network planning and design group. I worked in the international group but among other things we were responsible for the build out of AS701 into Canada, the exchange sites where AS701 connected to the various other international UUNET AS's and the PoP's where dedicated circuits for international customers who wished to connect directly to AS701 would be terminated. The point being that I am familiar with how AS701 was operated at that time. UUNET's reputation at the time might not have been sterling due to the business decision to basically be a safe haven for spammers but from a technical standpoint the network was operated at a high standard. The basic BGP filtering referenced in the article was certainly in place at the time and if this had happened then heads would have rolled.
- jedberg 7y agoThey most likely upgraded the equipment since the build out and never put the filtering back.
- w8vY7ER 7y agoAs they should, this is day-one BGP configuration stuff. Nothing advanced, nothing especially technical or time consuming. Speaks to a gross degree of professional negligence on the part of AS701. Really disappointing to see this degree of ignorance.
- jrockway 7y agoNetworking infrastructure / tools are in the stone age compared to what we have in the software world. I am inclined to cut them some slack.
- ikiris 7y agoI for one remember UUNET being expensive transit, but rock fucking solid.
- OBLIQUE_PILLAR 7y agoI hope we remember this thread the next time Shady Russians or Crafty Chinese make a BGP mistake and are accused of Active Measures.
- xen2xen1 7y agoThis is an obvious mistake made by a steel company. The BGP "errors" that make tons of traffic go to Russia or China OFTEN WORK COMPLETELY. Some have worked so well no one noticed for months. That's far, far different than an error, that a large section of a country's traffic being successfully hijacked to another country. Accidents usually break things. What Russia and China are too successful to be accidents.
- londons_explore 7y agoCloudflare managed to get an in-depth blog post, one which has incident details, points blame to other parties, and makes some really quite aggressive (for corporate blog posts) claims, all during an incident, and they did all that in 8 hours. I'm impressed. At most other similar size companies, this would take 4 days. And in something like Amazon, it would be 2 weeks of approvals, editing, and review before a watered down version with all specifics removed is published.
- jawns 7y agoRegarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.
- anonbutler 7y agoTom is based in London. So he had a good night's sleep and was well rested.
- btown 7y agoWe don’t know that he wasn’t up late fixing another bug that we thankfully never saw, and that his life hasn’t been like a season of 24 this past day.
- LoSboccacc 7y agowe don't know either if he's an alien coming from a planet that has a shorter day and his circadian cycle is being disrupted being on earth, but Occam would suggest no.
- deleted 7y ago[deleted]
- floatingatoll 7y ago
- wil421 7y ago>It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place. I see you haven’t had much experience with large Telcos. They are all like this.
- empath75 7y agoMy favorite outage when I worked for a voip company was when one of our tech support people told a new customer that she needed to ‘add our ip address to your router’, meaning add it to the firewall whitelist, but she repeated that verbatim to the telco tech who misunderstood and then escalated her way up the chain at a major telco until some engineer with the wrong rights said ‘fuck it’ and updated bgp to route all of our traffic down her T-1 line. That was a fun conference call, and listening to the lady on the phone I could see how the engineer got to that point.
- kadira 7y ago"Engineer"
- bogomipz 7y agoFrom the post: >"It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place." Is "sloppiness or laziness" really the only possible attribution here? I'm not a big fan of Verizon but I'm a big fan of civility and empathy, two qualities which your blog post lacks. Outages are a really unfortunate fact of life. We've seen them recently with Google, AWS, Dyn - all companies where technical competency is generally not questioned. It's quite possible the cause of of this outage was some "perfect storm" scenario such as an eBGP router rebooted and came up with a stale or incorrect config. "Perfect storm" scenarios even happen at companies with very rigorous engineering cultures as we saw with the most recent Google outage. Your attempt to shame an organization without knowing all the details reeks of immaturity and pettiness. Ditto for your willingness to turn this into yet another Cloudflare marketing opportunity. Have you forgotten about your own Cloudbleed incident? How would you feel if it a security company took that as an opportunity to shame you for "sloppiness or laziness"? Or some other company's CEO was offering to send people "Cloudbleed Support Group" T-Shirts on HN as your own CEO is doing in this thread? Lastly RPKI isn't a silver bullet, RPKI authorities can also be misconfigured and attacked[1][2]. This happened with the LACNIC incident in 2013[2]. It's also worth mentioning that RPKI potentially creates new threats[2]. But again it seems more important to you to use this as a marketing opportunity and promote yourself while throwing someone else under a bus while uttering pithy summations. Also from your post: >"And, in particular, we're looking at you Verizon — and still waiting on your reply." Although Verizon is the 400lb gorilla in the room, their NOC and network engineers are still regular people with kids and families and feelings. They are also people who have had a really shit day today. Why you can't extend just a bit of human compassion and feel compelled to try to shame is quite inexplicable. You may think that your blog post was a marketing coup but I see it as a massive failure in in both leadership and civility. As a thought exercise maybe Cloudflare leadership could think about how they would like the community to react the next time they are at fault. [1] https://www.cs.bu.edu/~goldbe/papers/hotRPKI.pdf https://www.cs.bu.edu/~goldbe/papers/hotRPKI.pdf [2] https://www.cs.bu.edu/~goldbe/papers/sigRPKI_full.pdf https://www.cs.bu.edu/~goldbe/papers/sigRPKI_full.pdf
- MrStonedOne 7y agoIt was gross negligence that leads to this. Nothing more.
- ralph84 7y agoNothing is ever new in IT. A similar incident happened 22 years ago. Buggy and/or misconfigured gear disaggregates and reannounces routes and creates a blackhole. https://en.wikipedia.org/wiki/AS_7007_incident https://en.wikipedia.org/wiki/AS_7007_incident
- InTheArena 7y agoI don’’t think Cloudflare is going to get any business from Verizon anytime soon. This may be as a professional of a “Hey Verizon, you don’t know what the fsck you are doing” as I’ve seen.
- nikisweeting 7y agoThey're both big enough that they probably can't live without each other, which makes for an interesting relationship because they can probably swear at each other all day long and nothing will come of it.
- stackzero 7y agoI don't fully understand all the networking protocols involved but who/what is responsible to manage this kind of failure in the network? How should this ideally be handled besides cloudflare engineers calling someone?
- kosmet 7y agoThank you Verizon for giving me a recent example to talk about while teaching BGP in Computer Networks class. My favorite example will still be Pakistan routing all YouTube traffic to itself while trying to restrict access to it. It is a little bit old though: https://www.cnet.com/news/how-pakistan-knocked-youtube-offline-and-how-to-make-sure-it-never-happens-again/ https://www.cnet.com/news/how-pakistan-knocked-youtube-offli... http://web.mit.edu/6.02/www/s2012/handouts/youtube-pt.pdf http://web.mit.edu/6.02/www/s2012/handouts/youtube-pt.pdf
- stunt 7y agoWhat are the concerns around RPKI? I'd like to know why some companies aren't implementing it if it solves similar problems. What kind of criticism does it receive?
- mvkel 7y agoHas Verizon responded in any capacity yet?