9 ms·

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can m

It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smaller, less-connected network. The crazy thing about BGP is that any router can announce that it's responsible for a block of IP addresses and, if it's trusted enough, that's what the map of the Internet will reflect. The long term solution is for networks to implement and enforce RPKI. AT&T, for instance, implemented RPKI and we did not see any drop in traffic to their network today. Verizon not only didn't implement RPKI, which would be the best-of-breed approach, but also didn't do even basic route filtering. It's as if a trusted traffic cop (Verizon) overheard from a random passing motorist that the main road was closed and, as a result, directed all traffic off a pier and into the ocean. More about RPKI if you're interested: https://blog.cloudflare.com/rpki/ https://blog.cloudflare.com/rpki/

ziddoap 7y ago
Are you able to comment on how a company like yourself, or the other companies which were affected, can pursue anything with Verizon? Or is Verizon free to continue with bad practices and have a repeat of this issue?
- deleted 7y ago
  [deleted]
psim1 7y ago
The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.
- ethbro 7y ago
  I can't seem to muster much sympathy for any publicly-traded US ISP not performing technical due diligence. If they can afford to lobby against non-profit competition and for local monopolies, they should damn well be able to staff a NOC for this type of issue.
- xrisk 7y ago
  Blame is due where blame is due.
- ljm 7y ago
  I don’t think Verizon is a victim here. They’re big enough to have figured it out. They didn’t, so they’re being held to account.
  - crankylinuxuser 7y ago
    A better question: If they are a malicious/malfeasant actor, can non-Verizon ASNs partition Verizon off the internet until they fix their shit?
    - mschuster91 7y ago
      That's a dangerous precedent to set - because then others will jump in and demand knocking other countries (think Cuba, Venezuela, Iran, Syria or any other country on the US sanction list) offline. IIRC the only cases where this has happened was when a couple of self-proclaimed "bulletproof hosters" were booted off of their uplinks, but even this wasn't a direct partition of the Internet.
    - ecnahc515 7y ago
      That would just result in Verizon customers being unable to access stuff which isn't good either. Their users don't have much say in the matter, and due to internet monopolies in the US, may even be the only option for some people. They literally cannot even vote with their wallet, and thus any type of repercussion to Verizon would mostly be affecting the users.
- 7y ago
yingw787 7y ago
Very nice writeup on RPKI! I don't know anything about network engineering, but it appears that RPKI will distribute trust from ISPs to RIRs (Regional Internet Registries) like ARIN and RIPE. As I understand it, the RIR will sign your IP allocation with RPKI, which means fat-fingering on your side will result in the ISP not finding you as it takes BGP announcement and RIR confirmation for the ISP to acknowledge your IP. Again, very nice and understandable writeup :) I guess this does shift the burden of trust from an ISP to the RIR, and the blog post mentions international law as RIR and ISP memberships can be part of different countries and only RIRs would know who has what IP address since only they are TAs (which empowers certain governments over others). So I guess the debate is whether the pain of BGP route leaks and such is greater than the stress of another country having your RPKI entry. I guess we'll just have to see how badly Verizon messes up in the future.
- codebje 7y ago
  The IRR is also controlled by a few entities that would be vulnerable to government intervention, but that's the tool we currently rely on. RPKI roots trust at the RIRs, and that is a vulnerability, but any government intervention would end that trust and end the use of the RIRs as trust anchors. It's pretty unlikely to ever be used that way. Disclaimer: I co-authored some of the drafts for RPKI and helped implement RPKI systems at an RIR.
- toast0 7y ago
  RPKI uses CAs at the RIRs because the RIRs are who make the IP allocations and have a relationship with the IP holders and can (at least in theory) authenticate the holders. Just as a RIR could issue a certificate for your IPs to someone else, they could change WHOIS, which is how IP delegations are generally cross referenced. You're welcome to accept (or propagate) someone's advertisements without RPKI in case of some dispute with their RIR, but expect to get called out for it if the routes are bogus if you don't answer your NOC phone or email or twitters. Actually, I don't think Cloudflare was even calling Verizon out for not doing RPKI, which is fairly new and has costs, it was more for not limiting prefix counts; a small customer should probably be limited to 2n + 4 prefixes where N is the average number of prefixes they've advertised over the past 30 days; or like they have to put their prefixes in a portal or something. Filtering customer advertisements with IRRs is also pretty normal. But really, you gotta answer the phone. The steel guys answered the phone.
sofaofthedamned 7y ago
I do love that the CEO of Cloudflare is throwing technical shade at Verizon and others here and on Twitter for being useless.
- bogomipz 7y ago
  Do you really think that "throwing shade" is what the internet needs? Is "throwing shade" an admirable quality in someone who is supposed to be demonstrating leadership? Anyone who has worked as a network engineer for a major ISP knows the internet is quite brittle. During my entire time in that profession I can't remember a time when attempting to shame people was used to resolve a routing issue or to improve relations in order to resolve future routing issues.
  - FussyZeus 7y ago
    Shame is one of the most effective tools in influencing human behavior, and from the sounds of this post and the other coverage on the incident, Verizon has earned far more ire than is directed at them in this blog post. A lot of people seem to conflate speaking professionally with speaking like a doormat. Verizon, specifically the team in charge of this system, fucked up. There are varying levels to that of course; if you mess up the fonts in the end of month report to your super and he calls you a fucking idiot, he's probably an unbalanced person in need of mental help. If on the other hand you knock dead 15% of GLOBAL Internet traffic out of sheer laziness, I'd say you've earned more than a few 'go fuck yourself's.
    - bogomipz 7y ago
      How about leaking session tokens and other sensitive data for millions of people during "Cloudbleed"? Were you advocating public shaming for Cloudflare then? Was that also "sheer laziness" and did they earn "more than a few" of your "go fuck yourself's"?
      - deleted 7y ago
        [deleted]
kortilla 7y ago
Note that RPKI won’t prevent outages caused by route leaks (because the leak has a valid signed origin).
- minou 7y ago
  This is a great call out, BTW. The fact that the original AS Origin is included here makes this even more weaponized. Brings it back to why doesn't the Noction platform "dirty" the injected announcements. For example, throwing out some Private ASNs or ASNs of "tier 1" providers to prevent those announcements from ever getting propagated around.
ilaksh 7y ago
It's critical for the internet to work. Actually, some types of emergency services rely on it. So this level of negligence is dangerous. Shouldn't there be criminal charges? Or at least some kind of legal action.
bogomipz 7y ago
Your comments indicate to me that you don't really understand RPKI beyond a marketing value because if you did you wouldn't be throwing it around as some silver bullet. RPKI although a step in the right direction is also susceptible to misconfiguration and attack by hostile entities[1]. Additionally outages as bad as any BGP misconfiguration are also possible if an RIR's RPKI repo becomes unavailable. This has already happened. See: https://www.arin.net/vault/announcements/2018/20181024_update.html https://www.arin.net/vault/announcements/2018/20181024_updat... and https://www.ripe.net/support/service-announcements/service-announcements/ripe-ncc-rpki-repository-outage https://www.ripe.net/support/service-announcements/service-a... There are also issues such a broken legacy ROAS: https://blog.apnic.net/2018/10/16/cleaning-up-roas-inconsistent-with-the-bgp-state/ https://blog.apnic.net/2018/10/16/cleaning-up-roas-inconsist... And the list goes on. Please stop with the hype and hand waving. [1] https://www.cs.bu.edu/~goldbe/papers/sigRPKI_full.pdf https://www.cs.bu.edu/~goldbe/papers/sigRPKI_full.pdf

ziddoap 7y ago

Are you able to comment on how a company like yourself, or the other companies which were affected, can pursue anything with Verizon? Or is Verizon free to continue with bad practices and have a repeat of this issue?

deleted 7y ago
[deleted]

psim1 7y ago

The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.

ethbro 7y ago
I can't seem to muster much sympathy for any publicly-traded US ISP not performing technical due diligence. If they can afford to lobby against non-profit competition and for local monopolies, they should damn well be able to staff a NOC for this type of issue.
xrisk 7y ago
Blame is due where blame is due.
ljm 7y ago
I don’t think Verizon is a victim here. They’re big enough to have figured it out. They didn’t, so they’re being held to account.
- crankylinuxuser 7y ago
  A better question: If they are a malicious/malfeasant actor, can non-Verizon ASNs partition Verizon off the internet until they fix their shit?
  - mschuster91 7y ago
    That's a dangerous precedent to set - because then others will jump in and demand knocking other countries (think Cuba, Venezuela, Iran, Syria or any other country on the US sanction list) offline. IIRC the only cases where this has happened was when a couple of self-proclaimed "bulletproof hosters" were booted off of their uplinks, but even this wasn't a direct partition of the Internet.
  - ecnahc515 7y ago
    That would just result in Verizon customers being unable to access stuff which isn't good either. Their users don't have much say in the matter, and due to internet monopolies in the US, may even be the only option for some people. They literally cannot even vote with their wallet, and thus any type of repercussion to Verizon would mostly be affecting the users.
7y ago

yingw787 7y ago

Very nice writeup on RPKI! I don't know anything about network engineering, but it appears that RPKI will distribute trust from ISPs to RIRs (Regional Internet Registries) like ARIN and RIPE. As I understand it, the RIR will sign your IP allocation with RPKI, which means fat-fingering on your side will result in the ISP not finding you as it takes BGP announcement and RIR confirmation for the ISP to acknowledge your IP. Again, very nice and understandable writeup :) I guess this does shift the burden of trust from an ISP to the RIR, and the blog post mentions international law as RIR and ISP memberships can be part of different countries and only RIRs would know who has what IP address since only they are TAs (which empowers certain governments over others). So I guess the debate is whether the pain of BGP route leaks and such is greater than the stress of another country having your RPKI entry. I guess we'll just have to see how badly Verizon messes up in the future.

codebje 7y ago
The IRR is also controlled by a few entities that would be vulnerable to government intervention, but that's the tool we currently rely on. RPKI roots trust at the RIRs, and that is a vulnerability, but any government intervention would end that trust and end the use of the RIRs as trust anchors. It's pretty unlikely to ever be used that way. Disclaimer: I co-authored some of the drafts for RPKI and helped implement RPKI systems at an RIR.
toast0 7y ago
RPKI uses CAs at the RIRs because the RIRs are who make the IP allocations and have a relationship with the IP holders and can (at least in theory) authenticate the holders. Just as a RIR could issue a certificate for your IPs to someone else, they could change WHOIS, which is how IP delegations are generally cross referenced. You're welcome to accept (or propagate) someone's advertisements without RPKI in case of some dispute with their RIR, but expect to get called out for it if the routes are bogus if you don't answer your NOC phone or email or twitters. Actually, I don't think Cloudflare was even calling Verizon out for not doing RPKI, which is fairly new and has costs, it was more for not limiting prefix counts; a small customer should probably be limited to 2n + 4 prefixes where N is the average number of prefixes they've advertised over the past 30 days; or like they have to put their prefixes in a portal or something. Filtering customer advertisements with IRRs is also pretty normal. But really, you gotta answer the phone. The steel guys answered the phone.

sofaofthedamned 7y ago

I do love that the CEO of Cloudflare is throwing technical shade at Verizon and others here and on Twitter for being useless.

bogomipz 7y ago
Do you really think that "throwing shade" is what the internet needs? Is "throwing shade" an admirable quality in someone who is supposed to be demonstrating leadership? Anyone who has worked as a network engineer for a major ISP knows the internet is quite brittle. During my entire time in that profession I can't remember a time when attempting to shame people was used to resolve a routing issue or to improve relations in order to resolve future routing issues.
- FussyZeus 7y ago
  Shame is one of the most effective tools in influencing human behavior, and from the sounds of this post and the other coverage on the incident, Verizon has earned far more ire than is directed at them in this blog post. A lot of people seem to conflate speaking professionally with speaking like a doormat. Verizon, specifically the team in charge of this system, fucked up. There are varying levels to that of course; if you mess up the fonts in the end of month report to your super and he calls you a fucking idiot, he's probably an unbalanced person in need of mental help. If on the other hand you knock dead 15% of GLOBAL Internet traffic out of sheer laziness, I'd say you've earned more than a few 'go fuck yourself's.
  - bogomipz 7y ago
    How about leaking session tokens and other sensitive data for millions of people during "Cloudbleed"? Were you advocating public shaming for Cloudflare then? Was that also "sheer laziness" and did they earn "more than a few" of your "go fuck yourself's"?
    - deleted 7y ago
      [deleted]

kortilla 7y ago

Note that RPKI won’t prevent outages caused by route leaks (because the leak has a valid signed origin).

minou 7y ago
This is a great call out, BTW. The fact that the original AS Origin is included here makes this even more weaponized. Brings it back to why doesn't the Noction platform "dirty" the injected announcements. For example, throwing out some Private ASNs or ASNs of "tier 1" providers to prevent those announcements from ever getting propagated around.

ilaksh 7y ago

It's critical for the internet to work. Actually, some types of emergency services rely on it. So this level of negligence is dangerous. Shouldn't there be criminal charges? Or at least some kind of legal action.

bogomipz 7y ago

Your comments indicate to me that you don't really understand RPKI beyond a marketing value because if you did you wouldn't be throwing it around as some silver bullet. RPKI although a step in the right direction is also susceptible to misconfiguration and attack by hostile entities[1]. Additionally outages as bad as any BGP misconfiguration are also possible if an RIR's RPKI repo becomes unavailable. This has already happened. See: https://www.arin.net/vault/announcements/2018/20181024_update.html https://www.arin.net/vault/announcements/2018/20181024_updat... and https://www.ripe.net/support/service-announcements/service-announcements/ripe-ncc-rpki-repository-outage https://www.ripe.net/support/service-announcements/service-a... There are also issues such a broken legacy ROAS: https://blog.apnic.net/2018/10/16/cleaning-up-roas-inconsistent-with-the-bgp-state/ https://blog.apnic.net/2018/10/16/cleaning-up-roas-inconsist... And the list goes on. Please stop with the hype and hand waving. [1] https://www.cs.bu.edu/~goldbe/papers/sigRPKI_full.pdf https://www.cs.bu.edu/~goldbe/papers/sigRPKI_full.pdf