9 ms·

Thank you for the summary. And, a sincere thank you for not mincing words when it comes to something as important as this. >However, against numerous best pra

Thank you for the summary. And, a sincere thank you for not mincing words when it comes to something as important as this. >However, against numerous best practices outlined below, Verizon’s lack of filtering turned this into a major incident that affected many Internet services such as Amazon, Fastly, Linode and Cloudflare. >IRR filtering would not have increased Verizon's costs or limited their service in any way. Again, the only explanation we can conceive of why it wasn't in place is sloppiness or laziness. In an attempt to find any statement given by Verizon, I found that The Register was able to get this amazing statement: "Verizon sent us the following baffling response to today's BGP cockup: "There was an intermittent disruption in internet service for some [Verizon] FiOS customers earlier this morning. Our engineers resolved the issue around 9am ET."" [1] [1]https://www.theregister.co.uk/2019/06/24/verizon_bgp_misconfiguration_cloudflare/ https://www.theregister.co.uk/2019/06/24/verizon_bgp_misconf...

yingw787 7y ago
Verizon's response seems very non-committal and it appears this type of incident may happen again if they don't take any action. Are there ways for companies like Google or Cloudflare to work around ISPs like Verizon without affecting ISP customers, or is this a blocker? Was the 10% of the re-routed traffic from Cloudflare 100% of the traffic from Verizon to Cloudflare?
- eastdakota 7y ago
  It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smaller, less-connected network. The crazy thing about BGP is that any router can announce that it's responsible for a block of IP addresses and, if it's trusted enough, that's what the map of the Internet will reflect. The long term solution is for networks to implement and enforce RPKI. AT&T, for instance, implemented RPKI and we did not see any drop in traffic to their network today. Verizon not only didn't implement RPKI, which would be the best-of-breed approach, but also didn't do even basic route filtering. It's as if a trusted traffic cop (Verizon) overheard from a random passing motorist that the main road was closed and, as a result, directed all traffic off a pier and into the ocean. More about RPKI if you're interested: https://blog.cloudflare.com/rpki/ https://blog.cloudflare.com/rpki/
  - ziddoap 7y ago
    Are you able to comment on how a company like yourself, or the other companies which were affected, can pursue anything with Verizon? Or is Verizon free to continue with bad practices and have a repeat of this issue?
    - deleted 7y ago
      [deleted]
  - psim1 7y ago
    The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.
    - ethbro 7y ago
      I can't seem to muster much sympathy for any publicly-traded US ISP not performing technical due diligence. If they can afford to lobby against non-profit competition and for local monopolies, they should damn well be able to staff a NOC for this type of issue.
    - xrisk 7y ago
      Blame is due where blame is due.
    - ljm 7y ago
      I don’t think Verizon is a victim here. They’re big enough to have figured it out. They didn’t, so they’re being held to account.
      - crankylinuxuser 7y ago
        A better question: If they are a malicious/malfeasant actor, can non-Verizon ASNs partition Verizon off the internet until they fix their shit?
- icedchai 7y ago
  The happens all the time to lesser degrees and is a fact of life on the Internet. No action will be taken.
throw7 7y ago
Back in the suspender wearing neckbeards days, the answer was simple... blackhole all Verizon routes.

yingw787 7y ago

Verizon's response seems very non-committal and it appears this type of incident may happen again if they don't take any action. Are there ways for companies like Google or Cloudflare to work around ISPs like Verizon without affecting ISP customers, or is this a blocker? Was the 10% of the re-routed traffic from Cloudflare 100% of the traffic from Verizon to Cloudflare?

eastdakota 7y ago
It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly). We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smaller, less-connected network. The crazy thing about BGP is that any router can announce that it's responsible for a block of IP addresses and, if it's trusted enough, that's what the map of the Internet will reflect. The long term solution is for networks to implement and enforce RPKI. AT&T, for instance, implemented RPKI and we did not see any drop in traffic to their network today. Verizon not only didn't implement RPKI, which would be the best-of-breed approach, but also didn't do even basic route filtering. It's as if a trusted traffic cop (Verizon) overheard from a random passing motorist that the main road was closed and, as a result, directed all traffic off a pier and into the ocean. More about RPKI if you're interested: https://blog.cloudflare.com/rpki/ https://blog.cloudflare.com/rpki/
- ziddoap 7y ago
  Are you able to comment on how a company like yourself, or the other companies which were affected, can pursue anything with Verizon? Or is Verizon free to continue with bad practices and have a repeat of this issue?
  - deleted 7y ago
    [deleted]
- psim1 7y ago
  The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.
  - ethbro 7y ago
    I can't seem to muster much sympathy for any publicly-traded US ISP not performing technical due diligence. If they can afford to lobby against non-profit competition and for local monopolies, they should damn well be able to staff a NOC for this type of issue.
  - xrisk 7y ago
    Blame is due where blame is due.
  - ljm 7y ago
    I don’t think Verizon is a victim here. They’re big enough to have figured it out. They didn’t, so they’re being held to account.
    - crankylinuxuser 7y ago
      A better question: If they are a malicious/malfeasant actor, can non-Verizon ASNs partition Verizon off the internet until they fix their shit?
icedchai 7y ago
The happens all the time to lesser degrees and is a fact of life on the Internet. No action will be taken.

throw7 7y ago

Back in the suspender wearing neckbeards days, the answer was simple... blackhole all Verizon routes.