7 ms·

Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as a

by jawns 7y ago

Regarding those really aggressive claims, I was a bit shocked by that as well. Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.

anonbutler 7y ago
Tom is based in London. So he had a good night's sleep and was well rested.
- btown 7y ago
  We don’t know that he wasn’t up late fixing another bug that we thankfully never saw, and that his life hasn’t been like a season of 24 this past day.
  - LoSboccacc 7y ago
    we don't know either if he's an alien coming from a planet that has a shorter day and his circadian cycle is being disrupted being on earth, but Occam would suggest no.
  - deleted 7y ago
    [deleted]
floatingatoll 7y ago
There’s no excuse for the personal slander here - “his beauty rest interrupted” is a mean-spirited, inappropriate way to discuss Cloudflare’s post. Please don’t insult people unnecessarily here.
- pwodhouse 7y ago
  Likewise, don't insinuate people are ugly or don't deserve sleep.
- arkades 7y ago
  Uh, I think the idiom just means “got woken up early.” I don’t see an attack there.
- harry8 7y ago
  I read it as a reasonable and slightly humorous way to acknowledge that the author is a human being and may have been (justifiably?) in a bad mood writing the post because lack of sleep does that. No idea of how sensible that suggestion is or isn't in this case but didn't really think it was derogatory. A human being is not a machine. Sanity-sleep might be a better term for it but nobody uses that whereas beauty-sleep is in common, albeit ironic, usage. Let's all take our sleep seriously, yeah? :-)
bogomipz 7y ago
>"Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them" Indeed. And that's not going to help them or their customer's in the least the next time they need Verizon's cooperation to resolve an issue. You would never see this type of behavior on the NANOG mailing list which has been on the front line of communications between ISPs and providers for BGP issues since the beginning of the commercial internet. It is very much a "community" with reciprocal respect and professionalism, things this blog post was devoid of.
- dogecoinbase 7y ago
  Cloudflare's bet is essentially that they can control so much of the internet infrastructure that they can behave however they like and we all simply have to deal with it.
  - owenmarshall 7y ago
    > behave however they like and we all simply have to deal with it So basically what Verizon did by looking at BCP194 and saying “nah, too much bother”??
    - KirinDave 7y ago
      Not really. You don't have to be a massive player to screw things up with BGP.
      - owenmarshall 7y ago
        No doubt. But it’s 2019 and I can’t muster up much sympathy for a tier 1 who can’t get inbound filters and a responsive NOC implemented correctly - things which were table stakes in 2009.
        gwright 7y ago
        2009? You are being generous. I'm pretty sure when I was managing BGP announcements for my small ISP in 1999 route filtering was a thing.
jgrahamc 7y ago
The sequence of events went a bit like this: Team in London started working the problem and called in reinforcements from elsewhere; Upper management (me and one other person) got involved as it was serious/not resolved fast; I spoke with the network team in London who seemed to have a good handle on the problem and how they were working to resolve but we decided to wake a couple of other smart folks up to make sure we had all the best people on it; Problem got resolved by the diligence of an engineer in London getting through to and talking with DQE; Some people went back to bed; Tom worked on writing our internal incident report so that details were captured fast and people had visibility. He then volunteered to be point on writing the public blog (1415 UTC); Folks in California woke up and got involved with the blog. Ton of people contributed to it from around the world with Tom fielding all the changes and ideas; Very senior people at Cloudflare (including legal) signed off and we posted (1958 UTC). No one had an axe to grind with Verizon. We were working a complex problem affecting a good chunk of our traffic and customers. Everyone was calm and collected and thoughtful throughout. Shout out to the Support team who handled an additional 1,000 support requests during the incident!
- lima 7y ago
  Thank you! CloudFlare's response is appropriate. The incident itself and lack of response (for HOURS) from Verizon's side is absolutely unacceptable. It's 2019, filtering ALL of your customer's routes according to - at least - the IRR (including the legacy ones connected to the old router in the closet) and having a responsive 24/7 NOC contact in PeeringDB are a matter of course. Proper carriers like NTT go above and beyond simple IRR filtering nowadays with things like peerlock (http://instituut.net/~job/peerlock_manual.pdf http://instituut.net/~job/peerlock_manual.pdf). AT&T uses RPKI and was completely unaffected: https://twitter.com/Jerome_UZ/status/1143276134907305984 https://twitter.com/Jerome_UZ/status/1143276134907305984
  - ksec 7y ago
    So basically we should avoid all Verizon Enterprise Product. EdgeCast used to be my favourite CDN. Not sure how they are doing now.
smsm42 7y ago
If Cloudflare is correct on the technical part (no idea, I don't have enough knowledge to evaluate) then they are completely within their rights to call out Verizon on not doing the right thing. It's not "aggressive" at all. They don't mean harm to Verizon, they just point out Verizon is not doing their job correctly, and should step up and fix it. And not having somebody to talk to for 8hr for a provider than is capable of downing significant part of the internet is also something worth calling out. It's not some mom-and-pop shop that the owner can just hang a sign on the door and go to the beach, it's a major provider that should always have open channels to communicate on things like this.

by jawns 7y ago

anonbutler 7y ago

Tom is based in London. So he had a good night's sleep and was well rested.

btown 7y ago
We don’t know that he wasn’t up late fixing another bug that we thankfully never saw, and that his life hasn’t been like a season of 24 this past day.
- LoSboccacc 7y ago
  we don't know either if he's an alien coming from a planet that has a shorter day and his circadian cycle is being disrupted being on earth, but Occam would suggest no.
- deleted 7y ago
  [deleted]

floatingatoll 7y ago

There’s no excuse for the personal slander here - “his beauty rest interrupted” is a mean-spirited, inappropriate way to discuss Cloudflare’s post. Please don’t insult people unnecessarily here.

pwodhouse 7y ago
Likewise, don't insinuate people are ugly or don't deserve sleep.
arkades 7y ago
Uh, I think the idiom just means “got woken up early.” I don’t see an attack there.
harry8 7y ago
I read it as a reasonable and slightly humorous way to acknowledge that the author is a human being and may have been (justifiably?) in a bad mood writing the post because lack of sleep does that. No idea of how sensible that suggestion is or isn't in this case but didn't really think it was derogatory. A human being is not a machine. Sanity-sleep might be a better term for it but nobody uses that whereas beauty-sleep is in common, albeit ironic, usage. Let's all take our sleep seriously, yeah? :-)

bogomipz 7y ago

>"Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them" Indeed. And that's not going to help them or their customer's in the least the next time they need Verizon's cooperation to resolve an issue. You would never see this type of behavior on the NANOG mailing list which has been on the front line of communications between ISPs and providers for BGP issues since the beginning of the commercial internet. It is very much a "community" with reciprocal respect and professionalism, things this blog post was devoid of.

dogecoinbase 7y ago
Cloudflare's bet is essentially that they can control so much of the internet infrastructure that they can behave however they like and we all simply have to deal with it.
- owenmarshall 7y ago
  > behave however they like and we all simply have to deal with it So basically what Verizon did by looking at BCP194 and saying “nah, too much bother”??
  - KirinDave 7y ago
    Not really. You don't have to be a massive player to screw things up with BGP.
    - owenmarshall 7y ago
      No doubt. But it’s 2019 and I can’t muster up much sympathy for a tier 1 who can’t get inbound filters and a responsive NOC implemented correctly - things which were table stakes in 2009.
      - gwright 7y ago
        2009? You are being generous. I'm pretty sure when I was managing BGP announcements for my small ISP in 1999 route filtering was a thing.

jgrahamc 7y ago

The sequence of events went a bit like this: Team in London started working the problem and called in reinforcements from elsewhere; Upper management (me and one other person) got involved as it was serious/not resolved fast; I spoke with the network team in London who seemed to have a good handle on the problem and how they were working to resolve but we decided to wake a couple of other smart folks up to make sure we had all the best people on it; Problem got resolved by the diligence of an engineer in London getting through to and talking with DQE; Some people went back to bed; Tom worked on writing our internal incident report so that details were captured fast and people had visibility. He then volunteered to be point on writing the public blog (1415 UTC); Folks in California woke up and got involved with the blog. Ton of people contributed to it from around the world with Tom fielding all the changes and ideas; Very senior people at Cloudflare (including legal) signed off and we posted (1958 UTC). No one had an axe to grind with Verizon. We were working a complex problem affecting a good chunk of our traffic and customers. Everyone was calm and collected and thoughtful throughout. Shout out to the Support team who handled an additional 1,000 support requests during the incident!

lima 7y ago
Thank you! CloudFlare's response is appropriate. The incident itself and lack of response (for HOURS) from Verizon's side is absolutely unacceptable. It's 2019, filtering ALL of your customer's routes according to - at least - the IRR (including the legacy ones connected to the old router in the closet) and having a responsive 24/7 NOC contact in PeeringDB are a matter of course. Proper carriers like NTT go above and beyond simple IRR filtering nowadays with things like peerlock (http://instituut.net/~job/peerlock_manual.pdf http://instituut.net/~job/peerlock_manual.pdf). AT&T uses RPKI and was completely unaffected: https://twitter.com/Jerome_UZ/status/1143276134907305984 https://twitter.com/Jerome_UZ/status/1143276134907305984
- ksec 7y ago
  So basically we should avoid all Verizon Enterprise Product. EdgeCast used to be my favourite CDN. Not sure how they are doing now.

smsm42 7y ago

If Cloudflare is correct on the technical part (no idea, I don't have enough knowledge to evaluate) then they are completely within their rights to call out Verizon on not doing the right thing. It's not "aggressive" at all. They don't mean harm to Verizon, they just point out Verizon is not doing their job correctly, and should step up and fix it. And not having somebody to talk to for 8hr for a provider than is capable of downing significant part of the internet is also something worth calling out. It's not some mom-and-pop shop that the owner can just hang a sign on the door and go to the beach, it's a major provider that should always have open channels to communicate on things like this.