14 ms·

The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have sq

by psim1 7y ago

The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.

ethbro 7y ago
I can't seem to muster much sympathy for any publicly-traded US ISP not performing technical due diligence. If they can afford to lobby against non-profit competition and for local monopolies, they should damn well be able to staff a NOC for this type of issue.
xrisk 7y ago
Blame is due where blame is due.
ljm 7y ago
I don’t think Verizon is a victim here. They’re big enough to have figured it out. They didn’t, so they’re being held to account.
- crankylinuxuser 7y ago
  A better question: If they are a malicious/malfeasant actor, can non-Verizon ASNs partition Verizon off the internet until they fix their shit?
  - mschuster91 7y ago
    That's a dangerous precedent to set - because then others will jump in and demand knocking other countries (think Cuba, Venezuela, Iran, Syria or any other country on the US sanction list) offline. IIRC the only cases where this has happened was when a couple of self-proclaimed "bulletproof hosters" were booted off of their uplinks, but even this wasn't a direct partition of the Internet.
  - ecnahc515 7y ago
    That would just result in Verizon customers being unable to access stuff which isn't good either. Their users don't have much say in the matter, and due to internet monopolies in the US, may even be the only option for some people. They literally cannot even vote with their wallet, and thus any type of repercussion to Verizon would mostly be affecting the users.
youeeeeeediot 7y ago
This problem has been known for decades. As of April 2019, 56.1% of the world's population has internet access. Do you think it is acceptable for a major transit ISP to have no basic filters in 2019 let alone implement RPKI? Can you explain what part of the Cloudflare statement you consider to be posturing? A cursory review of the BGP announcements referenced in the article are pretty clear. Facts are facts regardless of how the message is delivered.
ikiris 7y ago
What the hell do you think cloudflare was supposed to have done here?
laughinghan 7y ago
I was thinking the exact same thing—right up until I got the part where they still haven't responded 8 hours later (to say nothing of apologizing), and played no role in fixing the problem (DQE did that, apparently). We all make mistakes. It's unreasonable to expect 100% uptime from anyone. But if you operate a service that so many people are relying on, and you make billions of dollars in profit each year (we're not talking about an unpaid volunteer open-source maintainer here), you absolutely have a responsibility to at least try to help fix it when there's a problem. It's brazenly irresponsible to go radio silent while your customer's other vendor fixes the problem.
ubercow13 7y ago
I think it makes them look pretty fine to very good
txcwpalpha 7y ago
I 100% agree with you, though I am unsurprised that HN is downvoting you. HN seems to revere Cloudflare bigtime despite the fact that Cloudflare often uses HN as their own corporate PR platform. I absolutely loathe Verizon and I'll be the first to line up for a good publish lashing of US ISPs, but even I feel like this blog post is unnecessarily unprofessional. What strikes me the most is that this whole "event" would have hardly even registered on anyone's radar (it affected less than 10% of their traffic during early hours of the morning. I saw one single news article about it, buried on The Verge, but other than that nothing), except for the fact that Cloudflare's CTO was on HN this morning fanning the flames of the one thread about it. It's like they dug their own hole drawing attention to the "Cloudflare outtage" headline, and now they're overcompensating by going to drastic measures to blame someone else. And now they keep harping on the fact that Verizon still hasn't responded? Sure, part of that is probably the fact that Verizon is a giant corporation that doesn't want to bother with this stuff, but the other part is that this "event" was hardly even big enough of a deal to register on VZ's PR team's radar, no matter how much CF whines about it. This blog post (and the accompanying HN comments from Cloudflare execs) just scream "immature company" to me. There's a reason that Cloudflare is the one making this blog post and devoting CEO time to it while the established behemoth is just going about their business as usual.
- SpicyLemonZest 7y ago
  The context (which isn't obvious, and I don't blame you for not knowing it) is that the Internet is held together by spit and duct tape. The only reason it works at all is that major participants are good actors, in the sense that: 1. They implement basic precautions to prevent dumb things from going wrong. 2. They're available 24/7, to immediately respond to and remediate whatever does go wrong. 3. Both of the above are core obligations, which supersede any questions of public relations or maturity or higher-ups not wanting to be bothered. If Verizon can't be trusted to properly operate their network, that's an immediate threat to the health of the Internet, and many people do need to be made aware of it. It's not just Cloudflare being salty because their customers yelled at them.
  - txcwpalpha 7y ago
sgentle 7y ago
I think your definition of unprofessional needs recalibrating if it applies more to the people calling out professional negligence than to the people committing it.

by psim1 7y ago

ethbro 7y ago

I can't seem to muster much sympathy for any publicly-traded US ISP not performing technical due diligence. If they can afford to lobby against non-profit competition and for local monopolies, they should damn well be able to staff a NOC for this type of issue.

xrisk 7y ago

Blame is due where blame is due.

ljm 7y ago

I don’t think Verizon is a victim here. They’re big enough to have figured it out. They didn’t, so they’re being held to account.

crankylinuxuser 7y ago
A better question: If they are a malicious/malfeasant actor, can non-Verizon ASNs partition Verizon off the internet until they fix their shit?
- mschuster91 7y ago
  That's a dangerous precedent to set - because then others will jump in and demand knocking other countries (think Cuba, Venezuela, Iran, Syria or any other country on the US sanction list) offline. IIRC the only cases where this has happened was when a couple of self-proclaimed "bulletproof hosters" were booted off of their uplinks, but even this wasn't a direct partition of the Internet.
- ecnahc515 7y ago
  That would just result in Verizon customers being unable to access stuff which isn't good either. Their users don't have much say in the matter, and due to internet monopolies in the US, may even be the only option for some people. They literally cannot even vote with their wallet, and thus any type of repercussion to Verizon would mostly be affecting the users.

youeeeeeediot 7y ago

This problem has been known for decades. As of April 2019, 56.1% of the world's population has internet access. Do you think it is acceptable for a major transit ISP to have no basic filters in 2019 let alone implement RPKI? Can you explain what part of the Cloudflare statement you consider to be posturing? A cursory review of the BGP announcements referenced in the article are pretty clear. Facts are facts regardless of how the message is delivered.

ikiris 7y ago

What the hell do you think cloudflare was supposed to have done here?

laughinghan 7y ago

I was thinking the exact same thing—right up until I got the part where they still haven't responded 8 hours later (to say nothing of apologizing), and played no role in fixing the problem (DQE did that, apparently). We all make mistakes. It's unreasonable to expect 100% uptime from anyone. But if you operate a service that so many people are relying on, and you make billions of dollars in profit each year (we're not talking about an unpaid volunteer open-source maintainer here), you absolutely have a responsibility to at least try to help fix it when there's a problem. It's brazenly irresponsible to go radio silent while your customer's other vendor fixes the problem.

ubercow13 7y ago

I think it makes them look pretty fine to very good

txcwpalpha 7y ago

I 100% agree with you, though I am unsurprised that HN is downvoting you. HN seems to revere Cloudflare bigtime despite the fact that Cloudflare often uses HN as their own corporate PR platform. I absolutely loathe Verizon and I'll be the first to line up for a good publish lashing of US ISPs, but even I feel like this blog post is unnecessarily unprofessional. What strikes me the most is that this whole "event" would have hardly even registered on anyone's radar (it affected less than 10% of their traffic during early hours of the morning. I saw one single news article about it, buried on The Verge, but other than that nothing), except for the fact that Cloudflare's CTO was on HN this morning fanning the flames of the one thread about it. It's like they dug their own hole drawing attention to the "Cloudflare outtage" headline, and now they're overcompensating by going to drastic measures to blame someone else. And now they keep harping on the fact that Verizon still hasn't responded? Sure, part of that is probably the fact that Verizon is a giant corporation that doesn't want to bother with this stuff, but the other part is that this "event" was hardly even big enough of a deal to register on VZ's PR team's radar, no matter how much CF whines about it. This blog post (and the accompanying HN comments from Cloudflare execs) just scream "immature company" to me. There's a reason that Cloudflare is the one making this blog post and devoting CEO time to it while the established behemoth is just going about their business as usual.

SpicyLemonZest 7y ago
The context (which isn't obvious, and I don't blame you for not knowing it) is that the Internet is held together by spit and duct tape. The only reason it works at all is that major participants are good actors, in the sense that: 1. They implement basic precautions to prevent dumb things from going wrong. 2. They're available 24/7, to immediately respond to and remediate whatever does go wrong. 3. Both of the above are core obligations, which supersede any questions of public relations or maturity or higher-ups not wanting to be bothered. If Verizon can't be trusted to properly operate their network, that's an immediate threat to the health of the Internet, and many people do need to be made aware of it. It's not just Cloudflare being salty because their customers yelled at them.
- txcwpalpha 7y ago

sgentle 7y ago

I think your definition of unprofessional needs recalibrating if it applies more to the people calling out professional negligence than to the people committing it.