5 ms·
From the post: >"It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness,
by bogomipz 7y ago
From the post:
>"It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place."
Is "sloppiness or laziness" really the only possible attribution here? I'm not a big fan of Verizon but I'm a big fan of civility and empathy, two qualities which your blog post lacks. Outages are a really unfortunate fact of life. We've seen them recently with Google, AWS, Dyn - all companies where technical competency is generally not questioned. It's quite possible the cause of of this outage was some "perfect storm" scenario such as an eBGP router rebooted and came up with a stale or incorrect config. "Perfect storm" scenarios even happen at companies with very rigorous engineering cultures as we saw with the most recent Google outage.
Your attempt to shame an organization without knowing all the details reeks of immaturity and pettiness. Ditto for your willingness to turn this into yet another Cloudflare marketing opportunity. Have you forgotten about your own Cloudbleed incident? How would you feel if it a security company took that as an opportunity to shame you for "sloppiness or laziness"? Or some other company's CEO was offering to send people "Cloudbleed Support Group" T-Shirts on HN as your own CEO is doing in this thread?
Lastly RPKI isn't a silver bullet, RPKI authorities can also be misconfigured and attacked[1][2]. This happened with the LACNIC incident in 2013[2]. It's also worth mentioning that RPKI potentially creates new threats[2]. But again it seems more important to you to use this as a marketing opportunity and promote yourself while throwing someone else under a bus while uttering pithy summations.
Also from your post:
>"And, in particular, we're looking at you Verizon — and still waiting on your reply."
Although Verizon is the 400lb gorilla in the room, their NOC and network engineers are still regular people with kids and families and feelings. They are also people who have had a really shit day today. Why you can't extend just a bit of human compassion and feel compelled to try to shame is quite inexplicable.
You may think that your blog post was a marketing coup but I see it as a massive failure in in both leadership and civility.
As a thought exercise maybe Cloudflare leadership could think about how they would like the community to react the next time they are at fault.
[1] https://www.cs.bu.edu/~goldbe/papers/hotRPKI.pdf https://www.cs.bu.edu/~goldbe/papers/hotRPKI.pdf
[2] https://www.cs.bu.edu/~goldbe/papers/sigRPKI_full.pdf https://www.cs.bu.edu/~goldbe/papers/sigRPKI_full.pdf
- MrStonedOne 7y agoIt was gross negligence that leads to this. Nothing more.
- bogomipz 7y agoDo you work for Verizon's NOC or Network Engineering department then? You have inside knowledge that it was negligence? Because I provided a specific scenario where it would not be "gross negligence."
- Cpoll 7y ago> Do you work for Verizon's NOC or Network Engineering department then If that's what it takes to be able to decide, then I guess we can safely declare that it wasn't negligence, because no-one that fits that description will ever publicly admit to it.
- wtallis 7y ago> Because I provided a specific scenario where it would not be "gross negligence." No, you didn't. You provided a vague conjecture for how the initial cause of the problem might not have been gross negligence, but offered no hypothesis for why Verizon isn't answering the Red Phone.
- bogomipz 7y agoYes I did. It's the part where I clearly state it's possible that a router that offline or rebooted came up with a stale or incorrect config. This actually happens occasionally. I've been on both ends of it. Clearly Verizon has inbound prefix filtering in place as this is not some common occurrence for AS 701. Verizon was in contact with people yesterday. I have spoken to two people from two other carriers who were in touch with them. And you are just parroting the idea that because Cloudflare didn't get a response that Verizon wasn't responding to anyone period. And that's just not true. The fact that you think there's some red phone that just anyone can call the NOC and magically speak to someone during a major outage shows you have no practical experience with thes things you are commenting on and criticizing.