6 ms·
BGP super-blunder: How Verizon sparked a ‘cascading catastrophic failure’
- mtmail 7y agoRelated HN discussion about the incident from June https://news.ycombinator.com/item?id=20267790 https://news.ycombinator.com/item?id=20267790
- Causality1 7y agoOdd how often Verizon's name comes up in these periodic stories about mass mis-routing of internet traffic.
- coldcode 7y agoSince they fail to filter these out, perhaps some enterprising person could route the entire internet through someone's closet ISP. Maybe, just maybe, they will then care.
- toast0 7y agoPart of this is Verizon's poor practices, and part of it is Verizon is a major transit provider; if a routing leak affects a large amount of the internet, it's almost certainly because a major transit provider accepted the announcement and propagated it.
- walrus01 7y agoAmong the many ASes of things they've acquired, Verizon is AS701 / UUNet, which is historically one of the largest and most widespread ASes on the planet. I still think of it as uunet.
- salawat 7y agoSteeling internet. Neat.
- mc32 7y agoForging the internet?
- deleted 7y ago[deleted]
- davidkuhta 7y agoMetaling with the internet!
- binarymax 7y agoOh the irony.
- captainbland 7y agoAt least they tried to prepare for this situation. The software they were using was very close to the metal.
- lapnitnelav 7y agoAh, yeah the good old Internet factory : https://twitter.com/cryptonomatron/status/1008397182213353472?lang=en https://twitter.com/cryptonomatron/status/100839718221335347...
- Isamu 7y agoI was interested to find out which steel mill, since they are almost non-existent in Pittsburgh these days, but this is about Allegheny Technologies, headquartered downtown, which owns a number of mills and specialty metals businesses scattered around. There is some steel produced around here still but it's nothing like the old days. The gigantic J&L plant was still in operation when we drove my brother to Carnegie Mellon.
- steveklabnik 7y agoI was also interested to find out which one, given that I'm from Pittsburgh. Turns out that that this is what Allegheny Ludlum is called these days; they pay my grandpa's pension, and several other relatives worked for them for a very long time as well. And today, I work for Cloudflare, though not on any of the stuff that was involved here. It's a small world.
- santaclaus 7y ago> I was interested to find out which steel mill, since they are almost non-existent in Pittsburgh these days As evidenced by the smell of sulphur yesterday morning the coke plants down in Clairton are still alive and well!
- donalhunt 7y agonot the first time and won't be the last time...
- mamoswine 7y agoIt's possible that the Chinese are involved in this with their hacking. The Chinese Communist Party has their spy tentacles in who knows where and I bet the steel mill IT guys haven't figured it out.
- ToFab123 7y agoIf you were the sys admin at that stell mill and you come into the office on that day. What would be different?
- justusthane 7y agoYou'd essentially be DDoSed. The internet wouldn't work, and your firewall logs would show that you're being flooded with inbound traffic. The firewall would probably be dropping the unsolicited inbound traffic, so internally you shouldn't see any impact. Unless as part of the misconfiguration that caused this, the firewall also thinks it knows how to route traffic to the affected prefixes, in which case it would be accepting the traffic and routing it, in which case segments of the internal LANs could be flooded too.
- icedchai 7y agoMore than likely this wouldn't even hit their firewall. BGP misconfigurations generally occur on the router(s), in front of their firewalls, that connect to the upstream providers. Packets come in one connection and go out the other because you're now the shortest path from A to B.
- deleted 7y ago[deleted]
- appleshore 7y agoThanks, this is why I’ll never know if I actually fixed the WiFi. My girlfriend says I haven’t.
- Domenic_S 7y agoTangentially related to Verizon: I lost Verizon cell and cell data service (along with my cable internet through a different provider) this weekend when a construction crew accidentally cut a fiber line. I had not realized that literally all modern communications were flowing through a single point of failure. It was a surreal experience.
- WrtCdEvrydy 7y agoYeah, redundant lines are expensive. Even multiple lines to the same DSLAM were considered expensive back in the day... so if you sliced a single line, you're SOL.
- neuronflux 7y agoBGP filters fix this. Only accept routes from customers for IP space that they control. I imagine Verizon has some sort of webUI with bad defaults (no filter) so that their helpdesk can setup new customers - Hanlon's razor being what it is.
- icedchai 7y agoI remember setting up a T1 in the mid 90's. Our upstream was InternetMCI. There was no Web UI. There was no documentation of any sort. The guy (MCI tech) just had me read him routes over the phone and he announced them. Fun times!
- yusyusyus 7y agoeasier explanation is that they were doing troubleshooting and had removed the filter to do that.
- icedchai 7y agoIn the early days, some upstream ISPs did no filtering. You could announce anything.
- OrgNet 7y agoso, who's going to get fined?