11 ms·
Grave flaws in BGP Error handling
- lexicality 3y agoVery comforting to know that most of the core underpinning of the internet doesn't follow the robustness principle by default. Still, I guess it keeps support contracts active and network engineers in a perpetual hell of on call alerts
- greyface- 3y ago> I worry about the state of networking vendors. > With that being said, I would like to thank the OpenBSD security team, who very rapidly acknowledged my report, and prepared a patch. My only regret with dealing with the OpenBSD team was reporting the issue to them too quickly, as they are uninterested in waiting for any kind of coordination in disclosure. Lesson to network engineers: switch your routers to OpenBSD/OpenBGPd and keep up on your syspatches to escape this particular hell. ;)
- literalAardvark 3y agoThat just sounds like creating a new level of hell
- martijnvds 3y agoOne full of daemons.
- appplication 3y agoI could have sworn I saw someone on here a few weeks ago saying how they switched their router to openBSD and ended up in some horrible broken state after a power outage or something. I can’t recall the details, maybe someone else would recall.
- basedrum 3y agoMaybe because you need to go to the data center to reboot open bsd
- binkHN 3y agoWho here is running a production system without redundant or backup power?
- appplication 3y agoI should have clarified: I think they said they did it on their home router.
- Shared404 3y agoHuh, I've been running an openbsd router as my prod home network router for ~ a year now without issue, and it has handled power outages and more without issues. I wonder if the outage happened during patching or something maybe?
- ninkendo 3y agoIt was probably me: https://news.ycombinator.com/item?id=36837184 https://news.ycombinator.com/item?id=36837184 It was only broken in the sense that it requires you to get out a keyboard and manually continue to a root prompt and type `fsck` (and then Y a bunch of times) and presumably some files may have been corrupted but i haven’t seen any. For a router I’d much prefer that it comes up on its own after I lose power, but the system isn’t totally broken or anything. It’s happened probably 3 times in the last year or so. To another commenter saying it’s crazy to not run a UPS with your router: this is residential. 99% of people don’t have a UPS for their router. They probably don’t even have a separate router and all and just use an all in one modem/router/wifi combo. None of these people are crazy, they’re just not made of money and a UPS is expensive. IMO it’s more crazy to not have a journaled file system in this day and age, no other systems have issues with sudden power loss at my house.
- appplication 3y ago
- Hikikomori 3y agoAny 100+ Terabit Openbsd routers out there?
- Nextgrid 3y agoI would very much doubt any of those carrier-grade routers run BGP in hardware. Chances are the control plane is handled by a bog-standard Linux/VxWorks/BSD that does all the logic of talking BGP/etc and then populates the right registers in the switching/routing hardware which handles the actual data plane. You could technically replace all the control and decision-making with whatever off-the-shelf computer platform you want, and have it drive the carrier-grade router and "manually" populate its routing table via some API/Telnet/SNMP. In fact, as long as the hardware packet-routing core is still good, this might be a good way to upgrade out-of-support equipment that is no longer getting updates to its control-plane computer.
- ExoticPearTree 3y ago> My only regret with dealing with the OpenBSD team was reporting the issue to them too quickly, as they are uninterested in waiting for any kind of coordination in disclosure. This BS with disclosure coordination and all the other crap vendors have been trying to train researchers to do for years needs to stop. You find a issue, you publish it, vendors need to deal with bad code and products in realtime, not when it suits them.
- mschuster91 3y agoWhile I agree in theory, in practice this will lead to bad actors ranging from trolls to cyberwar troops exploiting it immediately. There is IMHO no ethical alternative to responsible disclosure, and using actual people who can get harmed by having their information disclosed as pawns isn't the way to go forward.
- ExoticPearTree 3y agoYes, but have that pain inflicted on end customers long enough and they will eventually transfer it to the vendors one way or another.
- zbentley 3y ago> vendors need to deal with bad code and products in realtime One of the main reasons for coordinated disclosure is that, over time, it has become evident that vendors will not do that. We can say it until we're blue in the face, but vendor behavior will remain largely unchanged regardless (there are a few bright spots, but not nearly enough). So you go to war with the army you have. In this case, that means frequently deferring/coordinating security event communications. As much as I wish the market would appropriately penalize vendors that encourage researchers to sit on 0-days, that rarely happens, and the cost of insecurity is passed onto consumers instead. Tl;dr delaying disclosure does suck, but you should still do it.
- salawat 3y agoAlas, it is the consumer that apparently drives pain reception for vendors, as they have no interest in minimizing consumer pain, but their own pain. Their own pain is defined as lost business. 'Tis the world we live in, so might as well get used to it.
- karavelov 3y agowhy not switch to bird that was not affected at all? not just acknowledged and fixed. isn’t it “safe be default” better than “fixed after they pointed the flaw”
- treve 3y agoThe robustness principle: "be conservative in what you do, be liberal in what you accept from others", hardly applies here. Whether or not it was followed, a crash shouldn't happen. The robustness principle is also long considered to be not that great of an idea for protocols. There's a good document here going into this: https://datatracker.ietf.org/doc/html/rfc9413 https://datatracker.ietf.org/doc/html/rfc9413
- vermilingua 3y agoIt isn’t a crash, BGP terminates the session and retries because it considers the connection to be malformed. So, “be liberal in what you accept” certainly does apply.
- j16sdiz 3y agoThis "terminate and retry" behaviour is specified in RFC7606
- fanf2 3y agoThat is the exact opposite of what RFC 7606 says. As it explains in its introduction, it deprecates session reset in favour of treat-as-withdraw in most cases, unless there is an explicit exception to the general rule.
- forkerenok 3y agoThat RFC is a very insightful read. Thanks for sharing!
- hn8305823 3y agoThe software on big routers that run the Internet is far more reliable than it was 20 years ago. It's not perfect and each vendor has cycles of good and bad code releases but it's so much better than it used to be.
- 112233 3y agoThis is not helped by the RFCs that specify the behaviour of these core protocols. It is rare to find a network protocol rfc that is clear, specific and not self contradictory (IPSec rfcs come to mind as a positive example). Say, compare RFC3174 (sha1) and e.g. rfc4664 (l2vpn) the first is almost a copy-paste implementation, with tests and all. the second reads like a "write a presentation, ten pages minimum" or look at the base bgp4 rfc - that overspecified imaginary state machine is a good illustration of how bad things are there.
- JonChesterfield 3y agoDesign behaviour here was on bad input, shutdown. Some bad input happened, thus shutdown happened. Seems fine to me. I'm not seeing a grave flaw.
- vitiral 3y agoThe issue is how it interacts with other designs, such as forwarding of unknown attributes who are now poisoned
- Bluecobra 3y agoAgreed, the article title is a bit sensationalist. Really should be titled “RTFM: Juniper routers shutdown BGP on bad input”. That being said, I found it be a good deep dive and I appreciate the author looking into this. BGP isn’t perfect but we made it this far and I think it does a pretty good job for handling ~75K autonomous systems.
- gregw2 3y agoErr no, it was not just Juniper. Many other systems were fuzzed, half a dozen vendors were fine, another 5 weren’t and some remain unpatched and vulnerable. If it were just a Juniper implementation flaw it’d be less interesting than explaining an aspect of BGP error handling more generally which is indeed what the article was, for those of us aware of BGP but not familiar with specific protocol design decisions and implications. If you quibbled with the word “Grave” I might agree since it’s not industrywide and impacting major vendors like Cisco and Huawei and those implementing RFC 7606 (although there is no patch for Nokia or Extreme or Bird!) I’d counterpropose that a more neutral TLDR title/summary for an HN audience might be “Grave flaws in non compliant BGP error handling.” But on balance I think the author is better off with raising the alarm with a stronger headline; he clearly put his passion to good use and I learned a little something as a result.
- nsteel 3y agoAs I understand it, Nokia equipment already does the correct thing if you configure it correctly in fault-tolerance mode.
- basedrum 3y agoWhat is the response of the bird developers? Also, for those responding that you don't see the problem, it's simple: remote unsanitized input cause denial of service
- gorkish 3y agoBird is unaffected, so I assume their response is satisfaction for a job well done. FRR, on the other hand, appears to be less responsive to concerns -- which does actually concern me since I use FRR, though not presently for eBGP. Disclosure seems to have done the job thankfully, as there appears to have been a little scramble over at FRRouting/frr about an hour ago.
- zajio1am 3y agoBIRD was not affected, so we just found out today from Hacker News :-) We made some thorough code review w.r.t. RFC 7606 compliance years ago.
- hannob 3y agoNot surprised. A few years ago a team of researchers tried to use a feature attribute flag in BGP. Their experiment caused an important BGP software to crash. The BGP community reacted by thanking the researchers for uncovering that flaw and worked on improving stability. No, just kidding. They shouted at the researchers and asked them to stop, which they eventually did. I felt back then that this community had a very unhealthy relationship with the quality of their software and the work of security researchers. Sources: https://mailman.nanog.org/pipermail/nanog/2018-December/098458.html https://mailman.nanog.org/pipermail/nanog/2018-December/0984... https://mailman.nanog.org/pipermail/nanog/2019-January/098761.html https://mailman.nanog.org/pipermail/nanog/2019-January/09876... https://mailman.nanog.org/pipermail/nanog/2019-January/099187.html https://mailman.nanog.org/pipermail/nanog/2019-January/09918... https://mailman.nanog.org/pipermail/nanog/2019-January/099142.html https://mailman.nanog.org/pipermail/nanog/2019-January/09914...
- networkchad 3y ago[dead]
- deleted 3y ago[deleted]
- dogleash 3y ago> I felt back then that this community had a very unhealthy relationship with the quality of their software and the work of security researchers. I suspect a Network Operators mailing list is going to be full of people trying to keep production up, with the tools they've got, rather than the tools they'd like. Thinking the software quality on the routers they're using are dogshit is probably just called "Monday" and thinking their peers' routers are even worse is called "Tuesday". I can understand they don't appreciate "research" done on production systems that might cause issues when they already know their equipment sucks.
- themerone 3y agoNetwork operators are also the ones that pop into IPV6 discussions to voice their displeasure with having support another protocol.
- wawwow 3y ago[flagged]
- TheHappyOddish 3y agoNo, these are very unrelated things.
- salawat 3y agoHow do you figure? The IP address to route to in DNS is still an IP which from within that network segment will believe that prefix is provided by that AS if they aren't honoring RPKI. Therefore, within that routing zone, that traffic would be blackholed I'd think, unless I'm seriously missing some nuance that I'd love it if you would share.
- cratermoon 3y agoThis seems like a good time to mention Yuan et al., “Simple Testing Can Prevent Most Critical Failures.” “We found the majority of catastrophic failures could easily have been prevented by performing simple testing on error handling code – the last line of defense – even without an understanding of the software design.”
- sgjohnson 3y agoA small Brazilian network yesterday hijacked half the internet too, including several of my own prefixes, all of whom have IRR and RPKI set up. https://bgp.tools/as/266970?show-low-vis#prefixes https://bgp.tools/as/266970?show-low-vis#prefixes [bgp.tools Alert] for AS200676 A hijack on 2a06:a005:2730::/44 from AS266970 (NET WAY PROVEDOR DE INTERNET DE CACOAL LTDA) has been detected (based on your address policy you have configured) [bgp.tools Alert] for AS200676 A hijack on 2a0a:6040:b1f4::/48 from AS266970 (NET WAY PROVEDOR DE INTERNET DE CACOAL LTDA) has been detected (based on your address policy you have configured) [bgp.tools Alert] for AS200676 A hijack on 2a0a:6040:b1f1::/48 from AS266970 (NET WAY PROVEDOR DE INTERNET DE CACOAL LTDA) has been detected (based on your address policy you have configured) [bgp.tools Alert] for AS200676 A hijack on 2a0a:6040:b1f0::/48 from AS266970 (NET WAY PROVEDOR DE INTERNET DE CACOAL LTDA) has been detected (based on your address policy you have configured) BGP is a complete mess.
- mbo 3y agoSomewhat off-topic: how were the diagrams in this article made? I find the handdrawn animated wobble quite pleasing.