10 ms·
BGP handling bug causes widespread internet routing instability
- mgaunard 1y agoDoes multicast over the Internet even work? I thought BGP was only for private networks.
- FL410 1y agoBGP is very much not only for private networks - it is what the internet is built on. Maybe you are thinking of iBGP or something like OSPF?
- Alifatisk 1y agoNo, BGP is not only for private networks. ASNs use it to exchange messages between each other.
- bc569a80a344f9c 1y agoBGP is the routing protocol of the Internet. There effectively is no other choice of routing protocol between autonomous systems. A reasonable synonym for "Internet" is "the global BGP routing table". BGP also doesn't use multicast, you may be thinking of OSPF on multiaccess networks. BGP uses tcp/179 unicast to the IP addresses of its configured peers. That said, multicast works just fine over the Internet. It's not commonly used, certainly not by home users and not very often by enterprise users, and was phased out on Internet2 by 2021 (I think?), but there's absolutely nothing in principle that would make it not work.
- ai-christianson 1y agoAnycast is pretty useful on the Internet :)
- bc569a80a344f9c 1y agoAnycast is a very different beast, though. Anycast is just unicast but you announce the same IP space from multiple destinations, and the network figures out how to get a packet to the closest one. If one of those destinations fails, it just goes to the next closest one. Unicasts, multicasts, and broadcasts all actually work differently underneath and require specific handling by network equipment. Anycast is just a special case of unicast and generally speaking network equipment is completely unaware of it.
- deleted 1y ago[deleted]
- rnxrx 1y agoFWIW, 100% that BGP itself doesn't *use* multicast, but it can *propagate* multicast routing information. It's certainly technically possible to support multicast on the Internet (..thus the invention of MBGP) but in practice has been a non-starter for a whole bunch of reasons.
- toast0 1y ago> That said, multicast works just fine over the Internet. It's not commonly used, certainly not by home users and not very often by enterprise users, and was phased out on Internet2 by 2021 (I think?), but there's absolutely nothing in principle that would make it not work. In principle, no. In practice, I don't think many ISPs have equipment configured to forward multicast, except for those using multicast for TV and those probably don't interconnect with others.
- ta1243 1y agoMany years ago UK ISPs particpated in an MBone with BBC, ITV etc providing live broadcast https://www.bbc.co.uk/multicast/tv/channels.shtml https://www.bbc.co.uk/multicast/tv/channels.shtml Brandon Butterworths note about "why" https://support.bbc.co.uk/multicast/why.html https://support.bbc.co.uk/multicast/why.html Shows the growth of the backbone and CDNs: > The Olympic audience is expected to be around 50K streams, delivering 10Gbit+ is on the limit of sensible unicast delivery. In 2020 the BBC's internal CDN was delivering 100 times that [0] for 250k users, and 5 years later I suspect it's another order of magnitude given that iplayer does 5 million concurrent live views quite frequently [1,2] [0] https://medium.com/bbc-product-technology/bbc-online-2020-in-review-e47dada5df3f https://medium.com/bbc-product-technology/bbc-online-2020-in... [1] https://www.bbc.co.uk/mediacentre/2024/audiences-flock-to-bbc-for-major-sport-finals-euros-wimbledon https://www.bbc.co.uk/mediacentre/2024/audiences-flock-to-bb... [2] https://www.bbc.co.uk/mediacentre/2022/england-v-iran-bbc-live-streaming-record?utm_source=chatgpt.com https://www.bbc.co.uk/mediacentre/2022/england-v-iran-bbc-li... By 2035 and TV turnoff there's no reason to believe that the infrastructure won't have been able to scale another 100 fold and handle 500 million concurrent live streams. Makes no sense to multicast out 30 different formats, to people on phones and tablets and TVs hanging off wifi. It's a very different consumer experience than a PC wired into an ISP like it was in 2007.
- Bluecobra 1y agoYou can definitely route multicast over the Internet via some kind of tunnel, be it stunnel, VPN, GRE. The ICE exchange uses stunnel for dev/certification multicast.
- immibis 1y agoIt's sort of correct and sort of incorrect to say it's the only possible routing protocol between autonomous systems. Since many inter-AS connections are effectively made-to-order, they could use a different protocol, but in order to participate in the global BGP mesh, it would have to have semantics similar enough to BGP. Most notably, it would have to support the concept of AS-path. However I don't think there are any universal requirements beyond that. BGP isn't a single distributed algorithm like say OSPF - a completely separate instance of BGP runs between every pair of connected ASes and they share some data, indirectly forming a global system.
- AStonesThrow 1y agoIn fact, IPv4 and IPv6 both have a reserved range of multicast addresses (formerly called "Class D"): https://en.wikipedia.org/wiki/Multicast_address https://en.wikipedia.org/wiki/Multicast_address There were more than a few people who spotted how disused this range had become after mbone experiments, and sometimes suggested reclaiming the range as IPv4 address space was being exhausted. Interestingly, there are reserved multicast addresses (yes, addresses, not ports) not only for OSPF, but for many other interior routing protocols, as well as mDNS, LLMNR, and NTP. Conspicuously absent is any reservation for BGP.
- tonetegeatinst 1y agoUnsure about multicast...my gut would say no but I'd not trust that answer if I had to bet on it. BGP is the only way I know of that autonomous Systems can talk to each other and negotiate.
- patmorgan23 1y agoAll traffic over the Internet is routed by routes that have been propagated through BGP. It's how adjacent networks tell each other what IPs they originate, and what networks they are connected to.
- Hikikomori 1y agoThe "Internet" is a bunch of separate networks operated by different entities. BGP is what allows them to connect to each other by exchanging routing formation. There was something called the mbone back in the day. Nowadays you can't really send random multicast, but its very much in use by ISPs for IPTV.
- ta1243 1y agoThe standard approach is be liberal in what you accept and be specific in what you emit. You could 1) Filter the broken message 2) Drop the broken message 3) Ignore the broken attributes but pass them on 4) Break with the broken attributes To me, only 4 (Arista) is the really unacceptable behaviour. 3 (Juniper) isn't desirable but it's not a devastating behaviour. EDIT: Actually rereading it, Arista did 2 rather than 4. I think it just closed the connection as being invalid rather than completely crash. That's arguably acceptable, but not great for the users.
- klysm 1y agoI disagree with this approach. Being very very specific in what you accept and very very specific in what you emit seems better to me.
- tinco 1y agoThat's in conflict with the philosophy behind the internet. If you'd just drop anything because some part of it you don't understand, you lose a lot of flexibility. You have to keep in mind that some parts of the internet are running on 20 year old hardware, but some other parts might work so much better if some protocol is modified a little. Just like with web browsers, if everything is a little bit flexible in what they accept, you both improve the smoothness of the experience and create room for growth and innovation.
- bluGill 1y agoThere is a place for both. The accept everything model made some extensions better, but it also allowed for various malware when junk was accepted.
- colejohnson66 1y agoPostel's law doesn't mean "accept everything", but that you should accept de-facto rules people have created. If everyone says, "this is how we do it", you should ignore the RFC and just copy what others do.
- dakiol 1y agoIs it just me or BGP is something I never learnt about until I heard about it causing issues? It seems it's essential to the internet, just like TCP/IP, but nevertheless I learnt about the latter in the university, during my career, I read many books about TCP/IP... but nothing about BGP (not in the university, not at work, not in books, nothing). I can "play" with TCP/IP at home in dummy projects and learn more about it... but I have no idea how to "play" with BGP. In that regard, how does one learn about it at home?
- klysm 1y agoIt’s very much hidden, which I guess is a success of the design. You need not concern yourself with the web of ASNs when using IP.
- altilunium 1y agoI think you need to manage a real (and large) network that's connected to global internet traffic in order to "play" with BGP. Well, you can tinker with it at home, but only by using a network simulator.
- bc569a80a344f9c 1y agoYou can set up local BGP routers and peer them and play with it. Another fun thing is to log into publicly available looking glass servers. Most ISPs (including very, very, very large ones) operate routers that have their full view of the BGP routing tables. They either run web interfaces that let you query those tables (more common) or make public ssh or telnet credentials to log in with roles that have very limited access to the available commands, but have read rights to those tables.
- _gmax0 1y agoInternet researchers commonly use RIPE and RouteViews. https://www.routeviews.org/routeviews/about/ https://www.routeviews.org/routeviews/about/ https://stat.ripe.net/docs/02.data-api/ https://stat.ripe.net/docs/02.data-api/
- 1y ago
- sc68cal 1y agoI still remember the mad scramble we had to fix CVE-2023-4481 across our entire network. This class of bugs is going to be an absolute nightmare to deal with, and because of the way BGP has been designed & implemented, it is going to take a _long_ time to fix these kinds of behaviors.
- alecco 1y agoHGC Global Communications Limited, formerly known as Hutchison Global Communications Limited (abb. HGC), is an internet service provider of Hong Kong. https://en.wikipedia.org/wiki/HGC_Global_Communications https://en.wikipedia.org/wiki/HGC_Global_Communications
- mintflow 1y agoI was developping BGP feature in a telco vendor though it's decades ago. Still think BGP is too complex and people keeping add new features and vendors keeping implement it based on RFC standard or draft. And it seems BGP will never be deprecated so this sort of bugs will continue be found again and again...
- transitorykris 1y agoThere was certainly a period of time where folks like AT&T alongside Juniper and Cisco drove BGP into crazytown by way of MPLS and VPN related features. Terrifyingly complex (imo) but lucrative for some.
- incomingpain 1y agoOur IOS XR chassis' have gotten some of these packets. Corresponding with high bgp route advertisements. No idea what equipment upstream uses tbh. Makes me wonder if the BGP protocol is properly fuzzed. Perhaps its one of those things that everyone is scared to try to knock over given it's so important. I suppose it would be easy to write a fuzzer for bgp but very hard to diagnose crashes?
- benjojo12 1y ago(author of the post) Yes, this is exactly what I did in the post I linked to: https://blog.benjojo.co.uk/post/bgp-path-attributes-grave-error-handling https://blog.benjojo.co.uk/post/bgp-path-attributes-grave-er...
- incomingpain 1y agoBravo! This is great research!
- bhaney 1y ago> At 7AM (UTC) on Wednesday May 20th 2025 May 20th was a Tuesday, just sayin'
- benjojo12 1y agoGreat catch, there were like 3 mini incidents this week that I was keeping track of, so wires got crossed, will correct the post in a moment
- ta1243 1y agoAlthough at 7AM UTC in some parts of the world it was a Monday :D
- saghm 1y agoYes, but in those parts of the world it also wasn't May 20 at that time
- mschuster91 1y agoWell surprise, people cheat because an academic degree is yet another checkbox to be ticked in the ever more grueling list of tasks needed to be done to acquire a job that has even a remote chance of paying a living wage. Fix that, so that only those actually interested in academia per se and not just because they need a checkbox to tick remain, and the problem with cheating in academia will collapse.
- CSN3RD 1y agoSeveral vendors had this bug in the past https://www.kb.cert.org/vuls/id/347067 https://www.kb.cert.org/vuls/id/347067 CVE-2023-4481 (Juniper) CVE-2023-38802 (FRR) CVE-2023-38283 (OpenBGPd) CVE-2023-40457 (EXOS) Arista was not affected then.
- jiveturkey 1y agoGiven the impact of such bugs, I'm surprised there isn't a consortium with an interoperability test suite. Or maybe there is, and this specific issue isn't in the test suite. In which case, I'm surprised test cases aren't generated with a fuzzer and/or machine-generated full exploration of possible packet errors. I mean, it's fine if the suite takes hours or even days to run. I guess the author of the article here has written a fuzzer with some coverage, and has come across similar issues before. Astonishing that the vendors don't pick up on this work hungrily.
- Perenti 1y agoIt appears to this reader that BGP would be a lot more stable if the various hardware vendors agreed on a standard for handling these types of things. Is the real issue that each vendor wants lock-in, so won't standardise? DISCLAIMER: My understanding of BGP is hollow and shallow, I am not an expert.
- hoseja 1y agoHas there ever been anything so byzantine in scale and accidental complexity as internet plumbing.