4 ms·

Design behaviour here was on bad input, shutdown. Some bad input happened, thus shutdown happened. Seems fine to me. I'm not seeing a grave flaw.

vitiral 3y ago
The issue is how it interacts with other designs, such as forwarding of unknown attributes who are now poisoned
Bluecobra 3y ago
Agreed, the article title is a bit sensationalist. Really should be titled “RTFM: Juniper routers shutdown BGP on bad input”. That being said, I found it be a good deep dive and I appreciate the author looking into this. BGP isn’t perfect but we made it this far and I think it does a pretty good job for handling ~75K autonomous systems.
- gregw2 3y ago
  Err no, it was not just Juniper. Many other systems were fuzzed, half a dozen vendors were fine, another 5 weren’t and some remain unpatched and vulnerable. If it were just a Juniper implementation flaw it’d be less interesting than explaining an aspect of BGP error handling more generally which is indeed what the article was, for those of us aware of BGP but not familiar with specific protocol design decisions and implications. If you quibbled with the word “Grave” I might agree since it’s not industrywide and impacting major vendors like Cisco and Huawei and those implementing RFC 7606 (although there is no patch for Nokia or Extreme or Bird!) I’d counterpropose that a more neutral TLDR title/summary for an HN audience might be “Grave flaws in non compliant BGP error handling.” But on balance I think the author is better off with raising the alarm with a stronger headline; he clearly put his passion to good use and I learned a little something as a result.
  - nsteel 3y ago
    As I understand it, Nokia equipment already does the correct thing if you configure it correctly in fault-tolerance mode.
    - hellt 3y ago
      correct, Nokia implements RFC 7606, it needs to be added to the configuration, as it is not the default for now.
  - pdonis 3y ago
    Bird routers didn't have the problem, correct?
- basedrum 3y ago
  Sure if you didn't read and only skimmed the article...
- sgjohnson 3y ago
  > BGP isn’t perfect but we made it this far and I think it does a pretty good job for handling ~75K autonomous systems. Absolutely not. To me, it's a miracle that the internet even works.
buro9 3y ago
The impacted systems get stuck on a retry loop... they can neither forward the message blindly nor consume it. They break the BGP connection, reset, come back alive and get the same message and break the connection again. Effectively the impacted instances are now no longer able to do their job, they cannot propagate more rules. That's the flaw. When they receive something that they don't understand they need to propagate without breaking connection, and they do not.
- fanf2 3y ago
  Yes. It’s also important to emphasize that when a BGP session between two ISPs is lost, all connectivity between those ISPs on that path is also lost. This is why the author emphasized when you can reconfigure the routers for error tolerance without affecting active BGP sessions.
- throw0101c 3y ago
  > When they receive something that they don't understand they need to propagate without breaking connection, and they do not. And this is how things are done, per the article: > On 2 June 2023, a small Brazilian network (re)announced one of their internet routes with a small bit of information called an attribute that was corrupted. The information on this route was for a feature that had not finished standardisation, but was set up in such a way that if an intermediate router did not understand it, then the intermediate router would pass it on unchanged. > As many routers did not understand this attribute, this was no problem for them. They just took the information and propagated it along. However, the information was corrupted at the source, and those devices that did understand the attributed detected the corruption and rejected it: > However it turned out that Juniper routers running even slightly modern software did understand this attribute, and since the attribute was corrupted the software in its default configuration would respond by raising an error that would shut down the whole BGP session. The question is whether the session shutdown was appropriate (at least by default). I think Juniper's default is reasonable, as attributes can communicate important information about the intentions of the network operators, and rejecting a particular attribute and basically ignoring the intentions of the operators may also cause problems. I also think that thinking that perhaps allowing the session to continue but throwing a (loud) warning is also reasonable (and there is a flag for that).
capableweb 3y ago
That sounds like a typical denial of service attack, which you don't see a problem with?
- JonChesterfield 3y ago
  It sounds like a system designed to stop doing things on unexpected input. The designers chose correctness over robustness. It's not an accidental vulnerability. Maybe switch off on request is a dubious design choice for infrastructure that you'd rather stay alive, but it sounds like a new version of the spec has already been released to make that an option.
mnw21cam 3y ago
Bad input -> shutdown provides a method by which an external attacker can shut down your service by providing bad input, otherwise known as a denial of service attack.
- throw0101c 3y ago
  The input in question was a particular attribute: these attributes can have important information about the intentions of how the operators wish things to work. Simply ignoring the bad input and carrying on the BGP session without the attribute(s) in question may lead to working against the desired intentions of the network operators. I think Juniper's default is reasonable: rejecting a particular attribute and basically ignoring the intentions of the operators could also cause problems, and instead of trying to read the minds of those running the networks, the software doesn't try to be clever. I also think that thinking that perhaps allowing the session to continue but throwing a (loud) warning is also reasonable (and there is a flag for that).
  - tsimionescu 3y ago
    The problem is that one bad route in a BGP session caused the whole session to shut down. So if ISP1 advertises a new route with a corrupted attribute to ISP2 who doesn't understand it, and ISP2 advertises it to ISP3 who does understand the corrupted attribute, all traffic between ISP2 and ISP3 will shut down. So ISP1 has caused communication failure between ISP2 and ISP3 for all clients, not just for its own routes.
    - throw0101c 3y ago
      Yes, I know about BGP propagation works. My point still stands: attributes can have important information about the intentions of how a network operator wishes their network to be seen to the rest of the world. Having a corrupt attribute, and thinking it's okay to simply assume that without that attribute the rest of the advertisement is still valid is not fine IMHO: you're basically trying to guess/assume the intentions of the source of the advertisement. For all we know ignoring that attribute and accepting the rest of the advertisement as-is could have caused flood of traffic over the connection causing a just-as-effective-DoS.
remram 3y ago
Imagine if Gmail and Outlook refused to talk to each other and transmit any email anymore, because Gmail tried to deliver one email whose content was invalid (bad MIME, invalid attachment, etc). If you shutdown the whole session because of one bad forwarded message, you are indeed a broken implementation.

vitiral 3y ago

The issue is how it interacts with other designs, such as forwarding of unknown attributes who are now poisoned

Bluecobra 3y ago

Agreed, the article title is a bit sensationalist. Really should be titled “RTFM: Juniper routers shutdown BGP on bad input”. That being said, I found it be a good deep dive and I appreciate the author looking into this. BGP isn’t perfect but we made it this far and I think it does a pretty good job for handling ~75K autonomous systems.

gregw2 3y ago
Err no, it was not just Juniper. Many other systems were fuzzed, half a dozen vendors were fine, another 5 weren’t and some remain unpatched and vulnerable. If it were just a Juniper implementation flaw it’d be less interesting than explaining an aspect of BGP error handling more generally which is indeed what the article was, for those of us aware of BGP but not familiar with specific protocol design decisions and implications. If you quibbled with the word “Grave” I might agree since it’s not industrywide and impacting major vendors like Cisco and Huawei and those implementing RFC 7606 (although there is no patch for Nokia or Extreme or Bird!) I’d counterpropose that a more neutral TLDR title/summary for an HN audience might be “Grave flaws in non compliant BGP error handling.” But on balance I think the author is better off with raising the alarm with a stronger headline; he clearly put his passion to good use and I learned a little something as a result.
- nsteel 3y ago
  As I understand it, Nokia equipment already does the correct thing if you configure it correctly in fault-tolerance mode.
  - hellt 3y ago
    correct, Nokia implements RFC 7606, it needs to be added to the configuration, as it is not the default for now.
- pdonis 3y ago
  Bird routers didn't have the problem, correct?
basedrum 3y ago
Sure if you didn't read and only skimmed the article...
sgjohnson 3y ago
> BGP isn’t perfect but we made it this far and I think it does a pretty good job for handling ~75K autonomous systems. Absolutely not. To me, it's a miracle that the internet even works.

buro9 3y ago

The impacted systems get stuck on a retry loop... they can neither forward the message blindly nor consume it. They break the BGP connection, reset, come back alive and get the same message and break the connection again. Effectively the impacted instances are now no longer able to do their job, they cannot propagate more rules. That's the flaw. When they receive something that they don't understand they need to propagate without breaking connection, and they do not.

fanf2 3y ago
Yes. It’s also important to emphasize that when a BGP session between two ISPs is lost, all connectivity between those ISPs on that path is also lost. This is why the author emphasized when you can reconfigure the routers for error tolerance without affecting active BGP sessions.
throw0101c 3y ago
> When they receive something that they don't understand they need to propagate without breaking connection, and they do not. And this is how things are done, per the article: > On 2 June 2023, a small Brazilian network (re)announced one of their internet routes with a small bit of information called an attribute that was corrupted. The information on this route was for a feature that had not finished standardisation, but was set up in such a way that if an intermediate router did not understand it, then the intermediate router would pass it on unchanged. > As many routers did not understand this attribute, this was no problem for them. They just took the information and propagated it along. However, the information was corrupted at the source, and those devices that did understand the attributed detected the corruption and rejected it: > However it turned out that Juniper routers running even slightly modern software did understand this attribute, and since the attribute was corrupted the software in its default configuration would respond by raising an error that would shut down the whole BGP session. The question is whether the session shutdown was appropriate (at least by default). I think Juniper's default is reasonable, as attributes can communicate important information about the intentions of the network operators, and rejecting a particular attribute and basically ignoring the intentions of the operators may also cause problems. I also think that thinking that perhaps allowing the session to continue but throwing a (loud) warning is also reasonable (and there is a flag for that).

capableweb 3y ago

That sounds like a typical denial of service attack, which you don't see a problem with?

JonChesterfield 3y ago
It sounds like a system designed to stop doing things on unexpected input. The designers chose correctness over robustness. It's not an accidental vulnerability. Maybe switch off on request is a dubious design choice for infrastructure that you'd rather stay alive, but it sounds like a new version of the spec has already been released to make that an option.

mnw21cam 3y ago

Bad input -> shutdown provides a method by which an external attacker can shut down your service by providing bad input, otherwise known as a denial of service attack.

throw0101c 3y ago
The input in question was a particular attribute: these attributes can have important information about the intentions of how the operators wish things to work. Simply ignoring the bad input and carrying on the BGP session without the attribute(s) in question may lead to working against the desired intentions of the network operators. I think Juniper's default is reasonable: rejecting a particular attribute and basically ignoring the intentions of the operators could also cause problems, and instead of trying to read the minds of those running the networks, the software doesn't try to be clever. I also think that thinking that perhaps allowing the session to continue but throwing a (loud) warning is also reasonable (and there is a flag for that).
- tsimionescu 3y ago
  The problem is that one bad route in a BGP session caused the whole session to shut down. So if ISP1 advertises a new route with a corrupted attribute to ISP2 who doesn't understand it, and ISP2 advertises it to ISP3 who does understand the corrupted attribute, all traffic between ISP2 and ISP3 will shut down. So ISP1 has caused communication failure between ISP2 and ISP3 for all clients, not just for its own routes.
  - throw0101c 3y ago
    Yes, I know about BGP propagation works. My point still stands: attributes can have important information about the intentions of how a network operator wishes their network to be seen to the rest of the world. Having a corrupt attribute, and thinking it's okay to simply assume that without that attribute the rest of the advertisement is still valid is not fine IMHO: you're basically trying to guess/assume the intentions of the source of the advertisement. For all we know ignoring that attribute and accepting the rest of the advertisement as-is could have caused flood of traffic over the connection causing a just-as-effective-DoS.

remram 3y ago

Imagine if Gmail and Outlook refused to talk to each other and transmit any email anymore, because Gmail tried to deliver one email whose content was invalid (bad MIME, invalid attachment, etc). If you shutdown the whole session because of one bad forwarded message, you are indeed a broken implementation.