3 ms·

The impacted systems get stuck on a retry loop... they can neither forward the message blindly nor consume it. They break the BGP connection, reset, come back a

by buro9 3y ago

The impacted systems get stuck on a retry loop... they can neither forward the message blindly nor consume it. They break the BGP connection, reset, come back alive and get the same message and break the connection again. Effectively the impacted instances are now no longer able to do their job, they cannot propagate more rules. That's the flaw. When they receive something that they don't understand they need to propagate without breaking connection, and they do not.

fanf2 3y ago
Yes. It’s also important to emphasize that when a BGP session between two ISPs is lost, all connectivity between those ISPs on that path is also lost. This is why the author emphasized when you can reconfigure the routers for error tolerance without affecting active BGP sessions.
throw0101c 3y ago
> When they receive something that they don't understand they need to propagate without breaking connection, and they do not. And this is how things are done, per the article: > On 2 June 2023, a small Brazilian network (re)announced one of their internet routes with a small bit of information called an attribute that was corrupted. The information on this route was for a feature that had not finished standardisation, but was set up in such a way that if an intermediate router did not understand it, then the intermediate router would pass it on unchanged. > As many routers did not understand this attribute, this was no problem for them. They just took the information and propagated it along. However, the information was corrupted at the source, and those devices that did understand the attributed detected the corruption and rejected it: > However it turned out that Juniper routers running even slightly modern software did understand this attribute, and since the attribute was corrupted the software in its default configuration would respond by raising an error that would shut down the whole BGP session. The question is whether the session shutdown was appropriate (at least by default). I think Juniper's default is reasonable, as attributes can communicate important information about the intentions of the network operators, and rejecting a particular attribute and basically ignoring the intentions of the operators may also cause problems. I also think that thinking that perhaps allowing the session to continue but throwing a (loud) warning is also reasonable (and there is a flag for that).
- tsimionescu 3y ago
  The current behavior is akin to an HTTP server shutting down whenever it receives an invalid HTTP request. Or, for an even better analogy, it is like an HTTP server that, upon receiving a malformed HTTP request, drops all current and future traffic from that IP. A malicious user can then issue a malformed HTTP request through an open proxy, and deny all other connections to that server throgh that proxy for all other users who were sending proper requests.
  - throw0101c 3y ago
    > The current behavior is akin to an HTTP server shutting down whenever it receives an invalid HTTP request. It is nothing like that. HTTP requests do not propagate across an organization's entire network, or possibly the entire Internet. > Or, for an even better analogy, it is like an HTTP server that, upon receiving a malformed HTTP request, drops all current and future traffic from that IP. So just like (e.g.) fail2ban blocking IPs that continuously send bad login requests? That's awesome! I would want my web server to block bad clients. Heck, it would be nice if web browsers would do the same thing with content they downloaded. In the early days of the web there were all sorts of garbage files out there and instead of forcing people to correct their HTML they tried to be clever in mind reading: * https://en.wikipedia.org/wiki/Tag_soup https://en.wikipedia.org/wiki/Tag_soup For a time there was an entire 'movement' to get people to clean up their act: * https://en.wikipedia.org/wiki/W3C_Markup_Validation_Service https://en.wikipedia.org/wiki/W3C_Markup_Validation_Service * https://en.wikipedia.org/wiki/HTML_Tidy https://en.wikipedia.org/wiki/HTML_Tidy It's fine to have a flag to have stringency be a policy that can be toggled, but I think Postel's law ("be liberal in what you accept") can cause issues over the long-term: * https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle
    - fanf2 3y ago
      It’s like your origin web server unceremoniously dropping the http connection when its front end reverse proxy (fastly, cloudfront, akamai, etc) forwards an invalid request, and when your web server restarts the proxy retries the bad request.

by buro9 3y ago

fanf2 3y ago

Yes. It’s also important to emphasize that when a BGP session between two ISPs is lost, all connectivity between those ISPs on that path is also lost. This is why the author emphasized when you can reconfigure the routers for error tolerance without affecting active BGP sessions.

throw0101c 3y ago

> When they receive something that they don't understand they need to propagate without breaking connection, and they do not. And this is how things are done, per the article: > On 2 June 2023, a small Brazilian network (re)announced one of their internet routes with a small bit of information called an attribute that was corrupted. The information on this route was for a feature that had not finished standardisation, but was set up in such a way that if an intermediate router did not understand it, then the intermediate router would pass it on unchanged. > As many routers did not understand this attribute, this was no problem for them. They just took the information and propagated it along. However, the information was corrupted at the source, and those devices that did understand the attributed detected the corruption and rejected it: > However it turned out that Juniper routers running even slightly modern software did understand this attribute, and since the attribute was corrupted the software in its default configuration would respond by raising an error that would shut down the whole BGP session. The question is whether the session shutdown was appropriate (at least by default). I think Juniper's default is reasonable, as attributes can communicate important information about the intentions of the network operators, and rejecting a particular attribute and basically ignoring the intentions of the operators may also cause problems. I also think that thinking that perhaps allowing the session to continue but throwing a (loud) warning is also reasonable (and there is a flag for that).

tsimionescu 3y ago
The current behavior is akin to an HTTP server shutting down whenever it receives an invalid HTTP request. Or, for an even better analogy, it is like an HTTP server that, upon receiving a malformed HTTP request, drops all current and future traffic from that IP. A malicious user can then issue a malformed HTTP request through an open proxy, and deny all other connections to that server throgh that proxy for all other users who were sending proper requests.
- throw0101c 3y ago
  > The current behavior is akin to an HTTP server shutting down whenever it receives an invalid HTTP request. It is nothing like that. HTTP requests do not propagate across an organization's entire network, or possibly the entire Internet. > Or, for an even better analogy, it is like an HTTP server that, upon receiving a malformed HTTP request, drops all current and future traffic from that IP. So just like (e.g.) fail2ban blocking IPs that continuously send bad login requests? That's awesome! I would want my web server to block bad clients. Heck, it would be nice if web browsers would do the same thing with content they downloaded. In the early days of the web there were all sorts of garbage files out there and instead of forcing people to correct their HTML they tried to be clever in mind reading: * https://en.wikipedia.org/wiki/Tag_soup https://en.wikipedia.org/wiki/Tag_soup For a time there was an entire 'movement' to get people to clean up their act: * https://en.wikipedia.org/wiki/W3C_Markup_Validation_Service https://en.wikipedia.org/wiki/W3C_Markup_Validation_Service * https://en.wikipedia.org/wiki/HTML_Tidy https://en.wikipedia.org/wiki/HTML_Tidy It's fine to have a flag to have stringency be a policy that can be toggled, but I think Postel's law ("be liberal in what you accept") can cause issues over the long-term: * https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle
  - fanf2 3y ago
    It’s like your origin web server unceremoniously dropping the http connection when its front end reverse proxy (fastly, cloudfront, akamai, etc) forwards an invalid request, and when your web server restarts the proxy retries the bad request.