6 ms·

The standard approach is be liberal in what you accept and be specific in what you emit. You could 1) Filter the broken message 2) Drop the broken message 3

by ta1243 1y ago

The standard approach is be liberal in what you accept and be specific in what you emit. You could 1) Filter the broken message 2) Drop the broken message 3) Ignore the broken attributes but pass them on 4) Break with the broken attributes To me, only 4 (Arista) is the really unacceptable behaviour. 3 (Juniper) isn't desirable but it's not a devastating behaviour. EDIT: Actually rereading it, Arista did 2 rather than 4. I think it just closed the connection as being invalid rather than completely crash. That's arguably acceptable, but not great for the users.

klysm 1y ago
I disagree with this approach. Being very very specific in what you accept and very very specific in what you emit seems better to me.
- tinco 1y ago
 That's in conflict with the philosophy behind the internet. If you'd just drop anything because some part of it you don't understand, you lose a lot of flexibility. You have to keep in mind that some parts of the internet are running on 20 year old hardware, but some other parts might work so much better if some protocol is modified a little. Just like with web browsers, if everything is a little bit flexible in what they accept, you both improve the smoothness of the experience and create room for growth and innovation.
 - bluGill 1y ago
 There is a place for both. The accept everything model made some extensions better, but it also allowed for various malware when junk was accepted.
 - colejohnson66 1y ago
 Postel's law doesn't mean "accept everything", but that you should accept de-facto rules people have created. If everyone says, "this is how we do it", you should ignore the RFC and just copy what others do.
 - bluGill 1y ago
 There are several problems with that. One, if everyone is doing something different from the spec it is hard to figure out what they are really doing and what they mean. Long term you have confidence things will continue to work even when someone else writes their own version which otherwise might also deviate from the spec. Two, it is easier to modify the spec as more features are dreamed up if you have confidence that the spec is boss meaning someone else didn't already use that field for something different (which you may not have heard about yet). Three, if you agree to a spec you can audit it (think security), if nobody even knows what the spec is that is much harder. Following the spec is harder in the early days. You have to put more effort into the spec because you can't discover a problem and just patch it in code. However the internet is far past those days. We need a spec that is the rule that everyone follows exactly.
 - SAI_Peregrinus 1y ago
 Postel's Law is important, but it creates brittle systems. You can force them further from the ideal operating state before failure, but when they fail they tend to fail suddenly and catastrophically. I like to call it the "Hardness Principle" as opposed to the "Robustness Principle" in analogy to metallurgy.
 - emmelaich 1y ago
 Surely the opposite? If everything was very pedantic and strict, the 'net would be so brittle as to be non-functional. You're imagining a world where things get specified and implemented completely correctly. Which does not exist and probably can't!
 - immibis 1y ago
 That's what Postel thought. He was wrong. Allowing everything creates a brittle system because the system has to accept all the undocumented behaviour that other broken systems emit. If broken files were rejected quickly, nobody would generate them. There's a difference between unknown extensions following a known format, and data that's simply broken (e.g. offset pointer past end of data).
 emmelaich 1y ago
 You're not accounting for the incorrectly rejected file / protocols. And incomplete protocol specifications. And generally I think critics of Postel are lacking the context in which they were made. You and probably others would actually make similar decisions than Postel for many particular issues.
 SAI_Peregrinus 1y ago
 I disagree that I'd make similar decisions. Postel's Law is a big part of the reason Bleichenbacher attacks (adaptive chosen-ciphertext attacks)[1] stayed so common for so long. As an engineer responsible for the security I absolutely reject malformed inputs. https://en.wikipedia.org/wiki/Adaptive_chosen-ciphertext_attack https://en.wikipedia.org/wiki/Adaptive_chosen-ciphertext_att...
 - arccy 1y ago
 This is so wrong, read up on https://datatracker.ietf.org/doc/html/rfc9413 https://datatracker.ietf.org/doc/html/rfc9413 The internet is ossified because middleboxes stick their noses where they shouldn't. If they just route IP packets, we could have had nice things like SCTP...
- bolognafairy 1y ago
 Alright. See you over on the XHTML Internet. Oh, wait.
 - hinkley 1y ago
 If the iPhone had come out just a little bit later I think xhtml-basic would have gotten more traction. It was pretty nice to implement.
 - debugnik 1y ago
 XHTML still lives on in the epub spec. I kinda wish we had an "epub web".
 - oblio 1y ago
 HTML is a nightmare that had to be reverse engineered as in, rebuilt with proper engineering standards in mind, several times. HTML and CSS are both quite horrible.
 - hombre_fatal 1y ago
 Browsers are permissive not because it's technically superior but as a concession for the end user who still wants to be able to use a poorly built website, and they're competing with browsers who will bend over backwards to render that crappy website so that they look good and your browser looks bad. It's not a concession you want to make unless you really have to.
 - immibis 1y ago
 In other words, because the early Web followed Postel's law, we're now stuck in this local maximum.
 - hombre_fatal 1y ago
 Well, my point is that there's unique pressure for browsers to be permissive for practical reasons beyond Postel's law even if you were building a browser in 2025 and the whole internet reset to xhtml. And that's because the end-user is at the mercy of, but not party to, an over the air interface between the producer and consumer that you can't verify ahead of time. So if you're consuming a stream of supposed xhtml `foobar`, you have to decide if you want to screw the user for the producer's mistake for a single fuck up in the website's footer.
- erincandescent 1y ago
 Being that prescriptive is fundamentally unworkable in practice. Propagating unknown attributes is fundamentally what made the deployment of 32-bit AS numbers possible (originally RFC 4893; unaware routers pass the `AS4_PATH` attribute without needing to comprehend it), large communities (RFC 8092), the Only To Customer attribute (RFC 9234) and others. A BGP Update message is mostly just a container of Type-Length-Value attributes. As long as the TLV structure is intact, you should be able to just pass on those TLVs without problems to any peers that the route is destined for. The problem fundamentally is three things: 1. The original BGP RFC suggests tearing down the connection upon receiving an erroneous message. This is a terrible idea, especially for transitive attributes: you'll just reconnect and your peer will resend you the same message, flapping over and over, and the attribute is likely to not even be your peer's fault. The modern recommendation is Treat As Withdraw, i.e. remove any matching routes from the same peer from your routing table. 2. A lack of fuzz testing and similar by BGP implementers (Arista in this case) 3. Even for vendors which have done such testing, a number of have decided (IMO stupidly) to require you to turn on these robustness features explicitly.
 - hinkley 1y ago
 PNG solved this problem when BGP was still young: each section of an image document is marked as to whether understanding it is necessary to process the payload or not. So image transform and palette data is intrinsic, but metadata is not. Adding EXIF for instance is thus made trivial. No browser needs to understand it so it can be added without breaking the distribution mechanism.
 - zajio1am 1y ago
 This is also how BGP (mostly) solved it. Each attribute has 'transitive' bit. Unknown attributes with 'transitive' bit set are passed, one without are discarded.
 - immibis 1y ago
 ... Except for acTL, which is a special exception because it turns out that wasn't sufficient to ensure consistency in 100% of cases.
sc68cal 1y ago
The problem is that folks took advantage of the behavior of BGP where it would forward unknown attributes that the local device didn't understand, as a means to do all sorts of things throughout the network. People now rely on that behavior. Now, we're experiencing the downside of this "feature"
- spwa4 1y ago
  BGP has classes attributes that it forwards. While it is true that it forwards route attributes it doesn't know about, this was an attribute that it DID know about and knows it shouldn't forward. In fact it's a bit strange just how lenient Juniper's software was here. If a session is configured as IBGP on one end and EBGP on the other end, it should never get past the initial message. Juniper not only let it get past the connection establishment but forwarded obviously wrong routes.
  - sc68cal 1y ago
    Yes but you are seeing a symptom of what I believe is a fundamental design decision to be liberal in passing on data and then _later_ go through and build logic that stops certain things from being forwarded, and the result is that things slip through the cracks that shouldn't. Rather than the inverse where you only forward things explicitly and by default do not forward.
  - icehawk 1y ago
    As far as I'm aware "a session is configured as IBGP on one end and EBGP on the other end" isn't possible. You can't configure it like that, most of the BGP implementations I'm familiar with automatically treat the a same-AS neighbor as iBGP and a different-AS neighbor as eBGP. Juniper explicitly has 'internal' and 'external' neighbors, but you can't configure a different peer AS than your own on an internal neighbor or the same peer AS on an external neighbor. BGP sessions also have the AS of the neighbor specified in the local config, and will not bring up the session if it's not what's configured.
- zajio1am 1y ago
  Without this behavior it would be impossible to deploy newer BGP attributes globally.
ExoticPearTree 1y ago
Arista did 2, but it also dropped the whole connection which was probably bad. IMHO, just drop the broken attributes in the message and log them, and pass on the valid data if there's any left. If not, pretend you did not receive an UPDATE message from that particular peer. Monitoring will catch the offending originator and people can deal with this without having to deal with any network instability.
- eqvinox 1y ago
  In case you want to calibrate your sense of armchair-ness: you have completely missed the point that discarding an individual attribute can quite badly change the meaning of a route, and since we're talking about the DFZ here, such breakage can spread around the planet to literally every DFZ participant. The only safe thing you can do is to drop the entire route. Maybe there was a point to this being discussed at quite some length by very knowledgeable people, before 7606 became RFC ;) (I haven't downvoted your comment, but I can see why others would — you're making very simple and definite statements about very complicated problems, and you don't seem to be aware of the complications involved. Hence: your calibration is a bit off.)
  - ExoticPearTree 1y ago
    Funny enough, I actually have a few routers with a DFZ, so I have an idea or two about how BGP works. My point is that: - if you drop a connection, especially one through which you announce the full routing table, it is going to create a lot of churn to your downstreams. Depending on the kind of routers they use, it can create some network instability for quite a while. And if you drop it again when you receive that malformed route, the instability continues - removing only the malformed attribute maybe changes the way you treat traffic but you still route it. OK, you send it to maybe another interface, but no biggie - if you’re using a DFZ setup, dropping that single route could blackhole traffic to that destination if you’re the only upstream to another router
    - wbl 1y ago
      Blackholes are acceptable when it comes to broken attributes. Blackholes spreading is not.
billfor 1y ago
Author makes this point in a related post: At a glance this “feature” seems like an incredibly bad idea, as it allows possibly unknown information to propagate blindly through systems that do not understand the impact of what they are forwarding. However this feature has also allowed widespread deployment of things like Large Communities to happen faster, and has arguably made deployment of new BGP features possible at all.
zajio1am 1y ago
There is already RFC 7606 (Revised Error Handling for BGP UPDATE Messages), which specifies in detail how broken BGP messages should be handled. The most common approach is 'treat-as-withdraw', i.e. handle the update (announcement of a route) as if it was a withdraw (removal of previously announced route). You should not just drop the broken message as that whould lead to keeping old, no longer valid state.
- timewizard 1y ago
  [flagged]
eqvinox 1y ago
All of this is understood and has been discussed to death, it's just that Arista didn't implement the agreed-best approach (RFC7606) correctly.
hannob 1y ago
> The standard approach is be liberal in what you accept and be specific in what you emit. What you're paraphrasing here is the so-called "robustness principle", also known as "Poestel's law". It is an idea from the ancient history of the 1980s and 09s Internet. Today, it's widely understood that it is a misguided idea that has led to protocol ossification and countless security issues.
- senderista 1y ago
  Postel's Law certainly has led to a lot of problems, but is it really responsible for protocol ossification? Isn't the problem the opposite, e.g. that middleboxes are too strict in what they accept (say only the HTTP application protocol or only the TCP and UDP transport protocols)?
  - fc417fc802 1y ago
    Overly strict and overly liberal both lead to ossification. That's merely the observation that buggy behavior in either direction can potentially come to be relied on (or to be unpredictably forced on you, in the case of middleboxes filtering your traffic). I'd only expect security issues to result from being overly liberal but 1. I wouldn't expect it to be very common and 2. I'm not at all convinced that's a compelling argument to reduce the robustness of an implementation.
    - NooneAtAll3 1y ago
      what does ossification mean? open source something?
      - remexre 1y ago
        It's when existing implementations' inflexibility prevent protocol evolution. https://en.wikipedia.org/wiki/Protocol_ossification https://en.wikipedia.org/wiki/Protocol_ossification
      - SchemaLoad 1y ago
AdamJacobMuller 1y ago
I would perhaps argue that juniper's behavior is the preferable one. Remember the definition of this "drop the message I think is broken" not inherently "drop the broken message," it's entirely plausible that the message is fine but you have a bug which makes you THINK it's a broken message. There is also a huge difference between considering it a broken message and a broken session, which is what Arista did.

by ta1243 1y ago

klysm 1y ago

I disagree with this approach. Being very very specific in what you accept and very very specific in what you emit seems better to me.

tinco 1y ago
That's in conflict with the philosophy behind the internet. If you'd just drop anything because some part of it you don't understand, you lose a lot of flexibility. You have to keep in mind that some parts of the internet are running on 20 year old hardware, but some other parts might work so much better if some protocol is modified a little. Just like with web browsers, if everything is a little bit flexible in what they accept, you both improve the smoothness of the experience and create room for growth and innovation.
- bluGill 1y ago
  There is a place for both. The accept everything model made some extensions better, but it also allowed for various malware when junk was accepted.
  - colejohnson66 1y ago
    Postel's law doesn't mean "accept everything", but that you should accept de-facto rules people have created. If everyone says, "this is how we do it", you should ignore the RFC and just copy what others do.
    - bluGill 1y ago
      There are several problems with that. One, if everyone is doing something different from the spec it is hard to figure out what they are really doing and what they mean. Long term you have confidence things will continue to work even when someone else writes their own version which otherwise might also deviate from the spec. Two, it is easier to modify the spec as more features are dreamed up if you have confidence that the spec is boss meaning someone else didn't already use that field for something different (which you may not have heard about yet). Three, if you agree to a spec you can audit it (think security), if nobody even knows what the spec is that is much harder. Following the spec is harder in the early days. You have to put more effort into the spec because you can't discover a problem and just patch it in code. However the internet is far past those days. We need a spec that is the rule that everyone follows exactly.
- SAI_Peregrinus 1y ago
  Postel's Law is important, but it creates brittle systems. You can force them further from the ideal operating state before failure, but when they fail they tend to fail suddenly and catastrophically. I like to call it the "Hardness Principle" as opposed to the "Robustness Principle" in analogy to metallurgy.
  - emmelaich 1y ago
    Surely the opposite? If everything was very pedantic and strict, the 'net would be so brittle as to be non-functional. You're imagining a world where things get specified and implemented completely correctly. Which does not exist and probably can't!
    - immibis 1y ago
      That's what Postel thought. He was wrong. Allowing everything creates a brittle system because the system has to accept all the undocumented behaviour that other broken systems emit. If broken files were rejected quickly, nobody would generate them. There's a difference between unknown extensions following a known format, and data that's simply broken (e.g. offset pointer past end of data).
      - emmelaich 1y ago
        You're not accounting for the incorrectly rejected file / protocols. And incomplete protocol specifications. And generally I think critics of Postel are lacking the context in which they were made. You and probably others would actually make similar decisions than Postel for many particular issues.
        SAI_Peregrinus 1y ago
        I disagree that I'd make similar decisions. Postel's Law is a big part of the reason Bleichenbacher attacks (adaptive chosen-ciphertext attacks)[1] stayed so common for so long. As an engineer responsible for the security I absolutely reject malformed inputs. https://en.wikipedia.org/wiki/Adaptive_chosen-ciphertext_attack https://en.wikipedia.org/wiki/Adaptive_chosen-ciphertext_att...
- arccy 1y ago
  This is so wrong, read up on https://datatracker.ietf.org/doc/html/rfc9413 https://datatracker.ietf.org/doc/html/rfc9413 The internet is ossified because middleboxes stick their noses where they shouldn't. If they just route IP packets, we could have had nice things like SCTP...
bolognafairy 1y ago
Alright. See you over on the XHTML Internet. Oh, wait.
- hinkley 1y ago
 If the iPhone had come out just a little bit later I think xhtml-basic would have gotten more traction. It was pretty nice to implement.
- debugnik 1y ago
 XHTML still lives on in the epub spec. I kinda wish we had an "epub web".
- oblio 1y ago
 HTML is a nightmare that had to be reverse engineered as in, rebuilt with proper engineering standards in mind, several times. HTML and CSS are both quite horrible.
- hombre_fatal 1y ago
 Browsers are permissive not because it's technically superior but as a concession for the end user who still wants to be able to use a poorly built website, and they're competing with browsers who will bend over backwards to render that crappy website so that they look good and your browser looks bad. It's not a concession you want to make unless you really have to.
 - immibis 1y ago
 In other words, because the early Web followed Postel's law, we're now stuck in this local maximum.
 - hombre_fatal 1y ago
 Well, my point is that there's unique pressure for browsers to be permissive for practical reasons beyond Postel's law even if you were building a browser in 2025 and the whole internet reset to xhtml. And that's because the end-user is at the mercy of, but not party to, an over the air interface between the producer and consumer that you can't verify ahead of time. So if you're consuming a stream of supposed xhtml `foobar`, you have to decide if you want to screw the user for the producer's mistake for a single fuck up in the website's footer.
erincandescent 1y ago
Being that prescriptive is fundamentally unworkable in practice. Propagating unknown attributes is fundamentally what made the deployment of 32-bit AS numbers possible (originally RFC 4893; unaware routers pass the `AS4_PATH` attribute without needing to comprehend it), large communities (RFC 8092), the Only To Customer attribute (RFC 9234) and others. A BGP Update message is mostly just a container of Type-Length-Value attributes. As long as the TLV structure is intact, you should be able to just pass on those TLVs without problems to any peers that the route is destined for. The problem fundamentally is three things: 1. The original BGP RFC suggests tearing down the connection upon receiving an erroneous message. This is a terrible idea, especially for transitive attributes: you'll just reconnect and your peer will resend you the same message, flapping over and over, and the attribute is likely to not even be your peer's fault. The modern recommendation is Treat As Withdraw, i.e. remove any matching routes from the same peer from your routing table. 2. A lack of fuzz testing and similar by BGP implementers (Arista in this case) 3. Even for vendors which have done such testing, a number of have decided (IMO stupidly) to require you to turn on these robustness features explicitly.
- hinkley 1y ago
  PNG solved this problem when BGP was still young: each section of an image document is marked as to whether understanding it is necessary to process the payload or not. So image transform and palette data is intrinsic, but metadata is not. Adding EXIF for instance is thus made trivial. No browser needs to understand it so it can be added without breaking the distribution mechanism.
  - zajio1am 1y ago
    This is also how BGP (mostly) solved it. Each attribute has 'transitive' bit. Unknown attributes with 'transitive' bit set are passed, one without are discarded.
  - immibis 1y ago
    ... Except for acTL, which is a special exception because it turns out that wasn't sufficient to ensure consistency in 100% of cases.

sc68cal 1y ago

The problem is that folks took advantage of the behavior of BGP where it would forward unknown attributes that the local device didn't understand, as a means to do all sorts of things throughout the network. People now rely on that behavior. Now, we're experiencing the downside of this "feature"

spwa4 1y ago
BGP has classes attributes that it forwards. While it is true that it forwards route attributes it doesn't know about, this was an attribute that it DID know about and knows it shouldn't forward. In fact it's a bit strange just how lenient Juniper's software was here. If a session is configured as IBGP on one end and EBGP on the other end, it should never get past the initial message. Juniper not only let it get past the connection establishment but forwarded obviously wrong routes.
- sc68cal 1y ago
  Yes but you are seeing a symptom of what I believe is a fundamental design decision to be liberal in passing on data and then _later_ go through and build logic that stops certain things from being forwarded, and the result is that things slip through the cracks that shouldn't. Rather than the inverse where you only forward things explicitly and by default do not forward.
- icehawk 1y ago
  As far as I'm aware "a session is configured as IBGP on one end and EBGP on the other end" isn't possible. You can't configure it like that, most of the BGP implementations I'm familiar with automatically treat the a same-AS neighbor as iBGP and a different-AS neighbor as eBGP. Juniper explicitly has 'internal' and 'external' neighbors, but you can't configure a different peer AS than your own on an internal neighbor or the same peer AS on an external neighbor. BGP sessions also have the AS of the neighbor specified in the local config, and will not bring up the session if it's not what's configured.
zajio1am 1y ago
Without this behavior it would be impossible to deploy newer BGP attributes globally.

ExoticPearTree 1y ago

Arista did 2, but it also dropped the whole connection which was probably bad. IMHO, just drop the broken attributes in the message and log them, and pass on the valid data if there's any left. If not, pretend you did not receive an UPDATE message from that particular peer. Monitoring will catch the offending originator and people can deal with this without having to deal with any network instability.

eqvinox 1y ago
In case you want to calibrate your sense of armchair-ness: you have completely missed the point that discarding an individual attribute can quite badly change the meaning of a route, and since we're talking about the DFZ here, such breakage can spread around the planet to literally every DFZ participant. The only safe thing you can do is to drop the entire route. Maybe there was a point to this being discussed at quite some length by very knowledgeable people, before 7606 became RFC ;) (I haven't downvoted your comment, but I can see why others would — you're making very simple and definite statements about very complicated problems, and you don't seem to be aware of the complications involved. Hence: your calibration is a bit off.)
- ExoticPearTree 1y ago
  Funny enough, I actually have a few routers with a DFZ, so I have an idea or two about how BGP works. My point is that: - if you drop a connection, especially one through which you announce the full routing table, it is going to create a lot of churn to your downstreams. Depending on the kind of routers they use, it can create some network instability for quite a while. And if you drop it again when you receive that malformed route, the instability continues - removing only the malformed attribute maybe changes the way you treat traffic but you still route it. OK, you send it to maybe another interface, but no biggie - if you’re using a DFZ setup, dropping that single route could blackhole traffic to that destination if you’re the only upstream to another router
  - wbl 1y ago
    Blackholes are acceptable when it comes to broken attributes. Blackholes spreading is not.

billfor 1y ago

Author makes this point in a related post: At a glance this “feature” seems like an incredibly bad idea, as it allows possibly unknown information to propagate blindly through systems that do not understand the impact of what they are forwarding. However this feature has also allowed widespread deployment of things like Large Communities to happen faster, and has arguably made deployment of new BGP features possible at all.

zajio1am 1y ago

There is already RFC 7606 (Revised Error Handling for BGP UPDATE Messages), which specifies in detail how broken BGP messages should be handled. The most common approach is 'treat-as-withdraw', i.e. handle the update (announcement of a route) as if it was a withdraw (removal of previously announced route). You should not just drop the broken message as that whould lead to keeping old, no longer valid state.

timewizard 1y ago
[flagged]

eqvinox 1y ago

All of this is understood and has been discussed to death, it's just that Arista didn't implement the agreed-best approach (RFC7606) correctly.

hannob 1y ago

> The standard approach is be liberal in what you accept and be specific in what you emit. What you're paraphrasing here is the so-called "robustness principle", also known as "Poestel's law". It is an idea from the ancient history of the 1980s and 09s Internet. Today, it's widely understood that it is a misguided idea that has led to protocol ossification and countless security issues.

senderista 1y ago
Postel's Law certainly has led to a lot of problems, but is it really responsible for protocol ossification? Isn't the problem the opposite, e.g. that middleboxes are too strict in what they accept (say only the HTTP application protocol or only the TCP and UDP transport protocols)?
- fc417fc802 1y ago
  Overly strict and overly liberal both lead to ossification. That's merely the observation that buggy behavior in either direction can potentially come to be relied on (or to be unpredictably forced on you, in the case of middleboxes filtering your traffic). I'd only expect security issues to result from being overly liberal but 1. I wouldn't expect it to be very common and 2. I'm not at all convinced that's a compelling argument to reduce the robustness of an implementation.
  - NooneAtAll3 1y ago
    what does ossification mean? open source something?
    - remexre 1y ago
      It's when existing implementations' inflexibility prevent protocol evolution. https://en.wikipedia.org/wiki/Protocol_ossification https://en.wikipedia.org/wiki/Protocol_ossification
    - SchemaLoad 1y ago

AdamJacobMuller 1y ago

I would perhaps argue that juniper's behavior is the preferable one. Remember the definition of this "drop the message I think is broken" not inherently "drop the broken message," it's entirely plausible that the message is fine but you have a bug which makes you THINK it's a broken message. There is also a huge difference between considering it a broken message and a broken session, which is what Arista did.