4 ms·

I'm a big believer in Postel's Law (also known as the robustness principle) [1]. Basically be liberal in what you accept and conservative in what you send. The

by jmyeet 5y ago

I'm a big believer in Postel's Law (also known as the robustness principle) [1]. Basically be liberal in what you accept and conservative in what you send. The problem with TCP is that the actual Layer 2 and 3 infrastructure doesn't obey this principle. It's taken the opposite stance to simply reject anything that's weird or unexpected. There's a reason and history for why this is. The most defensible is security. Less defensible is things like deep packet inspection. This is a well-knwon problem and falls under the term (which I love) "ossification" [2]. As an example, much of the Internet stops working with MTUs above ~1500 (which this article mentions). Large packets (eg MTU ~9000) are almost a necessity for 10+ GbE but that often won't work outside of the LAN. So I guess my point is changes like this will take a long time (if ever) to get widespread support. This could still impact, say, data center and other "local" deployments but I guess you have to start somewhere. [1]: https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle [2]: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7738442 https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7738442

treesknees 5y ago
Part of this problem is there really isn't a layer 2 protocol for discovering the max MTU of a path. You have to use a higher protocol like ICMP to find it and then adjust your endpoint. The Internet will work with large L3/L4 packets assuming the path is able to create fragments. The article talked about the kernel's ability to hold fragments but it wasn't clear to me how Internet fragmentation comes into play or whether the performance gain is lost when using this on the Internet versus a LAN. Side tangent, this is one issue I have with the major cloud providers. Azure, AWS, et al all advertise high-bandwidth interfaces upwards of 100Gbps, but you can't saturate it anywhere near those rates with 1500 Byte packets from the Internet.
- jeffbee 5y ago
  Sure, but the amount of traffic inside AWS is way bigger than the amount of traffic that crosses its boundary. This is true for a lot of operators. Netflix and YouTube and similar loads that are virtually all egress are special cases.
valyagolev 5y ago
arguments against this principle were discussed here recently: https://news.ycombinator.com/item?id=30328446 https://news.ycombinator.com/item?id=30328446 I honestly don’t think an implementation of TCP that would try and follow this principle would bring anything but misery. I lived on the internet since IE5… imagine including “hacks for Cisco” in your packets
tomxor 5y ago
> As an example, much of the Internet stops working with MTUs above ~1500 (which this article mentions). I experienced this first hand recently with a lower MTU limit and it's pretty confusing. The annoying thing is that when it's only wrong at one end, most of the internet and web pages will actually carry on working, i.e it's only some HTTP requests with large enough request header that push past the MTU fragmentation threshold and disappear. Which results in websites half loading, or sometimes loading fine, but not other times. I have to use LTE to get decent internet in the city I live, and have been experimenting with different providers. Turns out most LTE networks use quite low MTUs, and so PMTUD (MTU discovery) _must_ work correctly otherwise you get the behaviour I described above... For the network with the best reception and backhaul, PMTUD just didn't seem to work, so it defaults to 1500, and all the wifi connections to the LTE router will be told to use 1500, so my wiregaurd sets itself to 1420 (less 80 bytes) and even that wont save itself when the UDP packets vanish. Now when the internet seems fishy my first response after a simple ping is to try ping at the network interface supposed MTU, e.g `ping -M do -s 1472 1.1.1.1` (-28 bytes for the header). Even more confusing is that for these LTE networks the effective MTU seems to change, possible as different combinations of cells are used for carrier aggregation... so if you are trying to use this for home internet you need to keep testing for the MTU over time to discover the lowest, then change your router wifi MTU (not even possible for most LTE routers) to the minimum for reliable internet.
- Sesse__ 5y ago
  PMTUD on the Internet, sadly, does not work anymore. It was wonky for a few years, then totally died when AWS and the other cloud providers started blocking all ICMP in the default configuration. (It is also notoriously hard to get to work through packet-based load balancers, since most routers won't include enough of the original packet in the “too big” ICMP to reliably send it the same way as all the data packets that form the flow.) The reason why you can browse the web at all with PMTU <1500 is due to the awful hack that is TCP MSS clamping, where the router will rewrite your TCP SYN packets to advertise a different MSS (which is roughly the same as MTU, only on layer 3) based on what it thinks the PMTU to the host is. Seriously, try browsing the Internet with MTU 1500, PMTU <1500 and a router in place that does _not_ do TCP MSS clamping. It's an incredibly frustrating experience.
drewg123 5y ago
If you read the actual LKML message, and not just the LWN article, it becomes clear this is really about bumping TSO/LRO sizes in linux, not about using link layer MTUs in excess of 64k. This is more akin to Microsoft's "LSOv2" (which is even mentioned in Erick's commit message for the mlx5 driver). I'm not sure if the LWN author just doesn't understand the distinction, or if they were trying to simplify things for their readers..
- Sesse__ 5y ago
  The article is clear that it's about TSO: > Modern network interfaces perform segmentation offloading, meaning that much of the work of creating individual packets is done within the interface itself. Making segmentation offloading work with jumbo packets tends to involve a small number of tweaks; a few drivers are updated in the patch set.
  - drewg123 5y ago
    I was keying off of this "The BIG TCP patch set adds the logic necessary to generate and accept jumbo packets when the maximum transmission unit (MTU) of a connection is set sufficiently high.", which made it sound to me like the author believed that the MTU was 185k
jbverschoor 5y ago
> https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle And I am totally against this. I agree for human / interactive input. But for automated things, be as strict as you can be, and hard fail on anything else. Specs are there for a reason, and not adhering them creates compatibility issues. > RFC 1122 (1989) expanded on Postel's principle by recommending that programmers "assume that the network is filled with malevolent entities that will send in packets designed to have the worst possible effect Well, that's gonna be a tough one if you try to make sense of garbage data (or data which doesn't adhere to the specs) > Protocols should allow for the addition of new codes for existing fields in future versions of protocols by accepting messages with unknown codes (possibly logging them). I agree with that. That does not mean "be liberal in what you accept". It means the protocol is designed in a better way. > Basically be liberal in what you accept and conservative in what you send That's not the same as "be conservative in what you do, be liberal in what you accept from others". "Doing" is a lot more than "sending". You know.. many programmers sometimes say that when things go wrong it's "a user error", or "a human error". But most of the time it is either a programming error, or a lazy programmer. It's so easy for a programmer to send the correct data. But we live in a world where usually the least correct/secure software is chosen, because it's slightly easier to get started, or allows for input errors (mysql data truncation, mongodb, etc, etc)
- david422 5y ago
  > And I am totally against this. I agree for human / interactive input. But for automated things, be as strict as you can be, and hard fail on anything else. Specs are there for a reason, and not adhering them creates compatibility issues. Totally agreed on this. When a human inputs a name with a trailing space, trim it for them. Or when they input a phone number with funny dashes, remove them. But when an API does this, it means the API client is simply incorrect. Incorrect requests should be rejected. Don't leave it up to the server to try to interpret it, this just leads to incorrect behavior.
sly010 5y ago
In this case I think the robustness principle is actually counter productive, here me out: MTU discovery is "robust". Most nodes already forward any packet they can handle. The problem is that _some_ nodes filter out ICMP packets (hence breaking MTU discovery) but the link still "kind-of-works" (tm). Nodes that break MTU discovery should be considered 100% broken, but instead of fixing them we "accommodate" them by statically (manually) lowering our outgoing MTU, hence proliferating the problem. Robustness works against us. If half-broken firewalls were considered fully-broken, they would be replaced instead of accommodated. Aside: the `Fragmentation Needed` packet should have been part of the IP header, not a separate protocol.
- sly010 5y ago
  > Most nodes already forward any packet they can handle On second thought, I guess I am wrong here, since the default MTU in Linux is a very low number (something like 1500). Robustness would be to have limit by default and fully rely on MTU discovery. Although the low default was probably established because of the aforementioned ICMP filtering issue.

by jmyeet 5y ago

treesknees 5y ago

Part of this problem is there really isn't a layer 2 protocol for discovering the max MTU of a path. You have to use a higher protocol like ICMP to find it and then adjust your endpoint. The Internet will work with large L3/L4 packets assuming the path is able to create fragments. The article talked about the kernel's ability to hold fragments but it wasn't clear to me how Internet fragmentation comes into play or whether the performance gain is lost when using this on the Internet versus a LAN. Side tangent, this is one issue I have with the major cloud providers. Azure, AWS, et al all advertise high-bandwidth interfaces upwards of 100Gbps, but you can't saturate it anywhere near those rates with 1500 Byte packets from the Internet.

jeffbee 5y ago
Sure, but the amount of traffic inside AWS is way bigger than the amount of traffic that crosses its boundary. This is true for a lot of operators. Netflix and YouTube and similar loads that are virtually all egress are special cases.

valyagolev 5y ago

arguments against this principle were discussed here recently: https://news.ycombinator.com/item?id=30328446 https://news.ycombinator.com/item?id=30328446 I honestly don’t think an implementation of TCP that would try and follow this principle would bring anything but misery. I lived on the internet since IE5… imagine including “hacks for Cisco” in your packets

tomxor 5y ago

> As an example, much of the Internet stops working with MTUs above ~1500 (which this article mentions). I experienced this first hand recently with a lower MTU limit and it's pretty confusing. The annoying thing is that when it's only wrong at one end, most of the internet and web pages will actually carry on working, i.e it's only some HTTP requests with large enough request header that push past the MTU fragmentation threshold and disappear. Which results in websites half loading, or sometimes loading fine, but not other times. I have to use LTE to get decent internet in the city I live, and have been experimenting with different providers. Turns out most LTE networks use quite low MTUs, and so PMTUD (MTU discovery) _must_ work correctly otherwise you get the behaviour I described above... For the network with the best reception and backhaul, PMTUD just didn't seem to work, so it defaults to 1500, and all the wifi connections to the LTE router will be told to use 1500, so my wiregaurd sets itself to 1420 (less 80 bytes) and even that wont save itself when the UDP packets vanish. Now when the internet seems fishy my first response after a simple ping is to try ping at the network interface supposed MTU, e.g `ping -M do -s 1472 1.1.1.1` (-28 bytes for the header). Even more confusing is that for these LTE networks the effective MTU seems to change, possible as different combinations of cells are used for carrier aggregation... so if you are trying to use this for home internet you need to keep testing for the MTU over time to discover the lowest, then change your router wifi MTU (not even possible for most LTE routers) to the minimum for reliable internet.

Sesse__ 5y ago
PMTUD on the Internet, sadly, does not work anymore. It was wonky for a few years, then totally died when AWS and the other cloud providers started blocking all ICMP in the default configuration. (It is also notoriously hard to get to work through packet-based load balancers, since most routers won't include enough of the original packet in the “too big” ICMP to reliably send it the same way as all the data packets that form the flow.) The reason why you can browse the web at all with PMTU <1500 is due to the awful hack that is TCP MSS clamping, where the router will rewrite your TCP SYN packets to advertise a different MSS (which is roughly the same as MTU, only on layer 3) based on what it thinks the PMTU to the host is. Seriously, try browsing the Internet with MTU 1500, PMTU <1500 and a router in place that does _not_ do TCP MSS clamping. It's an incredibly frustrating experience.

drewg123 5y ago

If you read the actual LKML message, and not just the LWN article, it becomes clear this is really about bumping TSO/LRO sizes in linux, not about using link layer MTUs in excess of 64k. This is more akin to Microsoft's "LSOv2" (which is even mentioned in Erick's commit message for the mlx5 driver). I'm not sure if the LWN author just doesn't understand the distinction, or if they were trying to simplify things for their readers..

Sesse__ 5y ago
The article is clear that it's about TSO: > Modern network interfaces perform segmentation offloading, meaning that much of the work of creating individual packets is done within the interface itself. Making segmentation offloading work with jumbo packets tends to involve a small number of tweaks; a few drivers are updated in the patch set.
- drewg123 5y ago
  I was keying off of this "The BIG TCP patch set adds the logic necessary to generate and accept jumbo packets when the maximum transmission unit (MTU) of a connection is set sufficiently high.", which made it sound to me like the author believed that the MTU was 185k

jbverschoor 5y ago

> https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle And I am totally against this. I agree for human / interactive input. But for automated things, be as strict as you can be, and hard fail on anything else. Specs are there for a reason, and not adhering them creates compatibility issues. > RFC 1122 (1989) expanded on Postel's principle by recommending that programmers "assume that the network is filled with malevolent entities that will send in packets designed to have the worst possible effect Well, that's gonna be a tough one if you try to make sense of garbage data (or data which doesn't adhere to the specs) > Protocols should allow for the addition of new codes for existing fields in future versions of protocols by accepting messages with unknown codes (possibly logging them). I agree with that. That does not mean "be liberal in what you accept". It means the protocol is designed in a better way. > Basically be liberal in what you accept and conservative in what you send That's not the same as "be conservative in what you do, be liberal in what you accept from others". "Doing" is a lot more than "sending". You know.. many programmers sometimes say that when things go wrong it's "a user error", or "a human error". But most of the time it is either a programming error, or a lazy programmer. It's so easy for a programmer to send the correct data. But we live in a world where usually the least correct/secure software is chosen, because it's slightly easier to get started, or allows for input errors (mysql data truncation, mongodb, etc, etc)

david422 5y ago
> And I am totally against this. I agree for human / interactive input. But for automated things, be as strict as you can be, and hard fail on anything else. Specs are there for a reason, and not adhering them creates compatibility issues. Totally agreed on this. When a human inputs a name with a trailing space, trim it for them. Or when they input a phone number with funny dashes, remove them. But when an API does this, it means the API client is simply incorrect. Incorrect requests should be rejected. Don't leave it up to the server to try to interpret it, this just leads to incorrect behavior.

sly010 5y ago

In this case I think the robustness principle is actually counter productive, here me out: MTU discovery is "robust". Most nodes already forward any packet they can handle. The problem is that _some_ nodes filter out ICMP packets (hence breaking MTU discovery) but the link still "kind-of-works" (tm). Nodes that break MTU discovery should be considered 100% broken, but instead of fixing them we "accommodate" them by statically (manually) lowering our outgoing MTU, hence proliferating the problem. Robustness works against us. If half-broken firewalls were considered fully-broken, they would be replaced instead of accommodated. Aside: the `Fragmentation Needed` packet should have been part of the IP header, not a separate protocol.

sly010 5y ago
> Most nodes already forward any packet they can handle On second thought, I guess I am wrong here, since the default MTU in Linux is a very low number (something like 1500). Robustness would be to have limit by default and fully rely on MTU discovery. Although the low default was probably established because of the aforementioned ICMP filtering issue.