3 ms·

As a quick summary, first of all the host at each end of the connection knows it's MTU and advertises this as MSS in the TCP connection so that in theory the c

As a quick summary, first of all the host at each end of the connection knows it's MTU and advertises this as MSS in the TCP connection so that in theory the connection is established at the right MTU but this is never tested until a large packet is sent (which may or may not arrive). (the MSS is actually smaller than the MTU since its the data size without headers and even on a given link can change e.g. if IP options are in use.. but in any case it's derived from the MTU.. RTFM for the gory details). Most consumer routers having an MTU of 1492 due to PPPoE will modify the MSS in-flight of the TCP connection to the MSS it knows for the next hop. This is not a standard feature in other routers though, and obviously wouldn't really work on the internet at large where paths can change mid connection. So it's best used at endpoint routers only. Ideally such routers might advertise the MTU using DHCP or similar, and it is possible to do this now (and this is commonly used e.g. openstack clouds) however it does unnecessarily then limit the MTU of the local network unnecessarily. In theory I guess it could send a mtu for the default route and local subnet separately but basically in practice no one does this afaik. Originally, a router along the path would know the MTU was smaller on the subsequent link, and would fragment the packet. For various reasons this is an expensive operation for all involved so we decided to instead take advantage of the "DF" (Don't Fragment) flag to instead generate an ICMP message back to the source asking them to please adjust their MTU/MSS and re-transmit. This flag specifically prevents the router fragmenting the message (that will never happen now) and is the default behavior at least on Linux and I assume most OS now. This can go wrong in two ways, firstly ICMP may be blocked somewhere so it never arrives even at the end host or by some over zealous firewall. Secondly, the MTU may be misconfigured somewhere at layer 2 so the packet is silently dropped rather than having an ICMP message generated. The ICMP message requires that a router knows* the MTU of the complete l2 path to the next hop. If it get's that wrong, it will just send the packet and assume it is OK, when in reality it is dropped silently. (This same problem can be seen on just a local network, if you configure e.g. 9000 MTU on two hosts but an MTU of 1500 on your switch, your connections will break. It's not purely a routing issue). There is a second protocol designed to detect PMTU "passively" to solve this problem called TCP PLPMD (Packetization Layer Path MTU Discovery). This basically uses heuristics to detect that packets are being dropped and then start probing the real path MTU by doing something like a binary search between some base MTU (e.g. 512) and the known max MTU (e.g. 1500). It tries various values until it settles on the actual MTU. The problem with this of course, is that packets may be dropped by packet loss rather than MTU issues and so there is a bit of luck and strategy to getting this right - and it also introduces some extra latency while some packets are being dropped. For whatever reason this option is currently off by default in Linux (e.g. Ubuntu) and if it was on, it would silently fix the MTU on a lot of connections that are currently not fixed. The option is sysctl net.ipv4.tcp_mtu_probing. This has been enabled by default since Windows Vista, which explains why we still see people with their MSS misconfigured since Windows Vista onwards will just fix it. In linux at least it also has two modes, one where it waits to detect an ICMP blackhole and another where it always uses the protocol from the start. The former results in a 1-2s delay before it kicks in to fix a connection, so not ideal if it's happening on every connection but OK for the odd broken connection. There is of course, like all topics, more to it than this.. but hopefully that helps.

lathiat 8y ago
It's worth noting the big downside to enabling PLPMD is of course that connections which are broken are unlikely to get fixed, but it can result in extra latency or lower initial connection performance which is then hard to diagnose. As opposed to being just broken and fixed by the administrator/network operator/user/etc in the first place. Most of the time the MTU issues are relatively local to the user, e.g. their router isn't configured with the actual MTU of their internet connection, etc. and can be fixed by them (as opposed to by the ISP etc)
toast0 8y ago
Thanks for the summary. I'm aware of all of this; but from the server side, when clients mysteriously don't send packets of a certain size, it's hard to know why --- because I'm only on the server side, and I haven't had any luck getting knowledgeable people on the other end. I can tell it's an MTU issue, because I get later packets, but I'm missing a block that happens to be an exact multiple of the MSS. It could be simple packet loss, except this pattern comes up too often, the client will sometimes send probing acks, but never the missing packets, and it never retries with smaller packets. This is especially bad when you're communicating with default Linux servers that always send syn+ack with their local MSS, because so many clients think they can send 1500 byte packets, but they can't. It's less bad when using most versions of FreeBSD [1] where it returns whichever is smaller of client side MSS or server MSS. There are a number of systems/networks that clamp outgoing SYNs but not incoming SYNs, so downloads mostly work, but uploads are broken. Mirroring the sent MSS isn't enough to make every connection work though, because of exciting things you mentioned -- somewhere in the middle, something is dropping large packets, and ICMPs aren't sent or otherwise don't make it to the client, so it just kind of sits around waiting forever. Edit to add: I wish the codified behavior was simply to truncate IP packets to the size available, rather than either fragment or alert. TCP Peers would be able to notice that only shorter packets get acked, and adjust, and in the meantime, some data would have gotten through. Not terribly great for UDP, but I dunno. [1] there's a change in -CURRENT to use the Linux behavior, unfortunately
- p1mrx 8y ago
  > I wish the codified behavior was simply to truncate IP packets to the size available If a router could replace the packet with a "truncation sentinel" that's compatible with most firewalls, detectable by new implementations, and doesn't cause data corruption for old implementations, then perhaps truncation could be incrementally deployed? Like a "packet too big" error, but in the forward direction. However, it might be impossible to construct a packet satisfying all of those conditions.

lathiat 8y ago

It's worth noting the big downside to enabling PLPMD is of course that connections which are broken are unlikely to get fixed, but it can result in extra latency or lower initial connection performance which is then hard to diagnose. As opposed to being just broken and fixed by the administrator/network operator/user/etc in the first place. Most of the time the MTU issues are relatively local to the user, e.g. their router isn't configured with the actual MTU of their internet connection, etc. and can be fixed by them (as opposed to by the ISP etc)

toast0 8y ago

Thanks for the summary. I'm aware of all of this; but from the server side, when clients mysteriously don't send packets of a certain size, it's hard to know why --- because I'm only on the server side, and I haven't had any luck getting knowledgeable people on the other end. I can tell it's an MTU issue, because I get later packets, but I'm missing a block that happens to be an exact multiple of the MSS. It could be simple packet loss, except this pattern comes up too often, the client will sometimes send probing acks, but never the missing packets, and it never retries with smaller packets. This is especially bad when you're communicating with default Linux servers that always send syn+ack with their local MSS, because so many clients think they can send 1500 byte packets, but they can't. It's less bad when using most versions of FreeBSD [1] where it returns whichever is smaller of client side MSS or server MSS. There are a number of systems/networks that clamp outgoing SYNs but not incoming SYNs, so downloads mostly work, but uploads are broken. Mirroring the sent MSS isn't enough to make every connection work though, because of exciting things you mentioned -- somewhere in the middle, something is dropping large packets, and ICMPs aren't sent or otherwise don't make it to the client, so it just kind of sits around waiting forever. Edit to add: I wish the codified behavior was simply to truncate IP packets to the size available, rather than either fragment or alert. TCP Peers would be able to notice that only shorter packets get acked, and adjust, and in the meantime, some data would have gotten through. Not terribly great for UDP, but I dunno. [1] there's a change in -CURRENT to use the Linux behavior, unfortunately

p1mrx 8y ago
> I wish the codified behavior was simply to truncate IP packets to the size available If a router could replace the packet with a "truncation sentinel" that's compatible with most firewalls, detectable by new implementations, and doesn't cause data corruption for old implementations, then perhaps truncation could be incrementally deployed? Like a "packet too big" error, but in the forward direction. However, it might be impossible to construct a packet satisfying all of those conditions.