3 ms·

> As an example, much of the Internet stops working with MTUs above ~1500 (which this article mentions). I experienced this first hand recently with a lower MT

by tomxor 5y ago

> As an example, much of the Internet stops working with MTUs above ~1500 (which this article mentions). I experienced this first hand recently with a lower MTU limit and it's pretty confusing. The annoying thing is that when it's only wrong at one end, most of the internet and web pages will actually carry on working, i.e it's only some HTTP requests with large enough request header that push past the MTU fragmentation threshold and disappear. Which results in websites half loading, or sometimes loading fine, but not other times. I have to use LTE to get decent internet in the city I live, and have been experimenting with different providers. Turns out most LTE networks use quite low MTUs, and so PMTUD (MTU discovery) _must_ work correctly otherwise you get the behaviour I described above... For the network with the best reception and backhaul, PMTUD just didn't seem to work, so it defaults to 1500, and all the wifi connections to the LTE router will be told to use 1500, so my wiregaurd sets itself to 1420 (less 80 bytes) and even that wont save itself when the UDP packets vanish. Now when the internet seems fishy my first response after a simple ping is to try ping at the network interface supposed MTU, e.g `ping -M do -s 1472 1.1.1.1` (-28 bytes for the header). Even more confusing is that for these LTE networks the effective MTU seems to change, possible as different combinations of cells are used for carrier aggregation... so if you are trying to use this for home internet you need to keep testing for the MTU over time to discover the lowest, then change your router wifi MTU (not even possible for most LTE routers) to the minimum for reliable internet.

Sesse__ 5y ago
PMTUD on the Internet, sadly, does not work anymore. It was wonky for a few years, then totally died when AWS and the other cloud providers started blocking all ICMP in the default configuration. (It is also notoriously hard to get to work through packet-based load balancers, since most routers won't include enough of the original packet in the “too big” ICMP to reliably send it the same way as all the data packets that form the flow.) The reason why you can browse the web at all with PMTU <1500 is due to the awful hack that is TCP MSS clamping, where the router will rewrite your TCP SYN packets to advertise a different MSS (which is roughly the same as MTU, only on layer 3) based on what it thinks the PMTU to the host is. Seriously, try browsing the Internet with MTU 1500, PMTU <1500 and a router in place that does _not_ do TCP MSS clamping. It's an incredibly frustrating experience.
- deleted 5y ago
  [deleted]
- toast0 5y ago
  > PMTUD on the Internet, sadly, does not work anymore. I don't think PMTUD ever really worked over the whole internet. Path MTU blackhole discovery can work, but it defaults to off on a lot of OSes (including Android!), and it's behavior isn't always great anyway. iOS actually does do a good job here (which doesn't change my overall opinion). I don't think Windows will pay attention to the MTU from DHCP, but I think it will try path mtu blackhole detection (but it may do it with rfc timers, which are way too slow) Also, most TCP stacks send out their MSS on SYN+ACK instead of sending max(received MSS, local MSS), which means that those networks that tried to fix their MTU problem by tweaking outgoing SYNs but don't tweak incoming SYN+ACKs don't get fixed. FreeBSD used to do the right thing, but it changed recently. People will argue that the RFC says to do it the way I'm calling the wrong way, but how many connections do you run into that can send a 1500 byte packet, but only receive a 1492 packet? Most/many large internet sites just send MSS indicating 1480 or less and call it a day. IMHO, something in-band would have been a lot more workable; let the router that can't forward the whole packet mark it, truncate, and forward it, because we can assume the router has a working forward path (otherwise there's no connection), but we can't assume the router has a working reverse path. For TCP, this would be relatively straight forward to manage; UDP wouldn't be great though.
  - tomxor 5y ago
    > Path MTU blackhole discovery can work, but it defaults to off on a lot of OSes (including Android!), and it's behavior isn't always great anyway. Interesting, I didn't know this existed, it's built into linux too but unfortunately it's only for TCP `man tcp`: tcp_mtu_probing (integer; default: 0; since Linux 2.6.17) This parameter controls TCP Packetization-Layer Path MTU Discovery. The following values may be assigned to the file: 0 Disabled 1 Disabled by default, enabled when an ICMP black hole detected 2 Always enabled, use initial MSS of tcp_base_mss. I get how it makes sense to do that at the TCP level for arbitrary paths, but I had the impression the main issue was high variability in end point MTU, i.e the particular gateway all your internet is being routed through, which is when it makes sense to do PMTU black hole discovery at the IP layer, which is essentially what I've been doing manually... Am I over generalising my problem?

by tomxor 5y ago

Sesse__ 5y ago

PMTUD on the Internet, sadly, does not work anymore. It was wonky for a few years, then totally died when AWS and the other cloud providers started blocking all ICMP in the default configuration. (It is also notoriously hard to get to work through packet-based load balancers, since most routers won't include enough of the original packet in the “too big” ICMP to reliably send it the same way as all the data packets that form the flow.) The reason why you can browse the web at all with PMTU <1500 is due to the awful hack that is TCP MSS clamping, where the router will rewrite your TCP SYN packets to advertise a different MSS (which is roughly the same as MTU, only on layer 3) based on what it thinks the PMTU to the host is. Seriously, try browsing the Internet with MTU 1500, PMTU <1500 and a router in place that does _not_ do TCP MSS clamping. It's an incredibly frustrating experience.

deleted 5y ago
[deleted]
toast0 5y ago
> PMTUD on the Internet, sadly, does not work anymore. I don't think PMTUD ever really worked over the whole internet. Path MTU blackhole discovery can work, but it defaults to off on a lot of OSes (including Android!), and it's behavior isn't always great anyway. iOS actually does do a good job here (which doesn't change my overall opinion). I don't think Windows will pay attention to the MTU from DHCP, but I think it will try path mtu blackhole detection (but it may do it with rfc timers, which are way too slow) Also, most TCP stacks send out their MSS on SYN+ACK instead of sending max(received MSS, local MSS), which means that those networks that tried to fix their MTU problem by tweaking outgoing SYNs but don't tweak incoming SYN+ACKs don't get fixed. FreeBSD used to do the right thing, but it changed recently. People will argue that the RFC says to do it the way I'm calling the wrong way, but how many connections do you run into that can send a 1500 byte packet, but only receive a 1492 packet? Most/many large internet sites just send MSS indicating 1480 or less and call it a day. IMHO, something in-band would have been a lot more workable; let the router that can't forward the whole packet mark it, truncate, and forward it, because we can assume the router has a working forward path (otherwise there's no connection), but we can't assume the router has a working reverse path. For TCP, this would be relatively straight forward to manage; UDP wouldn't be great though.
- tomxor 5y ago
  > Path MTU blackhole discovery can work, but it defaults to off on a lot of OSes (including Android!), and it's behavior isn't always great anyway. Interesting, I didn't know this existed, it's built into linux too but unfortunately it's only for TCP `man tcp`: tcp_mtu_probing (integer; default: 0; since Linux 2.6.17) This parameter controls TCP Packetization-Layer Path MTU Discovery. The following values may be assigned to the file: 0 Disabled 1 Disabled by default, enabled when an ICMP black hole detected 2 Always enabled, use initial MSS of tcp_base_mss. I get how it makes sense to do that at the TCP level for arbitrary paths, but I had the impression the main issue was high variability in end point MTU, i.e the particular gateway all your internet is being routed through, which is when it makes sense to do PMTU black hole discovery at the IP layer, which is essentially what I've been doing manually... Am I over generalising my problem?