4 ms·

> Path MTU discovery has not been enthusiastically embraced Ugh. I don't understand this. Especially passive PMTUD should just be rolled out everywhere. On

> Path MTU discovery has not been enthusiastically embraced Ugh. I don't understand this. Especially passive PMTUD should just be rolled out everywhere. On Linux it still defaults to disabled! https://sourcegraph.com/search?q=context%3Aglobal+repo%3A%5Egithub%5C.com%2Ftorvalds%2Flinux%24+tcp_mtu_probing&patternType=keyword&sm=0&__cc=1 https://sourcegraph.com/search?q=context%3Aglobal+repo%3A%5E...

mkj 2y ago
Would that help with UDP, or only TCP?
- deleted 2y ago
  [deleted]
- ajb 2y ago
  That particular one, only TCP. There is a different one for UDP applications: https://www.rfc-editor.org/rfc/rfc8899 https://www.rfc-editor.org/rfc/rfc8899 Because UDP is only a very thin layer, each layer on top (eg, QUIC) has to implement PLPMTUD; although, recently IETF standardised a way to extend UDP to have options and PLPTMUD is also specified for that: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-udp-options-dplpmtud/ https://datatracker.ietf.org/doc/draft-ietf-tsvwg-udp-option...
- cryptonector 2y ago
  You can implement passive PMTUD with UDP if you like. It's more work for you, but it's perfectly doable.
whiatp 2y ago
PMTU just doesn't feel reliable to me because of poorly behaved boxes in the middle. The worst offender I've had to deal with was AWS Transit Gateway, which just doesn't bother sending ICMP too big messages. The second worst offender is, IMO (data center and ISP) routers that generate ICMP replies in their CPU, meaning large packets hit a rate limited exception punt path out of the switch ASIC over to the cheapest CPU they could find to put in the box. If too many people are hitting that path at the same time, (maybe) no reply for you. More rare cases, but really frustrating to debug was when we had an L2 switch in the path with lower MTU than the routers it was joining together. Without an IP level stack, there is no generation of ICMP messages and that thing just ate larger packets. The even stranger case was when there was a Linux box doing forwarding that had segment offload left on. It was taking in several 1500 byte TCP packets from one side, smashing them into ~9000 byte monsters, and then tried to send those over a VPNish network interface that absolutely couldn't handle that. Even if the network in the middle bothered to generate the ICMP too big message, the source would have been thoroughly confused because it never sent anything over 1500.
- Hikikomori 2y ago
  They recently started supporting pmtud on tgw. But it wasn't a big deal really as it adjusted mss instead.
- immibis 2y ago
  L2 not generating errors is expected behaviour - all ports on the L2 network are supposed to have the same MTU set
- toast0 2y ago
  > The even stranger case was when there was a Linux box doing forwarding that had segment offload left on. It was taking in several 1500 byte TCP packets from one side, smashing them into ~9000 byte monsters, and then tried to send those over a VPNish network interface that absolutely couldn't handle that. Even if the network in the middle bothered to generate the ICMP too big message, the source would have been thoroughly confused because it never sent anything over 1500. This is an old Linux tcp offloading bug; large receive offload smooshes the inbound packet, then it's too big to forward. I had to track down the other side of this. FreeBSD used to resend the whole send queue if it got a too big message, even if the size did not change. Sending all at once made it pretty likely for the broken forwarder to get packets close enough to do LRO, which resulted in large enough packet sending to show up as network problems. I don't remember where the forwarder seemed to be, somewhere far away, IIRC.
- cryptonector 2y ago
  > PMTU just doesn't feel reliable to me because of poorly behaved boxes in the middle. The worst offender I've had to deal with was AWS Transit Gateway, which just doesn't bother sending ICMP too big messages. Passive PMTUD does NOT depend on ICMP messages.
JackSlateur 2y ago
path mtu discovery is worthless because the sending host does not control the path used So it is not compatible with anycast, for instance, which is massively used everywhere In the end, having no answer is better than having a most likely wrong answer

mkj 2y ago

Would that help with UDP, or only TCP?

deleted 2y ago
[deleted]
ajb 2y ago
That particular one, only TCP. There is a different one for UDP applications: https://www.rfc-editor.org/rfc/rfc8899 https://www.rfc-editor.org/rfc/rfc8899 Because UDP is only a very thin layer, each layer on top (eg, QUIC) has to implement PLPMTUD; although, recently IETF standardised a way to extend UDP to have options and PLPTMUD is also specified for that: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-udp-options-dplpmtud/ https://datatracker.ietf.org/doc/draft-ietf-tsvwg-udp-option...
cryptonector 2y ago
You can implement passive PMTUD with UDP if you like. It's more work for you, but it's perfectly doable.

whiatp 2y ago

PMTU just doesn't feel reliable to me because of poorly behaved boxes in the middle. The worst offender I've had to deal with was AWS Transit Gateway, which just doesn't bother sending ICMP too big messages. The second worst offender is, IMO (data center and ISP) routers that generate ICMP replies in their CPU, meaning large packets hit a rate limited exception punt path out of the switch ASIC over to the cheapest CPU they could find to put in the box. If too many people are hitting that path at the same time, (maybe) no reply for you. More rare cases, but really frustrating to debug was when we had an L2 switch in the path with lower MTU than the routers it was joining together. Without an IP level stack, there is no generation of ICMP messages and that thing just ate larger packets. The even stranger case was when there was a Linux box doing forwarding that had segment offload left on. It was taking in several 1500 byte TCP packets from one side, smashing them into ~9000 byte monsters, and then tried to send those over a VPNish network interface that absolutely couldn't handle that. Even if the network in the middle bothered to generate the ICMP too big message, the source would have been thoroughly confused because it never sent anything over 1500.

Hikikomori 2y ago
They recently started supporting pmtud on tgw. But it wasn't a big deal really as it adjusted mss instead.
immibis 2y ago
L2 not generating errors is expected behaviour - all ports on the L2 network are supposed to have the same MTU set
toast0 2y ago
> The even stranger case was when there was a Linux box doing forwarding that had segment offload left on. It was taking in several 1500 byte TCP packets from one side, smashing them into ~9000 byte monsters, and then tried to send those over a VPNish network interface that absolutely couldn't handle that. Even if the network in the middle bothered to generate the ICMP too big message, the source would have been thoroughly confused because it never sent anything over 1500. This is an old Linux tcp offloading bug; large receive offload smooshes the inbound packet, then it's too big to forward. I had to track down the other side of this. FreeBSD used to resend the whole send queue if it got a too big message, even if the size did not change. Sending all at once made it pretty likely for the broken forwarder to get packets close enough to do LRO, which resulted in large enough packet sending to show up as network problems. I don't remember where the forwarder seemed to be, somewhere far away, IIRC.
cryptonector 2y ago
> PMTU just doesn't feel reliable to me because of poorly behaved boxes in the middle. The worst offender I've had to deal with was AWS Transit Gateway, which just doesn't bother sending ICMP too big messages. Passive PMTUD does NOT depend on ICMP messages.

JackSlateur 2y ago

path mtu discovery is worthless because the sending host does not control the path used So it is not compatible with anycast, for instance, which is massively used everywhere In the end, having no answer is better than having a most likely wrong answer