5 ms·

This discussion is about truncating at L3, not L2; it assumes that the L2 MTU is fixed and known by every directly connected router, but a L3 packet transits ov

by cesarb 3y ago

This discussion is about truncating at L3, not L2; it assumes that the L2 MTU is fixed and known by every directly connected router, but a L3 packet transits over several L2 networks, each with its own MTU. A L2 network with variable MTU (or a MTU too small for Internet packets, like ATM) has its own independent segmentation and reassembly logic.

zamadatix 3y ago
Well that's the thing, for the real world the L3's MTU is limited by the L2 MTU. If you can guarantee L2 MTU will be fixed and known for every device along the path then there is not really a need for a variable L3 MTU anymore and you've solved the problem by convincing everyone in the world they'll never need or want a different MTU again. Then someone gets the bright idea they can do something with tunnels and it starts back up. If you want a world where you can rely on an L2 to have a way to handle the MTU mismatch then nothing is being solved by this other than the ability to say "fragmentation is not not in the current abstraction layer anymore, it's now an annoying problem on the layer below it instead" :).
- cesarb 3y ago
  > If you can guarantee L2 MTU will be fixed and known for every device along the path then there is not really a need for a variable L3 MTU anymore The L2 MTU is fixed and known for every device on that same L2 network. The Internet is a network of networks; the reason we need to find the path MTU on the L3 network is that a single packet transits over several L2 networks, and while each L2 network has a single MTU, the L3 path can vary.
  - zamadatix 3y ago
    The different L3 networks are connected by L2 networks in-between them, not just behind them. L2 doesn't disappear once you hit the router, you've still got to reach your peer's IP over L1/L2 somehow at some point. A great deal of internet peering is not even a L3 box <- direct L1+L2 connection -> L3 box connection, it's dumb L2 transport which provides the IP connectivity path for the BGP session to a more centralized router. Sometimes the path is a pseudowire doing the same with a functional MTU lower than the switched/routed path it rides on too. These underlying paths are not always static either, just because you have 9000 today doesn't mean when the path fails to a backup alternate tomorrow the MTU will be 9000. You have to get everyone involved on the internet to agree all links should now be 9000, now you can reliably set your router's outbound link to 9000 in all cases and rely on L3 fragmentation. Until someone wants to set their MTU to 12000 :). Even when I've had paid WAN transport with contracted MTU the MTU has lowered during carrier maintenance like firmware upgrades or unit replacements and I've had to call them up saying the MTU is broken because something like NFS servers will think they can statically set a known MTU on the path and it'll stay that way, a routed neighbor on the path would make the same error in this example.
- toast0 3y ago
  At the time of sending the L2 packet, the router must know the MTU of the link it's sending it on. This is the case with router based fragmentation, otherwise it wouldn't fragment. This is also the case where the router drops packets that are too big, otherwise what does it do; assuming you don't build routers where all MTUs on all interfaces are required to be the same. The proposal is that in the case that the router is forwarding a packet which won't fit in the egress MTU, it truncates it, rather than drops it. Hopefully maybe with a truncated bit (maybe we can reuse the evil bit for this). If a peer receives a truncated packet, which it should know because either the evil bit is set, or the received length doesn't match the length in the IP header, that would be processed depending on the protocol. For TCP, the receiver should send a duplicate ack with a newly specified option that indicates the length of the truncated packet it received; then the other peer would know the maximum length packet it could send. For UDP, the truncated packet would probably need to be sent to the application and it could figure out how to notify the peer to send shorter packets (and perhaps process the received truncated packet). Had this been specified in the before times, it would be reasonable to implement, and path mtu detection would work a lot better now. Some routers may still have been built that drop rather than truncate, but it's a lot more reasonable to send a smaller packet in some circumstances than to have the router fragment things (needs to send two packets sometimes instead of one) or to send a return packet (needs to create a packet and start it back through the routing process again). Additionally, all of the signalling is in-band, which makes it harder to lose the MTU signals when (small) data is otherwise flowing.
  - zamadatix 3y ago
    I already described the problem with assuming "the router must know the MTU of the link it's sending it on" implies guaranteed delivery across a switched path a comment higher in the chain: router A <-> switch E <-> switch C <-> router D Say router A and switch E have a 9000 byte MTU but switch C has an MTU of 1500 bytes. Router A gets a 12000 byte IP packet and sees to truncate the IP packet to 9000 bytes to be able to egress it to switch E, the designated path to its routed peer D. Switch E accepts this 9000 byte packet and bridges to switch C. Switch C only knows how to accept 1500 byte packets and so drops the 9000 byte packet. No truncated IP packet is ever delivered and the MTU is never learned. This scenario, and a few others, are already noted in the RFC mentioned in the article https://datatracker.ietf.org/doc/html/rfc8899 https://datatracker.ietf.org/doc/html/rfc8899 as problems classically run into when trying to do PMTUD. Your browser based PMTUD test relies on these principles when it sends packets of various size, that's the only foolproof PMTUD method there is - truncating at egress on only routed hops does not provide foolproof IP delivery across the internet so cannot provide foolproof PMTUD. Even when you can take the liberty to 100% guarantee A, C, E, and D will all support an MTU of 9000 bytes you can't guarantee nowhere else on your path across the internet will have the above issue on their switched paths between routers. Or that your packet won't go through a non-IP protocol tunnel as it goes through the internet and get dropped as too big in that encapsulation. Truncation is only 100% reliable when you're already 100% certain the real physical path across the internet can handle that size, which in practice requires knowing the real dynamic PMTU across multiple parties i.e. having PMTUD already.

by cesarb 3y ago

zamadatix 3y ago

Well that's the thing, for the real world the L3's MTU is limited by the L2 MTU. If you can guarantee L2 MTU will be fixed and known for every device along the path then there is not really a need for a variable L3 MTU anymore and you've solved the problem by convincing everyone in the world they'll never need or want a different MTU again. Then someone gets the bright idea they can do something with tunnels and it starts back up. If you want a world where you can rely on an L2 to have a way to handle the MTU mismatch then nothing is being solved by this other than the ability to say "fragmentation is not not in the current abstraction layer anymore, it's now an annoying problem on the layer below it instead" :).

cesarb 3y ago
> If you can guarantee L2 MTU will be fixed and known for every device along the path then there is not really a need for a variable L3 MTU anymore The L2 MTU is fixed and known for every device on that same L2 network. The Internet is a network of networks; the reason we need to find the path MTU on the L3 network is that a single packet transits over several L2 networks, and while each L2 network has a single MTU, the L3 path can vary.
- zamadatix 3y ago
  The different L3 networks are connected by L2 networks in-between them, not just behind them. L2 doesn't disappear once you hit the router, you've still got to reach your peer's IP over L1/L2 somehow at some point. A great deal of internet peering is not even a L3 box <- direct L1+L2 connection -> L3 box connection, it's dumb L2 transport which provides the IP connectivity path for the BGP session to a more centralized router. Sometimes the path is a pseudowire doing the same with a functional MTU lower than the switched/routed path it rides on too. These underlying paths are not always static either, just because you have 9000 today doesn't mean when the path fails to a backup alternate tomorrow the MTU will be 9000. You have to get everyone involved on the internet to agree all links should now be 9000, now you can reliably set your router's outbound link to 9000 in all cases and rely on L3 fragmentation. Until someone wants to set their MTU to 12000 :). Even when I've had paid WAN transport with contracted MTU the MTU has lowered during carrier maintenance like firmware upgrades or unit replacements and I've had to call them up saying the MTU is broken because something like NFS servers will think they can statically set a known MTU on the path and it'll stay that way, a routed neighbor on the path would make the same error in this example.
toast0 3y ago
At the time of sending the L2 packet, the router must know the MTU of the link it's sending it on. This is the case with router based fragmentation, otherwise it wouldn't fragment. This is also the case where the router drops packets that are too big, otherwise what does it do; assuming you don't build routers where all MTUs on all interfaces are required to be the same. The proposal is that in the case that the router is forwarding a packet which won't fit in the egress MTU, it truncates it, rather than drops it. Hopefully maybe with a truncated bit (maybe we can reuse the evil bit for this). If a peer receives a truncated packet, which it should know because either the evil bit is set, or the received length doesn't match the length in the IP header, that would be processed depending on the protocol. For TCP, the receiver should send a duplicate ack with a newly specified option that indicates the length of the truncated packet it received; then the other peer would know the maximum length packet it could send. For UDP, the truncated packet would probably need to be sent to the application and it could figure out how to notify the peer to send shorter packets (and perhaps process the received truncated packet). Had this been specified in the before times, it would be reasonable to implement, and path mtu detection would work a lot better now. Some routers may still have been built that drop rather than truncate, but it's a lot more reasonable to send a smaller packet in some circumstances than to have the router fragment things (needs to send two packets sometimes instead of one) or to send a return packet (needs to create a packet and start it back through the routing process again). Additionally, all of the signalling is in-band, which makes it harder to lose the MTU signals when (small) data is otherwise flowing.
- zamadatix 3y ago
  I already described the problem with assuming "the router must know the MTU of the link it's sending it on" implies guaranteed delivery across a switched path a comment higher in the chain: router A <-> switch E <-> switch C <-> router D Say router A and switch E have a 9000 byte MTU but switch C has an MTU of 1500 bytes. Router A gets a 12000 byte IP packet and sees to truncate the IP packet to 9000 bytes to be able to egress it to switch E, the designated path to its routed peer D. Switch E accepts this 9000 byte packet and bridges to switch C. Switch C only knows how to accept 1500 byte packets and so drops the 9000 byte packet. No truncated IP packet is ever delivered and the MTU is never learned. This scenario, and a few others, are already noted in the RFC mentioned in the article https://datatracker.ietf.org/doc/html/rfc8899 https://datatracker.ietf.org/doc/html/rfc8899 as problems classically run into when trying to do PMTUD. Your browser based PMTUD test relies on these principles when it sends packets of various size, that's the only foolproof PMTUD method there is - truncating at egress on only routed hops does not provide foolproof IP delivery across the internet so cannot provide foolproof PMTUD. Even when you can take the liberty to 100% guarantee A, C, E, and D will all support an MTU of 9000 bytes you can't guarantee nowhere else on your path across the internet will have the above issue on their switched paths between routers. Or that your packet won't go through a non-IP protocol tunnel as it goes through the internet and get dropped as too big in that encapsulation. Truncation is only 100% reliable when you're already 100% certain the real physical path across the internet can handle that size, which in practice requires knowing the real dynamic PMTU across multiple parties i.e. having PMTUD already.