5 ms·

Is there any place for IP fragmentation anymore? On surface level look, the main motivation seems to have been fitting over-sized dnssec responses in single dat

by zokier 3y ago

Is there any place for IP fragmentation anymore? On surface level look, the main motivation seems to have been fitting over-sized dnssec responses in single datagram, but with ecdsa/eddsa is that still a relevant concern? I realize some TXT etc data might be big, but presumably those could also be queried over TCP/HTTP/QUIC/etc; I see UDP more as the fast path for common A/AAAA queries, and those should fit in unfragmented packets just fine? What are other cases where fragmentation truly makes sense?

convolvatron 3y ago
well, if you read the article. the issue isn't really fragmentation as a mechanism, its finding the appropriate (dynamic) path MTU. that's kinda gonna be a problem forever given the stateless nature of IP. it would be fine if people hadn't decided that ICMP was .. undesirable. I guess you could just say '1480 for all time' and be done with it
tptacek 3y ago
There has never really been a place for it; "Fragmentation Considered Harmful" is one of the original and most famous "Considered Harmfuls" and it's from the late 1980s. A lot of protocol engineering, from MSS options to ENDS0, goes into ensuring that you never hit the fragmentation case in the first place. It's been kept around as a mechanism of last resort for weirdo hops with bizarro MTUs.
- rayiner 3y ago
  I’m surprised by the willful disregard of the feature, e.g., with some routers just dropping fragmented packets. Sending packets bigger than the MTU seems like a reasonable thing to want to do—especially given that IP by design won’t guarantee a stable path between endpoints. Was the reasoning that it’s always better handled at a higher protocol layer?
  - tptacek 3y ago
    Yes, that's the reasoning. Fragmentation is kind of a performative half-measure. You're not really enabling delivery across varying network links, in that the performance is so bad (necessarily!) that it alters the service model for many protocols. IPv6 moves to an end-to-end fragmentation model, but even then the right answer is to negotiate in a higher level protocol a maximum segment size for the path you're talking on, and then just avoid fragmentation entirely. Fragmentation is an absolutely wretched stream transport protocol!
    - raggi 3y ago
      the messy challenge here is that the trade-offs shifted, but persist contrarian arguments. packets are too small to be efficient without doing excessively tight loop optimizations, which is what makes fragmentation "slow" (it's not such a big slow down in a simple software implementation, it's a disaster in a highly common case optimized device). On the other side in order to move away from those excessively optimized systems while still delivering on user demand for higher throughput we need larger payloads - but we can't get to larger payloads without transparent solutions for fixup (ICMP, fragmentation, etc). Sprinkle in some stuff that has never been formally fixed (bad parts of ICMP concerns), and it's an ongoing recipe for ossified non-progress.
      - tptacek 3y ago
        Fragmentation isn't slow because routers are bad at fragmenting (though: they are), they're slow because the loss of any one fragment forces the discarding of the whole packet. Because you can't possibly make a transport protocol as dumb as fragmentation reassembly fast, forwarders don't optimize it.
        raggi 3y ago
        The picture in the case of losses is pretty complicated with dack, sack, congestion control, and so on. If loss is high enough that this is a main concern TCP performance is generally shot already.
  - zokier 3y ago
    I'm not a network engineer, but somehow I imagine that for at least some links fragmentation could be also handled transparently at lower layer, i.e. limit fragmentation to single hops and always reassemble packets before passing them on.
    - Vecr 3y ago
      It is. In my opinion that's what you should do if you can't provide 1500 MTU to your users directly.
      - tptacek 3y ago
        How is fragmenting and reassembling a better option than just communicating what the real MTU is so higher-layer protocols can adjust to it? What is the extra mechanism buying you?
        Vecr 3y ago
        At this point lots of things don't work right otherwise, and assuming your users want to run wireguard over this link that takes them down to 1420, and if you're already at 1420 it takes them down to 1340... Anyway, you can probably use a lot less header overhead if you do it at a lower layer too.
wolletd 3y ago
WireGuard! I had multiple issues with WireGuard related to MTU/fragmentation. Once, service reported that a system is pingable, but the management website didn't load. I was able to SSH into it, but similarily, the connection broke everytime I tried to run `ip addr` or view logs. Commands with short output worked, however – apparently, somewhere on the path the packets got fragmented and/or dropped without notice.
- mjg59 3y ago
  This is going to be true for any VPN - even if it were purely IP in IP encapsulation, you'd need an additional IP header to get the packet to the VPN endpoint, so your payload is smaller. But fragmentation probably isn't the right answer (especially since any packets marked DF will be dropped). If the MTU for your Wireguard interface is set correctly then anything trying to push a 1500 byte packet via a Wireguard interface should get back an ICMP packet telling it that that won't fit and adjust its packet size downwards appropriately. I actually hit this recently, where I wasn't able to stream videos from certain sites over Wireguard. I spend a while with tcpdump and figured out that they were using Fastly as a CDN, Fastly was sending 1500 byte packets marked with Don't Fragment, my Wireguard endpoint was returning a message saying that the maximum packet size for the link was 1460 bytes, and Fastly was then… sending another 1500 byte packet marked Don't Fragment. To their credit when I was able to provide their engineering with logs showing this was clearly a Fastly problem, they fixed it fairly promptly
  - kccqzy 3y ago
    Did they fix it just for you or for everyone? A proposed project at work involves using Fastly as CDN and I'm curious if this is something we need to keep in mind.
    - mjg59 3y ago
      Everyone, as far as I know
  - wolletd 3y ago
    In my case, the application was using the Wireguard MTU correctly, but the Wireguard packet itself was too big for something on the path. So, the Wireguard MTU wasn't set correctly, basically. But Wireguard packets aren't marked DF, so one would expect a worse, but still working connection in that case. However, that something on the path seemingly just dropped the packets without ICMP notice.
wcunning 3y ago
Automotive ethernet, particularly for high data rate sensors like LiDAR or radar in autonomous vehicles have a predefined 1500 byte fragment size and then expect fragmentation of larger datagrams to be done in sensor firmware. This prevents clogging the routers with huge packets and preventing time-critical messages from waiting in a queue too long while a large message sends.
- zokier 3y ago
  But why do it on IP level instead of higher level like everyone else is doing it?
  - aeonik 3y ago
    Why do it at the high level if the protocol supports it? Especially in an automotive system where you don't need to worry about rogue configurations as much.
    - Eduard 3y ago
      quite the opposite: CAN Injection: keyless car theft https://kentindell.github.io/2023/04/03/can-injection/ https://kentindell.github.io/2023/04/03/can-injection/
      - aeonik 3y ago
        I don't follow. I meant why not leverage fragmentation at the routing+transport protocol layer if configs can be controlled to support it and not drop fragmented packets. What advantage is there doing it at the application layer? I'm aware that all the layers can be messed with by rogue malicious actors. Though IIRC CAN doesn't really have a transport or routing layer; the ARB IDs are baked into the physical protocol, which is very cool, but it's fundamentally a bus architecture, and I'm not sure how it would apply to current thread.
- raggi 3y ago
  that concern with these numbers is very surprising, do you know if there’s somewhere with writing about this?
toast0 3y ago
If your path requires fragmentation to get to the wider world, there's a good chance it simply won't work. There's a small chance that it will work with some destinations. I don't think there's really a place for it in that sense. Maybe for UDP for protocols where the designers were really optimistic. IMHO, for DNS if you want it to work as much as possible, your queries and responses should fit in 512 bytes, or some people are going to have problems on some networks. DNS over TCP doesn't work everywhere, even if it's supposed to. It's more palatable to require DNS over TCP or working UDP over fragmented IP if it's for big txt records like for mail, but if your A and/or AAAA records don't fit in 512 bytes, expect lingering problems that are hard to track down.
Retr0id 3y ago
> with ecdsa/eddsa is that still a relevant concern Perhaps not, but with quantum-safe signature algorithms on the horizon, signatures are about to get much longer again.

by zokier 3y ago

convolvatron 3y ago

well, if you read the article. the issue isn't really fragmentation as a mechanism, its finding the appropriate (dynamic) path MTU. that's kinda gonna be a problem forever given the stateless nature of IP. it would be fine if people hadn't decided that ICMP was .. undesirable. I guess you could just say '1480 for all time' and be done with it

tptacek 3y ago

There has never really been a place for it; "Fragmentation Considered Harmful" is one of the original and most famous "Considered Harmfuls" and it's from the late 1980s. A lot of protocol engineering, from MSS options to ENDS0, goes into ensuring that you never hit the fragmentation case in the first place. It's been kept around as a mechanism of last resort for weirdo hops with bizarro MTUs.

rayiner 3y ago
I’m surprised by the willful disregard of the feature, e.g., with some routers just dropping fragmented packets. Sending packets bigger than the MTU seems like a reasonable thing to want to do—especially given that IP by design won’t guarantee a stable path between endpoints. Was the reasoning that it’s always better handled at a higher protocol layer?
- tptacek 3y ago
  Yes, that's the reasoning. Fragmentation is kind of a performative half-measure. You're not really enabling delivery across varying network links, in that the performance is so bad (necessarily!) that it alters the service model for many protocols. IPv6 moves to an end-to-end fragmentation model, but even then the right answer is to negotiate in a higher level protocol a maximum segment size for the path you're talking on, and then just avoid fragmentation entirely. Fragmentation is an absolutely wretched stream transport protocol!
  - raggi 3y ago
    the messy challenge here is that the trade-offs shifted, but persist contrarian arguments. packets are too small to be efficient without doing excessively tight loop optimizations, which is what makes fragmentation "slow" (it's not such a big slow down in a simple software implementation, it's a disaster in a highly common case optimized device). On the other side in order to move away from those excessively optimized systems while still delivering on user demand for higher throughput we need larger payloads - but we can't get to larger payloads without transparent solutions for fixup (ICMP, fragmentation, etc). Sprinkle in some stuff that has never been formally fixed (bad parts of ICMP concerns), and it's an ongoing recipe for ossified non-progress.
    - tptacek 3y ago
      Fragmentation isn't slow because routers are bad at fragmenting (though: they are), they're slow because the loss of any one fragment forces the discarding of the whole packet. Because you can't possibly make a transport protocol as dumb as fragmentation reassembly fast, forwarders don't optimize it.
      - raggi 3y ago
        The picture in the case of losses is pretty complicated with dack, sack, congestion control, and so on. If loss is high enough that this is a main concern TCP performance is generally shot already.
- zokier 3y ago
  I'm not a network engineer, but somehow I imagine that for at least some links fragmentation could be also handled transparently at lower layer, i.e. limit fragmentation to single hops and always reassemble packets before passing them on.
  - Vecr 3y ago
    It is. In my opinion that's what you should do if you can't provide 1500 MTU to your users directly.
    - tptacek 3y ago
      How is fragmenting and reassembling a better option than just communicating what the real MTU is so higher-layer protocols can adjust to it? What is the extra mechanism buying you?
      - Vecr 3y ago
        At this point lots of things don't work right otherwise, and assuming your users want to run wireguard over this link that takes them down to 1420, and if you're already at 1420 it takes them down to 1340... Anyway, you can probably use a lot less header overhead if you do it at a lower layer too.

wolletd 3y ago

WireGuard! I had multiple issues with WireGuard related to MTU/fragmentation. Once, service reported that a system is pingable, but the management website didn't load. I was able to SSH into it, but similarily, the connection broke everytime I tried to run `ip addr` or view logs. Commands with short output worked, however – apparently, somewhere on the path the packets got fragmented and/or dropped without notice.

mjg59 3y ago
This is going to be true for any VPN - even if it were purely IP in IP encapsulation, you'd need an additional IP header to get the packet to the VPN endpoint, so your payload is smaller. But fragmentation probably isn't the right answer (especially since any packets marked DF will be dropped). If the MTU for your Wireguard interface is set correctly then anything trying to push a 1500 byte packet via a Wireguard interface should get back an ICMP packet telling it that that won't fit and adjust its packet size downwards appropriately. I actually hit this recently, where I wasn't able to stream videos from certain sites over Wireguard. I spend a while with tcpdump and figured out that they were using Fastly as a CDN, Fastly was sending 1500 byte packets marked with Don't Fragment, my Wireguard endpoint was returning a message saying that the maximum packet size for the link was 1460 bytes, and Fastly was then… sending another 1500 byte packet marked Don't Fragment. To their credit when I was able to provide their engineering with logs showing this was clearly a Fastly problem, they fixed it fairly promptly
- kccqzy 3y ago
  Did they fix it just for you or for everyone? A proposed project at work involves using Fastly as CDN and I'm curious if this is something we need to keep in mind.
  - mjg59 3y ago
    Everyone, as far as I know
- wolletd 3y ago
  In my case, the application was using the Wireguard MTU correctly, but the Wireguard packet itself was too big for something on the path. So, the Wireguard MTU wasn't set correctly, basically. But Wireguard packets aren't marked DF, so one would expect a worse, but still working connection in that case. However, that something on the path seemingly just dropped the packets without ICMP notice.

wcunning 3y ago

Automotive ethernet, particularly for high data rate sensors like LiDAR or radar in autonomous vehicles have a predefined 1500 byte fragment size and then expect fragmentation of larger datagrams to be done in sensor firmware. This prevents clogging the routers with huge packets and preventing time-critical messages from waiting in a queue too long while a large message sends.

zokier 3y ago
But why do it on IP level instead of higher level like everyone else is doing it?
- aeonik 3y ago
  Why do it at the high level if the protocol supports it? Especially in an automotive system where you don't need to worry about rogue configurations as much.
  - Eduard 3y ago
    quite the opposite: CAN Injection: keyless car theft https://kentindell.github.io/2023/04/03/can-injection/ https://kentindell.github.io/2023/04/03/can-injection/
    - aeonik 3y ago
      I don't follow. I meant why not leverage fragmentation at the routing+transport protocol layer if configs can be controlled to support it and not drop fragmented packets. What advantage is there doing it at the application layer? I'm aware that all the layers can be messed with by rogue malicious actors. Though IIRC CAN doesn't really have a transport or routing layer; the ARB IDs are baked into the physical protocol, which is very cool, but it's fundamentally a bus architecture, and I'm not sure how it would apply to current thread.
raggi 3y ago
that concern with these numbers is very surprising, do you know if there’s somewhere with writing about this?

toast0 3y ago

If your path requires fragmentation to get to the wider world, there's a good chance it simply won't work. There's a small chance that it will work with some destinations. I don't think there's really a place for it in that sense. Maybe for UDP for protocols where the designers were really optimistic. IMHO, for DNS if you want it to work as much as possible, your queries and responses should fit in 512 bytes, or some people are going to have problems on some networks. DNS over TCP doesn't work everywhere, even if it's supposed to. It's more palatable to require DNS over TCP or working UDP over fragmented IP if it's for big txt records like for mail, but if your A and/or AAAA records don't fit in 512 bytes, expect lingering problems that are hard to track down.

Retr0id 3y ago

> with ecdsa/eddsa is that still a relevant concern Perhaps not, but with quantum-safe signature algorithms on the horizon, signatures are about to get much longer again.