11 ms·
So you think you understand IP fragmentation?
- Eduard 3y agodoes it even make sense to probe the path MTU on a packet-switched network such as the Internet? In general, the packets of a message could each go a different path with vastly different MTUs.
- metaxy2 3y ago> In my experience, it is rare for a network to correctly generate Time Exceeded messages for both IPv4 and IPv6. Doesn't that make it more one of those situations where the non-documented behavior has become the de facto standard, rather than "wrong" exactly? (I guess it depends on whether that decision is being made consciously by the implementors or just for lack of knowledge of the standards.)
- toast0 3y agoPeople who filter out all ICMP are probably unaware of the standard, but router implementors that limit ICMP rates are balancing transparent observability with the need to keep the equipment running. I guess you could provision the router cpus so they could send ICMPs for line rate incoming packets that must be dropped, but that doesn't seem like a good cost tradeoff.
- toast0 3y ago> Worse, fragments are more likely to be lost. Many routers and firewalls treat fragments as a security risk because they don't include the information from higher-level protocols like TCP or UDP and can't be filtered based on port, so they drop all IP fragments. I've seen worse than that. A firewall dropping the first fragment based on the UDP port number (which is available in the first fragment), but allowing further fragments. I'd love to see their new discovery algorithm get widely distributed, it's 2024, and a lot of stuff still breaks or suffers terrible delays if I don't apply the proper settings with my 1492 MTU.
- gpderetta 3y agoThat's seems a great way to bypass such a firewall.
- geraldhh 3y agoit's still useable because most ppl just don't care
- mzs 3y agoThat could be an old BSD bug. When there is no ARP entry the first fragment is dropped before the ARP entry is cached. That made it into many BDS based network software stacks before it was fixed. I just use 1024 since nobody seems to use SLIP anymore. It should fit under 1200 with headers and the logic is similar to DEC 512+64 limit arrived at initially. All the PMTU detection algos suffer from something lowering the MTU along a route for a long lasting connection.
- toast0 3y agoThis was chargen servers on random Windows boxes. Hitting me with tons of 64k UDP packets, so they would would get fragmented, but I almost never got the first fragment. This is, of course, nonsense on so many levels, but it was very effective, because my servers at the time had a ridiculous IP fragment reassembly buffer (autoscaled based on system ram size, with a formula written decades ago) and reassembly did a linear search of the buffer. No big deal in normal conditions when we got one or two fragmented packets a minute; immensely terrible when getting a couple GBps of fragmented chargen reflection that could never be reassembled. Setting the reassembly buffer to the minimum solved the problem; during DDoS attacks, legitimate fragments wouldn't be reassembled, but oh well, so many other services don't ever assemble fragments, we did the best we could.
- ardel95 3y agoOne of the biggest misses with IP fragmentation was not requiring each fragment to carry the higher protocol header. Or at least do that for UDP. That decision alone would’ve made fragments so much simpler on network devices and appliances, and much less likely for them to get dropped.
- wmf 3y agoThat would be a layering violation. IP routers don't necessarily know about higher protocols.
- tptacek 3y agoYou could design a network protocol that fragments by capturing a variable number of bytes from the next header, and ICMP already does something like that. (None of this would fix the real problem with fragmentation, which is that you can't efficiently segment out a large frame without having some kind of reliability layer).
- raggi 3y agoIf I was revisiting, I'd probably eradicate the layer and pick a fixed number of flow types with distinct headers and state machines. The layers were a reasonable choice given the understanding of the time, but in hindsight I think you can make a strong case they're cut at the wrong places.
- Hikikomori 3y agoTheres no strict rules about layers, most routers can and do read info in tcp/udp headers.
- p1mrx 3y agoIf I could go back in time, I would replace fragmentation with in-flight truncation. The receiver sees the exact MTU and can communicate it back to the sender as needed.
- zokier 3y agoWhy would that be preferable over having routers just drop oversized packets and send icmp message to sender?
- kevincox 3y ago1. ICMP messages often don't reliably make it back to the sender. This is especially true with low level load balancers. 2. You at least get some information. Maybe half a packet is better than no packet. 3. You get an exact size limit of the entire path the packet took. Rather than just a "too big" of the first hop that wouldn't pass it. Overall I think it is a pretty decent solution. Assuming that the IP header says the original size or similar so that the truncation can be easily detected. I do see some flaws though: 1. Layered protocols would have to be more complex to deal with packet truncation. (Although I guess they could simply treat truncation the same as dropped to avoid extra complexity.) 2. Checksums at the end of a packet being dropped likely make the whole packet useless. 3. It may be better to get a quick rejection from a local router rather than the possibly far away peer.
- amluto 3y agoI'll add an extra bonus: every now and then, an endpoint could append one extra bonus byte to a transmitted packet. That would probe for a potential increase in PMTU at very little cost.
- zamadatix 3y agoYeah, maybe a "contains MTU dummy data" bit at the start of the header which lets the receiving client know that this packet 1) May contain additional data after the real payload 2) If it does, send back an updated max packet size based on the amount that made it through. IETF folks wouldn't like that much state in the IP protocol and it doesn't solve intermediate L2 loss (the real reason you have IP MTU differences) when switches are involved in the path though. That said, might make in interesting ICMP probing method.
- egberts1 3y agoI grok IP-fragmentation because IDS/IPS/NDS/XDS all have to splice packets together to get to the payload data ... at 5Gbps. At that nosebleed speed, FPGA is designed to do such defragmentation of IP as well as assembly of UDP reassembly and TCP desegmentation. Very fast. Including a bulk of FPGA dedicated to the overlapping payload that hackers often play tricks on such IDS, et. al. That is probably the ONLY beef I have with RFCs covering IP/TCP/UDP: did not detail what to do in event of overlapping packets (first overlapped takes precedence or last overlapped overrides old data?)
- cesarb 3y ago> That is probably the ONLY beef I have with RFCs covering IP/TCP/UDP: did not detail what to do in event of overlapping packets (first overlapped takes precedence or last overlapped overrides old data?) For a correctly operating sender, it shouldn't matter, since the data should be identical. It would be the same as asking what should happen if the sender changes the data when retransmitting a packet (the only difference from overlapping fragments is that a retransmitted packet has 100% overlap). It's Undefined Behavior, the receiver is allowed to do anything it wants, and the sender can't complain since it's its own fault. (Also consider that packets are always allowed to be reordered in transit, so what you thought was the first packet might become the last packet on the next hop; even if the standard constrained the reassembly ordering, you might still not get the result you expected.)
- Arnavion 3y agoI believe egberts1's point is that leaving it unspecified invites different implementations to implement it differently. Some might choose to use the bytes from the first fragment, others might choose to use the bytes from the newer fragment. You could have a situation where the IDS / firewall / misc security appliance think the packets contain a benign request but the application server interprets them in a malicious way. Things like HTTP request smuggling rely on the same kind of mismatch at the application protocol layer, for example.
- raggi 3y ago
- zvmaz 3y ago> So you think you understand IP fragmentation? Titles like these sound unnecessarily arrogant to me and are off-putting.
- RustyRussell 3y agoPerhaps. I would have phrased it "Nobody understands IP fragmentation", but sometimes when delivering to a room of experts you need to increase the volume a little bit. Of course, it really depends on whether you can actually back this up! And in this case Val does, quite clearly.
- zvmaz 3y ago> Of course, it really depends on whether you can actually back this up! And in this case Val does, quite clearly. Yes indeed; and reading the article, she does not sound that arrogant: "Many networking experts think they know when IP fragmentation will happen, and I thought I did too—until I had to implement an algorithm for a VPN"
- zokier 3y agoIs there any place for IP fragmentation anymore? On surface level look, the main motivation seems to have been fitting over-sized dnssec responses in single datagram, but with ecdsa/eddsa is that still a relevant concern? I realize some TXT etc data might be big, but presumably those could also be queried over TCP/HTTP/QUIC/etc; I see UDP more as the fast path for common A/AAAA queries, and those should fit in unfragmented packets just fine? What are other cases where fragmentation truly makes sense?
- convolvatron 3y agowell, if you read the article. the issue isn't really fragmentation as a mechanism, its finding the appropriate (dynamic) path MTU. that's kinda gonna be a problem forever given the stateless nature of IP. it would be fine if people hadn't decided that ICMP was .. undesirable. I guess you could just say '1480 for all time' and be done with it
- tptacek 3y agoThere has never really been a place for it; "Fragmentation Considered Harmful" is one of the original and most famous "Considered Harmfuls" and it's from the late 1980s. A lot of protocol engineering, from MSS options to ENDS0, goes into ensuring that you never hit the fragmentation case in the first place. It's been kept around as a mechanism of last resort for weirdo hops with bizarro MTUs.
- rayiner 3y agoI’m surprised by the willful disregard of the feature, e.g., with some routers just dropping fragmented packets. Sending packets bigger than the MTU seems like a reasonable thing to want to do—especially given that IP by design won’t guarantee a stable path between endpoints. Was the reasoning that it’s always better handled at a higher protocol layer?
- tptacek 3y agoYes, that's the reasoning. Fragmentation is kind of a performative half-measure. You're not really enabling delivery across varying network links, in that the performance is so bad (necessarily!) that it alters the service model for many protocols. IPv6 moves to an end-to-end fragmentation model, but even then the right answer is to negotiate in a higher level protocol a maximum segment size for the path you're talking on, and then just avoid fragmentation entirely. Fragmentation is an absolutely wretched stream transport protocol!
- zamadatix 3y agoFragmentation/MTU is right up there with verifying delivery as one of those things that can sound like a really easy problem to solve until you start trying to solve it.
- peter_d_sherman 3y agoI wonder if IP fragmentation would/could have any effect on the ability for a Nation State's (i.e., China, Russia, North Korea, ?, ???) firewall to prevent packets from entering/leaving a blocked country... Why or why not? In fact, now that I think about it... what happens if a TCP/IP connection is initiated from another country to a TCP/IP address inside of an outwardly blocked country? Would that act similar to a NAT punchthrough, but on a Nation-State's firewall? Think about it this way... let's take an arbitrary example, China... And let's suppose for the purposes of discussion that what the western Mainstream Media says about China's Internet is completely true -- that outgoing connections to the West from China -- are blocked. OK, but there still is Alibaba -- the equivalent of China's Amazon.com. Connections FROM the West (e-commerce connections, "we buy from you", "you are a merchant to us", etc., etc.) TO Alibaba.com (in China) -- are not blocked, and would NOT be blocked by the Chinese government! Why? Because anywhere there's trade -- if the trade is beneficial to the government in question (anyone heard of taxes, anyone? They benefit the government you know!) -- there will be internal political pressure NOT to prevent that trade from occurring! So let's say that a TCP/IP connection that looked like a legitimate e-commerce connection -- was initiated FROM the West, to an IP address in China for a legitimate Chinese e-commerce site... Now, couldn't that connection, once established (since it is, after all bi-directional) -- be used to send data outside of China? Yes! But to only one outside/external IP address! But what if that IP address on the Western side (being in a non-firewalled country) -- now could somehow route that data to any other IP address around the world? Sort of like a VPN -- but the connection is opened up from an INCOMING connection, not an outgoing one... Would that be the equivalent of NAT punchthrough -- for a Nation State's firewall? ? Before you answer, I'm guessing that there are Deep State bots (from all countries!) that will try to derail this conversation... That's a testament to "always think for yourself" (let logic be your guide!) -- stay away from so called "expert" opinions, especially one or two sentence "no it can't be done" or "it would never work" replies by fake posters with fake names from fake accounts! But, I guess we'll open up the floor to the AI powered agenda-driven bots... foreign and domestic! :-) Also, note that I don't suggest/condone that anyone actually do any of the above -- this discussion is theoretical in scope only!
- vitus 3y ago> Connections FROM the West (e-commerce connections, "we buy from you", "you are a merchant to us", etc., etc.) TO Alibaba.com (in China) -- are not blocked, and would NOT be blocked by the Chinese government! Um, conspiracy theories aside, Alibaba has servers in the US.
- fch42 3y agoI'm amazed by this; call it "PMTUD Broadside" or some such. The idea is really a bit egg-of-columbus. Anyway, it tickles my geek senses.
- raggi 3y agoHaha, I like broadside. I had to lookup egg of columbus, TIL
- ay 3y agoIt was fun to see a lot of flak towards fragments, when really just about everything you access today is delivered in fragments called TCP segments. Sure, the delivery mechanisms are quite a bit different - but not that much if we were to speak of a case of a packet fitting into two fragments…
- deathanatos 3y agoWhat I'm gathering from the quiz is that the documentation for IP_MTU_DISCOVER is terribly written. > IP_PMTUDISC_WANT will fragment a datagram if needed according to the path MTU, or will set the don't-fragment flag otherwise. And then the quiz asks for the DF bit in the former case. The docs don't say; one could illogically assume the DF is unset in the former case … and in fact, that seems to be the case, though this leaves me wondering why it would do that, since with DF unset, it won't elicit the needed ICMP responses. There still remain some cases, of course, where we could do that (the latter case in the docs) … but this seems inefficient. In the former case, why wouldn't we fragment according to the PMTU and just set DF in all cases?
- zrm 3y agoIs it permissible to set DF on a datagram which is already fragmented? Or if not formally impermissible, does it trigger some interesting bugs in something that assumes DF set = not fragmented?
- LegionMammal978 3y agoThe IPv4 RFC doesn't have anything precluding DF on a fragment. Since each fragment is a datagram in its own right, adding a DF just means that it can't be fragmented even further. The example fragmentation algorithm only considers DF on a per-datagram basis, regardless of whether or not it is a fragment of some original datagram. (It does take care to forward the original offsets and MF flag correctly, to account for the datagram being a fragment.)