3 ms·

We were doing testing several years ago with a custom UDP based protocol to overcome some of the TCP issues, and ran into a perplexing problem. To meet the dat

We were doing testing several years ago with a custom UDP based protocol to overcome some of the TCP issues, and ran into a perplexing problem. To meet the data rate, we were sending datagrams that were 64KB in size (there was no system call at this time to send multiple datagrams in one go). The datagrams were being fragmented into MTU sized fragments by the OS and reassembled on the remote end. However, with the data rate and the fact that there is only a 16 bit IP ID in the IP header, the fragments would often get missassembled on the receiving end and passed up to the UDP layer. Usually, the UDP checksum would detect the problem and throw the datagram out. However, since the checksum is weak and we were sending a lot of data, it would sometimes pass the datagram up to the application layer. So, we would get corrupted data on the receiving end once in a while. It took quite a bit of time to figure this out, of course. The solution was to implement dynamic MTU discovery along with the protocol so we could avoid the fragmentation. The custom solution is still being used 20 years later!

r1ch 8y ago
Fragmentation by the IP layer is very unreliable in today's internet. I've seen plenty of consumer routers that throw away the whole datagram if a fragment arrives out-of-order, and who knows what kind of hell corporate middleboxes and other similar systems do to them. Pretty much every protocol these days goes to great lengths to avoid upper level fragmentation.
- jchb 8y ago
  IPv6 helps in that it mandates a MTU of at least 1280 bytes. In addition, in IPv6 only source nodes can break a packet into fragments - intermediary nodes such as firewalls and routers cannot. Sending back a ICMPv6 Packet Too Big for packets larger than the MTU is a must, which helps with path MTU discovery.
nly 8y ago
Sounds like a bad fragmentation implementation. At 64KB datagrams you have to send over 4GB of data before you need to recycle an ID Even if your packet ids are assigned poorly, on the receiving side Linux has a heuristic that takes in to account reordering to minimize the chance of reassembling bad packets. Basically if packets with the same frag id are too far apart then the whole datagram is chucked: https://marc.info/?l=linux-netdev&m=113267833031122&w=2 https://marc.info/?l=linux-netdev&m=113267833031122&w=2

r1ch 8y ago

Fragmentation by the IP layer is very unreliable in today's internet. I've seen plenty of consumer routers that throw away the whole datagram if a fragment arrives out-of-order, and who knows what kind of hell corporate middleboxes and other similar systems do to them. Pretty much every protocol these days goes to great lengths to avoid upper level fragmentation.

jchb 8y ago
IPv6 helps in that it mandates a MTU of at least 1280 bytes. In addition, in IPv6 only source nodes can break a packet into fragments - intermediary nodes such as firewalls and routers cannot. Sending back a ICMPv6 Packet Too Big for packets larger than the MTU is a must, which helps with path MTU discovery.

nly 8y ago

Sounds like a bad fragmentation implementation. At 64KB datagrams you have to send over 4GB of data before you need to recycle an ID Even if your packet ids are assigned poorly, on the receiving side Linux has a heuristic that takes in to account reordering to minimize the chance of reassembling bad packets. Basically if packets with the same frag id are too far apart then the whole datagram is chucked: https://marc.info/?l=linux-netdev&m=113267833031122&w=2 https://marc.info/?l=linux-netdev&m=113267833031122&w=2