4 ms·
A significant wrinkle in how NAT works is IP fragmentation. UDP datagrams can be larger than an IP packet. When that happens the payload is split into multiple
by colmmacc 1y ago
A significant wrinkle in how NAT works is IP fragmentation. UDP datagrams can be larger than an IP packet. When that happens the payload is split into multiple IP packets, but only the first packet has a UDP header in it. The NAT device needs to correlate these packets by looking at fragment IDs, and then rewrite the IP addresses in the headers.
That alone implies a second kind of state to maintain, but it gets worse. Fragments can arrive out of order. If the second or later packets arrive before the first, the NAT device has to buffer those fragments until they get the packet with the UDP header in it.
That might seem unlikely but it's surprisingly common. Modern protocols like DNSSEC do require fragmentation and in a large network with many paths fragments can end up taking different paths from each other.
Ordinarily when a network is using multiple links to load balance traffic, the routers will use flow steering. The routers look at the UDP or TCP header, make a hash of the connection/flow tuple, and then use that hash to pick a link to use. That way, all of the packets from the same connection or flow will be steered down the same link.
IP fragmentation breaks this too. Those second and subsequent packets don't have a UDP header in them, so they can't be flow steered statelessly. Smarter routers are clever enough to realize this from the beginning of the datagram and to only use a 3-tuple hash (source IP, dest IP, protocol) ... so the packets will still flow consistently. But many devices get this wrong - some just even assume there will be a UDP header and pick whatever values happen to be there.
The fragments end up taking different paths and if one link is more congested or latent enough than another, they'll ultimately arrive out of order.
This single wrinkle is probably responsible for half the complexity in a robust NAT implementation. Imagine having to solve for all of this in a highly-available and trasnactionally live-replicated implementation like managed NAT gateways.
Worst of all, this was all avoidable. If UDP datagrams were simply fragmented at the UDP layer, and every packet included a UDP header, none of this would be necessary. It's probably the worst mistake in TCP/IP. But obviously overall, it was a very successful design that brought on the Internet.
- EvanAnderson 1y ago> It's probably the worst mistake in TCP/IP. I vote for TCP/IP lacking a session layer as being the worst mistake. We wouldn't have IP mobility issues if there'd been an explicit session layer to decouple IP from the upper layer protocols.
- mindslight 1y agoThat's like the Nethead vs Bellhead argument though, and it's easy to say that with the benefit of several decades of adoption and development.
- EvanAnderson 1y agoI don't necessarily think a session layer protocol is automatically "Bell-headed". It's a natural place to plug-in per-byte billing and that ilk, for sure. I don't exactly know the timeline between the ITU protocol suite and/or DecNet (both of which have a concept of a session layer protocol) with IPv4. I think they were somewhat contemporaneous. Certainly, the idea of a session layer isn't something that came decades later than IPv4. Even just a host identifier, in lieu of the IP address of an interface, being used in the TCP tuple would have been so much better than what we have and probably would have been enough of a "session layer". It would be so amazing to have TCP connections that "just work" when clients or servers hop onto different IP networks, use different interfaces, etc. Edit: It has been mentioned that Vint Cerf regretted the decision to bind the IP into the TCP tuple, too. I don't have an exact quote but I know I've heard him mention it in a talk. Ref: https://argp.github.io/2006/03/05/vint-cerfs-talk/ https://argp.github.io/2006/03/05/vint-cerfs-talk/
- mindslight 1y ago> Even just a host identifier, in lieu of the IP address of an interface, being used in the TCP tuple would have been so much better What are you imagining as the implementation? Is it just in TCP, and IP (/ the network) is unchanged? I can see the benefit of that, but then there still needs to be some mechanism to change the binding of host->IP. And if it's not part of the core network, then it's not straightforward. There are also other more complex problems not solved by TCP (eg security). I'd rather have a host ID be a pubkey, than some small-namespace ID with a pubkey required on top of that. It feels like the real problem is the proliferation of different incompatible solutions to any of these problems, which was going to happen even if there was one less problem that needed to be solved. Another way of looking at it is that TCP got so entrenched because of NAT, and having a session ID within IP instead of (implicitly) within TCP/UDP might have allowed more flexibility with creating new protocols directly on top of IP. But 2+2 more bytes of addressing would have gone a long way too!
- Bluecobra 1y agoNot sure if I agree with it being the worst mistake. The beauty of UDP is its simplicity and you get the absolute minimum. (And that’s the way I like it!) I’ve worked on low latency financial networks that route 40+ Gb of UDP multicast daily and error free. Nobody is fragmenting UDP packets, and most packet sizes are less than 1000 bytes. All financial exchanges have their own proprietary format, but all use sequence numbers in the data gram to keep track of packets.
- tptacek 1y agoA UDP protocol that deliberately keeps datagram sizes below 1000 bytes to avoid fragmentation is essentially handling fragmentation itself, as Colm proposes UDP should have done to begin with.
- zokier 1y agoIP fragmentation does not really have anything to do with UDP, it can happen regardless of the inner protocol. > Worst of all, this was all avoidable. It is not that simple. To avoid fragmentation you need robust path mtu detection, which is another whole can of worms. Especially when packets can have multiple paths with different mtu.
- deleted 1y ago[deleted]
- zokier 1y ago> It's probably the worst mistake in TCP/IP. If you think fragmentation was mistake then what other alternative do you think would have been better while also feasible at the time when ipv4 was specified? IPv6 notably traded fragmentation for path mtu discovery, but I don't think requiring pmtud would have been realistic option in 1981.
- deleted 1y ago[deleted]
- viveknathani_ 1y agohi! thanks for explaining this bit in detail. i agree, fragmentation should be handled in the transport layer!
- commandersaki 1y agoIf UDP datagrams were simply fragmented at the UDP layer Agreed, but then it wouldn't be a datagram service anymore.