3 ms·

From a software point of view, it has always seemed to me that typical MTUs are an order of magnitude too small: Disk operations as well as CPU operations (aut

From a software point of view, it has always seemed to me that typical MTUs are an order of magnitude too small: Disk operations as well as CPU operations (authentication, encryption, e.g. TLS) all benefit from using larger block sizes of 65536 bytes and up. This amortizes system call overhead, enables larger sequential operations, enables more use of SIMD instructions, reduces instruction cache misses (working on one thing for longer), reduces context switches, and serves to increase the leverage that the control plane has on the data plane. Just one specific example, something like TCP needs to keep track of packets (for retransmission, acks etc.). When you're sending Gbps, small packets mean much more bookkeeping overhead (more method calls, more bits in ack packets, more hash table lookups) compared to large packets. Throughput should be much better with an MTU of 65536 (and latency sensitive applications don't have to send that much). I can imagine that in the past the cost of memory was radically different, implying a smaller MTU, but it seems like sticking with 1500 for IPv6 was an opportunity missed.

drewg123 8y ago
TCP overhead does indeed increase as MTU decreases, but not because "TCP needs to keep track of packets" -- TCP in fact keeps track of bytes, not packets. The reason that overhead is larger for smaller MTUs is simply because of the extra per-packet overheads (looking up connections, allocating memory for ethernet/ip packet headers, (uncontended) locking, repeated function calls to L3/L2 network layers and network drivers, etc). The fact that TCP tracks bytes rather than packets enables hardware offloads like TSO/LSO (TCP Segmentation Offload, known by some as TCP Large Send) and LRO (Large Receive Offload). TSO allows TCP to treat the effective MTU as large on the transmit side, and to send a single large packet (up to 64K everywhere but windows LSOv2) down to the network card, which then segments it into 1500 byte chunks. Similarly, on the receive side, LRO can make a small MTU appear large to TCP by transparently merging adjacent TCP segments into one much larger packet which is then passed to TCP. This can be done in hardware or software. Linux calls this GRO.
phicoh 8y ago
The problem here is ethernet. Even the latest greatest many gigabits per second ethernet can be bridged to plain old coax based 10 Mbps ethernet. Within the ethernet protocol there is no way to negotiate the maximum frame length between stations. So IEEE standard still has a maximum of 1500 octets for the payload. There are some RFCs that go beyond 1500, but mostly the IETF follows the IEEE specification. Note that the 1500 link MTU applies to ethernet (and similar link technologies). For anything else, you can have larger (or smaller) MTUs.
- akvadrako 8y ago
  Most network devices support jumbo frames¹, which are an IEEE standard, but they aren't the default. The point is not the IEEE standard, but what most devices are configured for. Transit operators and IXPs just don't want to configure their equipment to use jumbo frames. It's probably just a coordination problem.² [1] https://www.networkworld.com/article/2224722/cisco-subnet/jumbo-frames.html https://www.networkworld.com/article/2224722/cisco-subnet/ju... [2] https://lists.gt.net/nanog/users/187984 https://lists.gt.net/nanog/users/187984
  - zamadatix 8y ago
    > Most network devices support jumbo frames¹, which are an IEEE standard, but they aren't the default. Jumbo frames are not an IEEE standard.
majke 8y ago
There is a reason why MTU's are generally small. In early internet there was a clash (IMO unsettled) between fixed-packet-length and dynamic-packet-length networks. Compare ethernet to ATM. The benefit of ATM is bandwidth guarantees. You can precisely allocate bandwidth to each tenant. With ethernet you don't have such guarantees. Single tenant can potentially use all the resources. If you allowed jumbo frames in early slow ethernet, you could totally imagine one tenant blocking the line for a-very-long-time, and starving everyone else. Of course, times had changed, bandwidth shot up, and 1500 is indeed way to small. But there is still a limit. You probably don't want to allow gigantic 1.5MiB frames. Normal jumbo frames of 9KiB are fine. It's just a historical accident they aren't made more popular.

drewg123 8y ago

TCP overhead does indeed increase as MTU decreases, but not because "TCP needs to keep track of packets" -- TCP in fact keeps track of bytes, not packets. The reason that overhead is larger for smaller MTUs is simply because of the extra per-packet overheads (looking up connections, allocating memory for ethernet/ip packet headers, (uncontended) locking, repeated function calls to L3/L2 network layers and network drivers, etc). The fact that TCP tracks bytes rather than packets enables hardware offloads like TSO/LSO (TCP Segmentation Offload, known by some as TCP Large Send) and LRO (Large Receive Offload). TSO allows TCP to treat the effective MTU as large on the transmit side, and to send a single large packet (up to 64K everywhere but windows LSOv2) down to the network card, which then segments it into 1500 byte chunks. Similarly, on the receive side, LRO can make a small MTU appear large to TCP by transparently merging adjacent TCP segments into one much larger packet which is then passed to TCP. This can be done in hardware or software. Linux calls this GRO.

phicoh 8y ago

The problem here is ethernet. Even the latest greatest many gigabits per second ethernet can be bridged to plain old coax based 10 Mbps ethernet. Within the ethernet protocol there is no way to negotiate the maximum frame length between stations. So IEEE standard still has a maximum of 1500 octets for the payload. There are some RFCs that go beyond 1500, but mostly the IETF follows the IEEE specification. Note that the 1500 link MTU applies to ethernet (and similar link technologies). For anything else, you can have larger (or smaller) MTUs.

akvadrako 8y ago
Most network devices support jumbo frames¹, which are an IEEE standard, but they aren't the default. The point is not the IEEE standard, but what most devices are configured for. Transit operators and IXPs just don't want to configure their equipment to use jumbo frames. It's probably just a coordination problem.² [1] https://www.networkworld.com/article/2224722/cisco-subnet/jumbo-frames.html https://www.networkworld.com/article/2224722/cisco-subnet/ju... [2] https://lists.gt.net/nanog/users/187984 https://lists.gt.net/nanog/users/187984
- zamadatix 8y ago
  > Most network devices support jumbo frames¹, which are an IEEE standard, but they aren't the default. Jumbo frames are not an IEEE standard.

majke 8y ago

There is a reason why MTU's are generally small. In early internet there was a clash (IMO unsettled) between fixed-packet-length and dynamic-packet-length networks. Compare ethernet to ATM. The benefit of ATM is bandwidth guarantees. You can precisely allocate bandwidth to each tenant. With ethernet you don't have such guarantees. Single tenant can potentially use all the resources. If you allowed jumbo frames in early slow ethernet, you could totally imagine one tenant blocking the line for a-very-long-time, and starving everyone else. Of course, times had changed, bandwidth shot up, and 1500 is indeed way to small. But there is still a limit. You probably don't want to allow gigantic 1.5MiB frames. Normal jumbo frames of 9KiB are fine. It's just a historical accident they aren't made more popular.