5 ms·
The sad thing is, there's no reason for any modern system to require a small MTU. MTUs of only a thousand bytes made sense on sub-megabit networks (to limit jit
by tlb 5y ago
The sad thing is, there's no reason for any modern system to require a small MTU. MTUs of only a thousand bytes made sense on sub-megabit networks (to limit jitter), and when RAM was purchased by the kilobyte. But there's no reason why every network shouldn't support 64 kB MTUs today.
A required max of 64 kB with a default segment size of 16 kB would improve performance and allow any amount of headers to be added.
- drewg123 5y agoI have exactly the opposite view. With offloads like TSO and LRO, NICs can pretend the network supports very large "packets", while still segmenting traffic into standard 1500b chunks. From a hardware perspective, supporting jumbo frames makes everything harder. You need to have larger buffers. Especially when dealing with ethernet flow control. And latencies increase, because most NICs do store-and-forward, and need to hold onto the packet until the end arrives so they can calculate the CRC.
- jeffbee 5y agoWhat about the receive side? Suppose you have TSO but the receiver still gets a train of smaller frames, which consumes extra CPU. The solution to ethernet flow control is to disable it, of course.
- drewg123 5y agoThe solution on the receive side is LRO. I was doing 10GbE line rate with 1500b frames in 2006 or so on consumer-grade CPUs using software LRO (implemented in the driver). I worked for a NIC vendor (Myricom) that shipped our NIC drivers with a default 9000b MTU. That was fine when we just did HPC & everything reachable via layer 2 was another one of our NICs with a 9k MTU. But it was a mess when we started selling 10GbE NICs. I got pretty good at diagnosing MTU mis-matches..
- jeffbee 5y agoI don't doubt that you can hit 10g on a wimpy core, the question is how much is left over for your application to use. I recently analyzed jumbo vs. puny frames in EC2 with the ENA and the difference in softirq time amounted to one core out of the 32 cores on the instance type, at 12.5gbps line rate. That's pretty substantial cost when you have a lot of nodes. The ENA doesn't have a ton of offload features and EC2 tenants don't get to specify a better NIC.
- drewg123 5y agoIs this linux GRO, or LRO? I've found that GRO is can be less efficient than LRO, but its been 7 years since I paid much attention to Linux performance..