3 ms·

Are jumbo frames needed any more? Most NICs have various types of offloading now, especially if you get into 10+ GigE, so how much of a bottleneck is frame pro

Are jumbo frames needed any more? Most NICs have various types of offloading now, especially if you get into 10+ GigE, so how much of a bottleneck is frame processing nowadays? (Either on the send or receive side.)

eqvinox 4y ago
Disregarding everything else, you still lose a minor amount of performance to the IFG (inter-frame gap) and headers that need to be repeated. The offloads on the NIC — and switches/routers inbetween — also consume some small but non-zero amount of power per frame to do their work, which if you multiply by the number of servers in a Cloud DC…
zamadatix 4y ago
I'm not sure they were ever truly needed (just as 1500 isn't truly needed, 576 bytes works "almost the same" as well) but there is some nicety in certain payloads, such as the aforementioned NFS, either completely making it or completely not. At high enough speeds even offloads prefer frames not be tiny just because they can. Even most data center 100G+ switching is not line rate below 256 byte packets and the only thing it's designed to do is transport packets as quickly between ports it can without really inspecting them. Also encapsulations/overlays make the header loss ratio even greater across network to network links in the path. In the end though most places can do without jumbo without a noticeable impact. There can be buffer scheduling problems with ridiculously large frames as well.
touisteur 4y ago
Can confirm there's still a large performance difference in classical network methods sockets) when you're trying to reach 10Gb. All the nice things implemented down there have an interesting effect if you're using lots of parralel streams and applications, but for those of us streaming lots of data in real-time, jumbo frames are still very much useful. I noticed a 10 to 20% cpu load difference between the two. I've even forgone IP fragmentation for these kinds of streams, application-level packet fragmentation/reassembly with an acceptable loss rate. Now we're moving to 100Gb and soon 400-800Gb and anything helps up there. I'm waiting for someone to standardise some way to reassemble jumbo frames into large 1MB+ 'application packets' using custom headers and reassembly algorithms directly in NICs (hopefully something standard like P4) before DMA-dispatching them to user memory.

eqvinox 4y ago

Disregarding everything else, you still lose a minor amount of performance to the IFG (inter-frame gap) and headers that need to be repeated. The offloads on the NIC — and switches/routers inbetween — also consume some small but non-zero amount of power per frame to do their work, which if you multiply by the number of servers in a Cloud DC…

zamadatix 4y ago

I'm not sure they were ever truly needed (just as 1500 isn't truly needed, 576 bytes works "almost the same" as well) but there is some nicety in certain payloads, such as the aforementioned NFS, either completely making it or completely not. At high enough speeds even offloads prefer frames not be tiny just because they can. Even most data center 100G+ switching is not line rate below 256 byte packets and the only thing it's designed to do is transport packets as quickly between ports it can without really inspecting them. Also encapsulations/overlays make the header loss ratio even greater across network to network links in the path. In the end though most places can do without jumbo without a noticeable impact. There can be buffer scheduling problems with ridiculously large frames as well.

touisteur 4y ago

Can confirm there's still a large performance difference in classical network methods sockets) when you're trying to reach 10Gb. All the nice things implemented down there have an interesting effect if you're using lots of parralel streams and applications, but for those of us streaming lots of data in real-time, jumbo frames are still very much useful. I noticed a 10 to 20% cpu load difference between the two. I've even forgone IP fragmentation for these kinds of streams, application-level packet fragmentation/reassembly with an acceptable loss rate. Now we're moving to 100Gb and soon 400-800Gb and anything helps up there. I'm waiting for someone to standardise some way to reassemble jumbo frames into large 1MB+ 'application packets' using custom headers and reassembly algorithms directly in NICs (hopefully something standard like P4) before DMA-dispatching them to user memory.