3 ms·

(Intentional) jumbo frames at layer 2 and expanded MTUs at layer 3 are certainly available (as you may know). In fact it seems (I am, it should be obvious, not

by leoc 2y ago

(Intentional) jumbo frames at layer 2 and expanded MTUs at layer 3 are certainly available (as you may know). In fact it seems (I am, it should be obvious, not an expert) that using jumbo frames is more or less the common practice by now. There does in fact seem to have been some standards drama about this, too: I can't find it now, but IIRC in the '00s someone's proposal to extend the header protocols to allow the header to indicate a frame size of over 1500 bytes was rejected, and nothing seems to have been done since. At the moment it seems that the best way to indicate max. Ethernet frame sizes of over 1500 is an optional field in LLDP(!) https://www.juniper.net/documentation/us/en/software/junos/user-access/topics/topic-map/device-discovery-using-lldp-lldp-med.html https://www.juniper.net/documentation/us/en/software/junos/u... and the fall-back is sending successively larger pings and seeing when the network breaks(!) https://docs.oracle.com/cd/E36784_01/html/E36815/gmzds.html https://docs.oracle.com/cd/E36784_01/html/E36815/gmzds.html .

scifi 2y ago
I'm not certain what my point is, but I wanted to mention that jumbo frames don't work over the Internet. More of a LAN thing.
- toast0 2y ago
  My local internet exchange has a 1500 vlan and a 9000 vlan. My understanding is there are many fewer peers on the 9000 vlan, but it's not zero. If you want to use jumbo packets on the internet at large, you need to have working path MTU detection, which realistically means at least probing, but you really should have that at 1500 too, because there's still plenty of broken networks out there. My guess is you won't have many connections with an effective mtu above 1500, but you might have some.
  - zamadatix 2y ago
    Separate peering VLANs for those using 1500 byte peering and 9000 byte peering about sums up how much of a PITA it is to mix things and expect PMTUD to work. I'd be willing to bet my lunch more 10x more places have been moving down to assuming 1280 byte connections (since IPv6 guarantees it) than have been peering on the internet at >1500 (not counting 1504 for VLAN tags and whathaveyou).
kstrauser 2y ago
The common advice I've heard for jumbo frames is not to enable them unless you can do it for every devices on your LAN, and even then it's probably not worthwhile outside specific situations like a separate iSCSI network or such. I just now ran iperf3 from my Mac to my Synology without jumbo frames: [ ID] Interval Transfer Bitrate Retr [ 7] 0.00-10.00 sec 10.0 GBytes 8.61 Gbits/sec 0 sender [ 7] 0.00-10.00 sec 10.0 GBytes 8.60 Gbits/sec receiver Given how rarely I actually care to saturate the 10Gbit link, I'd rather use the slightly hypothetically slower default settings that are highly likely to work in all scenarios.
- akira2501 2y ago
  It seems much more effective on plain 1000Base. It's the difference between 850Mb/s and 975Mb/s for me.
  - kstrauser 2y ago
    That surprises me a little. From the Wikipedia article[0] I'd expected jumbo frames to be only about 5% more efficient. [0] https://en.wikipedia.org/wiki/Jumbo_frame https://en.wikipedia.org/wiki/Jumbo_frame
  - zamadatix 2y ago
    That difference is due to your devices relatively poor at handling high pps workloads and not really related to the particular speed it's running at. You can certainly get more than 850 mbps on a 1000BASE-T link with standard MTU, on the order of ~100 mbps more.
myrandomcomment 2y ago
The use of jumbo frames is NOT normal and is only used in very specific setups. In general something like a storage system that is isolated to its own layer2 network. At some point your jumbo network has to hit the reset of the network and packet fragmentation is done in software via the CPU which is very expensive and not linerate. The normal outcome is you break your network.

by leoc 2y ago

scifi 2y ago

I'm not certain what my point is, but I wanted to mention that jumbo frames don't work over the Internet. More of a LAN thing.

toast0 2y ago
My local internet exchange has a 1500 vlan and a 9000 vlan. My understanding is there are many fewer peers on the 9000 vlan, but it's not zero. If you want to use jumbo packets on the internet at large, you need to have working path MTU detection, which realistically means at least probing, but you really should have that at 1500 too, because there's still plenty of broken networks out there. My guess is you won't have many connections with an effective mtu above 1500, but you might have some.
- zamadatix 2y ago
  Separate peering VLANs for those using 1500 byte peering and 9000 byte peering about sums up how much of a PITA it is to mix things and expect PMTUD to work. I'd be willing to bet my lunch more 10x more places have been moving down to assuming 1280 byte connections (since IPv6 guarantees it) than have been peering on the internet at >1500 (not counting 1504 for VLAN tags and whathaveyou).

kstrauser 2y ago

The common advice I've heard for jumbo frames is not to enable them unless you can do it for every devices on your LAN, and even then it's probably not worthwhile outside specific situations like a separate iSCSI network or such. I just now ran iperf3 from my Mac to my Synology without jumbo frames: [ ID] Interval Transfer Bitrate Retr [ 7] 0.00-10.00 sec 10.0 GBytes 8.61 Gbits/sec 0 sender [ 7] 0.00-10.00 sec 10.0 GBytes 8.60 Gbits/sec receiver Given how rarely I actually care to saturate the 10Gbit link, I'd rather use the slightly hypothetically slower default settings that are highly likely to work in all scenarios.

akira2501 2y ago
It seems much more effective on plain 1000Base. It's the difference between 850Mb/s and 975Mb/s for me.
- kstrauser 2y ago
  That surprises me a little. From the Wikipedia article[0] I'd expected jumbo frames to be only about 5% more efficient. [0] https://en.wikipedia.org/wiki/Jumbo_frame https://en.wikipedia.org/wiki/Jumbo_frame
- zamadatix 2y ago
  That difference is due to your devices relatively poor at handling high pps workloads and not really related to the particular speed it's running at. You can certainly get more than 850 mbps on a 1000BASE-T link with standard MTU, on the order of ~100 mbps more.

myrandomcomment 2y ago

The use of jumbo frames is NOT normal and is only used in very specific setups. In general something like a storage system that is isolated to its own layer2 network. At some point your jumbo network has to hit the reset of the network and packet fragmentation is done in software via the CPU which is very expensive and not linerate. The normal outcome is you break your network.