4 ms·

IEEE 802 history is disappearing without a trace? Afaik it’s pretty well documented, you just need to be a member for some of the stuff. http://www.ieee802.or

IEEE 802 history is disappearing without a trace? Afaik it’s pretty well documented, you just need to be a member for some of the stuff. http://www.ieee802.org/ http://www.ieee802.org/ I feel like the last piece we’re missing in this story is the performance impact of fragmentation. Like why not just set all new hardware to an MTU of 9000 and wait ten years?

teddyh 7y ago
My favorite Ethernet resource is Charles Spurgeon’s Ethernet (IEEE 802.3) Web Site: http://www.ethermanage.com/resources/ http://www.ethermanage.com/resources/ It used to have even more stuff, but I think he removed a lot when he got his book published.
cesarb 7y ago
> Like why not just set all new hardware to an MTU of 9000 and wait ten years? The hardware in question is Ethernet NICs. However, for you to set the MTU on an Ethernet NIC to 9000, every device on the same Ethernet network (at least the same Ethernet VLAN), including all other NICs and switches, including ones which aren't connected yet, must also support and be configured for that MTU. And this also means you cannot use WiFi on that Ethernet network (since, at least last time I looked, WiFi cannot use a MTU that large).
- willis936 7y ago
  Sending a jumbo frame down a line that has hardware that doesn’t support jumbo frames somewhere along the way does not mean the packet gets dropped. The NIC that would send the jumbo frame fragments the packet down to the lower MTU. So what’s the performance impact of that fragmentation? If it isn’t higher than the difference in bandwidth overhead from headers of 9000 MTU traffic vs. 1500 MTU traffic then why not transition to 9000 MTU?
  - tyingq 7y ago
    It does mean packets sent to another local, non-routed, non-jumbo-frame interface would get lost. So you could, for example, maybe talk to the internet, but you couldn't print anything to the printer down the hall.
  - vlan0 7y ago
    PMTU-D will save their ass in some cases. But it's not safe to assume all routers in the path will respond to ICMP.
    - toast0 7y ago
      It doesn't matter that the routers respond to ICMP, it matters that they generate them, and that they're addressed properly, and that intermediate routers don't drop them. Some routers will generate the ICMPs, but are rate limited, and the underlying poor configuration means that the rate limits are hit continously and most connections are effectively in a path mtu blackhole.
      - vlan0 7y ago
        >It doesn't matter that the routers respond to ICMP, it matters that they generate them, and that they're addressed properly, and that intermediate routers don't drop them. >Some routers will generate the ICMPs, but are rate limited, and the underlying poor configuration means that the rate limits are hit continously and most connections are effectively in a path mtu blackhole. Sure. But I'm not about to sit here and name all the different reasons for folks. And since most here do not have a strong networking background running consumer grade routers at home, it seemed most applicable. I could have used a more encompassing term like PMTU-D blackhole, but I didn't.
  - sathackr 7y ago
    But how does the NIC know that, 11 hops away, there is a layer 2 device, which cannot communicate with the NIC(switches do not typically have the ability to communicate directly with the devices generating the packets), that only supports a 1500 byte frame? Now you need Path MTU discovery, which as the article indicates, has its own set of issues. (Overhead from trial and error, ICMP being blocked due to security concerns, etc...)
    - wbl 7y ago
      If you block ICMP you deserve what you get. Don't do this. (Edit: don't block ICMP)
      - oarsinsync 7y ago
        So now you're trying to communicate from your home machine to some random host on the internet (website, VPS, streaming service), and you're configured for MTU 9000, the remote service is also configured for MTU 9000, but some transit provider in the middle is not, and they've disabled ICMP for $reasons. They blocked ICMP, do you deserve what you get?
        wbl 7y ago
        Transit providers should push packets and generally do. With PMTU failures it's usually clueless network admins on firewalls nearer endpoints. And no, you don't and I wish the admin responsible could feel your pain.
        oarsinsync 7y ago
        > Transit providers should Agreed > and generally do Agreed. Now if you can make it 'will always just push packets', we'll be golden. Unfortunately, there are enough ATM/MPLS/SONET/etc networks being run by people who no longer understand what they're doing, that we're never going to get there. To make matters more entertaining, IPv6 depends on icmp6 even more.
    - willis936 7y ago
  - deleted 7y ago
    [deleted]
  - zamadatix 7y ago
    Fragmentation/reassembly is an l3 concept and not guaranteed to large MTUs when it is there.
  - cesarb 7y ago
    AFAIK, Ethernet has no support for fragmentation; I've never seen, in the Ethernet standards I've read (though I might have missed it), a field saying "this is a fragment of a larger frame". There's fragmentation in the IP layer, but it needs: (a) that the frame contains an IP packet; (b) that the IP packet can be fragmented (no "don't fragment" on IPv4, or a special header on IPv6); (c) that the sending host knows the receiving host's MTU; (d) that it's not a broadcast or multicast packet (which have no singular "receiving host"). You can have working fragmentation if you have two separate Ethernet segments, one for 1500 and the other for 9000, connected by an IP router; the cost (assuming no broken firewalls blocking the necessary ICMP packets, which sadly is still too common) is that the initial transmission will be resent since most modern IP stacks set the "don't fragment" bit (or don't include the extra header for IPv6 fragmentation).
  - toast0 7y ago
    > Sending a jumbo frame down a line that has hardware that doesn’t support jumbo frames somewhere along the way does not mean the packet gets dropped Almost all IP packets on the internet at large have the 'do not fragment' flag set. IP defragmentation performance ranges from pretty bad to an easy DDoS vector, so a lot of high traffic hosts drop fragments without processing them. If we had truncation (with a flag) instead of fragmentation, that might have been usable, because the endpoints could determine in-band the max size datagram and communicate it and use that; but that's not what we have.
- Avamander 7y ago
  The worst case I just recently encountered with Jumbo Frames was with NetworkManager trying to follow local DNS server's advertised MTU but when the local interface doesn't support Jumbo Frames it just dies and keeps looping. Even if you really want devices to use JF, some fail miserably because it's just not well thought out.
tyingq 7y ago
"why not just set all new hardware to an MTU of 9000" Routers can fragment the packets, switches can't. So that would be pretty chaotic for non-techie installed equipment.
- brutt 7y ago
  Last time I saw hardware Ethernet switch was 20 years ago. 8-[ ]
  - tyingq 7y ago
    There's one in your house probably. It won't frag packets between your wired PC and your wired printer. There are also certainly a shit load of them in closets and top-of-rack all over where I work.
- kitteh 7y ago
  Plenty of routers today that can't fragment packets. And they have rate limiters where they can only generate a small amount of ICMP 3/4s (maybe 50 a second).
vlan0 7y ago
>I feel like the last piece we’re missing in this story is the performance impact of fragmentation. Like why not just set all new hardware to an MTU of 9000 and wait ten years? Because a node with a MTU of 9000 will very likely be unable to determine the MTU of every link in it's path. At best, you'll see fragmentation. At worst, the node's packets will be registered as interface errors when it encounters an interface lower than 9k. Neither of those are desirable.
- deleted 7y ago
  [deleted]
phicoh 7y ago
The problem seems to be that both the IEEE and the IETF don't want to do anything. IEEE could define a way to support larger frames. 'just wait 10 years' doesn't strike me as the best solution, but at least it is a solution. In my opinion a better way be if all devices would report the max frame length they support. Bridges would just report the minimum over all ports on the same vlan. When there are legacy devices that don't report anything, just stay at 1500. IETF can also do someything today by having hosts probe the effective max. frame length. There are drafts but they don't go anywhere because too few people care.

teddyh 7y ago

My favorite Ethernet resource is Charles Spurgeon’s Ethernet (IEEE 802.3) Web Site: http://www.ethermanage.com/resources/ http://www.ethermanage.com/resources/ It used to have even more stuff, but I think he removed a lot when he got his book published.

cesarb 7y ago

> Like why not just set all new hardware to an MTU of 9000 and wait ten years? The hardware in question is Ethernet NICs. However, for you to set the MTU on an Ethernet NIC to 9000, every device on the same Ethernet network (at least the same Ethernet VLAN), including all other NICs and switches, including ones which aren't connected yet, must also support and be configured for that MTU. And this also means you cannot use WiFi on that Ethernet network (since, at least last time I looked, WiFi cannot use a MTU that large).

willis936 7y ago
Sending a jumbo frame down a line that has hardware that doesn’t support jumbo frames somewhere along the way does not mean the packet gets dropped. The NIC that would send the jumbo frame fragments the packet down to the lower MTU. So what’s the performance impact of that fragmentation? If it isn’t higher than the difference in bandwidth overhead from headers of 9000 MTU traffic vs. 1500 MTU traffic then why not transition to 9000 MTU?
- tyingq 7y ago
  It does mean packets sent to another local, non-routed, non-jumbo-frame interface would get lost. So you could, for example, maybe talk to the internet, but you couldn't print anything to the printer down the hall.
- vlan0 7y ago
  PMTU-D will save their ass in some cases. But it's not safe to assume all routers in the path will respond to ICMP.
  - toast0 7y ago
    It doesn't matter that the routers respond to ICMP, it matters that they generate them, and that they're addressed properly, and that intermediate routers don't drop them. Some routers will generate the ICMPs, but are rate limited, and the underlying poor configuration means that the rate limits are hit continously and most connections are effectively in a path mtu blackhole.
    - vlan0 7y ago
      >It doesn't matter that the routers respond to ICMP, it matters that they generate them, and that they're addressed properly, and that intermediate routers don't drop them. >Some routers will generate the ICMPs, but are rate limited, and the underlying poor configuration means that the rate limits are hit continously and most connections are effectively in a path mtu blackhole. Sure. But I'm not about to sit here and name all the different reasons for folks. And since most here do not have a strong networking background running consumer grade routers at home, it seemed most applicable. I could have used a more encompassing term like PMTU-D blackhole, but I didn't.
- sathackr 7y ago
  But how does the NIC know that, 11 hops away, there is a layer 2 device, which cannot communicate with the NIC(switches do not typically have the ability to communicate directly with the devices generating the packets), that only supports a 1500 byte frame? Now you need Path MTU discovery, which as the article indicates, has its own set of issues. (Overhead from trial and error, ICMP being blocked due to security concerns, etc...)
  - wbl 7y ago
    If you block ICMP you deserve what you get. Don't do this. (Edit: don't block ICMP)
    - oarsinsync 7y ago
      So now you're trying to communicate from your home machine to some random host on the internet (website, VPS, streaming service), and you're configured for MTU 9000, the remote service is also configured for MTU 9000, but some transit provider in the middle is not, and they've disabled ICMP for $reasons. They blocked ICMP, do you deserve what you get?
      - wbl 7y ago
        Transit providers should push packets and generally do. With PMTU failures it's usually clueless network admins on firewalls nearer endpoints. And no, you don't and I wish the admin responsible could feel your pain.
        oarsinsync 7y ago
        > Transit providers should Agreed > and generally do Agreed. Now if you can make it 'will always just push packets', we'll be golden. Unfortunately, there are enough ATM/MPLS/SONET/etc networks being run by people who no longer understand what they're doing, that we're never going to get there. To make matters more entertaining, IPv6 depends on icmp6 even more.
  - willis936 7y ago
- deleted 7y ago
  [deleted]
- zamadatix 7y ago
  Fragmentation/reassembly is an l3 concept and not guaranteed to large MTUs when it is there.
- cesarb 7y ago
  AFAIK, Ethernet has no support for fragmentation; I've never seen, in the Ethernet standards I've read (though I might have missed it), a field saying "this is a fragment of a larger frame". There's fragmentation in the IP layer, but it needs: (a) that the frame contains an IP packet; (b) that the IP packet can be fragmented (no "don't fragment" on IPv4, or a special header on IPv6); (c) that the sending host knows the receiving host's MTU; (d) that it's not a broadcast or multicast packet (which have no singular "receiving host"). You can have working fragmentation if you have two separate Ethernet segments, one for 1500 and the other for 9000, connected by an IP router; the cost (assuming no broken firewalls blocking the necessary ICMP packets, which sadly is still too common) is that the initial transmission will be resent since most modern IP stacks set the "don't fragment" bit (or don't include the extra header for IPv6 fragmentation).
- toast0 7y ago
  > Sending a jumbo frame down a line that has hardware that doesn’t support jumbo frames somewhere along the way does not mean the packet gets dropped Almost all IP packets on the internet at large have the 'do not fragment' flag set. IP defragmentation performance ranges from pretty bad to an easy DDoS vector, so a lot of high traffic hosts drop fragments without processing them. If we had truncation (with a flag) instead of fragmentation, that might have been usable, because the endpoints could determine in-band the max size datagram and communicate it and use that; but that's not what we have.
Avamander 7y ago
The worst case I just recently encountered with Jumbo Frames was with NetworkManager trying to follow local DNS server's advertised MTU but when the local interface doesn't support Jumbo Frames it just dies and keeps looping. Even if you really want devices to use JF, some fail miserably because it's just not well thought out.

tyingq 7y ago

"why not just set all new hardware to an MTU of 9000" Routers can fragment the packets, switches can't. So that would be pretty chaotic for non-techie installed equipment.

brutt 7y ago
Last time I saw hardware Ethernet switch was 20 years ago. 8-[ ]
- tyingq 7y ago
  There's one in your house probably. It won't frag packets between your wired PC and your wired printer. There are also certainly a shit load of them in closets and top-of-rack all over where I work.
kitteh 7y ago
Plenty of routers today that can't fragment packets. And they have rate limiters where they can only generate a small amount of ICMP 3/4s (maybe 50 a second).

vlan0 7y ago

>I feel like the last piece we’re missing in this story is the performance impact of fragmentation. Like why not just set all new hardware to an MTU of 9000 and wait ten years? Because a node with a MTU of 9000 will very likely be unable to determine the MTU of every link in it's path. At best, you'll see fragmentation. At worst, the node's packets will be registered as interface errors when it encounters an interface lower than 9k. Neither of those are desirable.

deleted 7y ago
[deleted]

phicoh 7y ago

The problem seems to be that both the IEEE and the IETF don't want to do anything. IEEE could define a way to support larger frames. 'just wait 10 years' doesn't strike me as the best solution, but at least it is a solution. In my opinion a better way be if all devices would report the max frame length they support. Bridges would just report the minimum over all ports on the same vlan. When there are legacy devices that don't report anything, just stay at 1500. IETF can also do someything today by having hosts probe the effective max. frame length. There are drafts but they don't go anywhere because too few people care.