7 ms·

The biggest single cause of network engineer hair loss must be MTU and fragmentation. We had some customers over a tunnel (IIRC, it was MPLS) that reduced the M

The biggest single cause of network engineer hair loss must be MTU and fragmentation. We had some customers over a tunnel (IIRC, it was MPLS) that reduced the MTU from 1500 (default Ethernet) to 1496. This happened while our vendor changed some equipment and forgot to update either MTU to accommodate the extra MPLS header. Of course, there was a misconfigured firewall that wouldn't fragment and wouldn't send "ICMP won't fragment". The result? Most DNS queries went through. All chat applications worked. SSH generally worked, except when you started a full-screen terminal application. Smaller web pages load. Larger web pages loaded only partially. Imagine non-technical users explaining their issue. "The Internet is half-broken. Please help." God only knows how much hair I would have today if the world had figured MTU and fragmentation properly.

iso1210 5y ago
The world did, then people came along and started blocking ICMP
- corty 5y ago
  Yes. There is a special circle in hell for braindead firewall admins filtering ICMP. Unfortunately happens often enough that "MTU problem" is the first thing that comes to mind at the grandparents description. Next in line would be "broken DNS"...
  - deleted 5y ago
    [deleted]
  - temp667 5y ago
    They are not braindead. ICMP used to be a bit broken and folks got sick of ping floods, sweeps, redirect attacks and tunneling hacks etc. Some pretty major places block ICMP as a result to keep their network reasoning and security analysis simpler.
    - korethr 5y ago
      Just because they are major places doesn't mean they are right or such practice buys any improvement in security worth the operational cost it imposes.
    - bcrl 5y ago
      ICMP ping of death was a thing in the 1990s. Those bugs have been fixed for more than 2 decades now.
    - viraptor 5y ago
      What do ping floods do that TCP syn floods don't? (I expect you run at least one service on the exposed hosts) Also what system even accepts ICMP based route redirects by default from the open internet?
- korethr 5y ago
  And when I am in my less charitable moods, I would claim that said people are unqualified to not just to configure networks, but to have opinions worth considering when it comes to setting network policy.
  - tinus_hn 5y ago
    Unfortunately the only qualification required is for you to want to communicate with their network.
- anticristi 5y ago
  In my utopic world, we wouldn't need "ICMP won't fragment". IPv4/IPv6 would have the MTU of exactly 4096 bytes. Why not a higher or lower level protocol? Because this is the lowest level protocol which is end-to-end and has a path MTU. Lower level protocols would need to somehow handle the MTU and higher level protocols would need to fit their segments within the MTU. Why 4096? Because this is a common page size on computers. Multiple of this would likely bring little benefits, due to GRO/GSO. Also, it's my utopic world, so if you don't like it create your own. :))
  - zamadatix 5y ago
    Not much of a utopia if all you do is say the same problem is now unsolved at a different layer ;). Nothing special about the layer below IP that allows it to fix the problem in a way IP couldn't be made to - it's just an annoying problem to fix efficiently.
  - alerighi 5y ago
    4096 can be too much for unstable connections, for example Wi-Fi or mobile. Also, 4096 bytes at what level? Layer 2? And when you are using a tunnel, you add bytes and thus have to go higher than 4096 or go lower and reduce MTU, or fragment the packet to keep 4096 bytes. You are just raising the limit, not solving the problem.
  - tinus_hn 5y ago
    This is the network version of ‘640k should be enough for anyone’. Don’t fixate this kind of policy in such a fundamental protocol.
  - noselasd 5y ago
    The issue is how the network are glued together, your X MTU will be embedded in a VPN tunnel, PPPOE - and that device needs to tell your device about its MTU
  - AlphaSite 5y ago
    How would this work for IP over IP networks?
rjsw 5y ago
I consider the time I spent implementing RFC 4638 to allow the PPPoE implementation I use to have a 1500 byte MTU to be a good investment.
- stock_toaster 5y ago
  I wish centurylink fiber supported baby jump frames. :/
  - toast0 5y ago
    Would make more sense for them to just do IP over fiber directly than fiddling with their PPPoE implementation. I have centurylink DSL with PPPoE and the thing that really bugs me is if your modem lost the PPPoE password, it can login with default credentials and ask for the passsword. CenturyLink clearly knows who I am without needing PPPoE authentication, so what do they get out of it?
    - praseodym 5y ago
      My local provider requires PPPoE over fiber but accepts any password. It’s probably some legacy artefact that’s too complicated to remove from the stack.
    - korethr 5y ago
      Dollars to doughnuts the PPPoE implementation is tied into a billing system.
    - bcrl 5y ago
      It makes wholesaling your network easy since you can tunnel switch based on the user@domain used during authentication.
  - the8472 5y ago
    baby jumbo frames.
    - 5y ago
korethr 5y ago
> Of course, there was a misconfigured firewall that wouldn't fragment and wouldn't send "ICMP won't fragment". Blocked or dropped ICMP has caused me heartburn as well. I am pretty sure my co-workers are used to my "blocking ICMP is Evil Bad and Wrong" rants by now.
- kureikain 5y ago
  AWS blocked ICMP by default though and cause a lot of traceroute problem for me...
  - est31 5y ago
    > If the maximum transmission unit (MTU) between hosts in your subnets is different, or your instances communicate with peers over the internet, you must add the following network ACL rule, both inbound and outbound. https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html#path_mtu_discovery https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network... If your instances communicate with peers over the internet. So by default, AWS was built to assume your instances do not communicate over the internet, lol.
- temp667 5y ago
  Some people don't want to deal with ping floods, sweeps, ICMP tunneling issued and the whole ICMP redirect attacks?
  - korethr 5y ago
    There's a difference between blanket blocking all ICMP and selectively blocking or rate-limiting a subset of ICMP messages according to an accurate threat model for your various networks. The latter, properly done, is unlikely to break MTU discovery or your own ability to troubleshoot and monitor your own hosts/networks. The former is what makes network engineers want to eviscerate you and use your gut as patch cables.
    - labawi 5y ago
rubatuga 5y ago
Websites half working should send alarm bells ringing that it’s an MTU problem. The reason google.com works is because they only support an MTU of 1280.
- X-Istence 5y ago
  The minimum size dictated by IPv6, and thus almost always guaranteed to work.
raspyberr 5y ago
Why would you reduce the MTU? I had an issue many years ago playing league of legends where log in would fail until I changed the MTU to be slightly lower. Something like 1486.
- jandrese 5y ago
  Tunnels always have the danger of reducing the MTU because they have to account for the header. Worse, is that if a tunnel doesn't reduce the MTU and its packets are dropped by a router further down the line the the ICMP TOO_FAT response back to the tunnel endpoint not the original host. The tunnel endpoint has zero clue what to do with it and drops it on the ground, leaving the original host in the dark. Even worse is when overzealous firewall admins start locking down everything they don't understand and that includes ICMP. "Some hackers could ping our networks!" Then you're truly up a creek. Luckily MTU breakage is fairly easy to spot once you know what to look for. It's easy enough to fix too if your local firewall admins aren't the block everything type. Just fix the MTU on the tunnel endpoint and suddenly everything will start working. One final note: If as a router or application developer you ever find yourself having to fragment packets, please, for the love of all that is holy, fragment them into two roughly equal sized packets. Don't create one huge packet and one runt from the leftovers. It hurts me to see packets coming in from some heavily tunneled source with 1400 byte initial packet followed by 3 or 4 tiny fragments. Or worse, 3 or 4 tiny out of order fragments followed by the 1400 byte bulk because the MAC prioritized small packets in its queue and send them first.
  - kazen44 5y ago
    > One final note: If as a router or application developer you ever find yourself having to fragment packets, Ideally, you should not want to fragment IP packets. It is far better to do MSS clamping in TCP to prevent the size of the TCP payload to grew above the size of the max MTU in the path. TCP can stitch together the data at the other node just fine, without routers in between having to fragment the IP packets, which will kill your bandwith in comparison.
    - 5y ago
kazen44 5y ago
what is even more annoying is that there is a perfectly fine solution for this problem called MTU pad discovery. sadly many people block the entirety of ICMP to prevent this from working properly. Also, some NAT devices just straight up lie about their MTUPD size.
cryptonector 5y ago
MTU mismatches are hard to solve too. This happens when you have nodes and/or switches that are either misconfigured or don't all support autodiscovery and then have divergent defaults. A typical symptom is that things (e.g., ssh) hang. This typically happens when you use a protocol that uses Kerberos in an Active Directory domain with very large Kerberos tickets. PQ crypto would do this too.
nn3 5y ago
I always saw ICMP blocking easy ability test for the firewall configuration person. If they block them this means they have no clue about the protocols they're configuring, and it's safe to assume that they likely got other things wrong too.
labawi 5y ago
I have an interesting uplink that drops packets of size about 1430-1470, but has no issue with smaller or bigger packets, even 1500. ICMP is rate-limited, and 1500 byte packets work, so PMTUD doesn't kick. Hence, the connections hang if you hit the magic size. Had to decrease MTU. Unfortunately, that doesn't solve remote-side UDP.

iso1210 5y ago

The world did, then people came along and started blocking ICMP

corty 5y ago
Yes. There is a special circle in hell for braindead firewall admins filtering ICMP. Unfortunately happens often enough that "MTU problem" is the first thing that comes to mind at the grandparents description. Next in line would be "broken DNS"...
- deleted 5y ago
  [deleted]
- temp667 5y ago
  They are not braindead. ICMP used to be a bit broken and folks got sick of ping floods, sweeps, redirect attacks and tunneling hacks etc. Some pretty major places block ICMP as a result to keep their network reasoning and security analysis simpler.
  - korethr 5y ago
    Just because they are major places doesn't mean they are right or such practice buys any improvement in security worth the operational cost it imposes.
  - bcrl 5y ago
    ICMP ping of death was a thing in the 1990s. Those bugs have been fixed for more than 2 decades now.
  - viraptor 5y ago
    What do ping floods do that TCP syn floods don't? (I expect you run at least one service on the exposed hosts) Also what system even accepts ICMP based route redirects by default from the open internet?
korethr 5y ago
And when I am in my less charitable moods, I would claim that said people are unqualified to not just to configure networks, but to have opinions worth considering when it comes to setting network policy.
- tinus_hn 5y ago
  Unfortunately the only qualification required is for you to want to communicate with their network.
anticristi 5y ago
In my utopic world, we wouldn't need "ICMP won't fragment". IPv4/IPv6 would have the MTU of exactly 4096 bytes. Why not a higher or lower level protocol? Because this is the lowest level protocol which is end-to-end and has a path MTU. Lower level protocols would need to somehow handle the MTU and higher level protocols would need to fit their segments within the MTU. Why 4096? Because this is a common page size on computers. Multiple of this would likely bring little benefits, due to GRO/GSO. Also, it's my utopic world, so if you don't like it create your own. :))
- zamadatix 5y ago
  Not much of a utopia if all you do is say the same problem is now unsolved at a different layer ;). Nothing special about the layer below IP that allows it to fix the problem in a way IP couldn't be made to - it's just an annoying problem to fix efficiently.
- alerighi 5y ago
  4096 can be too much for unstable connections, for example Wi-Fi or mobile. Also, 4096 bytes at what level? Layer 2? And when you are using a tunnel, you add bytes and thus have to go higher than 4096 or go lower and reduce MTU, or fragment the packet to keep 4096 bytes. You are just raising the limit, not solving the problem.
- tinus_hn 5y ago
  This is the network version of ‘640k should be enough for anyone’. Don’t fixate this kind of policy in such a fundamental protocol.
- noselasd 5y ago
  The issue is how the network are glued together, your X MTU will be embedded in a VPN tunnel, PPPOE - and that device needs to tell your device about its MTU
- AlphaSite 5y ago
  How would this work for IP over IP networks?

rjsw 5y ago

I consider the time I spent implementing RFC 4638 to allow the PPPoE implementation I use to have a 1500 byte MTU to be a good investment.

stock_toaster 5y ago
I wish centurylink fiber supported baby jump frames. :/
- toast0 5y ago
  Would make more sense for them to just do IP over fiber directly than fiddling with their PPPoE implementation. I have centurylink DSL with PPPoE and the thing that really bugs me is if your modem lost the PPPoE password, it can login with default credentials and ask for the passsword. CenturyLink clearly knows who I am without needing PPPoE authentication, so what do they get out of it?
  - praseodym 5y ago
    My local provider requires PPPoE over fiber but accepts any password. It’s probably some legacy artefact that’s too complicated to remove from the stack.
  - korethr 5y ago
    Dollars to doughnuts the PPPoE implementation is tied into a billing system.
  - bcrl 5y ago
    It makes wholesaling your network easy since you can tunnel switch based on the user@domain used during authentication.
- the8472 5y ago
  baby jumbo frames.
  - 5y ago

korethr 5y ago

> Of course, there was a misconfigured firewall that wouldn't fragment and wouldn't send "ICMP won't fragment". Blocked or dropped ICMP has caused me heartburn as well. I am pretty sure my co-workers are used to my "blocking ICMP is Evil Bad and Wrong" rants by now.

kureikain 5y ago
AWS blocked ICMP by default though and cause a lot of traceroute problem for me...
- est31 5y ago
  > If the maximum transmission unit (MTU) between hosts in your subnets is different, or your instances communicate with peers over the internet, you must add the following network ACL rule, both inbound and outbound. https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html#path_mtu_discovery https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network... If your instances communicate with peers over the internet. So by default, AWS was built to assume your instances do not communicate over the internet, lol.
temp667 5y ago
Some people don't want to deal with ping floods, sweeps, ICMP tunneling issued and the whole ICMP redirect attacks?
- korethr 5y ago
  There's a difference between blanket blocking all ICMP and selectively blocking or rate-limiting a subset of ICMP messages according to an accurate threat model for your various networks. The latter, properly done, is unlikely to break MTU discovery or your own ability to troubleshoot and monitor your own hosts/networks. The former is what makes network engineers want to eviscerate you and use your gut as patch cables.
  - labawi 5y ago

rubatuga 5y ago

Websites half working should send alarm bells ringing that it’s an MTU problem. The reason google.com works is because they only support an MTU of 1280.

X-Istence 5y ago
The minimum size dictated by IPv6, and thus almost always guaranteed to work.

raspyberr 5y ago

Why would you reduce the MTU? I had an issue many years ago playing league of legends where log in would fail until I changed the MTU to be slightly lower. Something like 1486.

jandrese 5y ago
Tunnels always have the danger of reducing the MTU because they have to account for the header. Worse, is that if a tunnel doesn't reduce the MTU and its packets are dropped by a router further down the line the the ICMP TOO_FAT response back to the tunnel endpoint not the original host. The tunnel endpoint has zero clue what to do with it and drops it on the ground, leaving the original host in the dark. Even worse is when overzealous firewall admins start locking down everything they don't understand and that includes ICMP. "Some hackers could ping our networks!" Then you're truly up a creek. Luckily MTU breakage is fairly easy to spot once you know what to look for. It's easy enough to fix too if your local firewall admins aren't the block everything type. Just fix the MTU on the tunnel endpoint and suddenly everything will start working. One final note: If as a router or application developer you ever find yourself having to fragment packets, please, for the love of all that is holy, fragment them into two roughly equal sized packets. Don't create one huge packet and one runt from the leftovers. It hurts me to see packets coming in from some heavily tunneled source with 1400 byte initial packet followed by 3 or 4 tiny fragments. Or worse, 3 or 4 tiny out of order fragments followed by the 1400 byte bulk because the MAC prioritized small packets in its queue and send them first.
- kazen44 5y ago
  > One final note: If as a router or application developer you ever find yourself having to fragment packets, Ideally, you should not want to fragment IP packets. It is far better to do MSS clamping in TCP to prevent the size of the TCP payload to grew above the size of the max MTU in the path. TCP can stitch together the data at the other node just fine, without routers in between having to fragment the IP packets, which will kill your bandwith in comparison.
  - 5y ago

kazen44 5y ago

what is even more annoying is that there is a perfectly fine solution for this problem called MTU pad discovery. sadly many people block the entirety of ICMP to prevent this from working properly. Also, some NAT devices just straight up lie about their MTUPD size.

cryptonector 5y ago

MTU mismatches are hard to solve too. This happens when you have nodes and/or switches that are either misconfigured or don't all support autodiscovery and then have divergent defaults. A typical symptom is that things (e.g., ssh) hang. This typically happens when you use a protocol that uses Kerberos in an Active Directory domain with very large Kerberos tickets. PQ crypto would do this too.

nn3 5y ago

I always saw ICMP blocking easy ability test for the firewall configuration person. If they block them this means they have no clue about the protocols they're configuring, and it's safe to assume that they likely got other things wrong too.

labawi 5y ago

I have an interesting uplink that drops packets of size about 1430-1470, but has no issue with smaller or bigger packets, even 1500. ICMP is rate-limited, and 1500 byte packets work, so PMTUD doesn't kick. Hence, the connections hang if you hit the magic size. Had to decrease MTU. Unfortunately, that doesn't solve remote-side UDP.