17 ms·

The main reason why Path MTU detection is fragile is that many people ignorantly filter all ICMP traffic (including type 3). But educating people about ICMP an

The main reason why Path MTU detection is fragile is that many people ignorantly filter all ICMP traffic (including type 3). But educating people about ICMP and its role in PMTUD [1] seems like a lost battle. I wish to be proven wrong but I have an impressions that younger developers, devops/SRE and even network engineers know about network protocols less in than 10-20 years ago. Nowadays I rarely meet people how know what Path MTU Discovery is. I even stopped asking about it on job interviews (for devs/ops) because no one can answer, it's a luck if a candidate can tell anything about ICMP at all. [1] http://www.znep.com/~marcs/mtu/ http://www.znep.com/~marcs/mtu/

jiggawatts 4y ago
The two things that boil my blood when dealing with SecOps people are: 1) Dropping all ICMP packets on the floor for "security" reasons, which means that basic diagnostics is now impossible, you get random issues with software that uses PING, and Path MTU Discovery is broken forever. 2) Leaving internal firewall ports with the default settings, which are intended for Internet-facing ports. For example, for "denied" incoming traffic from the Internet, the correct response is to silently drop the packet. Internally, the correct thing to do is to respond with a NACK to instantly close the connection. Without this, you spend days and days chasing down random and difficult-to-troubleshoot timeouts and weird 30-second delays all over the place. As a random example, Windows RDP makes a HTTP call out to the Internet from the server to verify a CRL. This is safe and secure. Without this, bad certificates can't be blocked. Unfortunately, this happens in the "SYSTEM" context, which tends not get proxy settings applied to it, so it is often blocked, with a 30-second timeout. This failure is cached for 24 hours on the server, which causes a maddening delay when you connect to servers. Every day. Every server. But you can't reproduce it, because the second time it won't happen. (It also won't happen if anyone else connects to the server right before you.) Years later, I still get angry remembering the snarky comments by the firewall guy saying that I'm just imagining things.
toast0 4y ago
That's one reason, but there are several other common issues. At high volume mtu bottlenecks, the routers dropping packets are likely to limit how many ICMPs they send; otherwise they'll run out of CPU. This wouldn't be terrible if ICMPs weren't so commonly dropped. Some bottlenecked routers don't have globally routable IP addresses, and may not be able to send ICMP at all. This can happen inside long haul networks, where there may be tunneling that reduces the effective MTU. This seems less common today, but I dunno? One 'recent' advance is that some PPPoE networks use 'mini jumbo' frames, with an ethernet MTU of 1508, so that the PPP mtu is 1500; that may be happening on other layers with tunnels too.
- citrin_ru 4y ago
  Yes, unfortunately it is not hard to find routers without globally routable IP on a loopback or interfaces (whatever is used as the source for ICMP unreach in a given configuration) but if they have any IP at all, even a private IP, PMTUD still may work as long as ICMP messages with private source IP are not blocked by a firewall. But when a packet is dropped inside a carrier network (and not on its edges) because of a MTA mismatch it looks like a gross misconfiguration to me.
vlovich123 4y ago
Isn’t it baked into IPv6 as something that happens automatically? I’m seeing better traction with ISPs seeming to finally hand out IPv6 addresses.
- citrin_ru 4y ago
  IPv6 still has PMTUD but ICMP blocking in v6 is less common because blocking of all ICMPv6 messages breaks network in more obvious ways than blocking of all ICMP v4 messages.

jiggawatts 4y ago

The two things that boil my blood when dealing with SecOps people are: 1) Dropping all ICMP packets on the floor for "security" reasons, which means that basic diagnostics is now impossible, you get random issues with software that uses PING, and Path MTU Discovery is broken forever. 2) Leaving internal firewall ports with the default settings, which are intended for Internet-facing ports. For example, for "denied" incoming traffic from the Internet, the correct response is to silently drop the packet. Internally, the correct thing to do is to respond with a NACK to instantly close the connection. Without this, you spend days and days chasing down random and difficult-to-troubleshoot timeouts and weird 30-second delays all over the place. As a random example, Windows RDP makes a HTTP call out to the Internet from the server to verify a CRL. This is safe and secure. Without this, bad certificates can't be blocked. Unfortunately, this happens in the "SYSTEM" context, which tends not get proxy settings applied to it, so it is often blocked, with a 30-second timeout. This failure is cached for 24 hours on the server, which causes a maddening delay when you connect to servers. Every day. Every server. But you can't reproduce it, because the second time it won't happen. (It also won't happen if anyone else connects to the server right before you.) Years later, I still get angry remembering the snarky comments by the firewall guy saying that I'm just imagining things.

toast0 4y ago

That's one reason, but there are several other common issues. At high volume mtu bottlenecks, the routers dropping packets are likely to limit how many ICMPs they send; otherwise they'll run out of CPU. This wouldn't be terrible if ICMPs weren't so commonly dropped. Some bottlenecked routers don't have globally routable IP addresses, and may not be able to send ICMP at all. This can happen inside long haul networks, where there may be tunneling that reduces the effective MTU. This seems less common today, but I dunno? One 'recent' advance is that some PPPoE networks use 'mini jumbo' frames, with an ethernet MTU of 1508, so that the PPP mtu is 1500; that may be happening on other layers with tunnels too.

citrin_ru 4y ago
Yes, unfortunately it is not hard to find routers without globally routable IP on a loopback or interfaces (whatever is used as the source for ICMP unreach in a given configuration) but if they have any IP at all, even a private IP, PMTUD still may work as long as ICMP messages with private source IP are not blocked by a firewall. But when a packet is dropped inside a carrier network (and not on its edges) because of a MTA mismatch it looks like a gross misconfiguration to me.

vlovich123 4y ago

Isn’t it baked into IPv6 as something that happens automatically? I’m seeing better traction with ISPs seeming to finally hand out IPv6 addresses.

citrin_ru 4y ago
IPv6 still has PMTUD but ICMP blocking in v6 is less common because blocking of all ICMPv6 messages breaks network in more obvious ways than blocking of all ICMP v4 messages.