3 ms·

Doing a public post mortem is nice. But there's a bit of, uh, "derp" in the post. 1) Conflating MTU and MSS. MTU is the maximum size of the layer 3 IP datagram

Doing a public post mortem is nice. But there's a bit of, uh, "derp" in the post. 1) Conflating MTU and MSS. MTU is the maximum size of the layer 3 IP datagram. MSS is the maximum size of the layer 4 TCP data section. Specifically MSS does not include TCP/IP headers etc. MSS whould be influenced by MTU, but theyre not interchangeable. 2) All hosts must accept minimum IPv4 datagram size of 576 bytes. The minimum fragment size is 68 bytes. See RFC 791 "Total Length" and "Fragmentation and Reassembly." 3) Not enabling pmtu discovery probing by default. A significant number (3%?) of clients on the internet have broken ICMP MTU detection and MTU < 1500. 4) Using a routing platform that can't ECMP on the data portion of ICMP packets. This is straight WTF. How are you doing dest unreachable/ttl exceeded without this, for example? 5) Hashing on the four tuple. Kinda maybe? More often I see 5 tuple (+proto) or 7 tuple (+proto, vlan, tos). And sweet jesus I hope you've disabled ingress interface in the hash. 6) Picking magic numbers of 1280 & 1024 byte MTU. Why not look at what your clients are advertising in practice? It's recorded on every socket. 7) Naming your hack "Path MTU Daemon." Holy name collisions batman, PMTUD is already a thing the rest of the internet knows about. And its not your version! 8) Conflating asymmetric anycast failures with PMTUD failure. If the dst can't get ICMP back to the original source why would any other IP make it back? I'm not even sure how you'd get in this state, outside of flaps. Someone ECMP per packet including proto to put ICMP on a different path which then has an intermediate node with a different AS path or terrible PBR? Bizarro, and still dont see how the cloudflare "pmtud" would help.

majke 12y ago
Thanks for feedback! 1) Indeed. I tried to speak only about MTU but well, MSS creeped in a few times. OTOH I don't want to get into details, I want to keep the article short. 2) Point taken. 3) It's more complex. In IPv4, as mentioned the problem has been known for much longer. Enabling RFC4821 is not perfect solution, better fix your ICMP. That's why it's off by default on linux. 4) What can I say. Hardware is being hardware? 5) Simplification. I did say "for TCP" though. 6) 1280 is the safe choice for ipv6. For IPv4 it's a compromise. We do listen to what the client says, 1024 is a fallback when RFC4821 kicks in. 8) I specifically avoided going into the details of this. Look for future blog posts.

majke 12y ago

Thanks for feedback! 1) Indeed. I tried to speak only about MTU but well, MSS creeped in a few times. OTOH I don't want to get into details, I want to keep the article short. 2) Point taken. 3) It's more complex. In IPv4, as mentioned the problem has been known for much longer. Enabling RFC4821 is not perfect solution, better fix your ICMP. That's why it's off by default on linux. 4) What can I say. Hardware is being hardware? 5) Simplification. I did say "for TCP" though. 6) 1280 is the safe choice for ipv6. For IPv4 it's a compromise. We do listen to what the client says, 1024 is a fallback when RFC4821 kicks in. 8) I specifically avoided going into the details of this. Look for future blog posts.