4 ms·

in DC Networks, 9k frames and pathmtu is very common. the issue however is that the MTU across the internet is 1500, and you cannot selectively use mtu's depend

in DC Networks, 9k frames and pathmtu is very common. the issue however is that the MTU across the internet is 1500, and you cannot selectively use mtu's depending on routes, so you end up with 9k MTU at edge routers which need to split that into 1500 MTU packets. this fragmentation is absolutely brutal to performance of edge routers.

toast0 5y ago
> you cannot selectively use mtu's depending on routes You absolutely can. Assume a single network port on the server. On BSD, it looks something like this: ifconfig foo 10.x.y.z/24 mtu 9000 route add default 10.x.y.1 -mtu 1500 route add 10.0.0.0/8 10.x.y.1 Linux has a mss attribute on routes which should work similarly. Roll your own IPV6 version, but radvd can sort of maybe offer network routes I think, so it might be slightly easier to config it that way.
- r1ch 5y ago
  Doing this requires fragmenting at the IP level, which is notoriously unreliable over the internet. The other option is to send PMTU exceeded messages telling the sender to drop their MTU for the path, adding significant overhead for every new connection (assuming the sender even listens to such messages).
  - toast0 5y ago
    How so? In my example, the default route has a MTU of 1500 and the localnet route has a MTU of whatever the interface is (9000 in this example) When you send a packet, the destination route is looked up to determine the MTU. If you send to the internet, you'll get small packets (not IP fragments), and 10. will get large packets.
    - r1ch 5y ago
      That works fine for locally generated traffic which has knowledge of the routes. For hosts going through a router, they have no idea what the outbound link MTU is, requiring PMTU discovery or messing with TCP MSS.
      - toast0 5y ago
        The claim I replied to was that you couldn't run one MTU for some routes and one for other routes. You absolutely can. But running your default route at 9000 is ill-advised as you mention. If your datacenter routes aren't simply rfc1918 addresses, it would be harder to set the end host routes, and the effort to get larger packets on the wire might not be worth the rather small benefits.
doublerabbit 5y ago
Would it not be possible to have another network device before the edge router that encapsulates the packet to 1500? And vice versa for inbound packets? I have a rack in a DC, I've always wanted to configure jumbo frames between the host and VMs. However I haven't because I am not sure of the recourse when serving traffic.
- kazen44 5y ago
  ofcourse this would be possible (with just another router) but this would still result in inefficienties. IP Fragmentation is something which you do not want (it was removed from IPv6 for a good reason). TCP does not like it when IP packets get fragmented, as this usually results in lost PDU's when there is even the slightest of packet loss. Which causes TCP to resend the PDU.

toast0 5y ago

> you cannot selectively use mtu's depending on routes You absolutely can. Assume a single network port on the server. On BSD, it looks something like this: ifconfig foo 10.x.y.z/24 mtu 9000 route add default 10.x.y.1 -mtu 1500 route add 10.0.0.0/8 10.x.y.1 Linux has a mss attribute on routes which should work similarly. Roll your own IPV6 version, but radvd can sort of maybe offer network routes I think, so it might be slightly easier to config it that way.

r1ch 5y ago
Doing this requires fragmenting at the IP level, which is notoriously unreliable over the internet. The other option is to send PMTU exceeded messages telling the sender to drop their MTU for the path, adding significant overhead for every new connection (assuming the sender even listens to such messages).
- toast0 5y ago
  How so? In my example, the default route has a MTU of 1500 and the localnet route has a MTU of whatever the interface is (9000 in this example) When you send a packet, the destination route is looked up to determine the MTU. If you send to the internet, you'll get small packets (not IP fragments), and 10. will get large packets.
  - r1ch 5y ago
    That works fine for locally generated traffic which has knowledge of the routes. For hosts going through a router, they have no idea what the outbound link MTU is, requiring PMTU discovery or messing with TCP MSS.
    - toast0 5y ago
      The claim I replied to was that you couldn't run one MTU for some routes and one for other routes. You absolutely can. But running your default route at 9000 is ill-advised as you mention. If your datacenter routes aren't simply rfc1918 addresses, it would be harder to set the end host routes, and the effort to get larger packets on the wire might not be worth the rather small benefits.

doublerabbit 5y ago

Would it not be possible to have another network device before the edge router that encapsulates the packet to 1500? And vice versa for inbound packets? I have a rack in a DC, I've always wanted to configure jumbo frames between the host and VMs. However I haven't because I am not sure of the recourse when serving traffic.

kazen44 5y ago
ofcourse this would be possible (with just another router) but this would still result in inefficienties. IP Fragmentation is something which you do not want (it was removed from IPv6 for a good reason). TCP does not like it when IP packets get fragmented, as this usually results in lost PDU's when there is even the slightest of packet loss. Which causes TCP to resend the PDU.