5 ms·

MTUs are fun. We (live video streaming platform) were recently investigating issues with users having trouble watching our live video streams when using a VPN.

MTUs are fun. We (live video streaming platform) were recently investigating issues with users having trouble watching our live video streams when using a VPN. Since we're using WebRTC on the viewer end we thought immediately it was just some WebRTC "protection" trying to not leak the users IP. Eventually we figured out that Open Broadcaster Software has a compiled configuration of a max MTU of 1392 for the UDP packets we're sending. Generally this is fine because most routers have a default MTU of 1500, however when coupled with some of the VPN technologies, it ends up pushing the MTU over the limit and the video packets get dropped. Overall MTUs seem to be slightly a not well understood thing, because answers on the internet wildly vary for appropriate ways of handling it. The consensus from some Google/WebRTC folks seems to be that 1200 is a safe and fast default. > Anyway, 1200 bytes is 1280 bytes minus the RTP headers minus some bytes for RTP header extensions minus a few "let's play it safe" bytes. It'll usually work. - https://groups.google.com/g/discuss-webrtc/c/gH5ysR3SoZI?pli=1 https://groups.google.com/g/discuss-webrtc/c/gH5ysR3SoZI?pli... - https://stackoverflow.com/questions/47635545/why-webrtc-chose-rtp-max-packet-size-to-1200-bytes https://stackoverflow.com/questions/47635545/why-webrtc-chos...

ignoramous 5y ago
Reminds me of: https://github.com/tailscale/tailscale/blob/d9a7205be/net/tstun/tun.go#L23 https://github.com/tailscale/tailscale/blob/d9a7205be/net/ts... // tunMTU is the MTU we set on tailscale's TUN interface. wireguard-go // defaults to 1420 bytes, which only works if the "outer" MTU is 1500 // bytes. This breaks on DSL connections (typically 1492 MTU) and on // GCE (1460 MTU?!). // // 1280 is the smallest MTU allowed for IPv6, which is a sensible // "probably works everywhere" setting until we develop proper PMTU // discovery. And: https://datatracker.ietf.org/doc/rfc9000/ https://datatracker.ietf.org/doc/rfc9000/ (sec 14) The maximum datagram size is defined as the largest size of UDP payload that can be sent across a network path using a single UDP datagram. QUIC MUST NOT be used if the network path cannot support a maximum datagram size of at least 1200 bytes. QUIC assumes a minimum IP packet size of at least 1280 bytes. This is the IPv6 minimum size [IPv6] and is also supported by most modern IPv4 networks. Assuming the minimum IP header size of 40 bytes for IPv6 and 20 bytes for IPv4 and a UDP header size of 8 bytes, this results in a maximum datagram size of 1232 bytes for IPv6 and 1252 bytes for IPv4. Thus, modern IPv4 and all IPv6 network paths are expected to be able to support QUIC. | Note: This requirement to support a UDP payload of 1200 bytes | limits the space available for IPv6 extension headers to 32 | bytes or IPv4 options to 52 bytes if the path only supports the | IPv6 minimum MTU of 1280 bytes. This affects Initial packets | and path validation. Any maximum datagram size larger than 1200 bytes can be discovered using Path Maximum Transmission Unit Discovery (PMTUD) (see Section 14.2.1) or Datagram Packetization Layer PMTU Discovery (DPLPMTUD) (see Section 14.3).
- yardstick 5y ago
  1200 MTU often in my experience does the trick. But, I don’t like the presumption in IPv6 that everything supports 1280. What if there is a VPN running over a 1280-MTU link, what is the MTU of the encrypted IPv6 payload? Now add in a couple more layers of VPNs for good measure.
  - kazen44 5y ago
    with IPv6, one could also do MTU-path discovery. Which would automatically set the correct MTU based on a ICMP test between nodes and all endpoints.
    - nieve 5y ago
      Wasn't it established in the article that PMTUD was broken by the asymmetric routing?
  - phicoh 5y ago
    1280 is a guarantee by the IPv6 standard. If you run a VPN over a link with 1280 MTU then the VPN protocol has to have a way of fragmenting the payload (which could be using fragmentation in IPv6, but application specific is likely to work better).
    - yardstick 5y ago
      Yes I mostly agree, the tricky part is what to do about packets marked as Don’t Fragment. Hopefully they are already well under the actual MTU, but if need be, should they be fragmented or dropped? (The answer is to fragment all except icmp imo, even if it’s against the wishes of the packet IP headers)
      - phicoh 5y ago
        Technically the packets don't get fragmented. It is the tunnel that fragments. For example, in the bad old days, when there was still a lot of ATM, packets would routinely get transmitted in 48 byte pieces (with a 5 byte header). Nobody wondered whether this kind of fragmentation should honor the DF bit. If I recall correctly, multi-link PPP could also split a packet in smaller pieces to transmit it over multiple links at the same time. So for IPv6, anything that is 1280 octets or smaller needs to forwarded without triggering a packet too big ICMP. Otherwise the link violates the IPv6 standard.

ignoramous 5y ago

Reminds me of: https://github.com/tailscale/tailscale/blob/d9a7205be/net/tstun/tun.go#L23 https://github.com/tailscale/tailscale/blob/d9a7205be/net/ts... // tunMTU is the MTU we set on tailscale's TUN interface. wireguard-go // defaults to 1420 bytes, which only works if the "outer" MTU is 1500 // bytes. This breaks on DSL connections (typically 1492 MTU) and on // GCE (1460 MTU?!). // // 1280 is the smallest MTU allowed for IPv6, which is a sensible // "probably works everywhere" setting until we develop proper PMTU // discovery. And: https://datatracker.ietf.org/doc/rfc9000/ https://datatracker.ietf.org/doc/rfc9000/ (sec 14) The maximum datagram size is defined as the largest size of UDP payload that can be sent across a network path using a single UDP datagram. QUIC MUST NOT be used if the network path cannot support a maximum datagram size of at least 1200 bytes. QUIC assumes a minimum IP packet size of at least 1280 bytes. This is the IPv6 minimum size [IPv6] and is also supported by most modern IPv4 networks. Assuming the minimum IP header size of 40 bytes for IPv6 and 20 bytes for IPv4 and a UDP header size of 8 bytes, this results in a maximum datagram size of 1232 bytes for IPv6 and 1252 bytes for IPv4. Thus, modern IPv4 and all IPv6 network paths are expected to be able to support QUIC. | Note: This requirement to support a UDP payload of 1200 bytes | limits the space available for IPv6 extension headers to 32 | bytes or IPv4 options to 52 bytes if the path only supports the | IPv6 minimum MTU of 1280 bytes. This affects Initial packets | and path validation. Any maximum datagram size larger than 1200 bytes can be discovered using Path Maximum Transmission Unit Discovery (PMTUD) (see Section 14.2.1) or Datagram Packetization Layer PMTU Discovery (DPLPMTUD) (see Section 14.3).

yardstick 5y ago
1200 MTU often in my experience does the trick. But, I don’t like the presumption in IPv6 that everything supports 1280. What if there is a VPN running over a 1280-MTU link, what is the MTU of the encrypted IPv6 payload? Now add in a couple more layers of VPNs for good measure.
- kazen44 5y ago
  with IPv6, one could also do MTU-path discovery. Which would automatically set the correct MTU based on a ICMP test between nodes and all endpoints.
  - nieve 5y ago
    Wasn't it established in the article that PMTUD was broken by the asymmetric routing?
- phicoh 5y ago
  1280 is a guarantee by the IPv6 standard. If you run a VPN over a link with 1280 MTU then the VPN protocol has to have a way of fragmenting the payload (which could be using fragmentation in IPv6, but application specific is likely to work better).
  - yardstick 5y ago
    Yes I mostly agree, the tricky part is what to do about packets marked as Don’t Fragment. Hopefully they are already well under the actual MTU, but if need be, should they be fragmented or dropped? (The answer is to fragment all except icmp imo, even if it’s against the wishes of the packet IP headers)
    - phicoh 5y ago
      Technically the packets don't get fragmented. It is the tunnel that fragments. For example, in the bad old days, when there was still a lot of ATM, packets would routinely get transmitted in 48 byte pieces (with a 5 byte header). Nobody wondered whether this kind of fragmentation should honor the DF bit. If I recall correctly, multi-link PPP could also split a packet in smaller pieces to transmit it over multiple links at the same time. So for IPv6, anything that is 1280 octets or smaller needs to forwarded without triggering a packet too big ICMP. Otherwise the link violates the IPv6 standard.