8 ms·

Operation Jumbo Drop: How sending large packets broke our AWS network

80 points by kiyanwang 5y ago

clone1018 5y ago
MTUs are fun. We (live video streaming platform) were recently investigating issues with users having trouble watching our live video streams when using a VPN. Since we're using WebRTC on the viewer end we thought immediately it was just some WebRTC "protection" trying to not leak the users IP. Eventually we figured out that Open Broadcaster Software has a compiled configuration of a max MTU of 1392 for the UDP packets we're sending. Generally this is fine because most routers have a default MTU of 1500, however when coupled with some of the VPN technologies, it ends up pushing the MTU over the limit and the video packets get dropped. Overall MTUs seem to be slightly a not well understood thing, because answers on the internet wildly vary for appropriate ways of handling it. The consensus from some Google/WebRTC folks seems to be that 1200 is a safe and fast default. > Anyway, 1200 bytes is 1280 bytes minus the RTP headers minus some bytes for RTP header extensions minus a few "let's play it safe" bytes. It'll usually work. - https://groups.google.com/g/discuss-webrtc/c/gH5ysR3SoZI?pli=1 https://groups.google.com/g/discuss-webrtc/c/gH5ysR3SoZI?pli... - https://stackoverflow.com/questions/47635545/why-webrtc-chose-rtp-max-packet-size-to-1200-bytes https://stackoverflow.com/questions/47635545/why-webrtc-chos...
- ignoramous 5y ago
  Reminds me of: https://github.com/tailscale/tailscale/blob/d9a7205be/net/tstun/tun.go#L23 https://github.com/tailscale/tailscale/blob/d9a7205be/net/ts... // tunMTU is the MTU we set on tailscale's TUN interface. wireguard-go // defaults to 1420 bytes, which only works if the "outer" MTU is 1500 // bytes. This breaks on DSL connections (typically 1492 MTU) and on // GCE (1460 MTU?!). // // 1280 is the smallest MTU allowed for IPv6, which is a sensible // "probably works everywhere" setting until we develop proper PMTU // discovery. And: https://datatracker.ietf.org/doc/rfc9000/ https://datatracker.ietf.org/doc/rfc9000/ (sec 14) The maximum datagram size is defined as the largest size of UDP payload that can be sent across a network path using a single UDP datagram. QUIC MUST NOT be used if the network path cannot support a maximum datagram size of at least 1200 bytes. QUIC assumes a minimum IP packet size of at least 1280 bytes. This is the IPv6 minimum size [IPv6] and is also supported by most modern IPv4 networks. Assuming the minimum IP header size of 40 bytes for IPv6 and 20 bytes for IPv4 and a UDP header size of 8 bytes, this results in a maximum datagram size of 1232 bytes for IPv6 and 1252 bytes for IPv4. Thus, modern IPv4 and all IPv6 network paths are expected to be able to support QUIC. | Note: This requirement to support a UDP payload of 1200 bytes | limits the space available for IPv6 extension headers to 32 | bytes or IPv4 options to 52 bytes if the path only supports the | IPv6 minimum MTU of 1280 bytes. This affects Initial packets | and path validation. Any maximum datagram size larger than 1200 bytes can be discovered using Path Maximum Transmission Unit Discovery (PMTUD) (see Section 14.2.1) or Datagram Packetization Layer PMTU Discovery (DPLPMTUD) (see Section 14.3).
  - yardstick 5y ago
    1200 MTU often in my experience does the trick. But, I don’t like the presumption in IPv6 that everything supports 1280. What if there is a VPN running over a 1280-MTU link, what is the MTU of the encrypted IPv6 payload? Now add in a couple more layers of VPNs for good measure.
    - kazen44 5y ago
      with IPv6, one could also do MTU-path discovery. Which would automatically set the correct MTU based on a ICMP test between nodes and all endpoints.
      - nieve 5y ago
        Wasn't it established in the article that PMTUD was broken by the asymmetric routing?
    - phicoh 5y ago
      1280 is a guarantee by the IPv6 standard. If you run a VPN over a link with 1280 MTU then the VPN protocol has to have a way of fragmenting the payload (which could be using fragmentation in IPv6, but application specific is likely to work better).
      - yardstick 5y ago
        Yes I mostly agree, the tricky part is what to do about packets marked as Don’t Fragment. Hopefully they are already well under the actual MTU, but if need be, should they be fragmented or dropped? (The answer is to fragment all except icmp imo, even if it’s against the wishes of the packet IP headers)
        phicoh 5y ago
        Technically the packets don't get fragmented. It is the tunnel that fragments. For example, in the bad old days, when there was still a lot of ATM, packets would routinely get transmitted in 48 byte pieces (with a 5 byte header). Nobody wondered whether this kind of fragmentation should honor the DF bit. If I recall correctly, multi-link PPP could also split a packet in smaller pieces to transmit it over multiple links at the same time. So for IPv6, anything that is 1280 octets or smaller needs to forwarded without triggering a packet too big ICMP. Otherwise the link violates the IPv6 standard.
wmf 5y ago
the AMIs AWS provides launch EC2 instances with the MTU set to 9001 ... A transit gateway supports an MTU of 8500 bytes ... The transit gateway does not generate the FRAG_NEEDED for ICMPv4 packet, or the Packet Too Big (PTB) for ICMPv6 packet. Therefore, the Path MTU Discovery (PMTUD) is not supported. Quite the footgun you've got there, AWS.
- tempnow987 5y ago
  It does set MSS so part of the reason they had this issue is I think related to asymmetric paths?
  - lima 5y ago
    MSS clamping is a nasty hack that works only for TCP.
    - tempnow987 5y ago
      Fair point. I'm not experienced, does path discovery work with PSH set?
      - sleepydog 5y ago
        Path discovery is agnostic of the transport protocol. Once an IP datagram with the "Don't Fragment" bit set reaches a device that cannot forward it because it is too large, the device must send an ICMP error back to the originating IP address with the correct MTU. The sender must adjust then adjust the MTU for the same destination for future datagrams. The sender may have to do this discovery multiple times, if each smaller packet gets progressively further, but doesn't reach the destination. This whole process is thwarted by overeager network administrators blocking ICMP traffic, or, apparently, by devices like this "transit gateway" that don't generate the ICMP packet at all. Edit: Sorry, I think I may have fallen into the trap of explaining something that you already knew and not answering your question. I realized this after jsnell's comment elsewhere that the kernel may not want to split packets with the PSH bit set. I wasn't able to find an answer to your question in linux/net/ipv4/tcp_output.c, hopefully someone more familiar with Linux's TCP can chime in.
deleted 5y ago
[deleted]
paleotrope 5y ago
I think someone posted about this same exact issue a few weeks ago. I wouldn't mix VPC peering and transit gateway if you avoid it.
- nijave 5y ago
  I guess if you're migrating from VPC peering to Transit Gateways you have to coordinate updating the routing tables in both VPCs at the same time (I'm guessing they probably updated routing tables in one VPC but hadn't updated them in the other yet) unless you adjust MTUs ahead of time. Seems to be a crappy migration path (execute these updates quickly and hopefully you don't drop too many packets before they apply)
  - deleted 5y ago
    [deleted]
jsnell 5y ago
I think the case is a little bit more subtle than described. Linux doesn't actually need ICMP-based path MTU discovery to figure out the safe MSS to use. It implements RFC 4821-style MTU probing, where after repeated retransmissions the sender will try to retransmit smaller prefixes of the segment and see if that makes it through. If it does, it can be deduced that the safe MSS is lower than advertised, and the MSS for the connection is (permanently) dropped to the newly discovered limit. So even if the ICMP return path isn't working, the connection should just have a hickup for a few seconds but then resume working. Why isn't MTU probing triggering here? I haven't checked the code, but would guess that it's refusing to split a segment with the PSH flag set.
- touisteur 5y ago
  Fun fact, was observing periodic latency peaks on some customer network. TCP analysis would show a resend after an icmp 'too big' packet from a middlebox. Fine. Once at connection establishment or rather at the first PSH over the default MTU, right? But why periodically? Well Linux re-raises the path mtu periodically, iirc in case of icmp spoofing... Is it different here for mss-based discovery? You say permanently?
  - jsnell 5y ago
    That was my recollection, but looking at the source I don't think that was correct.
    - touisteur 5y ago
      Ah thanks. Will have to open tcp.c, but it gets crazier every year it might take some time... It would make sense to keep it permanently if the justification is purely based on the risk of icmp spoofing. But TCP connections can be very long-lived and the network might change under your feet, even worst if you're doing some bonding/teaming or container-/connection-migration, it might be a good idea to probe the path mtu again. It's just a pain from a latency perspective. I wish I could skip a bit of a stream without re-establishing the full connection.
tlb 5y ago
The sad thing is, there's no reason for any modern system to require a small MTU. MTUs of only a thousand bytes made sense on sub-megabit networks (to limit jitter), and when RAM was purchased by the kilobyte. But there's no reason why every network shouldn't support 64 kB MTUs today. A required max of 64 kB with a default segment size of 16 kB would improve performance and allow any amount of headers to be added.
- drewg123 5y ago
  I have exactly the opposite view. With offloads like TSO and LRO, NICs can pretend the network supports very large "packets", while still segmenting traffic into standard 1500b chunks. From a hardware perspective, supporting jumbo frames makes everything harder. You need to have larger buffers. Especially when dealing with ethernet flow control. And latencies increase, because most NICs do store-and-forward, and need to hold onto the packet until the end arrives so they can calculate the CRC.
  - jeffbee 5y ago
    What about the receive side? Suppose you have TSO but the receiver still gets a train of smaller frames, which consumes extra CPU. The solution to ethernet flow control is to disable it, of course.
    - drewg123 5y ago
      The solution on the receive side is LRO. I was doing 10GbE line rate with 1500b frames in 2006 or so on consumer-grade CPUs using software LRO (implemented in the driver). I worked for a NIC vendor (Myricom) that shipped our NIC drivers with a default 9000b MTU. That was fine when we just did HPC & everything reachable via layer 2 was another one of our NICs with a 9k MTU. But it was a mess when we started selling 10GbE NICs. I got pretty good at diagnosing MTU mis-matches..
      - jeffbee 5y ago
        I don't doubt that you can hit 10g on a wimpy core, the question is how much is left over for your application to use. I recently analyzed jumbo vs. puny frames in EC2 with the ENA and the difference in softirq time amounted to one core out of the 32 cores on the instance type, at 12.5gbps line rate. That's pretty substantial cost when you have a lot of nodes. The ENA doesn't have a ton of offload features and EC2 tenants don't get to specify a better NIC.
ethbr0 5y ago
For those who don't get the '95 reference: https://m.youtube.com/watch?v=19JXZcmMJx0 https://m.youtube.com/watch?v=19JXZcmMJx0 (From a time when Disney made Vietnam War comedies with ensemble casts, instead of superhero movies)
IYasha 5y ago
Pumping up MTUs is mostly viable on a file transfer networks. Tried it on 10G NAS - worked pretty well. But that's LAN, where retransmits are pretty non-existent. On the internet larger damaged packets will cause larger retransfers. And one uncapable router/switch can ruin the whole profit.
- sneak 5y ago
  Fun fact: the default mainline kernel driver (or at least the one used by Raspbian and Ubuntu) for the Raspberry Pi gigabit interface does not support jumbo frames without patching and recompiling the kernel. :/ This is a colossal pain if you run your home lan at 9000 for the aforementioned storage/NAS reason.
  - IYasha 5y ago
    Wow. I haven't used Raspberry, but I hope someone fixes that. Another thing I forgot to mention - MTU mismatch is hard to debug. Everything works until you hit the packet limit. I had a very funny case when I worked through SSH console normally until I maximized its window )) Also, diferent devices have different ways of setting MTU up - with or without headers, etc. In my case there was linux-based NAS, proprietary managed switch, linux, win7 and android clients. And because of the Ethernet-to-USB adapter I had to revert the whole LAN MTU setup to minimum.
watersb 5y ago
So, not about kicking elephants out of perfectly flying airplanes. Which would also probably break an AWS network. https://www.spokesman.com/stories/1995/jul/31/elephant-airlift-the-story-behind-dumbo-drop-is/ https://www.spokesman.com/stories/1995/jul/31/elephant-airli...
pqcsam 5y ago
Did you try 9000? You are right around the nebulous jumbo frame mtu, and there are different implementations.
colanderman 5y ago
Does anyone know why AWS's jumbo frame MTU is 9001 octets? Every other system I've worked on used an MTU of 9000 for jumbo frames. The only reason I can speculate is to force the use of Path MTU Discovery when using Direct Connects (similar to how Microsoft added spaces to common paths to force correct filename handling).
- spaceprison 5y ago
  9216 is the typical supported mtu for ethernet. But earlier iterations of their network used devices whose jumbo mtu length was like 9100 bytes. Aws uses A LOT of different overlay technologies which are implemented through encapsulation(aka overhead). To enable cross region vpc connectivity, additional encryption and encapsulation can be layered on top of the existing encapsulation. 9001 bytes was the absolute max they could give to customers and still be able to do packet shenanigans.

80 points by kiyanwang 5y ago

MTUs are fun. We (live video streaming platform) were recently investigating issues with users having trouble watching our live video streams when using a VPN. Since we're using WebRTC on the viewer end we thought immediately it was just some WebRTC "protection" trying to not leak the users IP. Eventually we figured out that Open Broadcaster Software has a compiled configuration of a max MTU of 1392 for the UDP packets we're sending. Generally this is fine because most routers have a default MTU of 1500, however when coupled with some of the VPN technologies, it ends up pushing the MTU over the limit and the video packets get dropped. Overall MTUs seem to be slightly a not well understood thing, because answers on the internet wildly vary for appropriate ways of handling it. The consensus from some Google/WebRTC folks seems to be that 1200 is a safe and fast default. > Anyway, 1200 bytes is 1280 bytes minus the RTP headers minus some bytes for RTP header extensions minus a few "let's play it safe" bytes. It'll usually work. - https://groups.google.com/g/discuss-webrtc/c/gH5ysR3SoZI?pli=1 https://groups.google.com/g/discuss-webrtc/c/gH5ysR3SoZI?pli... - https://stackoverflow.com/questions/47635545/why-webrtc-chose-rtp-max-packet-size-to-1200-bytes https://stackoverflow.com/questions/47635545/why-webrtc-chos...

ignoramous 5y ago
Reminds me of: https://github.com/tailscale/tailscale/blob/d9a7205be/net/tstun/tun.go#L23 https://github.com/tailscale/tailscale/blob/d9a7205be/net/ts... // tunMTU is the MTU we set on tailscale's TUN interface. wireguard-go // defaults to 1420 bytes, which only works if the "outer" MTU is 1500 // bytes. This breaks on DSL connections (typically 1492 MTU) and on // GCE (1460 MTU?!). // // 1280 is the smallest MTU allowed for IPv6, which is a sensible // "probably works everywhere" setting until we develop proper PMTU // discovery. And: https://datatracker.ietf.org/doc/rfc9000/ https://datatracker.ietf.org/doc/rfc9000/ (sec 14) The maximum datagram size is defined as the largest size of UDP payload that can be sent across a network path using a single UDP datagram. QUIC MUST NOT be used if the network path cannot support a maximum datagram size of at least 1200 bytes. QUIC assumes a minimum IP packet size of at least 1280 bytes. This is the IPv6 minimum size [IPv6] and is also supported by most modern IPv4 networks. Assuming the minimum IP header size of 40 bytes for IPv6 and 20 bytes for IPv4 and a UDP header size of 8 bytes, this results in a maximum datagram size of 1232 bytes for IPv6 and 1252 bytes for IPv4. Thus, modern IPv4 and all IPv6 network paths are expected to be able to support QUIC. | Note: This requirement to support a UDP payload of 1200 bytes | limits the space available for IPv6 extension headers to 32 | bytes or IPv4 options to 52 bytes if the path only supports the | IPv6 minimum MTU of 1280 bytes. This affects Initial packets | and path validation. Any maximum datagram size larger than 1200 bytes can be discovered using Path Maximum Transmission Unit Discovery (PMTUD) (see Section 14.2.1) or Datagram Packetization Layer PMTU Discovery (DPLPMTUD) (see Section 14.3).
- yardstick 5y ago
  1200 MTU often in my experience does the trick. But, I don’t like the presumption in IPv6 that everything supports 1280. What if there is a VPN running over a 1280-MTU link, what is the MTU of the encrypted IPv6 payload? Now add in a couple more layers of VPNs for good measure.
  - kazen44 5y ago
    with IPv6, one could also do MTU-path discovery. Which would automatically set the correct MTU based on a ICMP test between nodes and all endpoints.
    - nieve 5y ago
      Wasn't it established in the article that PMTUD was broken by the asymmetric routing?
  - phicoh 5y ago
    1280 is a guarantee by the IPv6 standard. If you run a VPN over a link with 1280 MTU then the VPN protocol has to have a way of fragmenting the payload (which could be using fragmentation in IPv6, but application specific is likely to work better).
    - yardstick 5y ago
      Yes I mostly agree, the tricky part is what to do about packets marked as Don’t Fragment. Hopefully they are already well under the actual MTU, but if need be, should they be fragmented or dropped? (The answer is to fragment all except icmp imo, even if it’s against the wishes of the packet IP headers)
      - phicoh 5y ago
        Technically the packets don't get fragmented. It is the tunnel that fragments. For example, in the bad old days, when there was still a lot of ATM, packets would routinely get transmitted in 48 byte pieces (with a 5 byte header). Nobody wondered whether this kind of fragmentation should honor the DF bit. If I recall correctly, multi-link PPP could also split a packet in smaller pieces to transmit it over multiple links at the same time. So for IPv6, anything that is 1280 octets or smaller needs to forwarded without triggering a packet too big ICMP. Otherwise the link violates the IPv6 standard.

wmf 5y ago

the AMIs AWS provides launch EC2 instances with the MTU set to 9001 ... A transit gateway supports an MTU of 8500 bytes ... The transit gateway does not generate the FRAG_NEEDED for ICMPv4 packet, or the Packet Too Big (PTB) for ICMPv6 packet. Therefore, the Path MTU Discovery (PMTUD) is not supported. Quite the footgun you've got there, AWS.

tempnow987 5y ago
It does set MSS so part of the reason they had this issue is I think related to asymmetric paths?
- lima 5y ago
  MSS clamping is a nasty hack that works only for TCP.
  - tempnow987 5y ago
    Fair point. I'm not experienced, does path discovery work with PSH set?
    - sleepydog 5y ago
      Path discovery is agnostic of the transport protocol. Once an IP datagram with the "Don't Fragment" bit set reaches a device that cannot forward it because it is too large, the device must send an ICMP error back to the originating IP address with the correct MTU. The sender must adjust then adjust the MTU for the same destination for future datagrams. The sender may have to do this discovery multiple times, if each smaller packet gets progressively further, but doesn't reach the destination. This whole process is thwarted by overeager network administrators blocking ICMP traffic, or, apparently, by devices like this "transit gateway" that don't generate the ICMP packet at all. Edit: Sorry, I think I may have fallen into the trap of explaining something that you already knew and not answering your question. I realized this after jsnell's comment elsewhere that the kernel may not want to split packets with the PSH bit set. I wasn't able to find an answer to your question in linux/net/ipv4/tcp_output.c, hopefully someone more familiar with Linux's TCP can chime in.

deleted 5y ago

[deleted]

paleotrope 5y ago

I think someone posted about this same exact issue a few weeks ago. I wouldn't mix VPC peering and transit gateway if you avoid it.

nijave 5y ago
I guess if you're migrating from VPC peering to Transit Gateways you have to coordinate updating the routing tables in both VPCs at the same time (I'm guessing they probably updated routing tables in one VPC but hadn't updated them in the other yet) unless you adjust MTUs ahead of time. Seems to be a crappy migration path (execute these updates quickly and hopefully you don't drop too many packets before they apply)
- deleted 5y ago
  [deleted]

jsnell 5y ago

I think the case is a little bit more subtle than described. Linux doesn't actually need ICMP-based path MTU discovery to figure out the safe MSS to use. It implements RFC 4821-style MTU probing, where after repeated retransmissions the sender will try to retransmit smaller prefixes of the segment and see if that makes it through. If it does, it can be deduced that the safe MSS is lower than advertised, and the MSS for the connection is (permanently) dropped to the newly discovered limit. So even if the ICMP return path isn't working, the connection should just have a hickup for a few seconds but then resume working. Why isn't MTU probing triggering here? I haven't checked the code, but would guess that it's refusing to split a segment with the PSH flag set.

touisteur 5y ago
Fun fact, was observing periodic latency peaks on some customer network. TCP analysis would show a resend after an icmp 'too big' packet from a middlebox. Fine. Once at connection establishment or rather at the first PSH over the default MTU, right? But why periodically? Well Linux re-raises the path mtu periodically, iirc in case of icmp spoofing... Is it different here for mss-based discovery? You say permanently?
- jsnell 5y ago
  That was my recollection, but looking at the source I don't think that was correct.
  - touisteur 5y ago
    Ah thanks. Will have to open tcp.c, but it gets crazier every year it might take some time... It would make sense to keep it permanently if the justification is purely based on the risk of icmp spoofing. But TCP connections can be very long-lived and the network might change under your feet, even worst if you're doing some bonding/teaming or container-/connection-migration, it might be a good idea to probe the path mtu again. It's just a pain from a latency perspective. I wish I could skip a bit of a stream without re-establishing the full connection.

tlb 5y ago

The sad thing is, there's no reason for any modern system to require a small MTU. MTUs of only a thousand bytes made sense on sub-megabit networks (to limit jitter), and when RAM was purchased by the kilobyte. But there's no reason why every network shouldn't support 64 kB MTUs today. A required max of 64 kB with a default segment size of 16 kB would improve performance and allow any amount of headers to be added.

drewg123 5y ago
I have exactly the opposite view. With offloads like TSO and LRO, NICs can pretend the network supports very large "packets", while still segmenting traffic into standard 1500b chunks. From a hardware perspective, supporting jumbo frames makes everything harder. You need to have larger buffers. Especially when dealing with ethernet flow control. And latencies increase, because most NICs do store-and-forward, and need to hold onto the packet until the end arrives so they can calculate the CRC.
- jeffbee 5y ago
  What about the receive side? Suppose you have TSO but the receiver still gets a train of smaller frames, which consumes extra CPU. The solution to ethernet flow control is to disable it, of course.
  - drewg123 5y ago
    The solution on the receive side is LRO. I was doing 10GbE line rate with 1500b frames in 2006 or so on consumer-grade CPUs using software LRO (implemented in the driver). I worked for a NIC vendor (Myricom) that shipped our NIC drivers with a default 9000b MTU. That was fine when we just did HPC & everything reachable via layer 2 was another one of our NICs with a 9k MTU. But it was a mess when we started selling 10GbE NICs. I got pretty good at diagnosing MTU mis-matches..
    - jeffbee 5y ago
      I don't doubt that you can hit 10g on a wimpy core, the question is how much is left over for your application to use. I recently analyzed jumbo vs. puny frames in EC2 with the ENA and the difference in softirq time amounted to one core out of the 32 cores on the instance type, at 12.5gbps line rate. That's pretty substantial cost when you have a lot of nodes. The ENA doesn't have a ton of offload features and EC2 tenants don't get to specify a better NIC.

ethbr0 5y ago

For those who don't get the '95 reference: https://m.youtube.com/watch?v=19JXZcmMJx0 https://m.youtube.com/watch?v=19JXZcmMJx0 (From a time when Disney made Vietnam War comedies with ensemble casts, instead of superhero movies)

IYasha 5y ago

Pumping up MTUs is mostly viable on a file transfer networks. Tried it on 10G NAS - worked pretty well. But that's LAN, where retransmits are pretty non-existent. On the internet larger damaged packets will cause larger retransfers. And one uncapable router/switch can ruin the whole profit.

sneak 5y ago
Fun fact: the default mainline kernel driver (or at least the one used by Raspbian and Ubuntu) for the Raspberry Pi gigabit interface does not support jumbo frames without patching and recompiling the kernel. :/ This is a colossal pain if you run your home lan at 9000 for the aforementioned storage/NAS reason.
- IYasha 5y ago
  Wow. I haven't used Raspberry, but I hope someone fixes that. Another thing I forgot to mention - MTU mismatch is hard to debug. Everything works until you hit the packet limit. I had a very funny case when I worked through SSH console normally until I maximized its window )) Also, diferent devices have different ways of setting MTU up - with or without headers, etc. In my case there was linux-based NAS, proprietary managed switch, linux, win7 and android clients. And because of the Ethernet-to-USB adapter I had to revert the whole LAN MTU setup to minimum.

watersb 5y ago

So, not about kicking elephants out of perfectly flying airplanes. Which would also probably break an AWS network. https://www.spokesman.com/stories/1995/jul/31/elephant-airlift-the-story-behind-dumbo-drop-is/ https://www.spokesman.com/stories/1995/jul/31/elephant-airli...

pqcsam 5y ago

Did you try 9000? You are right around the nebulous jumbo frame mtu, and there are different implementations.

colanderman 5y ago

Does anyone know why AWS's jumbo frame MTU is 9001 octets? Every other system I've worked on used an MTU of 9000 for jumbo frames. The only reason I can speculate is to force the use of Path MTU Discovery when using Direct Connects (similar to how Microsoft added spaces to common paths to force correct filename handling).

spaceprison 5y ago
9216 is the typical supported mtu for ethernet. But earlier iterations of their network used devices whose jumbo mtu length was like 9100 bytes. Aws uses A LOT of different overlay technologies which are implemented through encapsulation(aka overhead). To enable cross region vpc connectivity, additional encryption and encapsulation can be layered on top of the existing encapsulation. 9001 bytes was the absolute max they could give to customers and still be able to do packet shenanigans.