4 ms·
> Meanwhile, the internet will be over here, still sending 1500 bytes at a time. Is this part true? Aren’t the larger data center interconnects using 9000 byte
by vlovich123 4y ago
> Meanwhile, the internet will be over here, still sending 1500 bytes at a time.
Is this part true? Aren’t the larger data center interconnects using 9000 bytes internally? Also, wouldn’t fat large distance links be using larger packets? I would have thought that a 4% wire overhead (and who knows how much CPU) would have pushed people away from 1500 for anything but last mile.
- patrakov 4y agoWith modern network cards, at the endpoints, the CPU overhead is zero because of TCP segmentation offload and generic receive offload. And routers use ASICs, not CPUs.
- shaklee3 4y agothis is not true at all. All modern routers use a combination of ASICs and CPUs. some of the data plane even runs through the CPU, depending on the protocol. also, TCP segmentation offload does not mean that the entire TCP protocol is offloaded onto the NIC. only a small part of it is and the rest of it still goes through CPU.
- richardwhiuk 4y agoWith TSO and GRO, the CPU deals with jumbo frames and the NIC handles splitting and combining them. You are correct that intermediate boxes all take a perf hit for the extra packets, because routers and other boxes are usually packets per second bound at some point, rather than bandwidth.
- tomohawk 4y agoHTTP/3 uses UDP. GSO can help with UDP, but is considerably more limited than TSO for TCP. The TCP interface to the kernel allows arbitrarily large buffers of data to be provided by the app, which the kernel/driver/card can segment. The UDP interface caps this at 64K
- shaklee3 4y agoYes, most large data centers use bigger frames internally. it's only when you get past the cgnat router that they go back to the compatible sizes.
- warent 4y agoI have CenturyLink business internet with a static IP. It has 1492 MTU. So technically 1500 isn't always true, sometimes it's smaller :)
- vlovich123 4y agoI know you’re just making a joke, but I was very careful about distinguishing MTU in the last mile from MTU within the routers powering the world’s interconnectivity.
- richardwhiuk 4y agoBy nature, interconnect effective MTU is governed by last mile MTU.
- CameronNemo 4y agoWe have two 100Gbps links with CenturyLink and we use an MTU of 1500.
- ikiris 4y agoNo one really peers at above 1500 without some really good reason. There be dragons.
- praptak 4y agoI remember observing my colleagues who had spent like a day troubleshooting an issue. The root cause was one end of a link sending super jumbo frames, which the other one was not configured to accept. It may sound simple but it was pretty hard for them to reproduce.
- passer_byer 4y agoPretty good if they did it in a day. Encountered the same issue and it took me at least three days to resolve. The issue presented itself as a database table not updating.
- dekhn 4y agoJumbo frames are just a guarantee that you'll have to debug a complicated problem in the future
- medellin 4y agoCan confirm. Although we used it fairly successfully in our datacenter. We had a few instances where setting the NIC config failed and someone ended up trying to debug a strange error who didn’t know anything about jumbo frames. It was normally only an hour before they reached out and we got it fixed. This normally only happened when servers arrived with new NICs and no one thought to give us a heads up.
- IYasha 4y agoCan you share a bit more detail regarding overall network performance before and after deploying jumbo frames in your DC?
- toast0 4y agoIt probably depends on who's running the DC, and how well traffic is separated. Path MTU detection is all sorts of fragile, so exposing a tcp maximum segment size implying a MTU above 1500 is asking for trouble; actually even just 1500 isn't always a good idea [1]. There are many networks where too big packets are dropped without notifications, and if the other end is also on a 9000 MTU LAN, then you can get stuck. sigh On the host side, packetization is not that big of a deal; nics can help, but it's also just not that many packets. Where hosts tend to run out of cpu is when you're dealing with much smaller packets. Larger packets could help routers of course. And larger packets would probably mean fewer acks, which would be nice too. But, it's unlikely to happen unless mtu probing is enabled in more places. [1] The best is to send the lesser of (theirs - 28) or your actual value. Well best is relative, this will result in the most successful connections, at the expense of more packets for networks where people aren't insane.
- citrin_ru 4y agoThe main reason why Path MTU detection is fragile is that many people ignorantly filter all ICMP traffic (including type 3). But educating people about ICMP and its role in PMTUD [1] seems like a lost battle. I wish to be proven wrong but I have an impressions that younger developers, devops/SRE and even network engineers know about network protocols less in than 10-20 years ago. Nowadays I rarely meet people how know what Path MTU Discovery is. I even stopped asking about it on job interviews (for devs/ops) because no one can answer, it's a luck if a candidate can tell anything about ICMP at all. [1] http://www.znep.com/~marcs/mtu/ http://www.znep.com/~marcs/mtu/
- jiggawatts 4y agoThe two things that boil my blood when dealing with SecOps people are: 1) Dropping all ICMP packets on the floor for "security" reasons, which means that basic diagnostics is now impossible, you get random issues with software that uses PING, and Path MTU Discovery is broken forever. 2) Leaving internal firewall ports with the default settings, which are intended for Internet-facing ports. For example, for "denied" incoming traffic from the Internet, the correct response is to silently drop the packet. Internally, the correct thing to do is to respond with a NACK to instantly close the connection. Without this, you spend days and days chasing down random and difficult-to-troubleshoot timeouts and weird 30-second delays all over the place. As a random example, Windows RDP makes a HTTP call out to the Internet from the server to verify a CRL. This is safe and secure. Without this, bad certificates can't be blocked. Unfortunately, this happens in the "SYSTEM" context, which tends not get proxy settings applied to it, so it is often blocked, with a 30-second timeout. This failure is cached for 24 hours on the server, which causes a maddening delay when you connect to servers. Every day. Every server. But you can't reproduce it, because the second time it won't happen. (It also won't happen if anyone else connects to the server right before you.) Years later, I still get angry remembering the snarky comments by the firewall guy saying that I'm just imagining things.
- dfex 4y agoLayer 2 (Ethernet) DCI will often be set up with 9000 (or larger) MTUs - the reason for this is to allow end-customers to carry their traffic using whatever encapsulation they want to internally (eg: Q-in-Q, MPLS, VXLAN, MACSEC, GRE or all of the above simultaneously), and the carrier/DC will transparently support it. But when it comes to external L3 peering between 3rd-parties (eg: the Internet), it's very rare that people will mess with the IP MTU. Unless you can guarantee that the IP PMTU is higher than 1500 bytes (across multiple segments that you may not control, including ISP last-mile) there is simply no benefit.
- hn8305823 4y agoSome interconnects between separate organizations are jumbo but those tend to be PNI or secondary vlans. The reason the Internet still runs exclusively on 1500 byte IP MTU is that any lower MTU in the path will effectively make the jumbo segments useless. This means any PMTU-D problems customers experience would have been unnecessary and avoidable by just using 1500 everywhere. Many public IX's (peering exchanges) have a jumbo vlan but it's almost always separate from the standard one that only allows 1500 byte MTU.