4 ms·

Hah, I just encountered this for a completed though similar reason. I'm still not entirely sure what was going on, but in Azure it seems like their network fabr

Hah, I just encountered this for a completed though similar reason. I'm still not entirely sure what was going on, but in Azure it seems like their network fabric that underpins VM networking stack were combining TCP packets into larger ones between two VMs. One of those was tunnelling the traffic elsewhere and was getting packets "from the network" with packet sizes up to 65k despite the MTU of the tunnel being set to 1400. The transmitting host was respecting the MTU. I had to force clamp the MSS to the PMTU in iptables to get things working again.

remar 6y ago
> I'm still not entirely sure what was going on, but in Azure FWIW, from someone who used to work on exactly this; neither did most of us :) The org had fired^Wtold most of the network engineers to either interview for dev positions or leave. That left the rest of us with only dev experience to have to learn DC networking from scratch (without any mentors) while also building the infra to automate it, because, well software engineers should be able to learn network engineering overnight right? I'll just say this, most of the devs I worked with had never coded before in a production environment, didn't know how to test their code (failing tests that would somehow pass in ways you'd think these were contest entries), and didn't even understand basic concepts in programming languages (I'm talking things like references vs values...). On top of this, the tooling and infrastructure was so bad, even if you wanted to be productive you'd be fighting things like your build environment just randomly breaking after a git pull and having to reinstall visual studio or reformatting windows to fix it. From what I've heard from people I still know on the inside, things haven't changed much (I'm very grateful for $MSFT though...)
bcrl 6y ago
The performance of modern network cards heavily leans on 2 so called stateless offloads: TCP/transmit segmentation offload and large receive offload (also call gso and gro). On transmit the kernel sends TCP packets in large chunks (~64KB) to the nic and the nic then puts them on the wire in MTU sized chunks as specified by the host. One receive, the nic tries to merge multiple small packets together into a single large packet. Both of these optimizations have the effect of reducing the number of packets per second the kernel and application have to handle, driving up throughput as a result. Naturally, this introduces a new class of subtle network behaviour that many folks aren't aware. This also results in tcpdump no longer showing what packets are being sent / received on the wire, making debugging tonnes of fun for the whole family!

remar 6y ago

> I'm still not entirely sure what was going on, but in Azure FWIW, from someone who used to work on exactly this; neither did most of us :) The org had fired^Wtold most of the network engineers to either interview for dev positions or leave. That left the rest of us with only dev experience to have to learn DC networking from scratch (without any mentors) while also building the infra to automate it, because, well software engineers should be able to learn network engineering overnight right? I'll just say this, most of the devs I worked with had never coded before in a production environment, didn't know how to test their code (failing tests that would somehow pass in ways you'd think these were contest entries), and didn't even understand basic concepts in programming languages (I'm talking things like references vs values...). On top of this, the tooling and infrastructure was so bad, even if you wanted to be productive you'd be fighting things like your build environment just randomly breaking after a git pull and having to reinstall visual studio or reformatting windows to fix it. From what I've heard from people I still know on the inside, things haven't changed much (I'm very grateful for $MSFT though...)

bcrl 6y ago

The performance of modern network cards heavily leans on 2 so called stateless offloads: TCP/transmit segmentation offload and large receive offload (also call gso and gro). On transmit the kernel sends TCP packets in large chunks (~64KB) to the nic and the nic then puts them on the wire in MTU sized chunks as specified by the host. One receive, the nic tries to merge multiple small packets together into a single large packet. Both of these optimizations have the effect of reducing the number of packets per second the kernel and application have to handle, driving up throughput as a result. Naturally, this introduces a new class of subtle network behaviour that many folks aren't aware. This also results in tcpdump no longer showing what packets are being sent / received on the wire, making debugging tonnes of fun for the whole family!