3 ms·

Could it not also be the problematic traffic getting routed to a Cloudfront node that's overloaded or otherwise misbehaving? Years ago, I had to troubleshoot a

Could it not also be the problematic traffic getting routed to a Cloudfront node that's overloaded or otherwise misbehaving? Years ago, I had to troubleshoot a very mysterious issue where sometimes a huge percentage of the PCs in the company I worked for would have trouble loading the company's e-commerce site. Everyone was going through the same array of 2-3 proxy servers, and the e-commerce site had Akamai caching in front of it. The proxy servers would hold onto a DNS lookup result or a TCP connection for longer than they were supposed to, and so they'd continue routing traffic to the old IP address even after Akamai had rotated a node out of service.

kevin_nisbet 4y ago
> Could it not also be the problematic traffic getting routed to a Cloudfront node that's overloaded or otherwise misbehaving? This is plausible. Nothing I saw in the packet capture specifically pointed in this direction, but I don't really have the information I'd want to rule this out totally. I've seen issues like this when the TCP stack on the server effectively runs out of allocated memory. This can cause the server not to use all of the available tcp window. The only reason I think this is less likely, is I'd expect a company like AWS to monitor and tune the memory on their CDN, and notice something like a kernel bug here or block an attack exploiting the kernel memory. Some network providers, especially wireless providers also deploy TCP acceleration equipment, so that network speeds are faster. On a wireless carrier if you've got temporary bad signal and drop some packets, if the carrier can retransmit them instead of going to the internet, this makes for a faster connection. Crappier implementations of these TCP accelerators, do a sort of transparent redirect to the kernel TCP stack. So if the ATT proxy has run out of kernel memory, it could be providing a slow connection. It wouldn't surprise me for a telecom carrier to miss and not tune or monitor kernel memory. If the issue was caused by a proxy, I'd expect more to be broken then a single server, and other customers to also be complaining. If ATT is using these accelerators, it also means the proxy could be hiding packet loss and delays between the proxy and upstream server or network. Because the client TCP connection is only to the proxy, we only get to see what happened between the client and proxy. So this precluded drawing strong conclusions from the client side capture only. There is one thing about the packet capture that suggests a shaper to me. And that's how even the traffic is from server to client. While I didn't spend a bunch of time timing out all the packets, it looked a bit like I'd expect to see for packets getting released by a token bucket algorithm. This isn't necessarily safe to conclude, because the client has an offload that is buffering and merging packets using a receive segment offload, which causes us to miss some timing information in the capture. So based on the above, shaping looks a little bit more likely to me, but I don't have the right information to rule out a broken or miss-configured server. An overloaded or otherwise broken network link like errors on a link doesn't appear likely to me at all.

kevin_nisbet 4y ago

> Could it not also be the problematic traffic getting routed to a Cloudfront node that's overloaded or otherwise misbehaving? This is plausible. Nothing I saw in the packet capture specifically pointed in this direction, but I don't really have the information I'd want to rule this out totally. I've seen issues like this when the TCP stack on the server effectively runs out of allocated memory. This can cause the server not to use all of the available tcp window. The only reason I think this is less likely, is I'd expect a company like AWS to monitor and tune the memory on their CDN, and notice something like a kernel bug here or block an attack exploiting the kernel memory. Some network providers, especially wireless providers also deploy TCP acceleration equipment, so that network speeds are faster. On a wireless carrier if you've got temporary bad signal and drop some packets, if the carrier can retransmit them instead of going to the internet, this makes for a faster connection. Crappier implementations of these TCP accelerators, do a sort of transparent redirect to the kernel TCP stack. So if the ATT proxy has run out of kernel memory, it could be providing a slow connection. It wouldn't surprise me for a telecom carrier to miss and not tune or monitor kernel memory. If the issue was caused by a proxy, I'd expect more to be broken then a single server, and other customers to also be complaining. If ATT is using these accelerators, it also means the proxy could be hiding packet loss and delays between the proxy and upstream server or network. Because the client TCP connection is only to the proxy, we only get to see what happened between the client and proxy. So this precluded drawing strong conclusions from the client side capture only. There is one thing about the packet capture that suggests a shaper to me. And that's how even the traffic is from server to client. While I didn't spend a bunch of time timing out all the packets, it looked a bit like I'd expect to see for packets getting released by a token bucket algorithm. This isn't necessarily safe to conclude, because the client has an offload that is buffering and merging packets using a receive segment offload, which causes us to miss some timing information in the capture. So based on the above, shaping looks a little bit more likely to me, but I don't have the right information to rule out a broken or miss-configured server. An overloaded or otherwise broken network link like errors on a link doesn't appear likely to me at all.