5 ms·

Taking a quick 5 minute look at the packet capture attached to the post, it looks to me like this is likely traffic shaping. While it's always difficult to be 1

Taking a quick 5 minute look at the packet capture attached to the post, it looks to me like this is likely traffic shaping. While it's always difficult to be 100% sure from just the client side capture, the capture looks relatively clean of errors, and the amount of data in flight doesn't appear to even approach the advertised window. We're getting some merging of segments likely from a segment offload, but I doubt that's throwing off the results. So there's a good chance there's a shaper letting through about 320kbit/s as it's relatively even throughout the capture.

acaloiar 4y ago
Thanks for the feedback. I was hoping to see more of this sort of analysis from the HN crowd :) There's one other person from HN doing some analysis with me via email. Maybe something will come of it. You both came to the same conclusion based on the pcap data. They're also analyzing the iphone tether pcap data I just provided in an update.
- kevin_nisbet 4y ago
  You're welcome. I took a very quick look at the iPhone capture. I don't see any notable issue (which is expected since this is "working" behavour), and the main noticeable item in the comparison is an apparent lower latency to the server. This appears to match the speculation I saw in the updates as well. One thing you might be able to try, if you're able to change the APN on the hotspot, is to see if you're able to connect with a different APN from the consumer side. This is pure speculation, but what some carriers do is use a different APN for their business services, that will route to enterprise only packet gateways. Some carriers may not necessarily disable access to the consumer APNs though that might go through a different P-GW that's closer to you. The iPhone looks like it's using a path about 120ms shorter than the hotspot. And just by chance, going through a different P-GW you might hit different traffic shaping equipment or configuration. Edit: although be aware that the carrier could also charge different APNs separately. I have no idea what AT&T does.
  - acaloiar 4y ago
    I considered switching APNs, but didn't want to run afoul of any AT&T rules. I went ahead and gave it a try anyway, switching from `broadband` to `NXTGENPHONE`. I wasn't able to authenticate with `NXTGENPHONE`, so I switched it back to `broadband`. After re-authenticating, I'm no longer being throttled. I'm both happy and deeply dissatisfied with this outcome because I think now I'm less likely to get to the bottom of it. I'll add this to my updates and keep monitoring the situation. Thanks, I think!
    - kevin_nisbet 4y ago
      You're welcome. This just reminded me of the IT Crowd... Have you tried turning it off and on again. By disconnecting / reconnecting by trying the APN switch, it might've just put you on a different P-GW that has a different traffic shaping configuration or current state.
      - acaloiar 4y ago
        It'll never get old or become poor advice :)
    - neurostimulant 4y ago
      The only way to get to the bottom of this is probably just applying for a relevant job at AT&T, get accepted by whatever means necessary (low asking salary, nepotism, etc), check their network configuration yourself, then immediately submit your resignation once you're satisfied. I think I read somewhere about someone who did exactly like this in a software company. A new employee showed up, immediately fixed a long-standing bug, then submit their resignation.
vitus 4y ago
I do see a couple of retransmissions early in that query (lost ACKs?) which could be upstream throttling but in any event certainly explains why the connection was slow to get off the ground. Agreed that segment offload is probably kernel-side coalescing. From shunting this into Wireshark, you can see that the sequence number grows fairly linearly (often a good indicator of a rate limiter) starting from 1.508s up until the 1.95s mark, but at a rate closer to 25-30 Mbps (1.5MB in 0.45s). https://i.imgur.com/s2JBrCf.png https://i.imgur.com/s2JBrCf.png edit: this was for iphone-capture.pcap. You see a similarly strong picture of shaping for capture.pcap [0], and indeed that corresponds to the ~320kbps range you've noted. [0] https://i.imgur.com/g9cN95d.png https://i.imgur.com/g9cN95d.png edit edit: of course, under normal situations, your bandwidth will get saturated, and steady-state will look fairly linear. If you're using "classic" congestion control like CUBIC or New Reno, you'll see this interspersed with drops (search for "tcp sawtooth"). Under BBR you'd expect to see drops in bandwidth due to latency spikes (per bufferbloat). Neither of those seem to be in play here.
blincoln 4y ago
Could it not also be the problematic traffic getting routed to a Cloudfront node that's overloaded or otherwise misbehaving? Years ago, I had to troubleshoot a very mysterious issue where sometimes a huge percentage of the PCs in the company I worked for would have trouble loading the company's e-commerce site. Everyone was going through the same array of 2-3 proxy servers, and the e-commerce site had Akamai caching in front of it. The proxy servers would hold onto a DNS lookup result or a TCP connection for longer than they were supposed to, and so they'd continue routing traffic to the old IP address even after Akamai had rotated a node out of service.
- kevin_nisbet 4y ago
  > Could it not also be the problematic traffic getting routed to a Cloudfront node that's overloaded or otherwise misbehaving? This is plausible. Nothing I saw in the packet capture specifically pointed in this direction, but I don't really have the information I'd want to rule this out totally. I've seen issues like this when the TCP stack on the server effectively runs out of allocated memory. This can cause the server not to use all of the available tcp window. The only reason I think this is less likely, is I'd expect a company like AWS to monitor and tune the memory on their CDN, and notice something like a kernel bug here or block an attack exploiting the kernel memory. Some network providers, especially wireless providers also deploy TCP acceleration equipment, so that network speeds are faster. On a wireless carrier if you've got temporary bad signal and drop some packets, if the carrier can retransmit them instead of going to the internet, this makes for a faster connection. Crappier implementations of these TCP accelerators, do a sort of transparent redirect to the kernel TCP stack. So if the ATT proxy has run out of kernel memory, it could be providing a slow connection. It wouldn't surprise me for a telecom carrier to miss and not tune or monitor kernel memory. If the issue was caused by a proxy, I'd expect more to be broken then a single server, and other customers to also be complaining. If ATT is using these accelerators, it also means the proxy could be hiding packet loss and delays between the proxy and upstream server or network. Because the client TCP connection is only to the proxy, we only get to see what happened between the client and proxy. So this precluded drawing strong conclusions from the client side capture only. There is one thing about the packet capture that suggests a shaper to me. And that's how even the traffic is from server to client. While I didn't spend a bunch of time timing out all the packets, it looked a bit like I'd expect to see for packets getting released by a token bucket algorithm. This isn't necessarily safe to conclude, because the client has an offload that is buffering and merging packets using a receive segment offload, which causes us to miss some timing information in the capture. So based on the above, shaping looks a little bit more likely to me, but I don't have the right information to rule out a broken or miss-configured server. An overloaded or otherwise broken network link like errors on a link doesn't appear likely to me at all.

acaloiar 4y ago

Thanks for the feedback. I was hoping to see more of this sort of analysis from the HN crowd :) There's one other person from HN doing some analysis with me via email. Maybe something will come of it. You both came to the same conclusion based on the pcap data. They're also analyzing the iphone tether pcap data I just provided in an update.

kevin_nisbet 4y ago
You're welcome. I took a very quick look at the iPhone capture. I don't see any notable issue (which is expected since this is "working" behavour), and the main noticeable item in the comparison is an apparent lower latency to the server. This appears to match the speculation I saw in the updates as well. One thing you might be able to try, if you're able to change the APN on the hotspot, is to see if you're able to connect with a different APN from the consumer side. This is pure speculation, but what some carriers do is use a different APN for their business services, that will route to enterprise only packet gateways. Some carriers may not necessarily disable access to the consumer APNs though that might go through a different P-GW that's closer to you. The iPhone looks like it's using a path about 120ms shorter than the hotspot. And just by chance, going through a different P-GW you might hit different traffic shaping equipment or configuration. Edit: although be aware that the carrier could also charge different APNs separately. I have no idea what AT&T does.
- acaloiar 4y ago
  I considered switching APNs, but didn't want to run afoul of any AT&T rules. I went ahead and gave it a try anyway, switching from `broadband` to `NXTGENPHONE`. I wasn't able to authenticate with `NXTGENPHONE`, so I switched it back to `broadband`. After re-authenticating, I'm no longer being throttled. I'm both happy and deeply dissatisfied with this outcome because I think now I'm less likely to get to the bottom of it. I'll add this to my updates and keep monitoring the situation. Thanks, I think!
  - kevin_nisbet 4y ago
    You're welcome. This just reminded me of the IT Crowd... Have you tried turning it off and on again. By disconnecting / reconnecting by trying the APN switch, it might've just put you on a different P-GW that has a different traffic shaping configuration or current state.
    - acaloiar 4y ago
      It'll never get old or become poor advice :)
  - neurostimulant 4y ago
    The only way to get to the bottom of this is probably just applying for a relevant job at AT&T, get accepted by whatever means necessary (low asking salary, nepotism, etc), check their network configuration yourself, then immediately submit your resignation once you're satisfied. I think I read somewhere about someone who did exactly like this in a software company. A new employee showed up, immediately fixed a long-standing bug, then submit their resignation.

vitus 4y ago

I do see a couple of retransmissions early in that query (lost ACKs?) which could be upstream throttling but in any event certainly explains why the connection was slow to get off the ground. Agreed that segment offload is probably kernel-side coalescing. From shunting this into Wireshark, you can see that the sequence number grows fairly linearly (often a good indicator of a rate limiter) starting from 1.508s up until the 1.95s mark, but at a rate closer to 25-30 Mbps (1.5MB in 0.45s). https://i.imgur.com/s2JBrCf.png https://i.imgur.com/s2JBrCf.png edit: this was for iphone-capture.pcap. You see a similarly strong picture of shaping for capture.pcap [0], and indeed that corresponds to the ~320kbps range you've noted. [0] https://i.imgur.com/g9cN95d.png https://i.imgur.com/g9cN95d.png edit edit: of course, under normal situations, your bandwidth will get saturated, and steady-state will look fairly linear. If you're using "classic" congestion control like CUBIC or New Reno, you'll see this interspersed with drops (search for "tcp sawtooth"). Under BBR you'd expect to see drops in bandwidth due to latency spikes (per bufferbloat). Neither of those seem to be in play here.

blincoln 4y ago

Could it not also be the problematic traffic getting routed to a Cloudfront node that's overloaded or otherwise misbehaving? Years ago, I had to troubleshoot a very mysterious issue where sometimes a huge percentage of the PCs in the company I worked for would have trouble loading the company's e-commerce site. Everyone was going through the same array of 2-3 proxy servers, and the e-commerce site had Akamai caching in front of it. The proxy servers would hold onto a DNS lookup result or a TCP connection for longer than they were supposed to, and so they'd continue routing traffic to the old IP address even after Akamai had rotated a node out of service.

kevin_nisbet 4y ago
> Could it not also be the problematic traffic getting routed to a Cloudfront node that's overloaded or otherwise misbehaving? This is plausible. Nothing I saw in the packet capture specifically pointed in this direction, but I don't really have the information I'd want to rule this out totally. I've seen issues like this when the TCP stack on the server effectively runs out of allocated memory. This can cause the server not to use all of the available tcp window. The only reason I think this is less likely, is I'd expect a company like AWS to monitor and tune the memory on their CDN, and notice something like a kernel bug here or block an attack exploiting the kernel memory. Some network providers, especially wireless providers also deploy TCP acceleration equipment, so that network speeds are faster. On a wireless carrier if you've got temporary bad signal and drop some packets, if the carrier can retransmit them instead of going to the internet, this makes for a faster connection. Crappier implementations of these TCP accelerators, do a sort of transparent redirect to the kernel TCP stack. So if the ATT proxy has run out of kernel memory, it could be providing a slow connection. It wouldn't surprise me for a telecom carrier to miss and not tune or monitor kernel memory. If the issue was caused by a proxy, I'd expect more to be broken then a single server, and other customers to also be complaining. If ATT is using these accelerators, it also means the proxy could be hiding packet loss and delays between the proxy and upstream server or network. Because the client TCP connection is only to the proxy, we only get to see what happened between the client and proxy. So this precluded drawing strong conclusions from the client side capture only. There is one thing about the packet capture that suggests a shaper to me. And that's how even the traffic is from server to client. While I didn't spend a bunch of time timing out all the packets, it looked a bit like I'd expect to see for packets getting released by a token bucket algorithm. This isn't necessarily safe to conclude, because the client has an offload that is buffering and merging packets using a receive segment offload, which causes us to miss some timing information in the capture. So based on the above, shaping looks a little bit more likely to me, but I don't have the right information to rule out a broken or miss-configured server. An overloaded or otherwise broken network link like errors on a link doesn't appear likely to me at all.