4 ms·

Fastly is the only official PyPI mirror, and I was once asked by a client to figure out why PyPI downloads weren't working, on a host that otherwise seemed to h

by dpifke 3y ago

Fastly is the only official PyPI mirror, and I was once asked by a client to figure out why PyPI downloads weren't working, on a host that otherwise seemed to have good connectivity (downloads of similarly-sized files from non-Fastly sites worked just fine). It turned out that Fastly ignores/blocks ICMP unreachable messages, which breaks Path MTU discovery[0]. The host was behind a VPN and I verified using Wireshark that it was asking Fastly to lower its MTU to accomodate the VPN packet overhead, but Fastly was ignoring the messages. Setting MSS on the host's default route (`ip route change default advmss 1380`) fixed it, but it seems basic networking tips like "don't block all ICMP" should be table stakes for a CDN. I nicely emailed Fastly about this, figuring they might appreciate knowing they had something misconfigured, but never heard back. Ironically, Cloudflare has a really good blog post about this issue[1], which I used when explaining the problem and solution to my client. [0]: https://en.wikipedia.org/wiki/Path_MTU_Discovery https://en.wikipedia.org/wiki/Path_MTU_Discovery [1]: https://blog.cloudflare.com/path-mtu-discovery-in-practice/ https://blog.cloudflare.com/path-mtu-discovery-in-practice/

r1ch 3y ago
Unfortunately you can't really rely on PMTU discovery these days. Not only is it regularly filtered, it slows every connection down since you add a RTT for the ICMP packets for every TCP host you talk to. In this case I would say the VPN client would be at fault for not adjusting the TCP MSS.
- devman0 3y ago
  Relying on MSS to not break layer 3 is going to be interesting when QUIC is more prevalent. PMTUD should be better supported.
  - r1ch 3y ago
    QUIC unfortunately defaults to 1370 bytes to avoid having to deal with PMTUD :(. Endpoints are free to negotiate higher and the protocol does provide a way to generate arbitrary sized ping frames, but it's likely most implementations will just stick with the default.
stock_toaster 3y ago
> using Wireshark that it was asking Fastly to lower its MTU to accomodate the VPN packet overhead, but Fastly was ignoring the messages It could also have been a middlebox somewhere between the vpn egress and fastly causing issues.
- dpifke 3y ago
  Looks like it's still reproducible. I just tried artificially lowering the MTU on the interface of a bastion host in a datacenter, with only one router between it and a major IX (no middleboxes), and running apt-get against the Fastly Debian mirror: 20:55:33.402893 IP 23.141.40.254.43004 > 151.101.10.132.443: Flags [.], ack 1, win 504, options [nop,nop,TS val 4088102127 ecr 3823759533,nop,nop,sack 1 {2889:3881}], length 0 20:55:33.580057 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], seq 1:1445, ack 397, win 285, options [nop,nop,TS val 3823759807 ecr 4088102127], length 1444 20:55:33.580314 IP 23.141.40.254 > 151.101.10.132: ICMP 23.141.40.254 unreachable - need to frag (mtu 1420), length 556 20:55:34.155995 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], seq 1:1445, ack 397, win 285, options [nop,nop,TS val 3823760383 ecr 4088102127], length 1444 20:55:34.156214 IP 23.141.40.254 > 151.101.10.132: ICMP 23.141.40.254 unreachable - need to frag (mtu 1420), length 556 20:55:35.308019 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], seq 1:1445, ack 397, win 285, options [nop,nop,TS val 3823761535 ecr 4088102127], length 1444 20:55:35.308275 IP 23.141.40.254 > 151.101.10.132: ICMP 23.141.40.254 unreachable - need to frag (mtu 1420), length 556 20:55:37.739916 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], seq 1:1445, ack 397, win 285, options [nop,nop,TS val 3823763967 ecr 4088102127], length 1444 20:55:37.740191 IP 23.141.40.254 > 151.101.10.132: ICMP 23.141.40.254 unreachable - need to frag (mtu 1420), length 556 20:55:42.347790 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], seq 1:1445, ack 397, win 285, options [nop,nop,TS val 3823768575 ecr 4088102127], length 1444 20:55:42.348049 IP 23.141.40.254 > 151.101.10.132: ICMP 23.141.40.254 unreachable - need to frag (mtu 1420), length 556 20:55:49.539608 IP 23.141.40.254.43004 > 151.101.10.132.443: Flags [F.], seq 397, ack 1, win 504, options [nop,nop,TS val 4088118265 ecr 3823759533,nop,nop,sack 1 {2889:3881}], length 0 20:55:49.659124 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], ack 398, win 285, options [nop,nop,TS val 3823775887 ecr 4088118265], length 0 20:55:49.659189 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [F.], seq 3881, ack 398, win 285, options [nop,nop,TS val 3823775887 ecr 4088118265], length 0 20:55:49.715144 IP 23.141.40.254.43004 > 151.101.10.132.443: Flags [R], seq 829588935, win 0, length 0 (FWIW, this thread nudged me to email noc@fastly.com again.)

by dpifke 3y ago

r1ch 3y ago

Unfortunately you can't really rely on PMTU discovery these days. Not only is it regularly filtered, it slows every connection down since you add a RTT for the ICMP packets for every TCP host you talk to. In this case I would say the VPN client would be at fault for not adjusting the TCP MSS.

devman0 3y ago
Relying on MSS to not break layer 3 is going to be interesting when QUIC is more prevalent. PMTUD should be better supported.
- r1ch 3y ago
  QUIC unfortunately defaults to 1370 bytes to avoid having to deal with PMTUD :(. Endpoints are free to negotiate higher and the protocol does provide a way to generate arbitrary sized ping frames, but it's likely most implementations will just stick with the default.

stock_toaster 3y ago

> using Wireshark that it was asking Fastly to lower its MTU to accomodate the VPN packet overhead, but Fastly was ignoring the messages It could also have been a middlebox somewhere between the vpn egress and fastly causing issues.

dpifke 3y ago
Looks like it's still reproducible. I just tried artificially lowering the MTU on the interface of a bastion host in a datacenter, with only one router between it and a major IX (no middleboxes), and running apt-get against the Fastly Debian mirror: 20:55:33.402893 IP 23.141.40.254.43004 > 151.101.10.132.443: Flags [.], ack 1, win 504, options [nop,nop,TS val 4088102127 ecr 3823759533,nop,nop,sack 1 {2889:3881}], length 0 20:55:33.580057 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], seq 1:1445, ack 397, win 285, options [nop,nop,TS val 3823759807 ecr 4088102127], length 1444 20:55:33.580314 IP 23.141.40.254 > 151.101.10.132: ICMP 23.141.40.254 unreachable - need to frag (mtu 1420), length 556 20:55:34.155995 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], seq 1:1445, ack 397, win 285, options [nop,nop,TS val 3823760383 ecr 4088102127], length 1444 20:55:34.156214 IP 23.141.40.254 > 151.101.10.132: ICMP 23.141.40.254 unreachable - need to frag (mtu 1420), length 556 20:55:35.308019 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], seq 1:1445, ack 397, win 285, options [nop,nop,TS val 3823761535 ecr 4088102127], length 1444 20:55:35.308275 IP 23.141.40.254 > 151.101.10.132: ICMP 23.141.40.254 unreachable - need to frag (mtu 1420), length 556 20:55:37.739916 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], seq 1:1445, ack 397, win 285, options [nop,nop,TS val 3823763967 ecr 4088102127], length 1444 20:55:37.740191 IP 23.141.40.254 > 151.101.10.132: ICMP 23.141.40.254 unreachable - need to frag (mtu 1420), length 556 20:55:42.347790 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], seq 1:1445, ack 397, win 285, options [nop,nop,TS val 3823768575 ecr 4088102127], length 1444 20:55:42.348049 IP 23.141.40.254 > 151.101.10.132: ICMP 23.141.40.254 unreachable - need to frag (mtu 1420), length 556 20:55:49.539608 IP 23.141.40.254.43004 > 151.101.10.132.443: Flags [F.], seq 397, ack 1, win 504, options [nop,nop,TS val 4088118265 ecr 3823759533,nop,nop,sack 1 {2889:3881}], length 0 20:55:49.659124 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], ack 398, win 285, options [nop,nop,TS val 3823775887 ecr 4088118265], length 0 20:55:49.659189 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [F.], seq 3881, ack 398, win 285, options [nop,nop,TS val 3823775887 ecr 4088118265], length 0 20:55:49.715144 IP 23.141.40.254.43004 > 151.101.10.132.443: Flags [R], seq 829588935, win 0, length 0 (FWIW, this thread nudged me to email noc@fastly.com again.)