4 ms·

WireGuard! I had multiple issues with WireGuard related to MTU/fragmentation. Once, service reported that a system is pingable, but the management website didn

WireGuard! I had multiple issues with WireGuard related to MTU/fragmentation. Once, service reported that a system is pingable, but the management website didn't load. I was able to SSH into it, but similarily, the connection broke everytime I tried to run `ip addr` or view logs. Commands with short output worked, however – apparently, somewhere on the path the packets got fragmented and/or dropped without notice.

mjg59 3y ago
This is going to be true for any VPN - even if it were purely IP in IP encapsulation, you'd need an additional IP header to get the packet to the VPN endpoint, so your payload is smaller. But fragmentation probably isn't the right answer (especially since any packets marked DF will be dropped). If the MTU for your Wireguard interface is set correctly then anything trying to push a 1500 byte packet via a Wireguard interface should get back an ICMP packet telling it that that won't fit and adjust its packet size downwards appropriately. I actually hit this recently, where I wasn't able to stream videos from certain sites over Wireguard. I spend a while with tcpdump and figured out that they were using Fastly as a CDN, Fastly was sending 1500 byte packets marked with Don't Fragment, my Wireguard endpoint was returning a message saying that the maximum packet size for the link was 1460 bytes, and Fastly was then… sending another 1500 byte packet marked Don't Fragment. To their credit when I was able to provide their engineering with logs showing this was clearly a Fastly problem, they fixed it fairly promptly
- kccqzy 3y ago
  Did they fix it just for you or for everyone? A proposed project at work involves using Fastly as CDN and I'm curious if this is something we need to keep in mind.
  - mjg59 3y ago
    Everyone, as far as I know
- wolletd 3y ago
  In my case, the application was using the Wireguard MTU correctly, but the Wireguard packet itself was too big for something on the path. So, the Wireguard MTU wasn't set correctly, basically. But Wireguard packets aren't marked DF, so one would expect a worse, but still working connection in that case. However, that something on the path seemingly just dropped the packets without ICMP notice.
- raggi 3y ago
  Very interesting, I bet this had come up before but remained undiagnosed, yay for packet captures
- geraldhh 3y ago
  > To their credit when I was able to provide their engineering with logs showing this was clearly a Fastly problem, they fixed it fairly promptly regarding credit; did you get paid for doing their homework?

mjg59 3y ago

This is going to be true for any VPN - even if it were purely IP in IP encapsulation, you'd need an additional IP header to get the packet to the VPN endpoint, so your payload is smaller. But fragmentation probably isn't the right answer (especially since any packets marked DF will be dropped). If the MTU for your Wireguard interface is set correctly then anything trying to push a 1500 byte packet via a Wireguard interface should get back an ICMP packet telling it that that won't fit and adjust its packet size downwards appropriately. I actually hit this recently, where I wasn't able to stream videos from certain sites over Wireguard. I spend a while with tcpdump and figured out that they were using Fastly as a CDN, Fastly was sending 1500 byte packets marked with Don't Fragment, my Wireguard endpoint was returning a message saying that the maximum packet size for the link was 1460 bytes, and Fastly was then… sending another 1500 byte packet marked Don't Fragment. To their credit when I was able to provide their engineering with logs showing this was clearly a Fastly problem, they fixed it fairly promptly

kccqzy 3y ago
Did they fix it just for you or for everyone? A proposed project at work involves using Fastly as CDN and I'm curious if this is something we need to keep in mind.
- mjg59 3y ago
  Everyone, as far as I know
wolletd 3y ago
In my case, the application was using the Wireguard MTU correctly, but the Wireguard packet itself was too big for something on the path. So, the Wireguard MTU wasn't set correctly, basically. But Wireguard packets aren't marked DF, so one would expect a worse, but still working connection in that case. However, that something on the path seemingly just dropped the packets without ICMP notice.
raggi 3y ago
Very interesting, I bet this had come up before but remained undiagnosed, yay for packet captures
geraldhh 3y ago
> To their credit when I was able to provide their engineering with logs showing this was clearly a Fastly problem, they fixed it fairly promptly regarding credit; did you get paid for doing their homework?