4 ms·
> This becomes noticeable when pipelines on IPv6 connected servers suddenly have random request/post failures to public services. Then either the whole service
by lxgr 4mo ago
> This becomes noticeable when pipelines on IPv6 connected servers suddenly have random request/post failures to public services. Then either the whole service is temporarily having issues or there are a few bad IPv6 endpoints while all the IPv4 endpoints are fine.
Do you have examples for this? I've never experienced this, and I've been using IPv6 for years.
Also, how can you be sure that the same request to IPv4 would have been fine? Did you actually see consistent failures on v6 and consistent success on v4? Otherwise, if a service has a reasonably low error rate, success on retry is the expected outcome, regardless of the path the retry takes.
- jck86 4mo agoThere were indeed consistent failures to specific IPv6 endpoints, clearly identifiable through curl, while all the IPv4 endpoints were ok. This happened with pypi (IPv6 BGP routing problem caused by a bad route from one of our peers combined with their fastly CDN not reply to us on IPv6 from the other side of the ocean for some weird reason), but also with yum and apt mirrors (seemingly random problems with the IPv6 service or firewall of the remote endpoint), and various other web resources accessed from pipelines. The solution always was to temporarily block the bad IPv6 endpoint(s) or temporarily completely disabling IPv6 on the server itself or on the squid proxy server for workloads without direct connectivity. Obviously it also can be the other way around, but in practice it appears to happen less often with IPv4, and if it does things get addressed quickly instead of taking hours or days or weeks.
- Dagger2 4mo agoTry `ip link set mtu 1280 dev eth0` (or equivalent for your OS). pMTUd breakage exists on v6 just like it exists on v4, and requires workarounds just like it does on v4. I get the impression a lot of people are applying a workaround on v4 but not on v6, then blaming the resulting failures on v6 without bothering to do any troubleshooting to figure out what's actually wrong.
- jck86 4mo agoThis was not a path mtu problem since it was random IPv6 endpoints having issues. And if all the IPv6 endpoints of a major service go down we should pin our mtu to the bare minimum? Nonsense. The inconvenient truth is that IPv4 will be more stable than IPv6 until the former gets demoted to second class fallback protocol instead of the major driver of the internet. And when that happens no one knows.
- Dagger2 4mo agoPeople running servers don't seem to be reliably capable of making sure that either pMTUd works or isn't needed on their network, so... it is indeed broken on random servers. We've mostly decided to go with TCP MSS clamping rather than using the minimum MTU, which is still nonsense but it's nonsense that we already decided to go with for the same problem in v4. Of course, I don't have enough info to tell if this was the problem or if it was something else, or a mix of the two, but pMTUd does seem to be a leading cause of "fails to load for some weird reason" problems.
- liveoneggs 4mo agoI saw HE stop routing to europe over ipv6 for an extended period of time two-ish years ago.
- Dagger2 4mo agoI'm in Europe and I use a tunnel from HE for v6. I feel like that's something I would have noticed if it was as widespread as you make it sound.
- toast0 4mo agoOpen source download mirrors often have much better targetting for v4 than v6. Just a few days ago, I was downloading installer images to check an issue and adding -4 to the command line reduced the download time significantly.