4 ms·

We are IPv6-only on our institute-internal CPU compute cluster based on slurm. Only the head node has an IPv4 address, so that it can be reached from IPv4 only

We are IPv6-only on our institute-internal CPU compute cluster based on slurm. Only the head node has an IPv4 address, so that it can be reached from IPv4 only clients (sadly, there are still quite a lot). All nodes inside the cluster talk over IPv6. And all other computers with IPv6 access use that to communicate to the head node. We are transitioning to IPv6-only for internal services and try to avoid using IPv4 addresses, only going back to it when something needs to be accessed from the outside. Sadly, our local ISP is still IPv4-only, meaning we cannot even access our IPv6 hosts while at home, so we need to fall-back to IPv4 quite a lot. Also, the Cisco VPN is still IPv4 only (because of lacking resources to add IPv6 support), so not even the VPN helps. We need to jump over some dual-stack host then. When speaking to the local ISP, they just reply that it's not planned soon, they don't have resources for it, and "they evaluated IPv6 and don't have a reason to support it". Me/us giving them reasons was not enough it seems.

marinesebastian 4y ago
With all these limitations, why did you prefer IPv6 over IPv4?
- maximaximal 4y ago
  Because we don't want to use 20 IPv4 addresses for the cluster of 20 nodes, when we only have so much addresses assigned to our institute. We could have gone the NAT route, but then we'd need to have some router. And if we designate the head node as router, all traffic would not go through the switch directly, but first through the head node and then out. This would mean that the nodes are less independent, as they have this one additional choke-point. Our university gave us a /64 for this network, so we just used that and it worked flawlessly, also for university-internal distro-package fetching and host-cluster connections. Also, research software is usually working nicely with IPv6. If we encounter something that would really need IPv4, we could update the thing and give it some local subnet - but currently we were lucky.
  - remram 4y ago
    But you still need NAT right? To support outbound connections from the worker nodes?
    - Dagger2 4y ago
      So long as you're using a /64 routed to you from your upstream, nope, you won't need NAT.
jeroenhd 4y ago
> Sadly, our local ISP is still IPv4-only, meaning we cannot even access our IPv6 hosts while at home, so we need to fall-back to IPv4 quite a lot. Have you perhaps looked into using tunnels to fix this? he.net can add IPv6 support to any routable IPv4 address that can respond to ICMP. If you need to stick to your internal ULA, something like NPTv6 can translate the /48 subnet HE provides to your own /48 subnet transparently. Requires some setting up, but so does maintaining VPNs.
- maximaximal 4y ago
  Yes, I did this on my personal network. I'm in Linz (Austria), the nearest tunnel to a german speaking country that's listed on tunnelbroker.net was Berlin (if I remember right). Everything worked rather well, but I then decided against it, because of the GeoIP implications (with one IP being in Austria and one in Germany) and the added latency. It's noticable when SSHing somewhere whether I'm Linz-Linz or Linz-Berlin-Linz, especially when a small lag spike happens, which is rather frequent with the cable-based network our ISP is offering. It wasn't worth it to half-slowdown our network just to not rely on jumping through some dual-stack node in the university. If anybody from Liwest (our local ISP) reads this - IPv6 would be really nice! :)
- epx 4y ago
  He.net needs a fixed non-CGNAT IPv4 address, with protocol 41 open, and this is quite difficult to get.
  - jeroenhd 4y ago
    That really depends on where you live. Every ISP I've ever had has fit this bill. I know IPv4 availability in developing countries is a problem and there's no fix for that other than "wait until IPv6 is available", but in those countries IPv6 is actually part of the solution to not have to rely on "communal" IPv4 CG-NAT.

marinesebastian 4y ago

With all these limitations, why did you prefer IPv6 over IPv4?

maximaximal 4y ago
Because we don't want to use 20 IPv4 addresses for the cluster of 20 nodes, when we only have so much addresses assigned to our institute. We could have gone the NAT route, but then we'd need to have some router. And if we designate the head node as router, all traffic would not go through the switch directly, but first through the head node and then out. This would mean that the nodes are less independent, as they have this one additional choke-point. Our university gave us a /64 for this network, so we just used that and it worked flawlessly, also for university-internal distro-package fetching and host-cluster connections. Also, research software is usually working nicely with IPv6. If we encounter something that would really need IPv4, we could update the thing and give it some local subnet - but currently we were lucky.
- remram 4y ago
  But you still need NAT right? To support outbound connections from the worker nodes?
  - Dagger2 4y ago
    So long as you're using a /64 routed to you from your upstream, nope, you won't need NAT.

jeroenhd 4y ago

> Sadly, our local ISP is still IPv4-only, meaning we cannot even access our IPv6 hosts while at home, so we need to fall-back to IPv4 quite a lot. Have you perhaps looked into using tunnels to fix this? he.net can add IPv6 support to any routable IPv4 address that can respond to ICMP. If you need to stick to your internal ULA, something like NPTv6 can translate the /48 subnet HE provides to your own /48 subnet transparently. Requires some setting up, but so does maintaining VPNs.

maximaximal 4y ago
Yes, I did this on my personal network. I'm in Linz (Austria), the nearest tunnel to a german speaking country that's listed on tunnelbroker.net was Berlin (if I remember right). Everything worked rather well, but I then decided against it, because of the GeoIP implications (with one IP being in Austria and one in Germany) and the added latency. It's noticable when SSHing somewhere whether I'm Linz-Linz or Linz-Berlin-Linz, especially when a small lag spike happens, which is rather frequent with the cable-based network our ISP is offering. It wasn't worth it to half-slowdown our network just to not rely on jumping through some dual-stack node in the university. If anybody from Liwest (our local ISP) reads this - IPv6 would be really nice! :)
epx 4y ago
He.net needs a fixed non-CGNAT IPv4 address, with protocol 41 open, and this is quite difficult to get.
- jeroenhd 4y ago
  That really depends on where you live. Every ISP I've ever had has fit this bill. I know IPv4 availability in developing countries is a problem and there's no fix for that other than "wait until IPv6 is available", but in those countries IPv6 is actually part of the solution to not have to rely on "communal" IPv4 CG-NAT.