5 ms·

We used to use RFC1918 (172.16/12 IIRC) addresses for the communication between internal nodes in a cluster-in-box system that I worked on, which worked great u

by wrigby 8y ago

We used to use RFC1918 (172.16/12 IIRC) addresses for the communication between internal nodes in a cluster-in-box system that I worked on, which worked great until we had a subnet collision on a customer's network. Leaves me wondering if link-local (169.254/16, fe80::/10) would have been a better option - while technically the customer could decide to make the external (customer-facing) network have a link-local interface, the chances of that configuration actually happening are pretty slim. I'm still not entirely sure what the best option is there. Maybe some clever use of network namespaces, with a named pipe to bridge between the "internal" and "external" universes? Just typing up that idea makes me cringe though.

derekp7 8y ago
BTW, for those wondering what this particular failure scenario is: Let's use Docker's default 172.17.0.0/16 subnet as an example. So your docker host has iptable DNAT rules that routes a given "external" IP address (10.0.1.15) to a given docker container (172.17.25.92). That works great, unless you have a workstation on a subnet such as 172.17.81.0/24. When that workstation sends a packet to 10.0.1.15, that packet gets routed to the destination container 172.17.25.92. That container goes to reply, but the reply packet never makes it out to the original workstation because the container host thinks it is bound for something else on its version of the 172.17 subnet. One workaround to this is to have the container host also put in an SNAT rule, so that anything that it forwards to a container would have the source IP address re-written to appear to come from the container host's IP, or the docker0 bridge IP (172.17.0.1/16)
- fyfy18 8y ago
  On a similar note, Docker for Mac assigns (or used to?) the IP address 192.168.99.100 to the VM that runs Docker. One day I was working in a coffee shop and got really confused as to why I couldn’t connect to my application, even though the server was running. Then I realised the coffee shop WiFi was using 192.168.99.0/24 for client IPs.
harshreality 8y ago
In a cluster-in-a-box scenario, you could modify the OS's network scripts to have the cluster-specific private interface start after the general LAN interface is up. Check both 10/8 and 172.16/12 to see if they're used by the public interface, and use whichever one isn't for the cluster network.
- derekp7 8y ago
  That only works if the host is on the conflicting network. But if the conflict is a couple hops away, Docker won't detect it.
  - deleted 8y ago
    [deleted]
kbaker 8y ago
IPv6 Unique Local Addressing is made for this scenario, with a low chance of collisions. fd: + 40 random bits, becomes your new /48. https://tools.ietf.org/html/rfc4193 https://tools.ietf.org/html/rfc4193
- dcow 8y ago
  I can't wait till the world comes around to the true advantages of IPv6. It's not just about adding more global addresses...nodes participate in multiple first class networks now (one of those networks is often the global internet). I'd be much more comfortable with smart devices in my home if they're on a universal local network with a public internet federation service for things like software updates. IPv6 makes this possible.
Symbiote 8y ago
The correct solution is to choose a small part of their allocated public range, and reserve it for this purpose. This is also the correct solution for your cluster.

by wrigby 8y ago

derekp7 8y ago

BTW, for those wondering what this particular failure scenario is: Let's use Docker's default 172.17.0.0/16 subnet as an example. So your docker host has iptable DNAT rules that routes a given "external" IP address (10.0.1.15) to a given docker container (172.17.25.92). That works great, unless you have a workstation on a subnet such as 172.17.81.0/24. When that workstation sends a packet to 10.0.1.15, that packet gets routed to the destination container 172.17.25.92. That container goes to reply, but the reply packet never makes it out to the original workstation because the container host thinks it is bound for something else on its version of the 172.17 subnet. One workaround to this is to have the container host also put in an SNAT rule, so that anything that it forwards to a container would have the source IP address re-written to appear to come from the container host's IP, or the docker0 bridge IP (172.17.0.1/16)

fyfy18 8y ago
On a similar note, Docker for Mac assigns (or used to?) the IP address 192.168.99.100 to the VM that runs Docker. One day I was working in a coffee shop and got really confused as to why I couldn’t connect to my application, even though the server was running. Then I realised the coffee shop WiFi was using 192.168.99.0/24 for client IPs.

harshreality 8y ago

In a cluster-in-a-box scenario, you could modify the OS's network scripts to have the cluster-specific private interface start after the general LAN interface is up. Check both 10/8 and 172.16/12 to see if they're used by the public interface, and use whichever one isn't for the cluster network.

derekp7 8y ago
That only works if the host is on the conflicting network. But if the conflict is a couple hops away, Docker won't detect it.
- deleted 8y ago
  [deleted]

kbaker 8y ago

IPv6 Unique Local Addressing is made for this scenario, with a low chance of collisions. fd: + 40 random bits, becomes your new /48. https://tools.ietf.org/html/rfc4193 https://tools.ietf.org/html/rfc4193

dcow 8y ago
I can't wait till the world comes around to the true advantages of IPv6. It's not just about adding more global addresses...nodes participate in multiple first class networks now (one of those networks is often the global internet). I'd be much more comfortable with smart devices in my home if they're on a universal local network with a public internet federation service for things like software updates. IPv6 makes this possible.

Symbiote 8y ago

The correct solution is to choose a small part of their allocated public range, and reserve it for this purpose. This is also the correct solution for your cluster.