4 ms·

What I am struggling to understand is why is it a good idea to to ip inside ip. Giving containers their own nic, and IP isn't hard, and in most cases is >> fas

What I am struggling to understand is why is it a good idea to to ip inside ip. Giving containers their own nic, and IP isn't hard, and in most cases is >> faster than spending time using some immature tunnelling protocol. I can see that doing multi-cloud deploys with transparent failover might benefit from a VPN, but short of operating on a hostile network I can't see its worth the heartache to deploy an overlay network.

navaati 7y ago
This a thousand ! You don't even need to give them actual nics, just veth pairs with routing (some CNI plugons do just that btw). I think a big part of why overlay nets are so common is that a lot of the engineers deploying these systems have a shaky or non existant understanding of networking and routing, they are stuck in the "local network and a default route" model and if you only have that, overlay it is...
- silasb 7y ago
  Do you know what CNI plugins support veth pairs?
  - boulos 7y ago
    The built-in bridge one does. Maybe you meant something else by your question though?
  - CameronNemo 7y ago
    https://github.com/containernetworking/plugins/blob/master/plugins/main/bridge/README.md https://github.com/containernetworking/plugins/blob/master/p...
  - elFarto 7y ago
    I've tried using Bridget[1] before, but didn't actually manage to get it working properly (although the problem was likely firewall related). [1] https://github.com/kvaps/bridget https://github.com/kvaps/bridget
  - navaati 7y ago
    It’s about routing more than veth pairs (I mean you can also do simple bridging as other comment say, but it’s more limited), so look for BGP using ones, mainly Calico, but kube-router too. I know there is simpler ones that don’t use BGP, but I don’t remember their names. For example there is one for AWS that uses ENI interfaces and pass them directly to containers (you’re obviously limited by the number of ENI you can attach to a particular host, then). There is also one that configures routes in the VPC routing table, and a similar one for GCP.
- KaiserPro 7y ago
  sorry, yes I should have said veth. at least that way you can dump them on an entirely different vlan, if you want separation. The best part is: you're using your cloud provider's networking tools (lets be honest it is a cloud and they are way more likely to work)
theojulienne 7y ago
This is actually interesting, we chose to use an overlay network because Kubernetes was a new, complex system that we initially didn't have experience with internally, and so we wanted to isolate problems it could create as much as possible, as well as ensure that teams working on Kubernetes deployment weren't blocked by needing network engineering time. This meant that we felt an overlay network was the most pragmatic (works out of the box) solution, even though IPIP has some drawbacks in adding complexity (within the Kubernetes cluster) and hashes across router ECMP / NIC RX queues poorly (neither look into the inner IP packet as part of hashing). It was definitely a concern of ours, though, considering we'd very intentionally avoided IPIP in our load balancer (https://github.blog/2018-08-08-glb-director-open-source-load-balancer/ https://github.blog/2018-08-08-glb-director-open-source-load... has a write-up of why). We've considered the alternative of announcing a /24 or /25 or similar via BGP on each Kubernetes node, which is supported by systems like Calico, but so far the limitations of IPIP haven't caused an issue because of our Kubernetes clusters being large enough to hash evenly regardless of the addition of an overlay network, so we haven't needed to migrate. It's definitely an interesting trade-off on complexity overall, though - simple routed networks are much easier to understand than another layer of encapsulation.
bbatha 7y ago
This is more or less how azure-cni works. Though you only have a single nic that’s assigned multiple ips by azure.

navaati 7y ago

This a thousand ! You don't even need to give them actual nics, just veth pairs with routing (some CNI plugons do just that btw). I think a big part of why overlay nets are so common is that a lot of the engineers deploying these systems have a shaky or non existant understanding of networking and routing, they are stuck in the "local network and a default route" model and if you only have that, overlay it is...

silasb 7y ago
Do you know what CNI plugins support veth pairs?
- boulos 7y ago
  The built-in bridge one does. Maybe you meant something else by your question though?
- CameronNemo 7y ago
  https://github.com/containernetworking/plugins/blob/master/plugins/main/bridge/README.md https://github.com/containernetworking/plugins/blob/master/p...
- elFarto 7y ago
  I've tried using Bridget[1] before, but didn't actually manage to get it working properly (although the problem was likely firewall related). [1] https://github.com/kvaps/bridget https://github.com/kvaps/bridget
- navaati 7y ago
  It’s about routing more than veth pairs (I mean you can also do simple bridging as other comment say, but it’s more limited), so look for BGP using ones, mainly Calico, but kube-router too. I know there is simpler ones that don’t use BGP, but I don’t remember their names. For example there is one for AWS that uses ENI interfaces and pass them directly to containers (you’re obviously limited by the number of ENI you can attach to a particular host, then). There is also one that configures routes in the VPC routing table, and a similar one for GCP.
KaiserPro 7y ago
sorry, yes I should have said veth. at least that way you can dump them on an entirely different vlan, if you want separation. The best part is: you're using your cloud provider's networking tools (lets be honest it is a cloud and they are way more likely to work)

theojulienne 7y ago

This is actually interesting, we chose to use an overlay network because Kubernetes was a new, complex system that we initially didn't have experience with internally, and so we wanted to isolate problems it could create as much as possible, as well as ensure that teams working on Kubernetes deployment weren't blocked by needing network engineering time. This meant that we felt an overlay network was the most pragmatic (works out of the box) solution, even though IPIP has some drawbacks in adding complexity (within the Kubernetes cluster) and hashes across router ECMP / NIC RX queues poorly (neither look into the inner IP packet as part of hashing). It was definitely a concern of ours, though, considering we'd very intentionally avoided IPIP in our load balancer (https://github.blog/2018-08-08-glb-director-open-source-load-balancer/ https://github.blog/2018-08-08-glb-director-open-source-load... has a write-up of why). We've considered the alternative of announcing a /24 or /25 or similar via BGP on each Kubernetes node, which is supported by systems like Calico, but so far the limitations of IPIP haven't caused an issue because of our Kubernetes clusters being large enough to hash evenly regardless of the addition of an overlay network, so we haven't needed to migrate. It's definitely an interesting trade-off on complexity overall, though - simple routed networks are much easier to understand than another layer of encapsulation.

bbatha 7y ago

This is more or less how azure-cni works. Though you only have a single nic that’s assigned multiple ips by azure.