4 ms·

My question is this. Why does the container world use NAT.. ( 3 layers to get out of container to base host in k8s ) ... and not use routing ? Is it just the c

by geggam 9y ago

My question is this. Why does the container world use NAT.. ( 3 layers to get out of container to base host in k8s ) ... and not use routing ? Is it just the container devs dont know routing ?

lima 9y ago
Look into Project Calico, they get it right: https://www.projectcalico.org/ https://www.projectcalico.org/
notzorbo3 9y ago
Because NATting encapsulates while routing doesn't? And encapsulation is the whole idea behind containers. Until everything is ready for IPv6 (lol, yeah right), NATting seems the only way to me.
- lima 9y ago
  You can encapsulate just fine with a routed architecture. You still need NAT to talk to the outside world (your services are behind a load balancer either way).
  - geggam 9y ago
    Why do you need NAT ?
falcolas 9y ago
A lot of it is due to an effort to make it work in as many environments with as few external dependencies (and environment control) as possible. The "simplest solution which could possibly work". Personally, I'd rather just bring on ipv6. But, in my case, we don't have enough people who understand ipv6 (and it's barely supported in AWS) to use it ourselves.
puzzle 9y ago
Kubernetes is the opposite. NAT is explicitly not required: https://kubernetes.io/docs/concepts/cluster-administration/networking/#kubernetes-model https://kubernetes.io/docs/concepts/cluster-administration/n... E.g. on AWS you might have all of a node's pod IPs on a bridge interface, then you talk to pods on other nodes thanks to VPC route table entries that the AWS cloud provider manages. NAT happens only when talking to the outside world or for traffic to Amazon DNS servers, which don't like source IP addresses other than those from the subnet they live in.
- geggam 9y ago
  This is an interesting article about k8s https://medium.com/google-cloud/kubernetes-from-load-balancer-to-pod-3f2399637b0c?imm_mid=0f75d4&cmp=em-webops-na-na-newsltr_20171020 https://medium.com/google-cloud/kubernetes-from-load-balance...
  - puzzle 9y ago
    That's inbound traffic coming from the outside world. You need NAT because the load balancer only knows about nodes, not individual pods (perhaps you can pull it off with e.g. ELBv2, but definitely not with v1). There's more iptables magic if you talk to a service's virtual cluster IP, because of the load balancing, but from pod to pod, which is what I thought you were referring to, NAT is usually not involved.
    - geggam 9y ago
      No point in having a service you cant use :)
      - puzzle 9y ago
        Are you referring to the service cluster IPs? Those are great for short lived or low volume connections. If you want to balance load over long lived connections or have high volume, you really want to know the addresses of all your backends, whether that's done in your code or in a sidecar like Istio's.
- falcolas 9y ago
  My memory is a touch fuzzy, but to route traffic out of a container in AWS, you have to either NAT thorough the instances network adapter, or attach an ENI to the container. However, you only get one ENI per vCPU in a VM (at least until Amazon finishes its custom NICs). What I'm really fuzzy on is whether the instance itself consumes one of those allocated ENIs. That is, if you're running off a m4.2xLarge instance, you get a maximum of 8 ENIs - 8 containers if you want to use only VPC routing. For some services, this may be OK, but for many others (most?), it's far too few.
takeda 9y ago
Because that's the easiest thing to do when you don't know anything about networking. Ironically this also makes everything else much more complex and failure prone.
- geggam 9y ago
  This is the answer surprisingly missing in the industry overall. It amazes me that I work with highly educated folks who cannot grasp some of the fundamental issues with k8s and the container ecosystem.

by geggam 9y ago

My question is this. Why does the container world use NAT.. ( 3 layers to get out of container to base host in k8s ) ... and not use routing ? Is it just the container devs dont know routing ?

lima 9y ago

Look into Project Calico, they get it right: https://www.projectcalico.org/ https://www.projectcalico.org/

notzorbo3 9y ago

Because NATting encapsulates while routing doesn't? And encapsulation is the whole idea behind containers. Until everything is ready for IPv6 (lol, yeah right), NATting seems the only way to me.

lima 9y ago
You can encapsulate just fine with a routed architecture. You still need NAT to talk to the outside world (your services are behind a load balancer either way).
- geggam 9y ago
  Why do you need NAT ?

falcolas 9y ago

A lot of it is due to an effort to make it work in as many environments with as few external dependencies (and environment control) as possible. The "simplest solution which could possibly work". Personally, I'd rather just bring on ipv6. But, in my case, we don't have enough people who understand ipv6 (and it's barely supported in AWS) to use it ourselves.

puzzle 9y ago

Kubernetes is the opposite. NAT is explicitly not required: https://kubernetes.io/docs/concepts/cluster-administration/networking/#kubernetes-model https://kubernetes.io/docs/concepts/cluster-administration/n... E.g. on AWS you might have all of a node's pod IPs on a bridge interface, then you talk to pods on other nodes thanks to VPC route table entries that the AWS cloud provider manages. NAT happens only when talking to the outside world or for traffic to Amazon DNS servers, which don't like source IP addresses other than those from the subnet they live in.

geggam 9y ago
This is an interesting article about k8s https://medium.com/google-cloud/kubernetes-from-load-balancer-to-pod-3f2399637b0c?imm_mid=0f75d4&cmp=em-webops-na-na-newsltr_20171020 https://medium.com/google-cloud/kubernetes-from-load-balance...
- puzzle 9y ago
  That's inbound traffic coming from the outside world. You need NAT because the load balancer only knows about nodes, not individual pods (perhaps you can pull it off with e.g. ELBv2, but definitely not with v1). There's more iptables magic if you talk to a service's virtual cluster IP, because of the load balancing, but from pod to pod, which is what I thought you were referring to, NAT is usually not involved.
  - geggam 9y ago
    No point in having a service you cant use :)
    - puzzle 9y ago
      Are you referring to the service cluster IPs? Those are great for short lived or low volume connections. If you want to balance load over long lived connections or have high volume, you really want to know the addresses of all your backends, whether that's done in your code or in a sidecar like Istio's.
falcolas 9y ago
My memory is a touch fuzzy, but to route traffic out of a container in AWS, you have to either NAT thorough the instances network adapter, or attach an ENI to the container. However, you only get one ENI per vCPU in a VM (at least until Amazon finishes its custom NICs). What I'm really fuzzy on is whether the instance itself consumes one of those allocated ENIs. That is, if you're running off a m4.2xLarge instance, you get a maximum of 8 ENIs - 8 containers if you want to use only VPC routing. For some services, this may be OK, but for many others (most?), it's far too few.

takeda 9y ago

Because that's the easiest thing to do when you don't know anything about networking. Ironically this also makes everything else much more complex and failure prone.

geggam 9y ago
This is the answer surprisingly missing in the industry overall. It amazes me that I work with highly educated folks who cannot grasp some of the fundamental issues with k8s and the container ecosystem.