4 ms·

Do they talk at all about what they're using to provide the VPC overlay? I have a DO k8s cluster and it uses Cilium for the CNI, which turns out to be quite us

Do they talk at all about what they're using to provide the VPC overlay? I have a DO k8s cluster and it uses Cilium for the CNI, which turns out to be quite useful, so I guess I'm wondering if they're also using Cilium for this. (Over in AWS land, they wrote a CNI for their own VPC networking. It turns out to have many strange limitations. For example, you can only run 17 pods on a certain type of node, because that node is only allowed to have 19 VPC addresses. I was quite surprised when pods stopped scheduling even though CPU and memory were available. Turns out internal IP addresses are a resource, too. DigitalOcean has the advantage of starting fresh, so might be able to use something open source that can be played with in a dev environment and extended with open source projects.)

andoriyu 7y ago
> Turns out internal IP addresses are a resource, too. That's not what is happening in AWS. IP address are resources (duh), but that's no the issue. With their CNI plugin each pod gets its own Elastic Network Interface. ENIs aren't just virtio's virtual network, it could be ENA (100Gbps) or Intel VF (10Gbs). It's a hardware limitation of amazon virtualization stack starting with previous generation instances. > I was quite surprised when pods stopped scheduling even though CPU and memory were available. This is well documented here: https://github.com/aws/amazon-vpc-cni-k8s https://github.com/aws/amazon-vpc-cni-k8s
dilyevsky 7y ago
Better way of doing natively addressable pods is assign whole subnets (like /25) as secondary interface and distribute that to pods via cni. I think gke pod network works that way. Not sure why eks decided 17 pods is ok lol
- tedk-42 7y ago
  Wow the ignorance. The AWS CNI which comes with EKS attaches ENIs (think of a NIC) to the instance. The number of these you can attach is dependent on the instance type. Pods get an IP in the CIDR range of the subnet the instance is in. There's no additional network overlay which does any NAT when traffic leaves the instance. If you want you can use your own CNI, but things can break.
  - dilyevsky 7y ago
    Thanks I’m aware that aws hacked the shit out of their inflexible legacy design to support this (as well as hid docs on github and continue to charge you for those ENIs). What else is new?
    - tedk-42 7y ago
      Umm you're not charged for using ENIs. You're thinking of elastic IPs. Please check your facts before making these claims.
      - dilyevsky 7y ago
        Hm I think you are right on the price (at least I don’t see it anymore). The fact that it’s ridiculously complex feature remains though. We ended up just running regular overlay since messing around and planning for ENIs is not worth it (I suppose not an option for eks nodes).
  - wolco 7y ago
    Parent poster sounds more credible. Maybe tone down a bit this is a forum for discussion not a lan party.
wmf 7y ago
It looks like each physical server in EC2 can have 750 IPs so if your VM is 1/Nth of the server you also get 1/Nth of the IPs.
- res0nat0r 7y ago
  It is actually based on the ec2 instance type you decide to boot, and generally the bigger the instance the more ENI's you can attach. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-en... The EC2 k8s network driver they wrote essentially will attach/detach extra ENI's on the fly and pre-allocate IP addresses to your EC2 host to allow for fast pod spin up/down. I found this article pretty helpful to explain some of the AWS differences: https://www.contino.io/insights/kubernetes-is-hard-why-eks-makes-it-easier-for-network-and-security-architects https://www.contino.io/insights/kubernetes-is-hard-why-eks-m...

andoriyu 7y ago

> Turns out internal IP addresses are a resource, too. That's not what is happening in AWS. IP address are resources (duh), but that's no the issue. With their CNI plugin each pod gets its own Elastic Network Interface. ENIs aren't just virtio's virtual network, it could be ENA (100Gbps) or Intel VF (10Gbs). It's a hardware limitation of amazon virtualization stack starting with previous generation instances. > I was quite surprised when pods stopped scheduling even though CPU and memory were available. This is well documented here: https://github.com/aws/amazon-vpc-cni-k8s https://github.com/aws/amazon-vpc-cni-k8s

dilyevsky 7y ago

Better way of doing natively addressable pods is assign whole subnets (like /25) as secondary interface and distribute that to pods via cni. I think gke pod network works that way. Not sure why eks decided 17 pods is ok lol

tedk-42 7y ago
Wow the ignorance. The AWS CNI which comes with EKS attaches ENIs (think of a NIC) to the instance. The number of these you can attach is dependent on the instance type. Pods get an IP in the CIDR range of the subnet the instance is in. There's no additional network overlay which does any NAT when traffic leaves the instance. If you want you can use your own CNI, but things can break.
- dilyevsky 7y ago
  Thanks I’m aware that aws hacked the shit out of their inflexible legacy design to support this (as well as hid docs on github and continue to charge you for those ENIs). What else is new?
  - tedk-42 7y ago
    Umm you're not charged for using ENIs. You're thinking of elastic IPs. Please check your facts before making these claims.
    - dilyevsky 7y ago
      Hm I think you are right on the price (at least I don’t see it anymore). The fact that it’s ridiculously complex feature remains though. We ended up just running regular overlay since messing around and planning for ENIs is not worth it (I suppose not an option for eks nodes).
- wolco 7y ago
  Parent poster sounds more credible. Maybe tone down a bit this is a forum for discussion not a lan party.

wmf 7y ago

It looks like each physical server in EC2 can have 750 IPs so if your VM is 1/Nth of the server you also get 1/Nth of the IPs.

res0nat0r 7y ago
It is actually based on the ec2 instance type you decide to boot, and generally the bigger the instance the more ENI's you can attach. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-en... The EC2 k8s network driver they wrote essentially will attach/detach extra ENI's on the fly and pre-allocate IP addresses to your EC2 host to allow for fast pod spin up/down. I found this article pretty helpful to explain some of the AWS differences: https://www.contino.io/insights/kubernetes-is-hard-why-eks-makes-it-easier-for-network-and-security-architects https://www.contino.io/insights/kubernetes-is-hard-why-eks-m...