4 ms·

Slight tangent inspired by the premise, but why do EKS etc cap the number of pods and other resources on a node? We try to enable our customers to start on 1 n

Slight tangent inspired by the premise, but why do EKS etc cap the number of pods and other resources on a node? We try to enable our customers to start on 1 node and scale up at their discretion, but found we had to into all sorts of contortions to make that work. Likewise, k8s is resource hungry. We can run the docker compose version on a much smaller laptop wrt CPU & RAM.

quantumcd 4y ago
It's been a while since I've used EKS, but I seem to recall it's primarily based on the maximum ENIs the instance type supports.
- mdaniel 4y ago
  I believe you're correct, although pedantically that would only apply if one is using their vpc-cni <https://github.com/aws/amazon-vpc-cni-k8s#readme https://github.com/aws/amazon-vpc-cni-k8s#readme> and not with a competing CNI. Kubelet offers a configurable for the number of Pods per Node <https://github.com/kubernetes/kubelet/blob/v0.26.2/config/v1beta1/types.go#L418-L422 https://github.com/kubernetes/kubelet/blob/v0.26.2/config/v1...> which defaults to 110 for what I would presume is CIDR or pid cgroups reasons and thus is unlikely to differ by instance size as the ENI limit you mention does (IIRC)
simianpirate 4y ago
Kubernetes has a default (and considered best practice) limit of 110 pods per node. Are you seeing them cap it lower than that? I've never checked this personally on EKS, TBH. Also, what do you mean by capping resources? In my experience with EKS, I haven't had any issues fully utilizing a nodes resources, even in EKS.
freshmatrix 4y ago
# of pods are essentially capped by the worker node choice. below excerpt from: https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt https://github.com/awslabs/amazon-eks-ami/blob/master/files/... # Mapping is calculated from AWS EC2 API using the following formula: # * First IP on each ENI is not used for pods # * +2 for the pods that use host-networking (AWS CNI and kube-proxy) # # # of ENI * (# of IPv4 per ENI - 1) + 2 # # https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI doc on EC2 instance types ("Amazon EKS recommended maximum pods for each Amazon EC2 instance type") https://docs.aws.amazon.com/eks/latest/userguide/choosing-instance-type.html https://docs.aws.amazon.com/eks/latest/userguide/choosing-in...
mac-chaffee 4y ago
Kubernetes is only tested against 110 pods per node: https://kubernetes.io/docs/setup/best-practices/cluster-large/ https://kubernetes.io/docs/setup/best-practices/cluster-larg... In my testing, it's a limitation in either the container runtime or kubelet in processing all the events that flow from pods. And since the container networking and container storage interfaces aren't part of Kubernetes, there are likely scaling issues in those pieces of software as well.

quantumcd 4y ago

It's been a while since I've used EKS, but I seem to recall it's primarily based on the maximum ENIs the instance type supports.

mdaniel 4y ago
I believe you're correct, although pedantically that would only apply if one is using their vpc-cni <https://github.com/aws/amazon-vpc-cni-k8s#readme https://github.com/aws/amazon-vpc-cni-k8s#readme> and not with a competing CNI. Kubelet offers a configurable for the number of Pods per Node <https://github.com/kubernetes/kubelet/blob/v0.26.2/config/v1beta1/types.go#L418-L422 https://github.com/kubernetes/kubelet/blob/v0.26.2/config/v1...> which defaults to 110 for what I would presume is CIDR or pid cgroups reasons and thus is unlikely to differ by instance size as the ENI limit you mention does (IIRC)

simianpirate 4y ago

Kubernetes has a default (and considered best practice) limit of 110 pods per node. Are you seeing them cap it lower than that? I've never checked this personally on EKS, TBH. Also, what do you mean by capping resources? In my experience with EKS, I haven't had any issues fully utilizing a nodes resources, even in EKS.

freshmatrix 4y ago

# of pods are essentially capped by the worker node choice. below excerpt from: https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt https://github.com/awslabs/amazon-eks-ami/blob/master/files/... # Mapping is calculated from AWS EC2 API using the following formula: # * First IP on each ENI is not used for pods # * +2 for the pods that use host-networking (AWS CNI and kube-proxy) # # # of ENI * (# of IPv4 per ENI - 1) + 2 # # https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI doc on EC2 instance types ("Amazon EKS recommended maximum pods for each Amazon EC2 instance type") https://docs.aws.amazon.com/eks/latest/userguide/choosing-instance-type.html https://docs.aws.amazon.com/eks/latest/userguide/choosing-in...

mac-chaffee 4y ago

Kubernetes is only tested against 110 pods per node: https://kubernetes.io/docs/setup/best-practices/cluster-large/ https://kubernetes.io/docs/setup/best-practices/cluster-larg... In my testing, it's a limitation in either the container runtime or kubelet in processing all the events that flow from pods. And since the container networking and container storage interfaces aren't part of Kubernetes, there are likely scaling issues in those pieces of software as well.