5 ms·

Suggestion to anybody reading this: don't use a DaemonSet for this. This really ought to be a Deployment of nginx-ingress resources behind a service exposed as

Suggestion to anybody reading this: don't use a DaemonSet for this. This really ought to be a Deployment of nginx-ingress resources behind a service exposed as `type: LoadBalancer` (if you're in a cloud-provider that supports LoadBalancer services). Then just create DNS aliases and configure nginx to do session affinity if needed, etc. Not only will it be able to scale with your load instead of cluster size, but you can actually update it in a rolling update already; DaemonSets cannot yet do that. Really the most important part, though, is that DaemonSets are for services that need to run on each host. Like a log collection service [1] or prometheus node exporter [2]. [1] https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch https://github.com/kubernetes/kubernetes/tree/master/cluster... [2] https://github.com/prometheus/node_exporter https://github.com/prometheus/node_exporter

fortytw2 10y ago
So I (the author) am a bit torn on this - a `type: LoadBalancer` service will create a NodePort underneath (yet another internal loadbalancer) and map those ports to a $cloud-platform-tcp-loadbalancer. By using a DaemonSet with a host port bound, you avoid a layer of internal routing. I'm not so sure if one approach is _particularly_ better than the other though.
- andrewstuart2 10y ago
  As far as the routing layer, that's going to be a very small gain as it's all handled by the kernel via iptables anyway. What you give up by using a DaemonSet is a constant overhead for every single worker node that gets created. You have to set aside capacity for traffic you might not have. A Deployment can be scaled by actual utilization of the pod, and in the near future (if not already) via custom metrics of your own design (e.g. prometheus). So if you have to spin up 50 new nodes to handle some batch machine learning work, you're going to wastefully create 50 new nginx instances when there's no new ingress traffic to handle. With a deployment, it just scales naturally as needed. So it's not about the marginal gains, it's about using the right tool for the job. :-)
  - fortytw2 10y ago
    Fair enough points :) I'll see about reworking the article, and our deployment a bit - keeping in mind that LoadBalancer services are platform dependent
  - xkarga00 10y ago
    FWIW, you can constrain a DaemonSet to run on a specific set of nodes by using nodeAffinity but you would need to label those nodes appropriately which you are going to do anyway in a production cluster (use a specific set of nodes for running infra components such as routers or registries).
  - mrkurt 10y ago
    Services are all handled with IP Tables, but in current versions of kubernetes you end up sending a lot of traffic to different nodes for no reason. There's an issue to prevent that, and send traffic to the pod on the local node if possible, but it's pretty gross right now. This sucks for performance/reliability reasons. It also makes it crazy difficult to keep track of visitors' source IPs.
caleblloyd 10y ago
In order to get the proper Source IP with TCP Load Balancing before Kubernetes 1.5, you used to need to use a Daemon Set with Host Networking. Kubernetes 1.5 introduced Source IP using Source NAT and Health Checks: https://kubernetes.io/docs/tutorials/services/source-ip/ https://kubernetes.io/docs/tutorials/services/source-ip/ You still must write scheduling rules so that pods are scheduled to have at most 1 instance running per node.
maktouch 10y ago
We started nginx-ingress as a deployment, and we converted it to a DaemonSet: - We rarely deploy new versions of the ingress controller - We can't (or don't know how to) choose in which nodes the pods will go. If I make a deployment with 10 replicas, there's a chance that it'll all go in the same node - Because we can't choose to distribute the pods, when a node containing the ingress pod went down, there was a noticeable blip of downtime (~27 seconds approx.). That's kinda unacceptable. - Nginx ingress is pretty light. I don't mind it having just 1 of them in each node. - Since we put our databases and stateful stuff outside kubernetes, we also decided to separate web facing kubernetes cluster and worker ones. This solves the problem of the "spin up 50 new nodes to handle some batch machine learning job". So far, so good. I would actually suggest that you use a DaemonSet for this, just like I suggest you convert Kube-DNS to a daemonset (it's not by default on GKE for some obscure reason).

fortytw2 10y ago

So I (the author) am a bit torn on this - a `type: LoadBalancer` service will create a NodePort underneath (yet another internal loadbalancer) and map those ports to a $cloud-platform-tcp-loadbalancer. By using a DaemonSet with a host port bound, you avoid a layer of internal routing. I'm not so sure if one approach is _particularly_ better than the other though.

andrewstuart2 10y ago
As far as the routing layer, that's going to be a very small gain as it's all handled by the kernel via iptables anyway. What you give up by using a DaemonSet is a constant overhead for every single worker node that gets created. You have to set aside capacity for traffic you might not have. A Deployment can be scaled by actual utilization of the pod, and in the near future (if not already) via custom metrics of your own design (e.g. prometheus). So if you have to spin up 50 new nodes to handle some batch machine learning work, you're going to wastefully create 50 new nginx instances when there's no new ingress traffic to handle. With a deployment, it just scales naturally as needed. So it's not about the marginal gains, it's about using the right tool for the job. :-)
- fortytw2 10y ago
  Fair enough points :) I'll see about reworking the article, and our deployment a bit - keeping in mind that LoadBalancer services are platform dependent
- xkarga00 10y ago
  FWIW, you can constrain a DaemonSet to run on a specific set of nodes by using nodeAffinity but you would need to label those nodes appropriately which you are going to do anyway in a production cluster (use a specific set of nodes for running infra components such as routers or registries).
- mrkurt 10y ago
  Services are all handled with IP Tables, but in current versions of kubernetes you end up sending a lot of traffic to different nodes for no reason. There's an issue to prevent that, and send traffic to the pod on the local node if possible, but it's pretty gross right now. This sucks for performance/reliability reasons. It also makes it crazy difficult to keep track of visitors' source IPs.

caleblloyd 10y ago

In order to get the proper Source IP with TCP Load Balancing before Kubernetes 1.5, you used to need to use a Daemon Set with Host Networking. Kubernetes 1.5 introduced Source IP using Source NAT and Health Checks: https://kubernetes.io/docs/tutorials/services/source-ip/ https://kubernetes.io/docs/tutorials/services/source-ip/ You still must write scheduling rules so that pods are scheduled to have at most 1 instance running per node.

maktouch 10y ago

We started nginx-ingress as a deployment, and we converted it to a DaemonSet: - We rarely deploy new versions of the ingress controller - We can't (or don't know how to) choose in which nodes the pods will go. If I make a deployment with 10 replicas, there's a chance that it'll all go in the same node - Because we can't choose to distribute the pods, when a node containing the ingress pod went down, there was a noticeable blip of downtime (~27 seconds approx.). That's kinda unacceptable. - Nginx ingress is pretty light. I don't mind it having just 1 of them in each node. - Since we put our databases and stateful stuff outside kubernetes, we also decided to separate web facing kubernetes cluster and worker ones. This solves the problem of the "spin up 50 new nodes to handle some batch machine learning job". So far, so good. I would actually suggest that you use a DaemonSet for this, just like I suggest you convert Kube-DNS to a daemonset (it's not by default on GKE for some obscure reason).