6 ms·

The first thing I do when I look at kubernetes tutorials is look at ingress or load-balancing... and usually I find it absent. It is not the fault of the autho

The first thing I do when I look at kubernetes tutorials is look at ingress or load-balancing... and usually I find it absent. It is not the fault of the author (and I commend them for taking the trouble), but Kubernetes is super complex for getting traffic inside the cluster. But here's a genuine appeal from someone who has suffered this before. In kubernetes, you have multiple types of ingresses - and you invariably need to have a combination of them to get anything done correctly. For example nginx-ingress will lose source-ip info and will not do L4 proxying very well. Which is why the unsaid point of all kubernetes deployments is "use ELB or GLB with proxy protocol and call it a day". I get why - K8s is trying to solve the most complex problems first and trickling it down. Which means for the time being, ingress is a very hard problem to solve in k8s. The second place where every one will get stuck is calico vs weave vs flannel vs whatever. Most tutorials say "use any one of them and let's move on". Unfortunately, that is not the case - choose your network plugins wisely. After spending a lot of time in k8s, I setup a Docker Swarm cluster in about 3 hours. It does not have some of the more complex usecases supported but it does three things super well - networking, ingress and secrets ... and for most beginners, that's all they need. P.S. i have the exact same stack running on kubernetes and docker swarm Docker Compose yml form 3.1 is brilliant - other than my docker files, i need a 30 line yml file to deploy my whole stack across 3 machines. Kubernetes on the other hand needs atleast 12 yml files ("deployments" and "services"). I'm not counting another 4 yml files for statefulsets and persistentvolumeclaim since there is nothing equivalent in docker swarm

predakanga 10y ago
Could you provide some more detail about your statement that "nginx-ingress will lose source-ip info"? I'm currently working towards a k8s-on-metal system using nginx for ingress and was led to believe that I could use either X-Forwarded-For or proxy_protocol to preserve the originating IP address. Is the traffic bounced through an SNAT rule on it's way to the ingress controller or similar?
- sandGorgon 10y ago
  you should google for proxy-protocol. so here's the thing - nginx cannot INJECT proxy protocol. I'm not an expert in nginx.. but atleast the versions used in k8s couldnt. It can read/pass-through proxy protocol headers. So if you have haproxy in front of nginx.. it will work fine (or ELB/GLB). Which is why there's another beta ingress - the "keepalived-vip" repo in k8s ingress. However, not a lot of people seem to be using haproxy primarily because haproxy cannot have a zero-downtime deploy. A couple of weeks back on k8s slack, I was discussing using Github multibinder to solve that one problem and move ahead with haproxy ingress. protip: please join the #sig-onpremise channel on slack. There's not a lot of momentum for bare metal deploys otherwise. https://githubengineering.com/glb-part-2-haproxy-zero-downtime-zero-delay-reloads-with-multibinder/ https://githubengineering.com/glb-part-2-haproxy-zero-downti...
  - predakanga 10y ago
    I'm familiar with proxy-protocol, and while I haven't injected it with nginx, it seems that it does support it: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_protocol https://nginx.org/en/docs/stream/ngx_stream_proxy_module.htm... For reference, the feature was added in nginx 1.9.2, and the ingress-nginx controller appears to use nginx 1.11.9: https://github.com/kubernetes/ingress/blob/0.9.1/images/nginx-slim/build.sh https://github.com/kubernetes/ingress/blob/0.9.1/images/ngin... Regardless, would I be right in assuming that your issues are mostly related to the ingress of TCP streams? My usecase is almost exclusively HTTP, so I might be missing the worst of it. Thanks for the pointer to the #sig-onpremise channel, I'll be sure to check it out. Been meaning to look into the SIGs since reading https://coreos.com/blog/self-hosted-kubernetes.html https://coreos.com/blog/self-hosted-kubernetes.html
    - sandGorgon 10y ago
      please double check - nginx's support for proxy protocol is to be able to read it. it cannot support injecting it. if nginx is the first thing your traffic hits, it will not inject proxy protocol headers. Again, I will not stake my reputation on this... but I remember going deep into this about 2 months back. if you are using nginx ingress, then you need to move all of your existing config into their own config file format. Since setting up full TLS internally in kubernetes is buggy, I didnt want to terminate my traffic on the ingress. Also I really did not want to move my (fairly complex) nginx configuration to the ingress. Which is why I'm a big believer in haproxy ingress - it was designed to interface with nginx. However, you are already one of the "enlightened" ones. You actually do know what an ingress is. A significant amount of k8s slack traffic is "how do I setup a loadbalancer/how to make the cluster available to the outside world"
shaklee3 10y ago
Maybe you or someone else can help me understand the use case for a load balancer. I have an application where it has a very high rate of ingress traffic. So much so that while a load balancer sounds like it's ideal, a single node (the balancer) would not be able to handle the traffic. Is the load balancer only useful for applications with a low ingress rate that you're trying spread across many servers?
- hashkb 10y ago
  Without a load balancer, how do you currently spread the load over multiple nodes in your app?
  - shaklee3 10y ago
    If you have control over the sender, you can, right? That's my use case and maybe that's why I don't see the value.
    - hashkb 10y ago
      I see. A lot of us I think assume we are talking about web apps or mobile apps where you don't.
- sandGorgon 10y ago
  great question! kubernetes pretty much mandates either an LB because it is offloading the ingress job to a machine which can talk to the outside world and the VPC simultaneously. Otherwise you either get into complex hostport/nodeport/hostnetwork semantics... or use an ingress. In the k8s world, there is not many simpler alternatives.
- lobster_johnson 10y ago
  How much traffic are we talking about? Google Cloud Platform's load balancers (which sit outside of Kubernetes and are replicated geographically) are designed for quite a high load of traffic. Either way, your client has to know what pod (group of Docker containers running your app) to send traffic to. The pods run on hosts and ports that aren't fixed in time. So if you want some kind of client-resolving load balancing without involving a central bottleneck on the server, you'd have to transport that inventory of pod-host-ports to the client, and then either provide a host-based proxy or open the ports on the nodes themselves. And of course you'd have to build the balancing/failover logic into the client (which would either have to be randomly distributed, or rely on resource utilization info fetched from Kubernetes). That may be great for some narrow use cases, but for most applications, the load balancing pattern is much simpler.
tamalsaha001 10y ago
You are absolutely right that getting traffic inside the cluster is probably one of most difficult tasks in Kubernetes. The standard ingress objects are too simple when you have a complex deployment. So, we wrote our own Ingress Third Party Resource to fit the needs of our deployment. We use HAProxy. We have been using this in prod for almost a year. You can find it here: https://github.com/appscode/voyager https://github.com/appscode/voyager If you find any issues or have questions, please file Github issues.
- sandGorgon 10y ago
  this is awesome! please please submit this to kubernetes-incubator
  - tamalsaha001 10y ago
    Thanks for the kind words. The incubation process seemed too complicated (process heavy?) for small teams like us. But I will look into that again.
- maktouch 10y ago
  I found this last week and I'm very impressed because it solves a lot of the problems we had with existing ingresses. I really like how you guys decided to do your own 3rd party manifests instead of using annotations. I'm going to play more with this :-)
koltaggar 10y ago
Couldnt agree more with the ingress sentiments
ahawkins 10y ago
Author here > The first thing I do when I look at kubernetes tutorials is look at ingress or load-balancing... and usually I find it absent. Yup! I dedicate a few minutes to this in the grab lesson because how important it is. > Docker Compose yml form 3.1 is brilliant - other than my docker files, i need a 30 line yml file to deploy my whole stack across 3 machines. Kubernetes on the other hand needs atleast 12 yml files ("deployments" and "services"). I'm not counting another 4 yml files for statefulsets and persistentvolumeclaim since there is nothing equivalent in docker swarm Interesting point. You may like helm (https://helm.sh https://helm.sh) as a tool for deploying Kubernetes applications.
- sandGorgon 10y ago
  actually, docker compose file is so awesome that there is now a top level incubator project called kompose that is trying to provide drop-in compatibility. https://github.com/kubernetes-incubator/kompose https://github.com/kubernetes-incubator/kompose you should check out the #kompose channel on slack.
atombender 10y ago
> Kubernetes on the other hand needs atleast 12 yml files You don't need to split your manifests into multiple files. You can use a list: apiVersion: v1 kind: List items: - apiVersion: extensions/v1beta1 kind: Deployment ... - apiVersion: v1 kind: Service ... - apiVersion: v1 kind: ConfigMap ... Edit: Or YAML's "multiple document" syntax: apiVersion: extensions/v1beta1 kind: Deployment ... --- apiVersion: v1 kind: Service ... --- apiVersion: v1 kind: ConfigMap ... Then you just use "kubectl apply -f myapp.yml" to create or update. Re ingress, I agree that it's probably the weakest point of Kubernetes at this point. It's particularly weak when it comes to internal load balancing. When you have lots of services that should only need to be available internally, you'll want to have an internal ingress; I don't know about AWS, but Google Cloud Platform's internal load balancer doesn't support pointing at Kubernetes [1]. I haven't found a better option than to run Traefik as a DaemonSet and rely on round-robin DNS (aka poor man's HA). [1] https://github.com/kubernetes/ingress/issues/112 https://github.com/kubernetes/ingress/issues/112
- crb 10y ago
  Or just put --- between various stanzas in one file: https://kubernetes.io/docs/user-guide/managing-deployments/#organizing-resource-configurations https://kubernetes.io/docs/user-guide/managing-deployments/#... (Google now has a L3 internal load balancer, but not a L7 LB. It's fair to say that this will be made much easier this year.)
  - lobster_johnson 10y ago
    True! I keep forgetting the multi-document support in YAML.
  - andrewstuart2 10y ago
    I really think the community ought to adopt this as best practice and actively discourage (or even deprecate) the List type. It's just too much noise and zero benefit IMO over raw slices or serialization as arrays (or document lists for YAML).
bdcravens 10y ago
> Docker Compose yml form 3.1 is brilliant What's nice about ECS is that it lets you deploy using Docker Compose files (though you often make minor modifications for ECS) ecs-cli compose --file ecs-compose.yml create However, there is the kompose project to bring Docker Compose to Kubernetes (it's still in Kubernetes Incubator at this point): http://blog.kubernetes.io/2016/11/kompose-tool-go-from-docker-compose-to-kubernetes.html http://blog.kubernetes.io/2016/11/kompose-tool-go-from-docke... https://github.com/kubernetes-incubator/kompose https://github.com/kubernetes-incubator/kompose

predakanga 10y ago

Could you provide some more detail about your statement that "nginx-ingress will lose source-ip info"? I'm currently working towards a k8s-on-metal system using nginx for ingress and was led to believe that I could use either X-Forwarded-For or proxy_protocol to preserve the originating IP address. Is the traffic bounced through an SNAT rule on it's way to the ingress controller or similar?

sandGorgon 10y ago
you should google for proxy-protocol. so here's the thing - nginx cannot INJECT proxy protocol. I'm not an expert in nginx.. but atleast the versions used in k8s couldnt. It can read/pass-through proxy protocol headers. So if you have haproxy in front of nginx.. it will work fine (or ELB/GLB). Which is why there's another beta ingress - the "keepalived-vip" repo in k8s ingress. However, not a lot of people seem to be using haproxy primarily because haproxy cannot have a zero-downtime deploy. A couple of weeks back on k8s slack, I was discussing using Github multibinder to solve that one problem and move ahead with haproxy ingress. protip: please join the #sig-onpremise channel on slack. There's not a lot of momentum for bare metal deploys otherwise. https://githubengineering.com/glb-part-2-haproxy-zero-downtime-zero-delay-reloads-with-multibinder/ https://githubengineering.com/glb-part-2-haproxy-zero-downti...
- predakanga 10y ago
  I'm familiar with proxy-protocol, and while I haven't injected it with nginx, it seems that it does support it: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_protocol https://nginx.org/en/docs/stream/ngx_stream_proxy_module.htm... For reference, the feature was added in nginx 1.9.2, and the ingress-nginx controller appears to use nginx 1.11.9: https://github.com/kubernetes/ingress/blob/0.9.1/images/nginx-slim/build.sh https://github.com/kubernetes/ingress/blob/0.9.1/images/ngin... Regardless, would I be right in assuming that your issues are mostly related to the ingress of TCP streams? My usecase is almost exclusively HTTP, so I might be missing the worst of it. Thanks for the pointer to the #sig-onpremise channel, I'll be sure to check it out. Been meaning to look into the SIGs since reading https://coreos.com/blog/self-hosted-kubernetes.html https://coreos.com/blog/self-hosted-kubernetes.html
  - sandGorgon 10y ago
    please double check - nginx's support for proxy protocol is to be able to read it. it cannot support injecting it. if nginx is the first thing your traffic hits, it will not inject proxy protocol headers. Again, I will not stake my reputation on this... but I remember going deep into this about 2 months back. if you are using nginx ingress, then you need to move all of your existing config into their own config file format. Since setting up full TLS internally in kubernetes is buggy, I didnt want to terminate my traffic on the ingress. Also I really did not want to move my (fairly complex) nginx configuration to the ingress. Which is why I'm a big believer in haproxy ingress - it was designed to interface with nginx. However, you are already one of the "enlightened" ones. You actually do know what an ingress is. A significant amount of k8s slack traffic is "how do I setup a loadbalancer/how to make the cluster available to the outside world"

shaklee3 10y ago

Maybe you or someone else can help me understand the use case for a load balancer. I have an application where it has a very high rate of ingress traffic. So much so that while a load balancer sounds like it's ideal, a single node (the balancer) would not be able to handle the traffic. Is the load balancer only useful for applications with a low ingress rate that you're trying spread across many servers?

hashkb 10y ago
Without a load balancer, how do you currently spread the load over multiple nodes in your app?
- shaklee3 10y ago
  If you have control over the sender, you can, right? That's my use case and maybe that's why I don't see the value.
  - hashkb 10y ago
    I see. A lot of us I think assume we are talking about web apps or mobile apps where you don't.
sandGorgon 10y ago
great question! kubernetes pretty much mandates either an LB because it is offloading the ingress job to a machine which can talk to the outside world and the VPC simultaneously. Otherwise you either get into complex hostport/nodeport/hostnetwork semantics... or use an ingress. In the k8s world, there is not many simpler alternatives.
lobster_johnson 10y ago
How much traffic are we talking about? Google Cloud Platform's load balancers (which sit outside of Kubernetes and are replicated geographically) are designed for quite a high load of traffic. Either way, your client has to know what pod (group of Docker containers running your app) to send traffic to. The pods run on hosts and ports that aren't fixed in time. So if you want some kind of client-resolving load balancing without involving a central bottleneck on the server, you'd have to transport that inventory of pod-host-ports to the client, and then either provide a host-based proxy or open the ports on the nodes themselves. And of course you'd have to build the balancing/failover logic into the client (which would either have to be randomly distributed, or rely on resource utilization info fetched from Kubernetes). That may be great for some narrow use cases, but for most applications, the load balancing pattern is much simpler.

tamalsaha001 10y ago

You are absolutely right that getting traffic inside the cluster is probably one of most difficult tasks in Kubernetes. The standard ingress objects are too simple when you have a complex deployment. So, we wrote our own Ingress Third Party Resource to fit the needs of our deployment. We use HAProxy. We have been using this in prod for almost a year. You can find it here: https://github.com/appscode/voyager https://github.com/appscode/voyager If you find any issues or have questions, please file Github issues.

sandGorgon 10y ago
this is awesome! please please submit this to kubernetes-incubator
- tamalsaha001 10y ago
  Thanks for the kind words. The incubation process seemed too complicated (process heavy?) for small teams like us. But I will look into that again.
maktouch 10y ago
I found this last week and I'm very impressed because it solves a lot of the problems we had with existing ingresses. I really like how you guys decided to do your own 3rd party manifests instead of using annotations. I'm going to play more with this :-)

koltaggar 10y ago

Couldnt agree more with the ingress sentiments

ahawkins 10y ago

Author here > The first thing I do when I look at kubernetes tutorials is look at ingress or load-balancing... and usually I find it absent. Yup! I dedicate a few minutes to this in the grab lesson because how important it is. > Docker Compose yml form 3.1 is brilliant - other than my docker files, i need a 30 line yml file to deploy my whole stack across 3 machines. Kubernetes on the other hand needs atleast 12 yml files ("deployments" and "services"). I'm not counting another 4 yml files for statefulsets and persistentvolumeclaim since there is nothing equivalent in docker swarm Interesting point. You may like helm (https://helm.sh https://helm.sh) as a tool for deploying Kubernetes applications.

sandGorgon 10y ago
actually, docker compose file is so awesome that there is now a top level incubator project called kompose that is trying to provide drop-in compatibility. https://github.com/kubernetes-incubator/kompose https://github.com/kubernetes-incubator/kompose you should check out the #kompose channel on slack.

atombender 10y ago

> Kubernetes on the other hand needs atleast 12 yml files You don't need to split your manifests into multiple files. You can use a list: apiVersion: v1 kind: List items: - apiVersion: extensions/v1beta1 kind: Deployment ... - apiVersion: v1 kind: Service ... - apiVersion: v1 kind: ConfigMap ... Edit: Or YAML's "multiple document" syntax: apiVersion: extensions/v1beta1 kind: Deployment ... --- apiVersion: v1 kind: Service ... --- apiVersion: v1 kind: ConfigMap ... Then you just use "kubectl apply -f myapp.yml" to create or update. Re ingress, I agree that it's probably the weakest point of Kubernetes at this point. It's particularly weak when it comes to internal load balancing. When you have lots of services that should only need to be available internally, you'll want to have an internal ingress; I don't know about AWS, but Google Cloud Platform's internal load balancer doesn't support pointing at Kubernetes [1]. I haven't found a better option than to run Traefik as a DaemonSet and rely on round-robin DNS (aka poor man's HA). [1] https://github.com/kubernetes/ingress/issues/112 https://github.com/kubernetes/ingress/issues/112

crb 10y ago
Or just put --- between various stanzas in one file: https://kubernetes.io/docs/user-guide/managing-deployments/#organizing-resource-configurations https://kubernetes.io/docs/user-guide/managing-deployments/#... (Google now has a L3 internal load balancer, but not a L7 LB. It's fair to say that this will be made much easier this year.)
- lobster_johnson 10y ago
  True! I keep forgetting the multi-document support in YAML.
- andrewstuart2 10y ago
  I really think the community ought to adopt this as best practice and actively discourage (or even deprecate) the List type. It's just too much noise and zero benefit IMO over raw slices or serialization as arrays (or document lists for YAML).

bdcravens 10y ago

> Docker Compose yml form 3.1 is brilliant What's nice about ECS is that it lets you deploy using Docker Compose files (though you often make minor modifications for ECS) ecs-cli compose --file ecs-compose.yml create However, there is the kompose project to bring Docker Compose to Kubernetes (it's still in Kubernetes Incubator at this point): http://blog.kubernetes.io/2016/11/kompose-tool-go-from-docker-compose-to-kubernetes.html http://blog.kubernetes.io/2016/11/kompose-tool-go-from-docke... https://github.com/kubernetes-incubator/kompose https://github.com/kubernetes-incubator/kompose