5 ms·

> We enhanced GLB, our internal load balancing service, to support Kubernetes NodePort Services. Everyone does this - because Kubernetes Achilles heel is its i

> We enhanced GLB, our internal load balancing service, to support Kubernetes NodePort Services. Everyone does this - because Kubernetes Achilles heel is its ingress. It is still built philosophically as a post-loadbalancing system . This is the single biggest reason why using Docker Swarm is so pleasant.

alpb 9y ago
Can you elaborate? What are the issues you're seeing with Ingress?
- mnutt 9y ago
  The service I'm working on porting to kubernetes has bandwidth requirements that far exceed a single machine, and though we're on AWS, we can't use ELB for various reasons. We ended up with a ReplicaSet of haproxies using HostPort, and a separate app that watches for haproxy service changes and updates Route53 round-robin DNS. We're somewhat fortunate that we only have a single service that needs to use the http/https HostPort. We could have just left our haproxies outside of kubernetes, and may eventually end up doing so if the network performance doesn't meet our needs. As it is, it all works but there are a ton of sidecar services all over the place.
  - yebyen 9y ago
    This sounds like what Deis Router was created to do. It sounds like you've already got it nailed down, but maybe like to have a look at this: https://github.com/deis/router https://github.com/deis/router (It's probably tightly coupled to Deis Controller, but something to look at anyway!)
orf 9y ago
Can you elaborate? I found Kubernetes ingresses to be one of the most pleasant parts - we use the `nginx-ingress` from helm and it works very well. Not exactly an industrial github-strength load balancer, but it will get you a long way surely?
- snug 9y ago
  I think he means on a global scale (multi datacenter/region/cloud)
kelseyhightower 9y ago
Any load balancer can be configured or modified to target routable Pod IP addresses and skip node ports altogether. You'll have to integrate with the Kubernetes Endpoints API[1] and support dynamic backends. Another option would be to leverage Kubernetes' DNS and the SRV records[2] backing each service. The reason node ports are used in the Cloud today is because most Cloud load balancing solutions only target VMs, not arbitrary endpoints such as containers, a limitation that will go away over time. [1] Envoy with Kubernetes Endpoints integration: https://github.com/kelseyhightower/kubernetes-envoy-sds https://github.com/kelseyhightower/kubernetes-envoy-sds [2] https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ https://kubernetes.io/docs/concepts/services-networking/dns-...
- codesuki 9y ago
  Sorry to turn this into a different direction but I wanted to ask you about packaging + deployment. I am about to try using helm for packaging my Kubernetes configs to make use of its templating. Being able to include Kubernetes changes in the release makes it less common to forget some new env variable etc. The only thing I don't like is that it replaces kubectl in a way. And some comments here speak of problems during deploy and rollback which makes me wary. What are some best practices around that topic? I was also thinking of deploying with spinnaker but could not confirm it works with helm. Any info is appreciated! P.S. have you thought about making a redis-cluster example repo? Redis 4.0 has a new `cluster_announce_ip` setting with which I made it work, but I still don't like my setup 100%.
  - deleted 9y ago
    [deleted]
  - bacongobbler 9y ago
    This is a question you should ask in #helm-users in kubernetes.slack.com. Lots of Helm users are more than happy to help answer questions like this. :)
- sandGorgon 9y ago
  Hi Kelsey, thanks for replying. You are definitely k8s secret weapon ;) I know about this. However, ultimately the question is that Kubernetes is not a gradual scale up solution for most people. I have to be prepared to deal with building my own load balancer. Basically, I cannot do an on-metal deployment very easily. Most of the questions on k8s slack for metal deployments were - how do I set this up with few tweaks like ssl pass through and source ip preservation. It is not easy. Either you build your own load balancer or you use a cloud provided one. Now, Ingresses are not pleasant. I'm not sure about the state of source ip preservation, but last I remember that the nginx ingress had still not surfaced ssl_preread_server_name to the ingress configuration. Now, what would have been nice is if it was ingress-all-the-way-down : ingress with something like istio/linkerd, maybe it is possible. Tl;Dr - I'm not github. I can't build my own load balancer. Give me something that works out of the box. Yes, I know it may go down - I'll survive. Docker Swarm does this.
movedx 9y ago
We use Kubernetes on AWS - it creates the ELBs for us when we provision a new service. We've never created a load balancer after deploying a service.
brown9-2 9y ago
What does Swarm do differently here?
- sandGorgon 9y ago
  swarm is a batteries-included system. you can use it in the way that kubernetes is used... or (if you dont have all these sophisticated load balancers), you can allow it to load balance for you. https://docs.docker.com/engine/swarm/ingress/#publish-a-port-for-a-service https://docs.docker.com/engine/swarm/ingress/#publish-a-port... What it means is that when you create a docker swarm - it starts working.
  - colemickens 9y ago
    I don't really know much about Swarm. Can you describe how that is different than NodePort with Kubernetes (which is the default for Services)?
  - brown9-2 9y ago
    Thanks for the link, but this sounds the same as Kubernetes' Service NodePort.
    - sandGorgon 9y ago
      I think I hashlinked directly. I wanted to post the main page. Ingress also use nodeport, the difference is not in nodeport..But in the ingress setup itself. Docker Swarm's inbuilt ingress is now trying to build in proxy protocol and ipip mode for default usage. Fwiw, you can use Swarm's inbuilt ingress with an external load balancer as well.
      - thockingoog 9y ago
        The link you posted highlights my exact concern with Swarm. If I use port 8080, then nobody else can use port 8080. That is a different tradeoff than Kubernetes is willing to make. Functionally, we have this in NodePort, but because it is exclusively managed, you're very unlikely to have a meaningful conflict. BUT it depends on traffic ingress being managed. If you just want to map port 8080 on every node into your kube Service, it's more complicated than Swarm. Granted. That's because we don't think you should do that - it doesn't scale. If you just want to map a port on every node (and you don't care which port), kube Services have you covered. Swarm's model is a nearly-direct clone of this.
thockingoog 9y ago
This is an apt point. Kubernetes models Borg, and Borg has no concept of ingress. That's an entirely different problem space. Obviously that doesn't fly if there isn't an equivalent open solution, so we did what we could with the system to make it not terrible. We can do more. The point about Swarm is interesting, and has been much on my mind. Some of Kubernetes' perceived complexity is because we go to great lengths to avoid ever having two users collide, with escape hatches for the people who really need "unfriendly" features. This is because, again, Kubernetes models Borg. Borg clusters are giant, shared, multi-user, multi-app animals, where the users are in different business units and chances of collisions are high. Swarm, on the other hand, thinks of a cluster more as an application construct. Sharing is not a big problem, and coordination is easy and local. This allows them to make different tradeoffs. I doubt very much that you can run a large number of similar apps in a single swarm without having collisions on things like ports. I still believe the large-shared-cluster model is right in the limit. There are so many efficiencies to be had. But there are legit reasons it is hard to achieve right now. I'm very interested in ways to make Kubernetes easier to use, ESPECIALLY in this regard. Real user feedback is critical.
- sandGorgon 9y ago
  that's an interesting perspective - however, Docker Swarm also does that. Docker Swarm secrets have been GA for longer thank k8s. The new UCP mechanism in the Docker Datacenter product is fairly interesting (has not made it to swarm yet). Its a paid product but makes RBAC pretty simple - https://success.docker.com/Architecture/Docker_Reference_Architecture%3A_Securing_Docker_EE_and_Security_Best_Practices https://success.docker.com/Architecture/Docker_Reference_Arc... . One very interesting tool that Docker makes available is https://store.docker.com/community/images/docker/docker-bench-security https://store.docker.com/community/images/docker/docker-benc... I think the issue with k8s is that it is competing with the "Ruby on Rails" of frameworks viz Docker Swarm. I think the pluggability of critical pieces like ingress and secrets was taken too far. > I doubt very much that you can run a large number of similar apps in a single swarm without having collisions on things like ports. I dont think that is true, it does manage its overlay networks pretty well. Which is FWIW, another place where k8s took the non-opinionatedness too far. I think the number of bugs on "my stuff doesnt work with flannel but works with calico" should tell you that. To be honest, Docker Swarm has some of these issues as well - https://github.com/moby/moby/issues/25526 https://github.com/moby/moby/issues/25526. But the fixes are included in the "batteries". On kuberenetes, I have to run behind upstream projects with heterogenous configuration (nginx vs haproxy ingress. or flannel vs calico configuration) to try and fix it.
hardymansen 9y ago
I have to pitch our solution to this problem also, a fork of the nginx ingress controller, External nginx ingress controller. https://github.com/unibet/ext_nginx https://github.com/unibet/ext_nginx Basically just handling nginx.conf from information in k8. We run in production with ECMP in our routers to load balance stateless over any number of nodes. Easy to understand and very scalable.

alpb 9y ago

Can you elaborate? What are the issues you're seeing with Ingress?

mnutt 9y ago
The service I'm working on porting to kubernetes has bandwidth requirements that far exceed a single machine, and though we're on AWS, we can't use ELB for various reasons. We ended up with a ReplicaSet of haproxies using HostPort, and a separate app that watches for haproxy service changes and updates Route53 round-robin DNS. We're somewhat fortunate that we only have a single service that needs to use the http/https HostPort. We could have just left our haproxies outside of kubernetes, and may eventually end up doing so if the network performance doesn't meet our needs. As it is, it all works but there are a ton of sidecar services all over the place.
- yebyen 9y ago
  This sounds like what Deis Router was created to do. It sounds like you've already got it nailed down, but maybe like to have a look at this: https://github.com/deis/router https://github.com/deis/router (It's probably tightly coupled to Deis Controller, but something to look at anyway!)

orf 9y ago

Can you elaborate? I found Kubernetes ingresses to be one of the most pleasant parts - we use the `nginx-ingress` from helm and it works very well. Not exactly an industrial github-strength load balancer, but it will get you a long way surely?

snug 9y ago
I think he means on a global scale (multi datacenter/region/cloud)

kelseyhightower 9y ago

Any load balancer can be configured or modified to target routable Pod IP addresses and skip node ports altogether. You'll have to integrate with the Kubernetes Endpoints API[1] and support dynamic backends. Another option would be to leverage Kubernetes' DNS and the SRV records[2] backing each service. The reason node ports are used in the Cloud today is because most Cloud load balancing solutions only target VMs, not arbitrary endpoints such as containers, a limitation that will go away over time. [1] Envoy with Kubernetes Endpoints integration: https://github.com/kelseyhightower/kubernetes-envoy-sds https://github.com/kelseyhightower/kubernetes-envoy-sds [2] https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ https://kubernetes.io/docs/concepts/services-networking/dns-...

codesuki 9y ago
Sorry to turn this into a different direction but I wanted to ask you about packaging + deployment. I am about to try using helm for packaging my Kubernetes configs to make use of its templating. Being able to include Kubernetes changes in the release makes it less common to forget some new env variable etc. The only thing I don't like is that it replaces kubectl in a way. And some comments here speak of problems during deploy and rollback which makes me wary. What are some best practices around that topic? I was also thinking of deploying with spinnaker but could not confirm it works with helm. Any info is appreciated! P.S. have you thought about making a redis-cluster example repo? Redis 4.0 has a new `cluster_announce_ip` setting with which I made it work, but I still don't like my setup 100%.
- deleted 9y ago
  [deleted]
- bacongobbler 9y ago
  This is a question you should ask in #helm-users in kubernetes.slack.com. Lots of Helm users are more than happy to help answer questions like this. :)
sandGorgon 9y ago
Hi Kelsey, thanks for replying. You are definitely k8s secret weapon ;) I know about this. However, ultimately the question is that Kubernetes is not a gradual scale up solution for most people. I have to be prepared to deal with building my own load balancer. Basically, I cannot do an on-metal deployment very easily. Most of the questions on k8s slack for metal deployments were - how do I set this up with few tweaks like ssl pass through and source ip preservation. It is not easy. Either you build your own load balancer or you use a cloud provided one. Now, Ingresses are not pleasant. I'm not sure about the state of source ip preservation, but last I remember that the nginx ingress had still not surfaced ssl_preread_server_name to the ingress configuration. Now, what would have been nice is if it was ingress-all-the-way-down : ingress with something like istio/linkerd, maybe it is possible. Tl;Dr - I'm not github. I can't build my own load balancer. Give me something that works out of the box. Yes, I know it may go down - I'll survive. Docker Swarm does this.

movedx 9y ago

We use Kubernetes on AWS - it creates the ELBs for us when we provision a new service. We've never created a load balancer after deploying a service.

brown9-2 9y ago

What does Swarm do differently here?

sandGorgon 9y ago
swarm is a batteries-included system. you can use it in the way that kubernetes is used... or (if you dont have all these sophisticated load balancers), you can allow it to load balance for you. https://docs.docker.com/engine/swarm/ingress/#publish-a-port-for-a-service https://docs.docker.com/engine/swarm/ingress/#publish-a-port... What it means is that when you create a docker swarm - it starts working.
- colemickens 9y ago
  I don't really know much about Swarm. Can you describe how that is different than NodePort with Kubernetes (which is the default for Services)?
- brown9-2 9y ago
  Thanks for the link, but this sounds the same as Kubernetes' Service NodePort.
  - sandGorgon 9y ago
    I think I hashlinked directly. I wanted to post the main page. Ingress also use nodeport, the difference is not in nodeport..But in the ingress setup itself. Docker Swarm's inbuilt ingress is now trying to build in proxy protocol and ipip mode for default usage. Fwiw, you can use Swarm's inbuilt ingress with an external load balancer as well.
    - thockingoog 9y ago
      The link you posted highlights my exact concern with Swarm. If I use port 8080, then nobody else can use port 8080. That is a different tradeoff than Kubernetes is willing to make. Functionally, we have this in NodePort, but because it is exclusively managed, you're very unlikely to have a meaningful conflict. BUT it depends on traffic ingress being managed. If you just want to map port 8080 on every node into your kube Service, it's more complicated than Swarm. Granted. That's because we don't think you should do that - it doesn't scale. If you just want to map a port on every node (and you don't care which port), kube Services have you covered. Swarm's model is a nearly-direct clone of this.

thockingoog 9y ago

This is an apt point. Kubernetes models Borg, and Borg has no concept of ingress. That's an entirely different problem space. Obviously that doesn't fly if there isn't an equivalent open solution, so we did what we could with the system to make it not terrible. We can do more. The point about Swarm is interesting, and has been much on my mind. Some of Kubernetes' perceived complexity is because we go to great lengths to avoid ever having two users collide, with escape hatches for the people who really need "unfriendly" features. This is because, again, Kubernetes models Borg. Borg clusters are giant, shared, multi-user, multi-app animals, where the users are in different business units and chances of collisions are high. Swarm, on the other hand, thinks of a cluster more as an application construct. Sharing is not a big problem, and coordination is easy and local. This allows them to make different tradeoffs. I doubt very much that you can run a large number of similar apps in a single swarm without having collisions on things like ports. I still believe the large-shared-cluster model is right in the limit. There are so many efficiencies to be had. But there are legit reasons it is hard to achieve right now. I'm very interested in ways to make Kubernetes easier to use, ESPECIALLY in this regard. Real user feedback is critical.

sandGorgon 9y ago
that's an interesting perspective - however, Docker Swarm also does that. Docker Swarm secrets have been GA for longer thank k8s. The new UCP mechanism in the Docker Datacenter product is fairly interesting (has not made it to swarm yet). Its a paid product but makes RBAC pretty simple - https://success.docker.com/Architecture/Docker_Reference_Architecture%3A_Securing_Docker_EE_and_Security_Best_Practices https://success.docker.com/Architecture/Docker_Reference_Arc... . One very interesting tool that Docker makes available is https://store.docker.com/community/images/docker/docker-bench-security https://store.docker.com/community/images/docker/docker-benc... I think the issue with k8s is that it is competing with the "Ruby on Rails" of frameworks viz Docker Swarm. I think the pluggability of critical pieces like ingress and secrets was taken too far. > I doubt very much that you can run a large number of similar apps in a single swarm without having collisions on things like ports. I dont think that is true, it does manage its overlay networks pretty well. Which is FWIW, another place where k8s took the non-opinionatedness too far. I think the number of bugs on "my stuff doesnt work with flannel but works with calico" should tell you that. To be honest, Docker Swarm has some of these issues as well - https://github.com/moby/moby/issues/25526 https://github.com/moby/moby/issues/25526. But the fixes are included in the "batteries". On kuberenetes, I have to run behind upstream projects with heterogenous configuration (nginx vs haproxy ingress. or flannel vs calico configuration) to try and fix it.

hardymansen 9y ago

I have to pitch our solution to this problem also, a fork of the nginx ingress controller, External nginx ingress controller. https://github.com/unibet/ext_nginx https://github.com/unibet/ext_nginx Basically just handling nginx.conf from information in k8. We run in production with ECMP in our routers to load balance stateless over any number of nodes. Easy to understand and very scalable.