4 ms·

One thing missing from the article is the "load balancer" aspect of the orchestrators. I believe K8S favours big cloud providers tremendously due to load balanc

One thing missing from the article is the "load balancer" aspect of the orchestrators. I believe K8S favours big cloud providers tremendously due to load balancer lock-in or handcuff or whatever else you want to call it. For Kubernetes there's a WIP Metallb but I personally haven't had confident success running on my own servers. K8S setup complexity is already highlighted but I'd add upgrade and maintenance pain to the mix as well. I've only recently started with Nomad and while it's so simple, I question my self at every step because coming from K8S I've gotten used to complexity and over configuration. Nomad allows you to bring your own loadbalancer, and i'm experimenting with Traefik & Nginx now.

verdverm 4y ago
k8s has good, non-cloud based ingress controllers. https://github.com/kubernetes/ingress-nginx https://github.com/kubernetes/ingress-nginx https://doc.traefik.io/traefik/providers/kubernetes-ingress/ https://doc.traefik.io/traefik/providers/kubernetes-ingress/
- bamazizi 4y ago
  To run k8s on bare metal server clusters there's only Metallb option to bring user traffic into clusters NodePort services, via externally exposed IP address(es). I wasn't talking about k8s internal load balancers and reverse proxies
  - verdverm 4y ago
    You were talking about cloud, now you are talking about bare metal? Of course you have to route traffic to your cluster, but you are implying some cloud based load balancer lockin which just isn't true
    - chucky_z 4y ago
      I believe the OP is talking about LoadBalancer (https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/ https://kubernetes.io/docs/tasks/access-application-cluster/...), and I believe you're talking about Ingress (https://kubernetes.io/docs/concepts/services-networking/ingress/ https://kubernetes.io/docs/concepts/services-networking/ingr...).
      - verdverm 4y ago
        It's still an optional, rarely used service config, and does not imply lockin
        buzer 4y ago
        It's very commonly used by the ingress controllers. You can get things working with hostNetwork or custom service port ranges, but that's a lot rarer than doing it via load balancer. Third option would be to expose pods directly in the network. EKS does this by default, but from my experience it's quite rare for people to leverage it.
        verdverm 4y ago
        Yes, an ingress resource will typically create a load balancer if you are in the cloud, but not a service resource.
  - deleted 4y ago
    [deleted]
  - hitpointdrew 4y ago
    This is not true at all, you can use a self-provisioned edge (load balancer) like haproxy. See https://kubernetes.github.io/ingress-nginx/deploy/baremetal/ https://kubernetes.github.io/ingress-nginx/deploy/baremetal/
dixie_land 4y ago
We ran a large number of services on a large number of k8s clusters handling 1M+ QPS What worked for us is to use k8s strictly for deployment/container orchestration, we have our own Envoy-based network layer on top of k8s and do not use Services or Ingress or Ingress Controllers
hitpointdrew 4y ago
>I believe K8S favours big cloud providers tremendously due to load balancer lock-in or handcuff or whatever else you want to call it. For Kubernetes there's a WIP Metallb but I personally haven't had confident success running on my own servers. > Nomad allows you to bring your own loadbalancer I think this is a widely held myth that you can't easily self-host k8s due to lack of access to a cloud load balancer. You can use your own load balancers with Kubernetes, it isn't that hard. I have production k8s clusters running on on-prem hardware right now using haproxy. I use kubeadm when setting up my cluster and point the control plane endpoint to the haproxy load balancer. Then when it comes to the ingress I use the nginx ingress and have it setup exactly as described in the "Using a self-provisioned edge" section of the documentation here: https://kubernetes.github.io/ingress-nginx/deploy/baremetal/ https://kubernetes.github.io/ingress-nginx/deploy/baremetal/

verdverm 4y ago

k8s has good, non-cloud based ingress controllers. https://github.com/kubernetes/ingress-nginx https://github.com/kubernetes/ingress-nginx https://doc.traefik.io/traefik/providers/kubernetes-ingress/ https://doc.traefik.io/traefik/providers/kubernetes-ingress/

bamazizi 4y ago
To run k8s on bare metal server clusters there's only Metallb option to bring user traffic into clusters NodePort services, via externally exposed IP address(es). I wasn't talking about k8s internal load balancers and reverse proxies
- verdverm 4y ago
  You were talking about cloud, now you are talking about bare metal? Of course you have to route traffic to your cluster, but you are implying some cloud based load balancer lockin which just isn't true
  - chucky_z 4y ago
    I believe the OP is talking about LoadBalancer (https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/ https://kubernetes.io/docs/tasks/access-application-cluster/...), and I believe you're talking about Ingress (https://kubernetes.io/docs/concepts/services-networking/ingress/ https://kubernetes.io/docs/concepts/services-networking/ingr...).
    - verdverm 4y ago
      It's still an optional, rarely used service config, and does not imply lockin
      - buzer 4y ago
        It's very commonly used by the ingress controllers. You can get things working with hostNetwork or custom service port ranges, but that's a lot rarer than doing it via load balancer. Third option would be to expose pods directly in the network. EKS does this by default, but from my experience it's quite rare for people to leverage it.
        verdverm 4y ago
        Yes, an ingress resource will typically create a load balancer if you are in the cloud, but not a service resource.
- deleted 4y ago
  [deleted]
- hitpointdrew 4y ago
  This is not true at all, you can use a self-provisioned edge (load balancer) like haproxy. See https://kubernetes.github.io/ingress-nginx/deploy/baremetal/ https://kubernetes.github.io/ingress-nginx/deploy/baremetal/

dixie_land 4y ago

We ran a large number of services on a large number of k8s clusters handling 1M+ QPS What worked for us is to use k8s strictly for deployment/container orchestration, we have our own Envoy-based network layer on top of k8s and do not use Services or Ingress or Ingress Controllers

hitpointdrew 4y ago

>I believe K8S favours big cloud providers tremendously due to load balancer lock-in or handcuff or whatever else you want to call it. For Kubernetes there's a WIP Metallb but I personally haven't had confident success running on my own servers. > Nomad allows you to bring your own loadbalancer I think this is a widely held myth that you can't easily self-host k8s due to lack of access to a cloud load balancer. You can use your own load balancers with Kubernetes, it isn't that hard. I have production k8s clusters running on on-prem hardware right now using haproxy. I use kubeadm when setting up my cluster and point the control plane endpoint to the haproxy load balancer. Then when it comes to the ingress I use the nginx ingress and have it setup exactly as described in the "Using a self-provisioned edge" section of the documentation here: https://kubernetes.github.io/ingress-nginx/deploy/baremetal/ https://kubernetes.github.io/ingress-nginx/deploy/baremetal/