8 ms·

Kubernetes Journey – How to setup the HAProxy Cluster with high availability

75 points by mvallim 7y ago

q3k 7y ago
I don't understand why you would use HAProxy here. Generally on k8s, you'd use a LoadBalancer (backed by eg. metallb on bare metal) to route external traffic into your cluster. This goes for both 'normal' payloads/services and the Kubernetes API endpoint itself. In addition, it seems like the quorum configuration shown here is pretty poor - only uses two nodes. This is a recipe for split-brain.
- pepemon 7y ago
  > Generally on k8s, you'd use a LoadBalancer Are you sure? All encountered k8s installations (including ones rolled from scratch by me) use Ingress controllers as the option for getting traffic into the cluster. Community NGINX Ingress controller is the de facto standard. One need to use LoadBalancer service type because of its managed origin (metallb is a different beast and I wouldn't recommend it, if you need to load balance traffic to your ingresses on on-premises infra, it's better to do it outside of k8s). Anyway, you lose all flexibility and observability of Ingress solutions with LoadBalancer service types if you use them directly as traffic routers to your backend.
  - q3k 7y ago
    They're complementary - LoadBalancer Services are L3, Ingresses are L4+. More often than not in-cluster Ingress providers (eg. nginx-ingress-controller) will in fact use LoadBalancers to actually route L3 traffic into the cluster in the first place. Without being able to create LoadBalancer services there's no easy way to get any traffic into your cluster other than using NodePorts, and these have tons of shortcomings.
    - pepemon 7y ago
      NodePort is a bad practice for a such case, you are right. You get additional unnecessary routing on CNI level between the cluster nodes just to get traffic into the Ingress controller pod, for example. The solution here is to use hostNetwork mode with the pool of workers solely dedicated to scheduling Ingress controllers. Traffic directly hits NGINX/Envoy/HAProxy/Traefik/whatever and gets into the cluster without additional intermediates. You need a load balancing solution for this pool of Ingress controllers, that's right, but as I said before, this setup gives you the flexibility to cook load balancing as you desire. BTW, community NGINX Ingress controller is able to ingress L3 (TCP) traffic into the cluster.
      - q3k 7y ago
        > You need a load balancing solution for this pool of Ingress controllers, that's right, but as I said before, this setup gives you the flexibility to cook load balancing as you desire. Sure, but this means that you cannot use LoadBalancers, which is painful. It means every payload has to be configured both at k8s level and then externally. That somewhat defeats the use of k8s as a self-service platform internally in an organization (other dev/ops teams need to go through a centralized ops channel to get traffic ingressing into the cluster if for some reason they can't use an Ingress). > BTW, community NGINX Ingress controller is able to ingress L3 (TCP) traffic into the cluster. Yes, but it's configurable via a single ConfigMap (which limits self-service if you're running a multi-tenant orga-wide cluster, unless you bring your own automation), and still you only have one namespace of ports, ie. a single external address for all ports - unless you complicate your 'external' LB system even further. With all these caveats, I really don't understand why not just run metallb.
  - nicolast 7y ago
    Not all services exposed to the outside are HTTP(S).
    - pepemon 7y ago
      Yes, Ingress resources implicitly assume that you want to get HTTP(S) traffic into the cluster, but as for example, ingress-nginx is able to expose gRPC via additional annotations and generic L3 via related ConfigMap.
  - DelightOne 7y ago
    You lead me down a rabbit hole, thank you! Do you know whether ingress controllers like nginx-ingress-controller honor the readiness status of pods from a service and so only sends traffic to ready pods from the concerned deployment?
    - llarsson 7y ago
      They send traffic to the Service that exposes your Pods, so yes, it only gets routed to Endpoints that work according to readiness tests.
      - DelightOne 7y ago
        hmm makes sense, thank you!
      - rcconf 7y ago
        I remember seeing documentation on this a while ago, but I can't seem to find any documentation from nginx-ingress that talks about how nginx-ingress uses readinessprobe or livenessprobe. I was looking into this because if I have a pod with 3 containers, if 1 container wasn't running, nginx-ingress stopped serving traffic. I actually want to continue serving traffic if a certain container is still running, not all 3 running necessarily Do you know or have any documentation on how nginx-ingress actually does its readiness/liveness checks?
        xnxn 7y ago
        ingress-nginx doesn't do readiness/liveness checks itself; the kubelet does[1]. If any of the containers in a Pod aren't ready, the endpoints controller[2] removes the Endpoint object corresponding to that Pod. ingress-nginx watches these Endpoints objects[3] to determine which Pods it should send traffic to. Edit: As to your use case, I think you should remove the readiness probe from that one container you don't care about. [1]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ https://kubernetes.io/docs/tasks/configure-pod-container/con... [2]: https://kubernetes.io/docs/concepts/overview/components/#kube-controller-manager https://kubernetes.io/docs/concepts/overview/components/#kub... [3]: https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#why-endpoints-and-not-services https://kubernetes.github.io/ingress-nginx/user-guide/miscel...
- anonsivalley652 7y ago
  Envoy or similar?
- deleted 7y ago
  [deleted]
- Spivak 7y ago
  > on k8s, you'd use a LoadBalancer Which will be internally backed by exactly the kind of infra the OP is describing. * MetalLB doesn't work in most cloud environments because the networking is weird. * MetalLB is extremely new while VIP+Nginx/HAP has been working forever and is the standard choice for implementing a HA LB. * MetalLB plays the same exact role as what the OP is laying just in a different style that has nothing to do with k8s. MetalLB could have been unicast VRRP for all it matters. The value is the code to instrument the LB from within k8s which can work with any external LB. In k8s load balancers are inherently external to the cluster and need to maintain their state. It doesn't really matter how this is accomplished but Pacemaker/Corosync is the "off the shelf and supported by your distro with good docs" option. "Why don't you just use $managed_k8s?" -- Because it defeats the purpose of learning how to administer it. Sibebar: Why is everybody treating k8s like it's something special and magical here and needs it's own solution? Your k8s cluster is a group of app servers with a little bit of networking sprinkle so that you can reach every app from any node and some API sprinkle to allow the app servers to instrument external resources. The k8s cluster needs something outside itself to route traffic to all the nodes. There isn't a way around this. From an infra perspective a k8s cluster is a black box of generic app servers that want to be able to control the LB with an API integration. Nothing else.
  - q3k 7y ago
    > MetalLB doesn't work in most cloud environments because the networking is weird. Sure, but most cloud environments are handled by other LoadBalancer controllers specific to that environment. See: Kubernetes Cloud Controller Manager. > MetalLB is extremely new while VIP+Nginx/HAP has been working forever and is the standard choice for implementing a HA LB. I do not disagree with that. Hell, I've filed/fixed metallb issues myself already. > MetalLB plays the same exact role as what the OP is laying just in a different style that has nothing to do with k8s. MetalLB could have been unicast VRRP for all it matters. The value is the code to instrument the LB from within k8s which can work with any external LB. Sure, but the author is not describing it to act as a Kube LoadBalancer, but as a separate entity with its own configuration. So to actually then use Kubernetes meaningfully they still need to bring something that actually fulfills LoadBalancer requests. Might as well use the same code for both cases. > In k8s load balancers are inherently external to the cluster and need to maintain their state. It doesn't really matter how this is accomplished but Pacemaker/Corosync is the "off the shelf and supported by your distro with good docs" option. Why? You can easily keep all state and components within k8s, as MetalLB does. > The k8s cluster needs something outside itself to route traffic to all the nodes. There isn't a way around this. Sure, that's called BGP to a ToR that can then ECMP-route traffic where needed, and that's what metallb gives you.
- rcconf 7y ago
  This isn't always true. On GKE for example, LoadBalancer is very expensive first of all. Second of all, if you use LetsEncrypt such as cert-manager, the GKE load balancer is awful to configure and takes forever to get a cert. Also by using the GKE load balancer for ingress, you lose out on a lot of things, like password protection, or certain rules you want with nginx. I've tried my best to stick with GKE LoadBalancer, but it's just an awful experience. Now I have GKE to load balance traffic to nginx-ingress, the level 3 load balancer is just not flexible enough and in general annoying to configure.
  - q3k 7y ago
    I think you're conflating the GKE Ingress Controller (which you don't have to use, and yes, confusingly it's named the GCLB controller, but that's because it configures L7 GCP LB's, not because it's there for k8s LBs) with the GKE load balancer controller/provider for LoadBalancer services (which you do have to use, even if you run nginx-ingress-controller). And yes, I agree that both are extremely slow to reconfigure and the that Ingress controller is inflexible. I'm not saying that you need to always use a LoadBalancer for all payloads. But you do need one to ingress traffic in a sensible manner if you're running your own N-I-C (which you have to do on bare metal, and which, as you said, you end up using even on GKE).
- deleted 7y ago
  [deleted]
- mvallim 7y ago
  Keep an open mind, you can add as many HAProxy as you like, here for practical reasons it is shown with only 2 but it could be 3, 5, 7 ... In time, this LB is for handling the availability and balance of the Control Plane, without direct exposure from the Master Nodes.
therealmarv 7y ago
everything old is new again? If there is not any good easy method in Kubernetes as q3k describes than why should somebody go into this overengineering approach?
- hijinks 7y ago
  as someone that works in operations.. This is pretty common think. Lets still do things how we did it on bare metal inside kubernetes. They are stuck in the same mindset and just think Kubernetes is a automation tool. I see it all day long as I interview people.
  - notyourday 7y ago
    > as someone that works in operations.. This is pretty common think. Lets still do things how we did it on bare metal inside kubernetes. Because haproxy and nginx are proven technologies with limited and very well known failure modes, which means there's exactly 5-6 well documented, well understood and very well known ways haproxy and nginx can fail. Experienced ops people understand that one does not optimize for the blue skies -- during "everything works wonderfully" all non completely broken technologies perform at approximately the same level. Rather these people optimize for the quick recovery from the "it is not working" state.
    - tyingq 7y ago
      The article, though, suggests also adding corosync and pacemaker. So 4 things on top of the already complex K8S. I bet someone later throws in a service mesh. Imagine troubleshooting all that.
      - wtarreau 7y ago
        That's exatcly in order to blow in this fog that you want to install a layer7 load balancer designed with observability in mind. Seeing what's happening is critical when everything changes by itself under your feed. With a component like haproxy you get accurate and detailed information about what's happening and the cause of occasional failures allowing you to address them before they become the new standard.
redwood 7y ago
Anyone wonder if k8s is about to enter its trough of disillusionment phase?
- Nux 7y ago
  Indeed. I'm still holding off, in the hope it will go away (like Openstack), but it does seem like it has some more stamina as a project.
  - arvinsim 7y ago
    Sticking to Docker for local projects for now
    - marshmellowtest 7y ago
      No thanks. It's a huge security dumpster fire as well.
      - vpEfljFL 7y ago
        Could you elaborate please and what approach do you recommend?
        zxcvou 7y ago
        containers in general are horrible wrt security because they are architecturally flawed - they pretend to have some sort of 'isolation' but that was crap docker marketing people just made up - there is no isolation - k8s pushes this agenda further by declaring that multi-tenant workloads are perfectly normal and ok for containers which they absolutely are not just look at the CVEs from recent years: * docker doomsday * escaping like a rkt * cryptojacking? - that didn't even exist until containers were here!
      - pepemon 7y ago
        Well if you're uncomfortable with its priviged daemon, you can always switch to CRI-O with Red Hat tooling for it. But for all my years with Docker as the container runtime, all security related problems have occured within the backend code, not Docker, not Linux cgroups, not Linux itself.
zxcvou 7y ago
k8s is a huge hunk of shit and everyone who uses it shouldn't be coding - they will all be reaped in the next culling containers and k8s are security dumpster fires as another poster stated, they are entirely too complex for the task at hand, and their performance is absolutely horrible on a different note - I think one of the problems with the virus that is k8s is the resurgence of developer relations && develeoper evangelists in the recent years - those with 10s to 100ks of twitter followers, yet, they literally don't write code - we as an industry need to fix this - if you can't code you shouldn't be posing as one on twitter
blyry 7y ago
When we migrated to k8s we stuck with haproxy instead of using ingress for some of the reasons others have outlined already -- we've been running haproxy for a decade. Our configurations are tuned for our applications, we know the cpu usage and failure modes, and haproxy 1.9/2x support for srv records made it really easy. Bring able to trust k8s and our health checks and drop our previous 3 vm + vrrp setup was a no brainer!
- blyry 7y ago
  That got us on k8s faster and gave us time to evaluate if we /really/ needed a service mesh or not (we don't, but the tracing is nice so we might still add istio yet). We may move to ingress based solutions eventually but our ecosystem is big and there's a lot more bigger fish to fry for now!
vinay_ys 7y ago
If you have a L3 IP network within your datacenter, doing a BGP anycast of the VIP to a bunch of HAP servers is the best way to go. With ECMP, you can have your cake and eat it too – all of your HAP nodes will be handling traffic and when one of the nodes die, other nodes pick up the load nicely.
deleted 7y ago
[deleted]
mvallim 7y ago
Please, before making your criticisms, understand the purpose of what was written. The official documentation itself addresses this. https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/#create-load-balancer-for-kube-apiserver https://kubernetes.io/docs/setup/production-environment/tool...

75 points by mvallim 7y ago

q3k 7y ago

I don't understand why you would use HAProxy here. Generally on k8s, you'd use a LoadBalancer (backed by eg. metallb on bare metal) to route external traffic into your cluster. This goes for both 'normal' payloads/services and the Kubernetes API endpoint itself. In addition, it seems like the quorum configuration shown here is pretty poor - only uses two nodes. This is a recipe for split-brain.

pepemon 7y ago
> Generally on k8s, you'd use a LoadBalancer Are you sure? All encountered k8s installations (including ones rolled from scratch by me) use Ingress controllers as the option for getting traffic into the cluster. Community NGINX Ingress controller is the de facto standard. One need to use LoadBalancer service type because of its managed origin (metallb is a different beast and I wouldn't recommend it, if you need to load balance traffic to your ingresses on on-premises infra, it's better to do it outside of k8s). Anyway, you lose all flexibility and observability of Ingress solutions with LoadBalancer service types if you use them directly as traffic routers to your backend.
- q3k 7y ago
  They're complementary - LoadBalancer Services are L3, Ingresses are L4+. More often than not in-cluster Ingress providers (eg. nginx-ingress-controller) will in fact use LoadBalancers to actually route L3 traffic into the cluster in the first place. Without being able to create LoadBalancer services there's no easy way to get any traffic into your cluster other than using NodePorts, and these have tons of shortcomings.
  - pepemon 7y ago
    NodePort is a bad practice for a such case, you are right. You get additional unnecessary routing on CNI level between the cluster nodes just to get traffic into the Ingress controller pod, for example. The solution here is to use hostNetwork mode with the pool of workers solely dedicated to scheduling Ingress controllers. Traffic directly hits NGINX/Envoy/HAProxy/Traefik/whatever and gets into the cluster without additional intermediates. You need a load balancing solution for this pool of Ingress controllers, that's right, but as I said before, this setup gives you the flexibility to cook load balancing as you desire. BTW, community NGINX Ingress controller is able to ingress L3 (TCP) traffic into the cluster.
    - q3k 7y ago
      > You need a load balancing solution for this pool of Ingress controllers, that's right, but as I said before, this setup gives you the flexibility to cook load balancing as you desire. Sure, but this means that you cannot use LoadBalancers, which is painful. It means every payload has to be configured both at k8s level and then externally. That somewhat defeats the use of k8s as a self-service platform internally in an organization (other dev/ops teams need to go through a centralized ops channel to get traffic ingressing into the cluster if for some reason they can't use an Ingress). > BTW, community NGINX Ingress controller is able to ingress L3 (TCP) traffic into the cluster. Yes, but it's configurable via a single ConfigMap (which limits self-service if you're running a multi-tenant orga-wide cluster, unless you bring your own automation), and still you only have one namespace of ports, ie. a single external address for all ports - unless you complicate your 'external' LB system even further. With all these caveats, I really don't understand why not just run metallb.
- nicolast 7y ago
  Not all services exposed to the outside are HTTP(S).
  - pepemon 7y ago
    Yes, Ingress resources implicitly assume that you want to get HTTP(S) traffic into the cluster, but as for example, ingress-nginx is able to expose gRPC via additional annotations and generic L3 via related ConfigMap.
- DelightOne 7y ago
  You lead me down a rabbit hole, thank you! Do you know whether ingress controllers like nginx-ingress-controller honor the readiness status of pods from a service and so only sends traffic to ready pods from the concerned deployment?
  - llarsson 7y ago
    They send traffic to the Service that exposes your Pods, so yes, it only gets routed to Endpoints that work according to readiness tests.
    - DelightOne 7y ago
      hmm makes sense, thank you!
    - rcconf 7y ago
      I remember seeing documentation on this a while ago, but I can't seem to find any documentation from nginx-ingress that talks about how nginx-ingress uses readinessprobe or livenessprobe. I was looking into this because if I have a pod with 3 containers, if 1 container wasn't running, nginx-ingress stopped serving traffic. I actually want to continue serving traffic if a certain container is still running, not all 3 running necessarily Do you know or have any documentation on how nginx-ingress actually does its readiness/liveness checks?
      - xnxn 7y ago
        ingress-nginx doesn't do readiness/liveness checks itself; the kubelet does[1]. If any of the containers in a Pod aren't ready, the endpoints controller[2] removes the Endpoint object corresponding to that Pod. ingress-nginx watches these Endpoints objects[3] to determine which Pods it should send traffic to. Edit: As to your use case, I think you should remove the readiness probe from that one container you don't care about. [1]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ https://kubernetes.io/docs/tasks/configure-pod-container/con... [2]: https://kubernetes.io/docs/concepts/overview/components/#kube-controller-manager https://kubernetes.io/docs/concepts/overview/components/#kub... [3]: https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#why-endpoints-and-not-services https://kubernetes.github.io/ingress-nginx/user-guide/miscel...
anonsivalley652 7y ago
Envoy or similar?
deleted 7y ago
[deleted]
Spivak 7y ago
> on k8s, you'd use a LoadBalancer Which will be internally backed by exactly the kind of infra the OP is describing. * MetalLB doesn't work in most cloud environments because the networking is weird. * MetalLB is extremely new while VIP+Nginx/HAP has been working forever and is the standard choice for implementing a HA LB. * MetalLB plays the same exact role as what the OP is laying just in a different style that has nothing to do with k8s. MetalLB could have been unicast VRRP for all it matters. The value is the code to instrument the LB from within k8s which can work with any external LB. In k8s load balancers are inherently external to the cluster and need to maintain their state. It doesn't really matter how this is accomplished but Pacemaker/Corosync is the "off the shelf and supported by your distro with good docs" option. "Why don't you just use $managed_k8s?" -- Because it defeats the purpose of learning how to administer it. Sibebar: Why is everybody treating k8s like it's something special and magical here and needs it's own solution? Your k8s cluster is a group of app servers with a little bit of networking sprinkle so that you can reach every app from any node and some API sprinkle to allow the app servers to instrument external resources. The k8s cluster needs something outside itself to route traffic to all the nodes. There isn't a way around this. From an infra perspective a k8s cluster is a black box of generic app servers that want to be able to control the LB with an API integration. Nothing else.
- q3k 7y ago
  > MetalLB doesn't work in most cloud environments because the networking is weird. Sure, but most cloud environments are handled by other LoadBalancer controllers specific to that environment. See: Kubernetes Cloud Controller Manager. > MetalLB is extremely new while VIP+Nginx/HAP has been working forever and is the standard choice for implementing a HA LB. I do not disagree with that. Hell, I've filed/fixed metallb issues myself already. > MetalLB plays the same exact role as what the OP is laying just in a different style that has nothing to do with k8s. MetalLB could have been unicast VRRP for all it matters. The value is the code to instrument the LB from within k8s which can work with any external LB. Sure, but the author is not describing it to act as a Kube LoadBalancer, but as a separate entity with its own configuration. So to actually then use Kubernetes meaningfully they still need to bring something that actually fulfills LoadBalancer requests. Might as well use the same code for both cases. > In k8s load balancers are inherently external to the cluster and need to maintain their state. It doesn't really matter how this is accomplished but Pacemaker/Corosync is the "off the shelf and supported by your distro with good docs" option. Why? You can easily keep all state and components within k8s, as MetalLB does. > The k8s cluster needs something outside itself to route traffic to all the nodes. There isn't a way around this. Sure, that's called BGP to a ToR that can then ECMP-route traffic where needed, and that's what metallb gives you.
rcconf 7y ago
This isn't always true. On GKE for example, LoadBalancer is very expensive first of all. Second of all, if you use LetsEncrypt such as cert-manager, the GKE load balancer is awful to configure and takes forever to get a cert. Also by using the GKE load balancer for ingress, you lose out on a lot of things, like password protection, or certain rules you want with nginx. I've tried my best to stick with GKE LoadBalancer, but it's just an awful experience. Now I have GKE to load balance traffic to nginx-ingress, the level 3 load balancer is just not flexible enough and in general annoying to configure.
- q3k 7y ago
  I think you're conflating the GKE Ingress Controller (which you don't have to use, and yes, confusingly it's named the GCLB controller, but that's because it configures L7 GCP LB's, not because it's there for k8s LBs) with the GKE load balancer controller/provider for LoadBalancer services (which you do have to use, even if you run nginx-ingress-controller). And yes, I agree that both are extremely slow to reconfigure and the that Ingress controller is inflexible. I'm not saying that you need to always use a LoadBalancer for all payloads. But you do need one to ingress traffic in a sensible manner if you're running your own N-I-C (which you have to do on bare metal, and which, as you said, you end up using even on GKE).
deleted 7y ago
[deleted]
mvallim 7y ago
Keep an open mind, you can add as many HAProxy as you like, here for practical reasons it is shown with only 2 but it could be 3, 5, 7 ... In time, this LB is for handling the availability and balance of the Control Plane, without direct exposure from the Master Nodes.

therealmarv 7y ago

everything old is new again? If there is not any good easy method in Kubernetes as q3k describes than why should somebody go into this overengineering approach?

hijinks 7y ago
as someone that works in operations.. This is pretty common think. Lets still do things how we did it on bare metal inside kubernetes. They are stuck in the same mindset and just think Kubernetes is a automation tool. I see it all day long as I interview people.
- notyourday 7y ago
  > as someone that works in operations.. This is pretty common think. Lets still do things how we did it on bare metal inside kubernetes. Because haproxy and nginx are proven technologies with limited and very well known failure modes, which means there's exactly 5-6 well documented, well understood and very well known ways haproxy and nginx can fail. Experienced ops people understand that one does not optimize for the blue skies -- during "everything works wonderfully" all non completely broken technologies perform at approximately the same level. Rather these people optimize for the quick recovery from the "it is not working" state.
  - tyingq 7y ago
    The article, though, suggests also adding corosync and pacemaker. So 4 things on top of the already complex K8S. I bet someone later throws in a service mesh. Imagine troubleshooting all that.
    - wtarreau 7y ago
      That's exatcly in order to blow in this fog that you want to install a layer7 load balancer designed with observability in mind. Seeing what's happening is critical when everything changes by itself under your feed. With a component like haproxy you get accurate and detailed information about what's happening and the cause of occasional failures allowing you to address them before they become the new standard.

redwood 7y ago

Anyone wonder if k8s is about to enter its trough of disillusionment phase?

Nux 7y ago
Indeed. I'm still holding off, in the hope it will go away (like Openstack), but it does seem like it has some more stamina as a project.
- arvinsim 7y ago
  Sticking to Docker for local projects for now
  - marshmellowtest 7y ago
    No thanks. It's a huge security dumpster fire as well.
    - vpEfljFL 7y ago
      Could you elaborate please and what approach do you recommend?
      - zxcvou 7y ago
        containers in general are horrible wrt security because they are architecturally flawed - they pretend to have some sort of 'isolation' but that was crap docker marketing people just made up - there is no isolation - k8s pushes this agenda further by declaring that multi-tenant workloads are perfectly normal and ok for containers which they absolutely are not just look at the CVEs from recent years: * docker doomsday * escaping like a rkt * cryptojacking? - that didn't even exist until containers were here!
    - pepemon 7y ago
      Well if you're uncomfortable with its priviged daemon, you can always switch to CRI-O with Red Hat tooling for it. But for all my years with Docker as the container runtime, all security related problems have occured within the backend code, not Docker, not Linux cgroups, not Linux itself.

zxcvou 7y ago

k8s is a huge hunk of shit and everyone who uses it shouldn't be coding - they will all be reaped in the next culling containers and k8s are security dumpster fires as another poster stated, they are entirely too complex for the task at hand, and their performance is absolutely horrible on a different note - I think one of the problems with the virus that is k8s is the resurgence of developer relations && develeoper evangelists in the recent years - those with 10s to 100ks of twitter followers, yet, they literally don't write code - we as an industry need to fix this - if you can't code you shouldn't be posing as one on twitter

blyry 7y ago

When we migrated to k8s we stuck with haproxy instead of using ingress for some of the reasons others have outlined already -- we've been running haproxy for a decade. Our configurations are tuned for our applications, we know the cpu usage and failure modes, and haproxy 1.9/2x support for srv records made it really easy. Bring able to trust k8s and our health checks and drop our previous 3 vm + vrrp setup was a no brainer!

blyry 7y ago
That got us on k8s faster and gave us time to evaluate if we /really/ needed a service mesh or not (we don't, but the tracing is nice so we might still add istio yet). We may move to ingress based solutions eventually but our ecosystem is big and there's a lot more bigger fish to fry for now!

vinay_ys 7y ago

If you have a L3 IP network within your datacenter, doing a BGP anycast of the VIP to a bunch of HAP servers is the best way to go. With ECMP, you can have your cake and eat it too – all of your HAP nodes will be handling traffic and when one of the nodes die, other nodes pick up the load nicely.

deleted 7y ago

[deleted]

mvallim 7y ago

Please, before making your criticisms, understand the purpose of what was written. The official documentation itself addresses this. https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/#create-load-balancer-for-kube-apiserver https://kubernetes.io/docs/setup/production-environment/tool...