5 ms·

I think kubernetes is still lacking a load balancer that can run on bare metal. Do you want to get a couple of VPS severs to run a cluster? Forget about it or i

I think kubernetes is still lacking a load balancer that can run on bare metal. Do you want to get a couple of VPS severs to run a cluster? Forget about it or install Nginx and manually do reverse proxy to appropriate web app. I was hoping that MetalLB would solve it but these tools are written for service providers with their own routing etc. There is nothing for a small guy just wanting to deploy his blog and don't want to pay Amazon or Google for setting up a cluster.

q3k 6y ago
nginx-ingress-controller running on a NodePort service should work well enough here - just set DNS records to round-robin onto your machines. This will work on any network topology, and already works on k8s. If you want any better load balancing and/or failover, you'll have to either have something provided by your cloud provider (an LBaaS, and then tie that into Kubernetes via some controller), or have a network topology that can be exploited for this (shared L2 with a VIP or L3 with BGP, so that you can use something like MetalLB). This is not a Kubernetes problem, but a networking problem.
- intricatedetail 6y ago
  This requires fair bit of fiddling and never got it working properly. Kind of defeats the purpose of automated infrastructure.
manigandham 6y ago
Load balancers have never been part of Kubernetes. Service and Ingress are just routing rules. Service with ClusterIP creates an internal IP that redirects traffic to that service's pods. Service with NodePort creates a CluserIP and creates a port on every node that lets external traffic also reach those pods. Service with LoadBalancer creates a ClusterIP, NodePort, and a whatever load balancer implementation exists in that cloud to route external traffic to those K8S nodes and nodeport port numbers. This is the part that's missing in your bare metal setup, but you can create a nodeport service using port 80/443 and have DNS pointing to your nodes' public addresses. That's all it takes to get cluster access and you can point this Service to another downstream Service/Ingress if you want the routing rules to stay in YAML too.
- intricatedetail 6y ago
  That's why it is kind of pointless on self hosted environment, as you can't create an IP and a hostname that can be accessed from outside world. You have to manually do a reverse proxy to whatever ip has been added in the private network. If you use K8s on AWS or GCP then your services automatically are accessible from outside world. There were attempts to make Nginx or similar LB working with virtual hosts but never seen that working in practice. It seems like developers keep it enterprisey so that people keep using and be locked in to the cloud.
  - manigandham 6y ago
    What does self-hosted mean? Do any of your nodes have a public IP address? If so then I explained exactly how they can be accessed. If all of your nodes are private and behind some kind of firewall or virtual network then yes you'll have to make a public bridge, otherwise just point the DNS to your nodes. You can use a Deployment with the standard ports 80/443 to avoid any port translation too. That's all the clouds do anyway, they just run a bunch of load balancers that move external traffic to the nodes of your K8S cluster with the port specified in the Ingress/Service and K8S takes it from there. There's no shortcut, the traffic has to be routed somehow so if you don't use a managed service then you have to do it yourself. It's got nothing to do with "developers keep it enterprisey". Here's an answer I wrote about skipping GKE's cloud LB to accomplish the same thing: https://stackoverflow.com/a/54297777/173322 https://stackoverflow.com/a/54297777/173322 Also try using Ambassador as a reverse proxy/ingress/LB. It uses Envoy and is much faster and more configurable. You can set it to use the host network on your nodes and skip the K8S cluster mapping: https://www.getambassador.io/ https://www.getambassador.io/
    - intricatedetail 6y ago
      None of these examples work for self hosted scenario where you have one public IP (or a few statically assigned to your server). If you want to host a service the traditional way, you create a virtual host and a reverse proxy to your container (that is on a private network) or a service bound to a local IP. Currently I couldn't find a way to automate that. Let's say that could be done by a LoadBalancer, that is in fact controlling virtual hosts and reverse proxy entries for the pool of available IP addresses. For hostnames, there would be a pool of domains configured to resolve any host to that one IP.
rektide 6y ago
Also the team randomly decided NodePorts can only[1]: 1. be run in a specific range 2. you can expand that range, but if you do, any port in that range might be randomly assigned, so your service might start up on port 80. Changing alone would make using Kubernetes 10x easier for small operators. But one bomb-throwing curmudgeon drops by with entirely content-free unsubstantiated anger-posting: > Unless I'm misunderstanding the proposal, it involves unpredictable and difficult to diagnose failure of services, which seems like a complete non-starter to me. And the small-operators who just want some ability to open http or dns ports get nothing. Maddening beyond belief. [1] https://github.com/kubernetes/kubernetes/issues/9995 https://github.com/kubernetes/kubernetes/issues/9995
- intricatedetail 6y ago
  I would like something that would reconfigure Nginx reverse proxy to whatever private ip and port service is given, so that it would work like for big cloud providers without the need of BGP access level to a network.
  - manigandham 6y ago
    Have you seen this: https://www.getambassador.io/docs/latest/topics/install/bare-metal/ https://www.getambassador.io/docs/latest/topics/install/bare...
    - intricatedetail 6y ago
      This won't work as it doesn't support virtual hosts. For example you cannot provision two services using same port and an IP address. If you have a bare metal server with one IP address you would still have to do e.g. reverse proxy manually which defeats the purpose of having k8s.
      - deleted 6y ago
        [deleted]
- manigandham 6y ago

q3k 6y ago

nginx-ingress-controller running on a NodePort service should work well enough here - just set DNS records to round-robin onto your machines. This will work on any network topology, and already works on k8s. If you want any better load balancing and/or failover, you'll have to either have something provided by your cloud provider (an LBaaS, and then tie that into Kubernetes via some controller), or have a network topology that can be exploited for this (shared L2 with a VIP or L3 with BGP, so that you can use something like MetalLB). This is not a Kubernetes problem, but a networking problem.

intricatedetail 6y ago
This requires fair bit of fiddling and never got it working properly. Kind of defeats the purpose of automated infrastructure.

manigandham 6y ago

Load balancers have never been part of Kubernetes. Service and Ingress are just routing rules. Service with ClusterIP creates an internal IP that redirects traffic to that service's pods. Service with NodePort creates a CluserIP and creates a port on every node that lets external traffic also reach those pods. Service with LoadBalancer creates a ClusterIP, NodePort, and a whatever load balancer implementation exists in that cloud to route external traffic to those K8S nodes and nodeport port numbers. This is the part that's missing in your bare metal setup, but you can create a nodeport service using port 80/443 and have DNS pointing to your nodes' public addresses. That's all it takes to get cluster access and you can point this Service to another downstream Service/Ingress if you want the routing rules to stay in YAML too.

intricatedetail 6y ago
That's why it is kind of pointless on self hosted environment, as you can't create an IP and a hostname that can be accessed from outside world. You have to manually do a reverse proxy to whatever ip has been added in the private network. If you use K8s on AWS or GCP then your services automatically are accessible from outside world. There were attempts to make Nginx or similar LB working with virtual hosts but never seen that working in practice. It seems like developers keep it enterprisey so that people keep using and be locked in to the cloud.
- manigandham 6y ago
  What does self-hosted mean? Do any of your nodes have a public IP address? If so then I explained exactly how they can be accessed. If all of your nodes are private and behind some kind of firewall or virtual network then yes you'll have to make a public bridge, otherwise just point the DNS to your nodes. You can use a Deployment with the standard ports 80/443 to avoid any port translation too. That's all the clouds do anyway, they just run a bunch of load balancers that move external traffic to the nodes of your K8S cluster with the port specified in the Ingress/Service and K8S takes it from there. There's no shortcut, the traffic has to be routed somehow so if you don't use a managed service then you have to do it yourself. It's got nothing to do with "developers keep it enterprisey". Here's an answer I wrote about skipping GKE's cloud LB to accomplish the same thing: https://stackoverflow.com/a/54297777/173322 https://stackoverflow.com/a/54297777/173322 Also try using Ambassador as a reverse proxy/ingress/LB. It uses Envoy and is much faster and more configurable. You can set it to use the host network on your nodes and skip the K8S cluster mapping: https://www.getambassador.io/ https://www.getambassador.io/
  - intricatedetail 6y ago
    None of these examples work for self hosted scenario where you have one public IP (or a few statically assigned to your server). If you want to host a service the traditional way, you create a virtual host and a reverse proxy to your container (that is on a private network) or a service bound to a local IP. Currently I couldn't find a way to automate that. Let's say that could be done by a LoadBalancer, that is in fact controlling virtual hosts and reverse proxy entries for the pool of available IP addresses. For hostnames, there would be a pool of domains configured to resolve any host to that one IP.

rektide 6y ago

Also the team randomly decided NodePorts can only[1]: 1. be run in a specific range 2. you can expand that range, but if you do, any port in that range might be randomly assigned, so your service might start up on port 80. Changing alone would make using Kubernetes 10x easier for small operators. But one bomb-throwing curmudgeon drops by with entirely content-free unsubstantiated anger-posting: > Unless I'm misunderstanding the proposal, it involves unpredictable and difficult to diagnose failure of services, which seems like a complete non-starter to me. And the small-operators who just want some ability to open http or dns ports get nothing. Maddening beyond belief. [1] https://github.com/kubernetes/kubernetes/issues/9995 https://github.com/kubernetes/kubernetes/issues/9995

intricatedetail 6y ago
I would like something that would reconfigure Nginx reverse proxy to whatever private ip and port service is given, so that it would work like for big cloud providers without the need of BGP access level to a network.
- manigandham 6y ago
  Have you seen this: https://www.getambassador.io/docs/latest/topics/install/bare-metal/ https://www.getambassador.io/docs/latest/topics/install/bare...
  - intricatedetail 6y ago
    This won't work as it doesn't support virtual hosts. For example you cannot provision two services using same port and an IP address. If you have a bare metal server with one IP address you would still have to do e.g. reverse proxy manually which defeats the purpose of having k8s.
    - deleted 6y ago
      [deleted]
manigandham 6y ago