5 ms·

We’re in the same situation — we’ve engineered for minimum provider-specific dependencies but GKE LoadBalancers were where they got us via arm twisting as well.

We’re in the same situation — we’ve engineered for minimum provider-specific dependencies but GKE LoadBalancers were where they got us via arm twisting as well. There is no way to expose a cluster to the outside world in a production environment otherwise.

tmpz22 7y ago
Do you also have occasional outages because the load balancer gets into a confused state and changes take 10+ minutes to propagate with no re-course other then than to destroy and re-create the entire resource?
- deleted 7y ago
  [deleted]
lvh 7y ago
It's kind of ridiculous internal load balancers can't get automatic certs. We've had to do a stupid dance just to get certs via the LE DNS challenge out of band, and then regularly install them on internal LBs.
- tmpz22 7y ago
  Same! I still manually provision some certificates just because LEGO/etc. just don't work with GCP + Google Cloud Load balancer! And the docs for the entire subject are useless..
- rolleiflex 7y ago
  We paid for long-lasting wildcard certs because of that. Which Apple killed a few days ago. It’s going to be fun when they are close to expiry.
- zomglings 7y ago
  Does cert-manager not for your needs? https://github.com/jetstack/cert-manager https://github.com/jetstack/cert-manager
  - tmpz22 7y ago
    > As this project is pre-1.0, we do not currently offer strong guarantees around our API stability. Notably, we may choose to make breaking changes to our API specification (i.e. the Issuer, ClusterIssuer and Certificate resources) in new minor releases.
    - yebyen 7y ago
      In practice, the cert-manager team has made breaking changes in probably close to 1/3 of minor releases (which is really fine pre 1.0, IMHO), there has been comprehensive guidance to lead users or cluster admins through upgrading, that walks through exactly what steps are needed, and followed well does not interrupt your cluster's service in any way. It's not dark magic, it might make building off of it in the form of integrations prohibitive, but they have done a great job making sure users can upgrade one release to the next. It is a little bit of a treadmill, but it certainly beats manually renewing certificates!
- hitpointdrew 7y ago
  Maybe I don't understand your problem, but can't you just use Traefik (https://docs.traefik.io/user-guides/crd-acme/ https://docs.traefik.io/user-guides/crd-acme/). It will get certs from letsencrypt for you.
  - lvh 7y ago
    Key word: "internal" -- these aren't on the internet, Traefik does ALPN, which means the LB itself has to be on the Internet. (Or something else that leaks the cert to the LB, but that doesn't sound any less complicated than using the DNS challenge.)
  - jrockway 7y ago
    TCP coming into your cluster means that you practically have to go through kube-proxy (because the load balancer and the Kubernetes scheduler aren't perfectly synchronized) and that the load-balancer can't balance per-request, only per-connection. If the load balancer terminates TLS, then it can just watch cluster endpoints and automatically route to the right node without any extra hop through kube-proxy, and it can also split large individual requests out of HTTP/2 and GRPC streams. I'm guessing 99% of workloads won't notice either of these issues, but it is an actual issue.
uberduper 7y ago
There are ways to expose your cluster to public and/or run your own load balancers on GKE (or any other cloud k8s deployment).
- deleted 7y ago
  [deleted]
manigandham 7y ago
Load balancers are created by the pre-installed controller that each cloud provides to let external traffic reach the nodes. You don't have to use it. It's no different than running your own load balancer like HAProxy pointed at the nodes which forward to a node-port service. There's also MetalLB if you're running your own hardware: https://metallb.universe.tf/ https://metallb.universe.tf/

tmpz22 7y ago

Do you also have occasional outages because the load balancer gets into a confused state and changes take 10+ minutes to propagate with no re-course other then than to destroy and re-create the entire resource?

deleted 7y ago
[deleted]

lvh 7y ago

It's kind of ridiculous internal load balancers can't get automatic certs. We've had to do a stupid dance just to get certs via the LE DNS challenge out of band, and then regularly install them on internal LBs.

tmpz22 7y ago
Same! I still manually provision some certificates just because LEGO/etc. just don't work with GCP + Google Cloud Load balancer! And the docs for the entire subject are useless..
rolleiflex 7y ago
We paid for long-lasting wildcard certs because of that. Which Apple killed a few days ago. It’s going to be fun when they are close to expiry.
zomglings 7y ago
Does cert-manager not for your needs? https://github.com/jetstack/cert-manager https://github.com/jetstack/cert-manager
- tmpz22 7y ago
  > As this project is pre-1.0, we do not currently offer strong guarantees around our API stability. Notably, we may choose to make breaking changes to our API specification (i.e. the Issuer, ClusterIssuer and Certificate resources) in new minor releases.
  - yebyen 7y ago
    In practice, the cert-manager team has made breaking changes in probably close to 1/3 of minor releases (which is really fine pre 1.0, IMHO), there has been comprehensive guidance to lead users or cluster admins through upgrading, that walks through exactly what steps are needed, and followed well does not interrupt your cluster's service in any way. It's not dark magic, it might make building off of it in the form of integrations prohibitive, but they have done a great job making sure users can upgrade one release to the next. It is a little bit of a treadmill, but it certainly beats manually renewing certificates!
hitpointdrew 7y ago
Maybe I don't understand your problem, but can't you just use Traefik (https://docs.traefik.io/user-guides/crd-acme/ https://docs.traefik.io/user-guides/crd-acme/). It will get certs from letsencrypt for you.
- lvh 7y ago
  Key word: "internal" -- these aren't on the internet, Traefik does ALPN, which means the LB itself has to be on the Internet. (Or something else that leaks the cert to the LB, but that doesn't sound any less complicated than using the DNS challenge.)
- jrockway 7y ago
  TCP coming into your cluster means that you practically have to go through kube-proxy (because the load balancer and the Kubernetes scheduler aren't perfectly synchronized) and that the load-balancer can't balance per-request, only per-connection. If the load balancer terminates TLS, then it can just watch cluster endpoints and automatically route to the right node without any extra hop through kube-proxy, and it can also split large individual requests out of HTTP/2 and GRPC streams. I'm guessing 99% of workloads won't notice either of these issues, but it is an actual issue.

uberduper 7y ago

There are ways to expose your cluster to public and/or run your own load balancers on GKE (or any other cloud k8s deployment).

deleted 7y ago
[deleted]

manigandham 7y ago

Load balancers are created by the pre-installed controller that each cloud provides to let external traffic reach the nodes. You don't have to use it. It's no different than running your own load balancer like HAProxy pointed at the nodes which forward to a node-port service. There's also MetalLB if you're running your own hardware: https://metallb.universe.tf/ https://metallb.universe.tf/