5 ms·

Too many people drank the cloud kool-aid. The move from day one was to create provider agnostic cloud architectures and repent the use of provider-specific serv

by tmpz22 7y ago

Too many people drank the cloud kool-aid. The move from day one was to create provider agnostic cloud architectures and repent the use of provider-specific services. That said they do make it damn hard. Our k8s cluster is as basic as it comes, no databases, simple deployments, but we do still have a dependency on Google Cloud Loadbalancer (which we hate). If pricing goes up too much from this we'll move, but the GCL dependency will be a PITA :/

rolleiflex 7y ago
We’re in the same situation — we’ve engineered for minimum provider-specific dependencies but GKE LoadBalancers were where they got us via arm twisting as well. There is no way to expose a cluster to the outside world in a production environment otherwise.
- tmpz22 7y ago
  Do you also have occasional outages because the load balancer gets into a confused state and changes take 10+ minutes to propagate with no re-course other then than to destroy and re-create the entire resource?
  - deleted 7y ago
    [deleted]
- lvh 7y ago
  It's kind of ridiculous internal load balancers can't get automatic certs. We've had to do a stupid dance just to get certs via the LE DNS challenge out of band, and then regularly install them on internal LBs.
  - tmpz22 7y ago
    Same! I still manually provision some certificates just because LEGO/etc. just don't work with GCP + Google Cloud Load balancer! And the docs for the entire subject are useless..
  - rolleiflex 7y ago
    We paid for long-lasting wildcard certs because of that. Which Apple killed a few days ago. It’s going to be fun when they are close to expiry.
  - zomglings 7y ago
    Does cert-manager not for your needs? https://github.com/jetstack/cert-manager https://github.com/jetstack/cert-manager
    - tmpz22 7y ago
      > As this project is pre-1.0, we do not currently offer strong guarantees around our API stability. Notably, we may choose to make breaking changes to our API specification (i.e. the Issuer, ClusterIssuer and Certificate resources) in new minor releases.
      - yebyen 7y ago
        In practice, the cert-manager team has made breaking changes in probably close to 1/3 of minor releases (which is really fine pre 1.0, IMHO), there has been comprehensive guidance to lead users or cluster admins through upgrading, that walks through exactly what steps are needed, and followed well does not interrupt your cluster's service in any way. It's not dark magic, it might make building off of it in the form of integrations prohibitive, but they have done a great job making sure users can upgrade one release to the next. It is a little bit of a treadmill, but it certainly beats manually renewing certificates!
  - hitpointdrew 7y ago
    Maybe I don't understand your problem, but can't you just use Traefik (https://docs.traefik.io/user-guides/crd-acme/ https://docs.traefik.io/user-guides/crd-acme/). It will get certs from letsencrypt for you.
    - lvh 7y ago
      Key word: "internal" -- these aren't on the internet, Traefik does ALPN, which means the LB itself has to be on the Internet. (Or something else that leaks the cert to the LB, but that doesn't sound any less complicated than using the DNS challenge.)
    - jrockway 7y ago
      TCP coming into your cluster means that you practically have to go through kube-proxy (because the load balancer and the Kubernetes scheduler aren't perfectly synchronized) and that the load-balancer can't balance per-request, only per-connection. If the load balancer terminates TLS, then it can just watch cluster endpoints and automatically route to the right node without any extra hop through kube-proxy, and it can also split large individual requests out of HTTP/2 and GRPC streams. I'm guessing 99% of workloads won't notice either of these issues, but it is an actual issue.
- uberduper 7y ago
  There are ways to expose your cluster to public and/or run your own load balancers on GKE (or any other cloud k8s deployment).
  - deleted 7y ago
    [deleted]
- manigandham 7y ago
  Load balancers are created by the pre-installed controller that each cloud provides to let external traffic reach the nodes. You don't have to use it. It's no different than running your own load balancer like HAProxy pointed at the nodes which forward to a node-port service. There's also MetalLB if you're running your own hardware: https://metallb.universe.tf/ https://metallb.universe.tf/
pojzon 7y ago
How about managing own k8s running on VMs / bare-metal? Pretty much anyone who works in ops longer understood from the go that its impossible to be totally provider-agnostic. K8S is just a nice api on top of provider api that still requires provider specific configuration.
- jrockway 7y ago
  The big problem with running your own cluster is the extra machines you need for a high-availability control plane, which is expensive. That is why Amazon and now Google feel like they can charge for this; you can't really do it any better yourself.
- takeda 7y ago
  From what I've seen looks like managing k8s on your own often ends up requiring a dedicated team to keep with their insane release cycle.
  - freedomben 7y ago
    Can confirm. Depending on your cluster size you will need at least 2 dedicated people on the "Kubernetes" team. You'll probably also end up rolling-your-own deployment tools because K8s API is a little overwhelming for most devs.
    - mleonhard 7y ago
      I started learning Kubernetes and was overwhelmed. The biggest problem was the missing docs. I filed a Github issue asking for missing Kubernetes YAML docs: https://github.com/kubernetes/website/issues/19139 https://github.com/kubernetes/website/issues/19139 Google will ignore it like all of the tickets I file. The fact is that Google is in the business of making money and they are focused on enterprise users. Enterprise users are not sensitive to integration difficulty since they can just throw people at any problems. So eventually everything Google makes will become extremely time-consuming to learn and difficult to use. They're becoming another Oracle.
Glyptodon 7y ago
Doing load balance, DNS, and egress has been way uglier in the Google Cloud K8s than I expected. Pushes projects towards doing it themselves in cluster IMO.

by tmpz22 7y ago

rolleiflex 7y ago

We’re in the same situation — we’ve engineered for minimum provider-specific dependencies but GKE LoadBalancers were where they got us via arm twisting as well. There is no way to expose a cluster to the outside world in a production environment otherwise.

tmpz22 7y ago
Do you also have occasional outages because the load balancer gets into a confused state and changes take 10+ minutes to propagate with no re-course other then than to destroy and re-create the entire resource?
- deleted 7y ago
  [deleted]
lvh 7y ago
It's kind of ridiculous internal load balancers can't get automatic certs. We've had to do a stupid dance just to get certs via the LE DNS challenge out of band, and then regularly install them on internal LBs.
- tmpz22 7y ago
  Same! I still manually provision some certificates just because LEGO/etc. just don't work with GCP + Google Cloud Load balancer! And the docs for the entire subject are useless..
- rolleiflex 7y ago
  We paid for long-lasting wildcard certs because of that. Which Apple killed a few days ago. It’s going to be fun when they are close to expiry.
- zomglings 7y ago
  Does cert-manager not for your needs? https://github.com/jetstack/cert-manager https://github.com/jetstack/cert-manager
  - tmpz22 7y ago
    > As this project is pre-1.0, we do not currently offer strong guarantees around our API stability. Notably, we may choose to make breaking changes to our API specification (i.e. the Issuer, ClusterIssuer and Certificate resources) in new minor releases.
    - yebyen 7y ago
      In practice, the cert-manager team has made breaking changes in probably close to 1/3 of minor releases (which is really fine pre 1.0, IMHO), there has been comprehensive guidance to lead users or cluster admins through upgrading, that walks through exactly what steps are needed, and followed well does not interrupt your cluster's service in any way. It's not dark magic, it might make building off of it in the form of integrations prohibitive, but they have done a great job making sure users can upgrade one release to the next. It is a little bit of a treadmill, but it certainly beats manually renewing certificates!
- hitpointdrew 7y ago
  Maybe I don't understand your problem, but can't you just use Traefik (https://docs.traefik.io/user-guides/crd-acme/ https://docs.traefik.io/user-guides/crd-acme/). It will get certs from letsencrypt for you.
  - lvh 7y ago
    Key word: "internal" -- these aren't on the internet, Traefik does ALPN, which means the LB itself has to be on the Internet. (Or something else that leaks the cert to the LB, but that doesn't sound any less complicated than using the DNS challenge.)
  - jrockway 7y ago
    TCP coming into your cluster means that you practically have to go through kube-proxy (because the load balancer and the Kubernetes scheduler aren't perfectly synchronized) and that the load-balancer can't balance per-request, only per-connection. If the load balancer terminates TLS, then it can just watch cluster endpoints and automatically route to the right node without any extra hop through kube-proxy, and it can also split large individual requests out of HTTP/2 and GRPC streams. I'm guessing 99% of workloads won't notice either of these issues, but it is an actual issue.
uberduper 7y ago
There are ways to expose your cluster to public and/or run your own load balancers on GKE (or any other cloud k8s deployment).
- deleted 7y ago
  [deleted]
manigandham 7y ago
Load balancers are created by the pre-installed controller that each cloud provides to let external traffic reach the nodes. You don't have to use it. It's no different than running your own load balancer like HAProxy pointed at the nodes which forward to a node-port service. There's also MetalLB if you're running your own hardware: https://metallb.universe.tf/ https://metallb.universe.tf/

pojzon 7y ago

How about managing own k8s running on VMs / bare-metal? Pretty much anyone who works in ops longer understood from the go that its impossible to be totally provider-agnostic. K8S is just a nice api on top of provider api that still requires provider specific configuration.

jrockway 7y ago
The big problem with running your own cluster is the extra machines you need for a high-availability control plane, which is expensive. That is why Amazon and now Google feel like they can charge for this; you can't really do it any better yourself.
takeda 7y ago
From what I've seen looks like managing k8s on your own often ends up requiring a dedicated team to keep with their insane release cycle.
- freedomben 7y ago
  Can confirm. Depending on your cluster size you will need at least 2 dedicated people on the "Kubernetes" team. You'll probably also end up rolling-your-own deployment tools because K8s API is a little overwhelming for most devs.
  - mleonhard 7y ago
    I started learning Kubernetes and was overwhelmed. The biggest problem was the missing docs. I filed a Github issue asking for missing Kubernetes YAML docs: https://github.com/kubernetes/website/issues/19139 https://github.com/kubernetes/website/issues/19139 Google will ignore it like all of the tickets I file. The fact is that Google is in the business of making money and they are focused on enterprise users. Enterprise users are not sensitive to integration difficulty since they can just throw people at any problems. So eventually everything Google makes will become extremely time-consuming to learn and difficult to use. They're becoming another Oracle.

Glyptodon 7y ago

Doing load balance, DNS, and egress has been way uglier in the Google Cloud K8s than I expected. Pushes projects towards doing it themselves in cluster IMO.