5 ms·

- 60% of the Kubernetes ecosystem is half-baked alpha software - Maintaining 200+ clusters for 10 small applications - Cloud bills - Autoscaling never workin

- 60% of the Kubernetes ecosystem is half-baked alpha software - Maintaining 200+ clusters for 10 small applications - Cloud bills - Autoscaling never working well - Trying to untangle Terraform state without taking down Prod

k8sToGo 4y ago
We are using karpenter.sh for autoscaling. Works just fine.
- JustinGarrison 4y ago
  I work with the karpenter team. Glad to hear you like it. Would love it if you added your info to our public reference adopters.md file https://github.com/aws/karpenter/blob/main/ADOPTERS.md https://github.com/aws/karpenter/blob/main/ADOPTERS.md
garciasn 4y ago
We use GKE. 1. I don't know about any of this; we don't seem to have problems. 2. This sounds like an architecture issue, not a k8s issue. 3. Our entire GKE infrastructure costs less than $50 a month. 4. You're right here; it doesn't work 'well', but it works 'well enough' for our use cases. 5. I'm sure you're talking about some event that was far more complex than the few times we've had to drain our pool, but we did what we needed to do without downtime in production. While annoyingly esoteric, I thought it worked pretty fucking well compared to our alternatives.
- ColdHeat 4y ago
  With regards to 3, isn’t the management plane alone $72 for GKE without considering the cost for the nodes? How are your costs so low?
  - seized 4y ago
    GKE gives you one free cluster. https://cloud.google.com/kubernetes-engine/pricing https://cloud.google.com/kubernetes-engine/pricing
  - garciasn 4y ago
    As noted by another HNer, we are using one cluster and up to 10 nodes as the top end, with only one running 90% of the time and only up to 3 covering the other 9%. We set 10 to the upper limit to ensure we're not going to have borked runs in case some crazy random ML model isn't going to take everything out and force killed jobs. The vast majority of workloads running on the cluster are HIGHLY variable with most running on Monday morning/afternoon and Month or Quarter starts + 3 days. We are running many hundreds of jobs on those peak days with only a handful running on any other day. While many bring up examples where 24/7 infrastructure from a single box is more than plenty, we find that we can run micro VMs in this configuration and not have to worry about resource contention as our jobs run. Pre-GKE, we were managing the timing manually, which was fine until we started to scale, but we found this to be a far better situation. Particularly because we simply don't have to think about it. YMMV.
- JamesSwift 4y ago
  > Our entire GKE infrastructure costs less than $50 a month. Uhhh what? I mean even my personal DO based cluster runs about $40 a month. I'm skeptical a production cluster is at $50.
  - garciasn 4y ago
    Service Compute Engine Kubernetes Engine Cost. $191.78 $62.34 Discounts ($13.96) ($62.34) Promotions and others $0.00 $0.00 Subtotal $177.83 $0.00 The GCE instance cost includes some of our 24/7 VMs which are the lion's share of that line item, not the micro VMs we use for the cluster.
- throwaway894345 4y ago
  Agreed. We've had only one autoscaling issue and it was on Google's end (datacenter ran out of nodes of a particular type and thus failed to scale up). Our GKE infrastructure costs a lot more, but we do a lot of heavy compute.
  - garciasn 4y ago
    Most of our 'heavy compute' happens in BigQuery so we pay for it there, instead, but we make sure the rest of our analytics infrastructure needs are engineered to be as light as possible to keep costs low. We are an ESOP so, as employee owners, it behooves us to be as cost-conscious as possible.
- otabdeveloper4 4y ago
  > 3. Our entire GKE infrastructure costs less than $50 a month. At this scale, you don't need Kubernetes, invest in a pocket calculator instead.
sleepybrett 4y ago
Tell me you barely understand kubernetes without saying you barely understand kuberentes.
- spicyusername 4y ago
  Well, that is a rude thing to say.
  - sleepybrett 4y ago
    Call'um like i see'um
- rglover 4y ago
  > 60% of the Kubernetes ecosystem is half-baked alpha software This one is fair. Wasted a lot of time trying to find the "correct" dependencies—I remember the Nginx Ingress Controller specifically being a headache—only to find a maze of deprecations, poorly written documentation, or stuff that just flat out didn't work. That was ~18 months ago (I set up my cluster to run sites for my business and have basically left it alone) so things may have changed but at the time I remember being surprised after hearing so much hype.
  - throwaway894345 4y ago
    Pretty sure nginx ingress controller is one of the more solid and widely-used pieces. I've had a lot more trouble with cert-manager, but it seems to be in a stable state on my cluster now and anyway similar solutions in the bare-VM world are just as painful (IIRC I gave up trying to get terraform to do the handshake for AWS ACM).
    - rglover 4y ago
      What I can say definitively is that having gone from not doing any infra work to using k8s and then over the past few months trying my hand at a bare-metal setup, just spinning up a Linux box and hand installing deps via apt or snap was far more enjoyable/easy to follow. Primarily because there was very little obscurity (i.e., config files that automate away a lot of thinking or Dockerfiles/containers doing the same). It also left me feeling more confident about stability because if something isn't working, it's pretty clear what I broke/forgot. Worst "bug" I ran into was a snap server hanging when installing a dependency.
nunez 4y ago
literally heaps of resources used in Kubernetes cluster are using API versions with alpha in the name!

k8sToGo 4y ago

We are using karpenter.sh for autoscaling. Works just fine.

JustinGarrison 4y ago
I work with the karpenter team. Glad to hear you like it. Would love it if you added your info to our public reference adopters.md file https://github.com/aws/karpenter/blob/main/ADOPTERS.md https://github.com/aws/karpenter/blob/main/ADOPTERS.md

garciasn 4y ago

We use GKE. 1. I don't know about any of this; we don't seem to have problems. 2. This sounds like an architecture issue, not a k8s issue. 3. Our entire GKE infrastructure costs less than $50 a month. 4. You're right here; it doesn't work 'well', but it works 'well enough' for our use cases. 5. I'm sure you're talking about some event that was far more complex than the few times we've had to drain our pool, but we did what we needed to do without downtime in production. While annoyingly esoteric, I thought it worked pretty fucking well compared to our alternatives.

ColdHeat 4y ago
With regards to 3, isn’t the management plane alone $72 for GKE without considering the cost for the nodes? How are your costs so low?
- seized 4y ago
  GKE gives you one free cluster. https://cloud.google.com/kubernetes-engine/pricing https://cloud.google.com/kubernetes-engine/pricing
- garciasn 4y ago
  As noted by another HNer, we are using one cluster and up to 10 nodes as the top end, with only one running 90% of the time and only up to 3 covering the other 9%. We set 10 to the upper limit to ensure we're not going to have borked runs in case some crazy random ML model isn't going to take everything out and force killed jobs. The vast majority of workloads running on the cluster are HIGHLY variable with most running on Monday morning/afternoon and Month or Quarter starts + 3 days. We are running many hundreds of jobs on those peak days with only a handful running on any other day. While many bring up examples where 24/7 infrastructure from a single box is more than plenty, we find that we can run micro VMs in this configuration and not have to worry about resource contention as our jobs run. Pre-GKE, we were managing the timing manually, which was fine until we started to scale, but we found this to be a far better situation. Particularly because we simply don't have to think about it. YMMV.
JamesSwift 4y ago
> Our entire GKE infrastructure costs less than $50 a month. Uhhh what? I mean even my personal DO based cluster runs about $40 a month. I'm skeptical a production cluster is at $50.
- garciasn 4y ago
  Service Compute Engine Kubernetes Engine Cost. $191.78 $62.34 Discounts ($13.96) ($62.34) Promotions and others $0.00 $0.00 Subtotal $177.83 $0.00 The GCE instance cost includes some of our 24/7 VMs which are the lion's share of that line item, not the micro VMs we use for the cluster.
throwaway894345 4y ago
Agreed. We've had only one autoscaling issue and it was on Google's end (datacenter ran out of nodes of a particular type and thus failed to scale up). Our GKE infrastructure costs a lot more, but we do a lot of heavy compute.
- garciasn 4y ago
  Most of our 'heavy compute' happens in BigQuery so we pay for it there, instead, but we make sure the rest of our analytics infrastructure needs are engineered to be as light as possible to keep costs low. We are an ESOP so, as employee owners, it behooves us to be as cost-conscious as possible.
otabdeveloper4 4y ago
> 3. Our entire GKE infrastructure costs less than $50 a month. At this scale, you don't need Kubernetes, invest in a pocket calculator instead.

sleepybrett 4y ago

Tell me you barely understand kubernetes without saying you barely understand kuberentes.

spicyusername 4y ago
Well, that is a rude thing to say.
- sleepybrett 4y ago
  Call'um like i see'um
rglover 4y ago
> 60% of the Kubernetes ecosystem is half-baked alpha software This one is fair. Wasted a lot of time trying to find the "correct" dependencies—I remember the Nginx Ingress Controller specifically being a headache—only to find a maze of deprecations, poorly written documentation, or stuff that just flat out didn't work. That was ~18 months ago (I set up my cluster to run sites for my business and have basically left it alone) so things may have changed but at the time I remember being surprised after hearing so much hype.
- throwaway894345 4y ago
  Pretty sure nginx ingress controller is one of the more solid and widely-used pieces. I've had a lot more trouble with cert-manager, but it seems to be in a stable state on my cluster now and anyway similar solutions in the bare-VM world are just as painful (IIRC I gave up trying to get terraform to do the handshake for AWS ACM).
  - rglover 4y ago
    What I can say definitively is that having gone from not doing any infra work to using k8s and then over the past few months trying my hand at a bare-metal setup, just spinning up a Linux box and hand installing deps via apt or snap was far more enjoyable/easy to follow. Primarily because there was very little obscurity (i.e., config files that automate away a lot of thinking or Dockerfiles/containers doing the same). It also left me feeling more confident about stability because if something isn't working, it's pretty clear what I broke/forgot. Worst "bug" I ran into was a snap server hanging when installing a dependency.

nunez 4y ago

literally heaps of resources used in Kubernetes cluster are using API versions with alpha in the name!