5 ms·

We use GKE. 1. I don't know about any of this; we don't seem to have problems. 2. This sounds like an architecture issue, not a k8s issue. 3. Our entire GKE

We use GKE. 1. I don't know about any of this; we don't seem to have problems. 2. This sounds like an architecture issue, not a k8s issue. 3. Our entire GKE infrastructure costs less than $50 a month. 4. You're right here; it doesn't work 'well', but it works 'well enough' for our use cases. 5. I'm sure you're talking about some event that was far more complex than the few times we've had to drain our pool, but we did what we needed to do without downtime in production. While annoyingly esoteric, I thought it worked pretty fucking well compared to our alternatives.

ColdHeat 4y ago
With regards to 3, isn’t the management plane alone $72 for GKE without considering the cost for the nodes? How are your costs so low?
- seized 4y ago
  GKE gives you one free cluster. https://cloud.google.com/kubernetes-engine/pricing https://cloud.google.com/kubernetes-engine/pricing
- garciasn 4y ago
  As noted by another HNer, we are using one cluster and up to 10 nodes as the top end, with only one running 90% of the time and only up to 3 covering the other 9%. We set 10 to the upper limit to ensure we're not going to have borked runs in case some crazy random ML model isn't going to take everything out and force killed jobs. The vast majority of workloads running on the cluster are HIGHLY variable with most running on Monday morning/afternoon and Month or Quarter starts + 3 days. We are running many hundreds of jobs on those peak days with only a handful running on any other day. While many bring up examples where 24/7 infrastructure from a single box is more than plenty, we find that we can run micro VMs in this configuration and not have to worry about resource contention as our jobs run. Pre-GKE, we were managing the timing manually, which was fine until we started to scale, but we found this to be a far better situation. Particularly because we simply don't have to think about it. YMMV.
JamesSwift 4y ago
> Our entire GKE infrastructure costs less than $50 a month. Uhhh what? I mean even my personal DO based cluster runs about $40 a month. I'm skeptical a production cluster is at $50.
- garciasn 4y ago
  Service Compute Engine Kubernetes Engine Cost. $191.78 $62.34 Discounts ($13.96) ($62.34) Promotions and others $0.00 $0.00 Subtotal $177.83 $0.00 The GCE instance cost includes some of our 24/7 VMs which are the lion's share of that line item, not the micro VMs we use for the cluster.
throwaway894345 4y ago
Agreed. We've had only one autoscaling issue and it was on Google's end (datacenter ran out of nodes of a particular type and thus failed to scale up). Our GKE infrastructure costs a lot more, but we do a lot of heavy compute.
- garciasn 4y ago
  Most of our 'heavy compute' happens in BigQuery so we pay for it there, instead, but we make sure the rest of our analytics infrastructure needs are engineered to be as light as possible to keep costs low. We are an ESOP so, as employee owners, it behooves us to be as cost-conscious as possible.
otabdeveloper4 4y ago
> 3. Our entire GKE infrastructure costs less than $50 a month. At this scale, you don't need Kubernetes, invest in a pocket calculator instead.

ColdHeat 4y ago

With regards to 3, isn’t the management plane alone $72 for GKE without considering the cost for the nodes? How are your costs so low?

seized 4y ago
GKE gives you one free cluster. https://cloud.google.com/kubernetes-engine/pricing https://cloud.google.com/kubernetes-engine/pricing
garciasn 4y ago
As noted by another HNer, we are using one cluster and up to 10 nodes as the top end, with only one running 90% of the time and only up to 3 covering the other 9%. We set 10 to the upper limit to ensure we're not going to have borked runs in case some crazy random ML model isn't going to take everything out and force killed jobs. The vast majority of workloads running on the cluster are HIGHLY variable with most running on Monday morning/afternoon and Month or Quarter starts + 3 days. We are running many hundreds of jobs on those peak days with only a handful running on any other day. While many bring up examples where 24/7 infrastructure from a single box is more than plenty, we find that we can run micro VMs in this configuration and not have to worry about resource contention as our jobs run. Pre-GKE, we were managing the timing manually, which was fine until we started to scale, but we found this to be a far better situation. Particularly because we simply don't have to think about it. YMMV.

JamesSwift 4y ago

> Our entire GKE infrastructure costs less than $50 a month. Uhhh what? I mean even my personal DO based cluster runs about $40 a month. I'm skeptical a production cluster is at $50.

garciasn 4y ago
Service Compute Engine Kubernetes Engine Cost. $191.78 $62.34 Discounts ($13.96) ($62.34) Promotions and others $0.00 $0.00 Subtotal $177.83 $0.00 The GCE instance cost includes some of our 24/7 VMs which are the lion's share of that line item, not the micro VMs we use for the cluster.

throwaway894345 4y ago

Agreed. We've had only one autoscaling issue and it was on Google's end (datacenter ran out of nodes of a particular type and thus failed to scale up). Our GKE infrastructure costs a lot more, but we do a lot of heavy compute.

garciasn 4y ago
Most of our 'heavy compute' happens in BigQuery so we pay for it there, instead, but we make sure the rest of our analytics infrastructure needs are engineered to be as light as possible to keep costs low. We are an ESOP so, as employee owners, it behooves us to be as cost-conscious as possible.

otabdeveloper4 4y ago

> 3. Our entire GKE infrastructure costs less than $50 a month. At this scale, you don't need Kubernetes, invest in a pocket calculator instead.