4 ms·

Most often I’m using ‘cordon’ and then ‘drain’ to remove nodes from a cluster that has auto-scaled up, but is being hesitant to scale down. Pretty nuts to thin

Most often I’m using ‘cordon’ and then ‘drain’ to remove nodes from a cluster that has auto-scaled up, but is being hesitant to scale down. Pretty nuts to think I’ve saved six figures in VM costs just doing that once or so a month.

xcambar 4y ago
I think I misunderstand. You seem to say you're using cordon+drain to prevent a cluster from scaling down, and it saves money? How that, it seems counterintuitive to me?
- dijit 4y ago
  Imagine that kubernetes isn’t moving workloads to nodes that have spare space that would fit. Or, think of it as defragging. You basically have enough work for 3 nodes but it’s spread over 5 nodes with some spare capacity on each. Kubernetes is not killing the extra workloads to consolidate it all on the 3 nodes it would have fit on; kubernetes only does bin-packing on new workloads.
  - embik 4y ago
    This might be true for "stock" Kubernetes, but it seems likely that the grandparent post is using cluster autoscaling (since they mention "auto-scaled up"), for which this post is not accurate. The cluster-autoscaler component does remove nodes with low utilization and Pods fitting onto other nodes, effectively defragging the cluster. There's just plenty of reasons for it to not drain a node (as per its FAQ[1]). But it's highly configurable, so I wonder if the grandparent post just needed some configuration changes to suit their scenario better. [1] https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node https://github.com/kubernetes/autoscaler/blob/master/cluster...
    - dijit 4y ago
      There’s a lot of autoscalers too to be perfectly honest. I can’t be sure which one the GP is referring to, but GKE definitely operates the way they described.
      - embik 4y ago
        Most autoscaling functionality is implemented on top of the upstream cluster-autoscaler component, and GKE documentation claims to remove nodes when pods fit onto other nodes as well[1], so it does have the same "defragging" functionality. It might not be configured aggressive enough for their liking though (but even then, GKE has a more aggressive autoscaling profile that might do the trick). My point is that while the original post is reporting problems with autoscaling in Kubernetes without enough details to go by, cluster-autoscaler is fairly good at its job of downsizing clusters if you take the time to a) optimize its configuration and b) make sure your workloads are configured in a way that allows downsizing clusters. [1] https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler?hl=en#how_cluster_autoscaler_works https://cloud.google.com/kubernetes-engine/docs/concepts/clu...
    - tveita 4y ago
      In particular an ephemeral directory mounted on e.g. /tmp counts as "local storage", so in common configurations nothing will get evicted until you work around it by adding "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" to all deployments. Silly default IMO but apparently "intentional". https://github.com/kubernetes/autoscaler/issues/2048 https://github.com/kubernetes/autoscaler/issues/2048
  - uberduper 4y ago
    The cluster autoscaler will do this for you if you configure it.
- remram 4y ago
  No, he's using it to make it scale down. Draining a node will allow it to shut down.
deleted 4y ago
[deleted]
snoth 4y ago
You might want to take a look at descheduler: https://github.com/kubernetes-sigs/descheduler https://github.com/kubernetes-sigs/descheduler
uberduper 4y ago
Maybe configure your cluster autoscaler for scale down resource thresholds?

xcambar 4y ago

I think I misunderstand. You seem to say you're using cordon+drain to prevent a cluster from scaling down, and it saves money? How that, it seems counterintuitive to me?

dijit 4y ago
Imagine that kubernetes isn’t moving workloads to nodes that have spare space that would fit. Or, think of it as defragging. You basically have enough work for 3 nodes but it’s spread over 5 nodes with some spare capacity on each. Kubernetes is not killing the extra workloads to consolidate it all on the 3 nodes it would have fit on; kubernetes only does bin-packing on new workloads.
- embik 4y ago
  This might be true for "stock" Kubernetes, but it seems likely that the grandparent post is using cluster autoscaling (since they mention "auto-scaled up"), for which this post is not accurate. The cluster-autoscaler component does remove nodes with low utilization and Pods fitting onto other nodes, effectively defragging the cluster. There's just plenty of reasons for it to not drain a node (as per its FAQ[1]). But it's highly configurable, so I wonder if the grandparent post just needed some configuration changes to suit their scenario better. [1] https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node https://github.com/kubernetes/autoscaler/blob/master/cluster...
  - dijit 4y ago
    There’s a lot of autoscalers too to be perfectly honest. I can’t be sure which one the GP is referring to, but GKE definitely operates the way they described.
    - embik 4y ago
      Most autoscaling functionality is implemented on top of the upstream cluster-autoscaler component, and GKE documentation claims to remove nodes when pods fit onto other nodes as well[1], so it does have the same "defragging" functionality. It might not be configured aggressive enough for their liking though (but even then, GKE has a more aggressive autoscaling profile that might do the trick). My point is that while the original post is reporting problems with autoscaling in Kubernetes without enough details to go by, cluster-autoscaler is fairly good at its job of downsizing clusters if you take the time to a) optimize its configuration and b) make sure your workloads are configured in a way that allows downsizing clusters. [1] https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler?hl=en#how_cluster_autoscaler_works https://cloud.google.com/kubernetes-engine/docs/concepts/clu...
  - tveita 4y ago
    In particular an ephemeral directory mounted on e.g. /tmp counts as "local storage", so in common configurations nothing will get evicted until you work around it by adding "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" to all deployments. Silly default IMO but apparently "intentional". https://github.com/kubernetes/autoscaler/issues/2048 https://github.com/kubernetes/autoscaler/issues/2048
- uberduper 4y ago
  The cluster autoscaler will do this for you if you configure it.
remram 4y ago
No, he's using it to make it scale down. Draining a node will allow it to shut down.

deleted 4y ago

[deleted]

snoth 4y ago

You might want to take a look at descheduler: https://github.com/kubernetes-sigs/descheduler https://github.com/kubernetes-sigs/descheduler

uberduper 4y ago

Maybe configure your cluster autoscaler for scale down resource thresholds?