9 ms·

2020 indicator would be great. I recently looked closer into my billing data for GKE (Google managed Kubernetes) and was astounded by the overhead taken up jus

2020 indicator would be great. I recently looked closer into my billing data for GKE (Google managed Kubernetes) and was astounded by the overhead taken up just by the Kubernetes internals. Something like 10-20%. It might be better if using bigger node types. how does Nomad compares on this front?

vngzs 5y ago
Nomad is very lightweight. It also does nothing near as much as what Kubernetes does. But if all you need is to schedule containers (or other binaries) it does so quite nicely. You generally want to pair it with Consul (as Cloudflare has done) to get service discovery. Consul is also quite lightweight.
Spivak 5y ago
For work I can totally understand this trade-off but this fact makes k8s pretty much a non-starter on side projects or personal sites because running a small cluster on small nodes leaves you so little headroom for your code that it's just not worth it. Swarm/Nomad/pacemaker all sip resources by comparison.
- egberts1 5y ago
  ^this^
- hinkley 5y ago
  I got all set up with a Pi-clone cluster and then K3s decided that an extra 100MB of memory savings was not worth maintaining a separate implementation of part of the services. On a 16GB machine that's not a big deal. On a 2GB machine that's a significant fraction of your disk cache, for a very slow disk. On a 1GB machine that's just make or break. ETA: And you may think that these toys don't matter, but people have to learn and buy-in somewhere, and the fact of the matter is that I have production services right now that mostly need bandwidth, not CPU or memory, and once I tune them to the most appropriate AWS instance for that workload, the Kubernetes overhead would become the plurality of resource usage on these boxes (if I were using k8s, which we are not at present). K8s only scales by getting into bin packing, where this latency-sensitive service runs with substantially more potentially noisy neighbors.
- hobofan 5y ago
  I mean for GKE, there is a free tier that covers the cost of a autopilot cluster[0], which means basically no overhead cost of running on GKE instead of a Compute Engine node. [0]: https://cloud.google.com/kubernetes-engine/pricing#cluster_management_fee_and_free_tier https://cloud.google.com/kubernetes-engine/pricing#cluster_m...
- p_l 5y ago
  The history of that is actually a bit funny. You see, there used to be no such "fixed overhead" in early versions (including post 1.0) of k8s. This turned out to be a worse idea for less involved or experienced operators than fixed overhead, because it turned out that people would run k8s nodes way underpowered, load them with a ton of workload, then have lots of outages as they starved critical system components of resources. Because of that a (settable) "overhead minimum" was added to calculation done about available resources, iirc originally going for 0.9 cpu core and I don't recall how much memory. This allowed to still run a 2 core node (though it's really not recommended) for experimentation, while greatly lowering chances of obscure for newbies issues. It didn't prevent them (PSA: If another team provisions your cluster, check what resources they provisioned...) but it makes it much harder to fail.
- sandGorgon 5y ago
  actually that is not really true - i strongly urge you to try out http://k3s.io/ http://k3s.io/ or https://k0sproject.io/ https://k0sproject.io/ these are full-fledged, certified k8s distributions that run on raspberry pi as well as all the way in production. https://www.youtube.com/results?search_query=raspberry+pi+k3s https://www.youtube.com/results?search_query=raspberry+pi+k3...
AaronFriel 5y ago
Having run Kubernetes (GKE) in production and AKS/EKS in trials, that overhead is a fixed marginal cost, not one that grows with node size. When running a small cluster with 1-3 nodes, there are reasons to scale out first (reliability), but margin will improve more quickly by scaling up nodes. GCP makes that conveniently pretty easy with their pricing model for CPU and memory, so it is possible to do this incrementally.
- hinkley 5y ago
  For capacity planning you have to look at worst case behaviors, and we have a lot of Pollyannas running around crowing about best and average case behaviors. A consensus protocol has O(logn) behavior on any network that displays any of the 8 fallacies of distributed computing. But the larger the cluster the more fallacies you're likely to have to deal with in a given day, and O(logn) is too optimistic. If the cost is ever 'marginal', it's in little islands of stability that won't last long. What I see over and over again is people expending huge opportunity costs trying to keep their brittle system in one of these local maxima as long as they can. I think because they fear that once they slip out of that comfy spot people will see they aren't some miracle worker, they're just slightly above average and really good at story telling.
  - marcinzm 5y ago
    I'm honestly not sure what your point is. The cloud provider literally charges a fixed cost irrespective of the number of nodes for managed kubernetes. Sure they may need to deal with scaling issues and costs but that's not the concern of a user who pays a flat fee.
    - hinkley 5y ago
      If the hosting dollar figure for K8s is the only cost you can think of, you're down at the bottom of a well of magic thinking that I'm not qualified to hoist you out of.
      - marcinzm 5y ago
GordonS 5y ago
I spun up a 3-node k8s cluster a year back, and was shocked by how much CPU it used when doing nothing. It just felt so utterly wasteful when compared to the likes of Docker Swarm. I totally understand there are use cases for which k8s is a great fit because it has so many capabilities, but I often see folk on HN advocating for using it everywhere, because it's "so simple" once you understand it. I just don't get it.

vngzs 5y ago

Nomad is very lightweight. It also does nothing near as much as what Kubernetes does. But if all you need is to schedule containers (or other binaries) it does so quite nicely. You generally want to pair it with Consul (as Cloudflare has done) to get service discovery. Consul is also quite lightweight.

Spivak 5y ago

For work I can totally understand this trade-off but this fact makes k8s pretty much a non-starter on side projects or personal sites because running a small cluster on small nodes leaves you so little headroom for your code that it's just not worth it. Swarm/Nomad/pacemaker all sip resources by comparison.

egberts1 5y ago
^this^
hinkley 5y ago
I got all set up with a Pi-clone cluster and then K3s decided that an extra 100MB of memory savings was not worth maintaining a separate implementation of part of the services. On a 16GB machine that's not a big deal. On a 2GB machine that's a significant fraction of your disk cache, for a very slow disk. On a 1GB machine that's just make or break. ETA: And you may think that these toys don't matter, but people have to learn and buy-in somewhere, and the fact of the matter is that I have production services right now that mostly need bandwidth, not CPU or memory, and once I tune them to the most appropriate AWS instance for that workload, the Kubernetes overhead would become the plurality of resource usage on these boxes (if I were using k8s, which we are not at present). K8s only scales by getting into bin packing, where this latency-sensitive service runs with substantially more potentially noisy neighbors.
hobofan 5y ago
I mean for GKE, there is a free tier that covers the cost of a autopilot cluster[0], which means basically no overhead cost of running on GKE instead of a Compute Engine node. [0]: https://cloud.google.com/kubernetes-engine/pricing#cluster_management_fee_and_free_tier https://cloud.google.com/kubernetes-engine/pricing#cluster_m...
p_l 5y ago
The history of that is actually a bit funny. You see, there used to be no such "fixed overhead" in early versions (including post 1.0) of k8s. This turned out to be a worse idea for less involved or experienced operators than fixed overhead, because it turned out that people would run k8s nodes way underpowered, load them with a ton of workload, then have lots of outages as they starved critical system components of resources. Because of that a (settable) "overhead minimum" was added to calculation done about available resources, iirc originally going for 0.9 cpu core and I don't recall how much memory. This allowed to still run a 2 core node (though it's really not recommended) for experimentation, while greatly lowering chances of obscure for newbies issues. It didn't prevent them (PSA: If another team provisions your cluster, check what resources they provisioned...) but it makes it much harder to fail.
sandGorgon 5y ago
actually that is not really true - i strongly urge you to try out http://k3s.io/ http://k3s.io/ or https://k0sproject.io/ https://k0sproject.io/ these are full-fledged, certified k8s distributions that run on raspberry pi as well as all the way in production. https://www.youtube.com/results?search_query=raspberry+pi+k3s https://www.youtube.com/results?search_query=raspberry+pi+k3...

AaronFriel 5y ago

Having run Kubernetes (GKE) in production and AKS/EKS in trials, that overhead is a fixed marginal cost, not one that grows with node size. When running a small cluster with 1-3 nodes, there are reasons to scale out first (reliability), but margin will improve more quickly by scaling up nodes. GCP makes that conveniently pretty easy with their pricing model for CPU and memory, so it is possible to do this incrementally.

hinkley 5y ago
For capacity planning you have to look at worst case behaviors, and we have a lot of Pollyannas running around crowing about best and average case behaviors. A consensus protocol has O(logn) behavior on any network that displays any of the 8 fallacies of distributed computing. But the larger the cluster the more fallacies you're likely to have to deal with in a given day, and O(logn) is too optimistic. If the cost is ever 'marginal', it's in little islands of stability that won't last long. What I see over and over again is people expending huge opportunity costs trying to keep their brittle system in one of these local maxima as long as they can. I think because they fear that once they slip out of that comfy spot people will see they aren't some miracle worker, they're just slightly above average and really good at story telling.
- marcinzm 5y ago
  I'm honestly not sure what your point is. The cloud provider literally charges a fixed cost irrespective of the number of nodes for managed kubernetes. Sure they may need to deal with scaling issues and costs but that's not the concern of a user who pays a flat fee.
  - hinkley 5y ago
    If the hosting dollar figure for K8s is the only cost you can think of, you're down at the bottom of a well of magic thinking that I'm not qualified to hoist you out of.
    - marcinzm 5y ago

GordonS 5y ago

I spun up a 3-node k8s cluster a year back, and was shocked by how much CPU it used when doing nothing. It just felt so utterly wasteful when compared to the likes of Docker Swarm. I totally understand there are use cases for which k8s is a great fit because it has so many capabilities, but I often see folk on HN advocating for using it everywhere, because it's "so simple" once you understand it. I just don't get it.