5 ms·

> That means the computers in my home server rack, the DigitalOcean VM I have, and the couple other servers dotted around the internet. These should all be part

by stevefan1999 6y ago

> That means the computers in my home server rack, the DigitalOcean VM I have, and the couple other servers dotted around the internet. These should all be part of one cluster, and behave as such. YES! This is exactly what we should have had been able to do in K8S swiftly, but I would like to suggest one additional goal: to get a highly-distributed, fault tolerant computer cluster over different network without a hassle (as a mean to spread risk), while also having a as-low-as-best-as-possible TCO through the use of heterogeneous architecture. So this means I can have used AWS for their Graviton instances for high performance, near-bare-metal microservices, and can also run typical x86 workload on various cheap VPS providers such as Vultr, DO and any other cheap hosts out in the wild to handle normal stuff that typically wouldn't run well on ARM, such as GitLab, Prometheus, Keycloak, in a sense that we associate x86 to have an affinity with "heavy stuff" runner and this implies ARM to be the Mr. "lightweight stuff" guy. This is not really possible by today's shape of the k8s ecosystem, given that the majority of the images in Docker Hub is predominately x86/amd64, and my wild guess is around 90%. By the way I use ServerHunter[1] to scavenge such cheap x86 servers. Also I'm a k3s[2] user and I have attempted to do this pretty well. Given your kernel support, you can even strap WireGuard on natively (albeit my VPS host internet bill tells me this can hit quite heavy) While I can definitely control k8s by using kubeadm but I just liked k3s's philosophy of being battery-packed: It will install Traefik (although my recent needs to run nginx-ingress ultimately let me turn it off), CoreDNS, Flannel, a simple local storage and extra goodies all out of the box in your cluster, and you can opt them out whenever you want if you think they are insufficient for your kind of workload (whilst they are adequate for most of 90% of the thing actually). To be honest, this is how k8s should have had been this simple in the very beginning to say the least. [1]: https://www.serverhunter.com/ https://www.serverhunter.com/ [2]: https://k3s.io https://k3s.io

q3k 6y ago
Counter-argument: having your entire world-wide deployment operate under a single control-plane is a recipe for global outages. There should no single command that one can fat-finger that will bring down your system globally. One-cluster-per-region (with some tie-in into one region being its own failure domain, both at the underlying infrastructure and application level) is the way to go for reliability.
- throwaway894345 6y ago
  The model put forth in TFA seems to address this in that pods (or “sub clusters” can run for weeks at a time without communication with the toplevel cluster. It’s pretty hand-wavy and probably can’t solve for all possible outage scenarios, but it seems like it would help dramatically.
yowlingcat 6y ago
One potential gotcha in your model is egress costs. For the big three, it's anywhere from $0.03/GB to $0.10/GB. The million dollar question suggested by your model is how do you you store and backup your persistent state in a manner which is not just conceptually robust but cost optimal?
dnautics 6y ago
> YES! This is exactly what we should have had been able to do in K8S swiftly, but I would like to suggest one additional goal: to get a highly-distributed, fault tolerant computer cluster over different network without a hassle (as a mean to spread risk) In practice this is going to be tricky unless your services are completely stateless. For 80% of people that's going to be true, but if you have customers with large datasets, (I'm thinking mostly media) you do not want to be schlepping those things between clouds or between cloud and on prem or even between on prems.
chucky_z 6y ago
I do this today with Nomad/Consul. Hashicorp's Raft+Serf implementation allows for shocking amounts of latency between servers/clients. I have several centralized server clusters (the "servers" _should_ be geographically close, the "agents" can be far and wide), and agents more than 200ms+ away, and across multiple clouds/on-prem. Everything works just fine. I've legitimately considered running some kind of simple SaaS and behind the scenes running a Nomad cluster and having the remote SaaS agent just be a Nomad client.

q3k 6y ago

Counter-argument: having your entire world-wide deployment operate under a single control-plane is a recipe for global outages. There should no single command that one can fat-finger that will bring down your system globally. One-cluster-per-region (with some tie-in into one region being its own failure domain, both at the underlying infrastructure and application level) is the way to go for reliability.

throwaway894345 6y ago
The model put forth in TFA seems to address this in that pods (or “sub clusters” can run for weeks at a time without communication with the toplevel cluster. It’s pretty hand-wavy and probably can’t solve for all possible outage scenarios, but it seems like it would help dramatically.

yowlingcat 6y ago

One potential gotcha in your model is egress costs. For the big three, it's anywhere from $0.03/GB to $0.10/GB. The million dollar question suggested by your model is how do you you store and backup your persistent state in a manner which is not just conceptually robust but cost optimal?

dnautics 6y ago

> YES! This is exactly what we should have had been able to do in K8S swiftly, but I would like to suggest one additional goal: to get a highly-distributed, fault tolerant computer cluster over different network without a hassle (as a mean to spread risk) In practice this is going to be tricky unless your services are completely stateless. For 80% of people that's going to be true, but if you have customers with large datasets, (I'm thinking mostly media) you do not want to be schlepping those things between clouds or between cloud and on prem or even between on prems.

chucky_z 6y ago

I do this today with Nomad/Consul. Hashicorp's Raft+Serf implementation allows for shocking amounts of latency between servers/clients. I have several centralized server clusters (the "servers" _should_ be geographically close, the "agents" can be far and wide), and agents more than 200ms+ away, and across multiple clouds/on-prem. Everything works just fine. I've legitimately considered running some kind of simple SaaS and behind the scenes running a Nomad cluster and having the remote SaaS agent just be a Nomad client.