4 ms·

The comparison is a bit misleading. As someone that has used both nomad and k8s at scale -- - Nomad is a scheduler. Clean and focused. It is very fast. I was a

The comparison is a bit misleading. As someone that has used both nomad and k8s at scale -- - Nomad is a scheduler. Clean and focused. It is very fast. I was an early user and encountered a number of bumps, but that's software. The people at Hashicorp are super sharp and lovely. - K8s is a lot more. It includes a scheduler, but in the simplest sense, it is a complete control-plane based on the control-loop pattern. You have an API, a scheduler, a db, various controllers, etc. Forget for a moment that most people use it to orchestrate containers -- it's really designed to orchestrate anything. Its API is extensible, you can add and compose controllers -- there are many possibilities once you wrap your head around it. This is all opinionated and includes a lot of capability. It's just very different. You can stitch together nomad, consul, vault, and various glue to create a container orchestration system... but when you start wanting to manage the control-planes as though they are the "kind" (the container for example) with meta-control-planes, and you start wanting to orchestrate network, storage, and other dependencies... all while doing this in a multi-tenant environment, then things get interesting. -charles.

still_grokking 5y ago
I'm not sure it's a good idea to use Google's "monolithic cloud operation system". Such a monoculture has the same issues as MS Windows had, even for the same reasons. The "Unix Way" of simple tools interacting seems more reasonable. Especially when it comes to lock-in effects.
- m00x 5y ago
  K8s itself is divided in multiple parts, where you can customize to your own liking, and you can swap parts if you'd like as long as the APIs are similar. It's very much built the UNIX way.
  - still_grokking 5y ago
    Where can I find those alternative elements that can be swapped? If this is true there should be a lot of them, right?
    - narism 5y ago
      If your complaint about Kubernetes is that it doesn’t provide you enough choice/extensibility, you’re probably not looking hard enough. Runtimes (CRI) - Docker shim (deprecated), containerd, CRI-O, kata Networking (CNI) - Flannel, Calico, Cilium, cloud specific ones, many more Storage (CSI) - way too many device plugins - GPU, TPU, RDMA/SRIOV NICs Data store - etcd, dqlite in microk8s, SQLite/postgresql/mysql in k3s DNS - kube-dns (somewhat deprecated), CoreDNS Ingress - NGINX (multiple), HAProxy, Envoy (many), etc Kubelet -https://github.com/virtual-kubelet/virtual-kubelet https://github.com/virtual-kubelet/virtual-kubelet Kube-proxy - Cilium can act as a replacement Cloud-controller-manager for each cloud provider kube-scheduler - https://kubernetes.io/docs/tasks/extend-kubernetes/configure-multiple-schedulers/ https://kubernetes.io/docs/tasks/extend-kubernetes/configure... kube-apiserver and kube-controller-manager are two parts where I’m not aware of any other implementations but a) they are kind of the heart of k8s and b) can be easily extended with CRDs/operators.
- nonameiguess 5y ago
  This is so far from the truth, I'm having a hard time imagining why you even think this. Kubernetes is practically the distributed embodiment of the Unix philosophy. You have a core set of interfaces and components that need to offer a particular API, and other than that, whether it is one program or many, written by one developer or hundreds, by a private company or via volunteers contributing to open source projects, is totally up to how you want to do it. You're free to use the original reference implementation that used to be owned by Google a decade ago before they open sourced it and donated it to the CNCF, but you certainly don't have to. Others have mentioned k3s, which is the busybox to the reference kubernetes GNU coreutils, all Kubernetes, plus ingress and network overlay, in a single binary, with am embedded sqlite db as the backing store instead of etcd. But k3s is still "Kubernetes." Kubernetes is a standard, much like POSIX. It's maybe unfortunate that the original reference implementation is also also named "kubernetes" because a lot of people seem to think that one is the only one you can use, and it has historically been complex to set up, but the reason for the complexity is it doesn't make any choices for you. Imagine if you wanted to use a Unix operating system, but instead of choosing a Linux distro, you just read the POSIX standard and went out and found every required utility, plus a kernel, and had to figure out on your own how to get those to work together and create a system that can run application-level software. If you just go to kubernetes.io and follow the instructions on how to get up and running with the reference implementation, that is what you're doing. It makes no decisions at all for you. You can run external etcd, or use kubeadm to set it up for you. You can run it HA or on a single node. You can add whatever overlay network you want. You can use whatever container runtime engine you want. You can use whatever ingress controller you want, or none at all, and not have any external networking, just as you can install Linux From Scratch and not even bother to include networking if you want a disconnected system for some reason. You have pretty much complete user freedom, and that is, in fact, the source and reason for a whole lot of complaints. Application developers and even most system administrators don't want to have to make that many decisions before they can even get to hello world. I believe Kelsey Hightower commented on this a while back, saying something to the effect that Kubernetes is not meant to be a developer platform. It's a framework for creating platforms. Application developers, startups, and small business should almost never be using Kubernetes directly unless they're actually developing a platform product. Whether you use a "distro" like RKE2 or k3s or a managed service from a cloud provider, building out your own cluster using the reference kubernetes is the modern day equivalent of deploying a LAMP stack but doing it on top of Linux From Scratch.

still_grokking 5y ago

I'm not sure it's a good idea to use Google's "monolithic cloud operation system". Such a monoculture has the same issues as MS Windows had, even for the same reasons. The "Unix Way" of simple tools interacting seems more reasonable. Especially when it comes to lock-in effects.

m00x 5y ago
K8s itself is divided in multiple parts, where you can customize to your own liking, and you can swap parts if you'd like as long as the APIs are similar. It's very much built the UNIX way.
- still_grokking 5y ago
  Where can I find those alternative elements that can be swapped? If this is true there should be a lot of them, right?
  - narism 5y ago
    If your complaint about Kubernetes is that it doesn’t provide you enough choice/extensibility, you’re probably not looking hard enough. Runtimes (CRI) - Docker shim (deprecated), containerd, CRI-O, kata Networking (CNI) - Flannel, Calico, Cilium, cloud specific ones, many more Storage (CSI) - way too many device plugins - GPU, TPU, RDMA/SRIOV NICs Data store - etcd, dqlite in microk8s, SQLite/postgresql/mysql in k3s DNS - kube-dns (somewhat deprecated), CoreDNS Ingress - NGINX (multiple), HAProxy, Envoy (many), etc Kubelet -https://github.com/virtual-kubelet/virtual-kubelet https://github.com/virtual-kubelet/virtual-kubelet Kube-proxy - Cilium can act as a replacement Cloud-controller-manager for each cloud provider kube-scheduler - https://kubernetes.io/docs/tasks/extend-kubernetes/configure-multiple-schedulers/ https://kubernetes.io/docs/tasks/extend-kubernetes/configure... kube-apiserver and kube-controller-manager are two parts where I’m not aware of any other implementations but a) they are kind of the heart of k8s and b) can be easily extended with CRDs/operators.
nonameiguess 5y ago
This is so far from the truth, I'm having a hard time imagining why you even think this. Kubernetes is practically the distributed embodiment of the Unix philosophy. You have a core set of interfaces and components that need to offer a particular API, and other than that, whether it is one program or many, written by one developer or hundreds, by a private company or via volunteers contributing to open source projects, is totally up to how you want to do it. You're free to use the original reference implementation that used to be owned by Google a decade ago before they open sourced it and donated it to the CNCF, but you certainly don't have to. Others have mentioned k3s, which is the busybox to the reference kubernetes GNU coreutils, all Kubernetes, plus ingress and network overlay, in a single binary, with am embedded sqlite db as the backing store instead of etcd. But k3s is still "Kubernetes." Kubernetes is a standard, much like POSIX. It's maybe unfortunate that the original reference implementation is also also named "kubernetes" because a lot of people seem to think that one is the only one you can use, and it has historically been complex to set up, but the reason for the complexity is it doesn't make any choices for you. Imagine if you wanted to use a Unix operating system, but instead of choosing a Linux distro, you just read the POSIX standard and went out and found every required utility, plus a kernel, and had to figure out on your own how to get those to work together and create a system that can run application-level software. If you just go to kubernetes.io and follow the instructions on how to get up and running with the reference implementation, that is what you're doing. It makes no decisions at all for you. You can run external etcd, or use kubeadm to set it up for you. You can run it HA or on a single node. You can add whatever overlay network you want. You can use whatever container runtime engine you want. You can use whatever ingress controller you want, or none at all, and not have any external networking, just as you can install Linux From Scratch and not even bother to include networking if you want a disconnected system for some reason. You have pretty much complete user freedom, and that is, in fact, the source and reason for a whole lot of complaints. Application developers and even most system administrators don't want to have to make that many decisions before they can even get to hello world. I believe Kelsey Hightower commented on this a while back, saying something to the effect that Kubernetes is not meant to be a developer platform. It's a framework for creating platforms. Application developers, startups, and small business should almost never be using Kubernetes directly unless they're actually developing a platform product. Whether you use a "distro" like RKE2 or k3s or a managed service from a cloud provider, building out your own cluster using the reference kubernetes is the modern day equivalent of deploying a LAMP stack but doing it on top of Linux From Scratch.