4 ms·

How does Keights compare to Kops ? https://github.com/kubernetes/kops https://github.com/kubernetes/kops

Nullabillity 8y ago
Kops is (intentionally) very out-of-date. This is a blog post about K8s 1.14 being released, but the latest stable Kops is still based on K8s 1.11.x.
- conradk 8y ago
  Ah yes, OK. What about Kubespray ? https://github.com/kubernetes-sigs/kubespray https://github.com/kubernetes-sigs/kubespray
  - Nullabillity 8y ago
    Kubespray SSH:s in and does what you'd do manually, while Kops deploys a prebuilt image. That means Kubespray is pretty much cloud-agnostic, and much easier to customize, but it's also not really compatible with auto-scaling groups and the like.
- slyall 8y ago
  I don't think they intend to be as out-of-date as they are. They are supposed to be around a version behind. So they should really be releasing 1.13 soon rather than 1.12. They have had a few big tech changes in the 1.9/1.10/1.11 timeline so hopefully they will be able to catchup a little.
  - justinsb 8y ago
    Yes, the reality is probably somewhere in between. We've heard that users like the idea that kops releases are more ready, where the .0 releases of k8s might have more issues particularly with ecosystem components. So we do lag a little behind - ideally one release. 1.12 has been a particularly tricky release getting everyone from etcd2 -> etcd3, but we've finally turned the corner on that one, so that should now let us catch up a bit. Finally, we've also heard that users want more of a choice, so we're going to start doing e.g. 1.13-alpha and 1.14 alphas much sooner. We'll still wait to do kops 1.14.0 until everything is ready, but for users that want to run k8s 1.14 sooner, they will have an option that isn't building from source. And hopefully this also gets more people using pre-release versions of kops (in non-production environments) and also helps stabilize the releases more rapidly.
joseph 8y ago
Where Kops is a command line tool with subcommands like create-cluster and update-cluster, Keights uses Ansible roles with CloudFormation underneath. This works well to manage the cluster from CI/CD, where updating the cluster is just updating the version in requirements.yml and retriggering the build with Ansible. Creating or updating the cluster is the same process. Keights also relies on kubeadm, which is released along with Kubernetes, so it can be up to date much quicker than Kops, which is several releases behind. Kops is a surprisingly large amount of code and probably needs significant changes between each release, whereas changing Keights to support a new release usually just means modifying the Kubeadm config files and changing dependency versions. At the last two companies where I worked, they couldn't use Kops because it checked for an internet gateway on the VPC and failed if one was not found. A lot of companies have centrally controlled Amazon accounts with locked down VPCs and no ability for most teams to modify them. Keights was created to work even in such environments and assumes that your network is already set up how you want it.
- conradk 8y ago
  Thanks for explaining. It sounds like Keights is the same as Kubespray then, isn't it ? Kubespray also uses kubeadm underneath, as far as I'm aware.
  - joseph 8y ago
    They both use kubeadm, but the design is very different. Kubespray runs Ansible against the hosts in the cluster, whereas Keights uses Ansible to build CloudFormation stacks, and all the instances then bootstrap themselves with kubeadm.
- justinsb 8y ago
  Thanks for the input on kops. There's a lot of functionality in kops - e.g. we can manage CNI providers, etcd etc. In some places this manifests as more code, but my gut is that if you add up all the code you're relying on it turns out to be a wash. And you can definitely use kops from CI, driving it from yaml files that are checked in to git - that seems to be a great configuration particularly for people managing large numbers of clusters. The long-term strategy is to get most of the kops functionality upstreamed into standalone community projects - and we're making progress with etcdadm, addon-operators, cluster-api etc. Then it will be easy to write your own tooling if you don't like some of the kops decisions, but still benefit from the community investment in e.g. etcd management etc. kops itself becomes a thinner shim around those shared common pieces. A lot of the decisions that are now generally agreed (e.g. dynamically attaching etcd volumes) weren't as well accepted when we started off, so it was harder to get them going as community efforts! We do have support for "phases" in kops which should allow you to use a provided VPC, but to be honest it's still not as easy as the rest of kops is. We also have a few PRs in-flight that to allow you to specify an alternative to an IGW e.g. a VPN, but it's hard to reach consensus (but I guess we should based on your input!). The big trade-off here is that once you start allowing arbitrary configuration, you lose the ability to validate things, and so for some fraction of people there are going to be mistakes. That works great for small community projects, it is really great if your business model is paid support, but for a large community project it really can be problematic. I don't think we've got the balance totally correct in kops, but that's the trade-off we wrestle with.

Kops is (intentionally) very out-of-date. This is a blog post about K8s 1.14 being released, but the latest stable Kops is still based on K8s 1.11.x.

conradk 8y ago
Ah yes, OK. What about Kubespray ? https://github.com/kubernetes-sigs/kubespray https://github.com/kubernetes-sigs/kubespray
- Nullabillity 8y ago
  Kubespray SSH:s in and does what you'd do manually, while Kops deploys a prebuilt image. That means Kubespray is pretty much cloud-agnostic, and much easier to customize, but it's also not really compatible with auto-scaling groups and the like.
slyall 8y ago
I don't think they intend to be as out-of-date as they are. They are supposed to be around a version behind. So they should really be releasing 1.13 soon rather than 1.12. They have had a few big tech changes in the 1.9/1.10/1.11 timeline so hopefully they will be able to catchup a little.
- justinsb 8y ago
  Yes, the reality is probably somewhere in between. We've heard that users like the idea that kops releases are more ready, where the .0 releases of k8s might have more issues particularly with ecosystem components. So we do lag a little behind - ideally one release. 1.12 has been a particularly tricky release getting everyone from etcd2 -> etcd3, but we've finally turned the corner on that one, so that should now let us catch up a bit. Finally, we've also heard that users want more of a choice, so we're going to start doing e.g. 1.13-alpha and 1.14 alphas much sooner. We'll still wait to do kops 1.14.0 until everything is ready, but for users that want to run k8s 1.14 sooner, they will have an option that isn't building from source. And hopefully this also gets more people using pre-release versions of kops (in non-production environments) and also helps stabilize the releases more rapidly.

joseph 8y ago

Where Kops is a command line tool with subcommands like create-cluster and update-cluster, Keights uses Ansible roles with CloudFormation underneath. This works well to manage the cluster from CI/CD, where updating the cluster is just updating the version in requirements.yml and retriggering the build with Ansible. Creating or updating the cluster is the same process. Keights also relies on kubeadm, which is released along with Kubernetes, so it can be up to date much quicker than Kops, which is several releases behind. Kops is a surprisingly large amount of code and probably needs significant changes between each release, whereas changing Keights to support a new release usually just means modifying the Kubeadm config files and changing dependency versions. At the last two companies where I worked, they couldn't use Kops because it checked for an internet gateway on the VPC and failed if one was not found. A lot of companies have centrally controlled Amazon accounts with locked down VPCs and no ability for most teams to modify them. Keights was created to work even in such environments and assumes that your network is already set up how you want it.

conradk 8y ago
Thanks for explaining. It sounds like Keights is the same as Kubespray then, isn't it ? Kubespray also uses kubeadm underneath, as far as I'm aware.
- joseph 8y ago
  They both use kubeadm, but the design is very different. Kubespray runs Ansible against the hosts in the cluster, whereas Keights uses Ansible to build CloudFormation stacks, and all the instances then bootstrap themselves with kubeadm.
justinsb 8y ago
Thanks for the input on kops. There's a lot of functionality in kops - e.g. we can manage CNI providers, etcd etc. In some places this manifests as more code, but my gut is that if you add up all the code you're relying on it turns out to be a wash. And you can definitely use kops from CI, driving it from yaml files that are checked in to git - that seems to be a great configuration particularly for people managing large numbers of clusters. The long-term strategy is to get most of the kops functionality upstreamed into standalone community projects - and we're making progress with etcdadm, addon-operators, cluster-api etc. Then it will be easy to write your own tooling if you don't like some of the kops decisions, but still benefit from the community investment in e.g. etcd management etc. kops itself becomes a thinner shim around those shared common pieces. A lot of the decisions that are now generally agreed (e.g. dynamically attaching etcd volumes) weren't as well accepted when we started off, so it was harder to get them going as community efforts! We do have support for "phases" in kops which should allow you to use a provided VPC, but to be honest it's still not as easy as the rest of kops is. We also have a few PRs in-flight that to allow you to specify an alternative to an IGW e.g. a VPN, but it's hard to reach consensus (but I guess we should based on your input!). The big trade-off here is that once you start allowing arbitrary configuration, you lose the ability to validate things, and so for some fraction of people there are going to be mistakes. That works great for small community projects, it is really great if your business model is paid support, but for a large community project it really can be problematic. I don't think we've got the balance totally correct in kops, but that's the trade-off we wrestle with.