4 ms·

Where Kops is a command line tool with subcommands like create-cluster and update-cluster, Keights uses Ansible roles with CloudFormation underneath. This works

by joseph 8y ago

Where Kops is a command line tool with subcommands like create-cluster and update-cluster, Keights uses Ansible roles with CloudFormation underneath. This works well to manage the cluster from CI/CD, where updating the cluster is just updating the version in requirements.yml and retriggering the build with Ansible. Creating or updating the cluster is the same process. Keights also relies on kubeadm, which is released along with Kubernetes, so it can be up to date much quicker than Kops, which is several releases behind. Kops is a surprisingly large amount of code and probably needs significant changes between each release, whereas changing Keights to support a new release usually just means modifying the Kubeadm config files and changing dependency versions. At the last two companies where I worked, they couldn't use Kops because it checked for an internet gateway on the VPC and failed if one was not found. A lot of companies have centrally controlled Amazon accounts with locked down VPCs and no ability for most teams to modify them. Keights was created to work even in such environments and assumes that your network is already set up how you want it.

conradk 8y ago
Thanks for explaining. It sounds like Keights is the same as Kubespray then, isn't it ? Kubespray also uses kubeadm underneath, as far as I'm aware.
- joseph 8y ago
  They both use kubeadm, but the design is very different. Kubespray runs Ansible against the hosts in the cluster, whereas Keights uses Ansible to build CloudFormation stacks, and all the instances then bootstrap themselves with kubeadm.
justinsb 8y ago
Thanks for the input on kops. There's a lot of functionality in kops - e.g. we can manage CNI providers, etcd etc. In some places this manifests as more code, but my gut is that if you add up all the code you're relying on it turns out to be a wash. And you can definitely use kops from CI, driving it from yaml files that are checked in to git - that seems to be a great configuration particularly for people managing large numbers of clusters. The long-term strategy is to get most of the kops functionality upstreamed into standalone community projects - and we're making progress with etcdadm, addon-operators, cluster-api etc. Then it will be easy to write your own tooling if you don't like some of the kops decisions, but still benefit from the community investment in e.g. etcd management etc. kops itself becomes a thinner shim around those shared common pieces. A lot of the decisions that are now generally agreed (e.g. dynamically attaching etcd volumes) weren't as well accepted when we started off, so it was harder to get them going as community efforts! We do have support for "phases" in kops which should allow you to use a provided VPC, but to be honest it's still not as easy as the rest of kops is. We also have a few PRs in-flight that to allow you to specify an alternative to an IGW e.g. a VPN, but it's hard to reach consensus (but I guess we should based on your input!). The big trade-off here is that once you start allowing arbitrary configuration, you lose the ability to validate things, and so for some fraction of people there are going to be mistakes. That works great for small community projects, it is really great if your business model is paid support, but for a large community project it really can be problematic. I don't think we've got the balance totally correct in kops, but that's the trade-off we wrestle with.
- joseph 8y ago
  Hi Justin, thanks for the feedback. I'll take another look at kops as it's been a while. I do think it could benefit from having a knob to turn off some of the pre-validation, or for it to do functional checks instead of checking for the existence of specific AWS resources. Many companies treat their AWS VPCs as an extension of the private corporate network, and they come up with lots of creative ways to route traffic to the internet (or select portions of it) without the VPC necessarily having an IGW or VPN. AWS is also making a growing number of services available from private endpoints within the VPC, so there is often no need for internet access. I understand the value of validating the environment, but an "expert mode" toggle would also be nice.

by joseph 8y ago

conradk 8y ago

Thanks for explaining. It sounds like Keights is the same as Kubespray then, isn't it ? Kubespray also uses kubeadm underneath, as far as I'm aware.

joseph 8y ago
They both use kubeadm, but the design is very different. Kubespray runs Ansible against the hosts in the cluster, whereas Keights uses Ansible to build CloudFormation stacks, and all the instances then bootstrap themselves with kubeadm.

justinsb 8y ago

Thanks for the input on kops. There's a lot of functionality in kops - e.g. we can manage CNI providers, etcd etc. In some places this manifests as more code, but my gut is that if you add up all the code you're relying on it turns out to be a wash. And you can definitely use kops from CI, driving it from yaml files that are checked in to git - that seems to be a great configuration particularly for people managing large numbers of clusters. The long-term strategy is to get most of the kops functionality upstreamed into standalone community projects - and we're making progress with etcdadm, addon-operators, cluster-api etc. Then it will be easy to write your own tooling if you don't like some of the kops decisions, but still benefit from the community investment in e.g. etcd management etc. kops itself becomes a thinner shim around those shared common pieces. A lot of the decisions that are now generally agreed (e.g. dynamically attaching etcd volumes) weren't as well accepted when we started off, so it was harder to get them going as community efforts! We do have support for "phases" in kops which should allow you to use a provided VPC, but to be honest it's still not as easy as the rest of kops is. We also have a few PRs in-flight that to allow you to specify an alternative to an IGW e.g. a VPN, but it's hard to reach consensus (but I guess we should based on your input!). The big trade-off here is that once you start allowing arbitrary configuration, you lose the ability to validate things, and so for some fraction of people there are going to be mistakes. That works great for small community projects, it is really great if your business model is paid support, but for a large community project it really can be problematic. I don't think we've got the balance totally correct in kops, but that's the trade-off we wrestle with.

joseph 8y ago
Hi Justin, thanks for the feedback. I'll take another look at kops as it's been a while. I do think it could benefit from having a knob to turn off some of the pre-validation, or for it to do functional checks instead of checking for the existence of specific AWS resources. Many companies treat their AWS VPCs as an extension of the private corporate network, and they come up with lots of creative ways to route traffic to the internet (or select portions of it) without the VPC necessarily having an IGW or VPN. AWS is also making a growing number of services available from private endpoints within the VPC, so there is often no need for internet access. I understand the value of validating the environment, but an "expert mode" toggle would also be nice.