6 ms·

The fork is my fault. I agree it's confusing. I wrote (and still maintain) kubecfg. Heptio joined the project and started adding lots of great stuff. Eventu

The fork is my fault. I agree it's confusing. I wrote (and still maintain) kubecfg. Heptio joined the project and started adding lots of great stuff. Eventually it was clear they wanted to take it in a direction that was different to the original borgcfg-like vision. I suggested we split that new functionality into a different tool so we could keep exploring both directions without trying to mash both into the same cli flags. Hence ksonnet/kubecfg and ksonnet/ksonnet. They both use jsonnet internally, generate the same k8s resources in the end, and have that common code heritage, so have many similarities. ksonnet/ksonnet has a bunch of extra ("rails-like") tooling to hold your hand while you generate jsonnet, and assumes it uses the k.libsonnet library (https://github.com/ksonnet/ksonnet-lib https://github.com/ksonnet/ksonnet-lib). ksonnet/kubecfg is much dumber and really just conceptually `jsonnet | kubectl apply`. In particular kubecfg avoids having any opinion about which jsonnet libraries you use. We use it extensively with https://github.com/bitnami-labs/kube-libsonnet https://github.com/bitnami-labs/kube-libsonnet, but you can also use it with https://github.com/ksonnet/ksonnet-lib https://github.com/ksonnet/ksonnet-lib or anything else that is valid jsonnet.

atombender 8y ago
Thanks for the explanation. Is Kubecfg like Helm in that it will apply a "destructive" diff (i.e. delete resources belonging to a deploy that are not part of the new deploy)? I took a look at Ksonnet a while back, and it seemed to have way too many bells and whistles. In particular, I did not particularly like the sheer amount of files, in something like three separate folders, that you have to maintain for each project. I love the idea of declaring a schema that you're generating data for, but Ksonnet seems to have adopted a fairly complex structure to accomplish this. We use Helm right now, though purely as a templating engine and destructive deployer of charts that we store in the same repo as the app itself. The package management isn't useful for our own apps, and really only gets in the way. We're looking at alternatives that don't use text-based templating and come with slightly higher-level concepts of releases. Right now, we wrap Helm with our own little CLI so that we can, for example, automatically build all the template inputs from a set of (defaults, environment-specific stuff, local overrides). Our tool also presents a diff if you want, and records things like git revision info and deployer name in the annotations. Thus, when you ask our tool to deploy something, it can look at what the currently deployed revision is, then do a "git log" to find what's changed, display that history with nice colours on the terminal, etc. before deploying. All things that, in my opinion, a Kubernetes deploy tool should have.
- ithkuil 8y ago
  (EDIT: the nick rang a bell: thanks for ktail!) kubecfg has the --gc-tag flag, which you explicitly pass so that it can know which of the existing resources in the k8s API server used to be created by this set of kubecfg maintained config (this "system", this "application", it's up to you to decide the grain of your modelling) and thus be able to delete the resources are are no longer output by the evaluation of the current configs. This catches the cases where you delete resources but also when you rename resources, or when you "move" them between namespaces. It's implemented without any in-cluster component (no "tiller") by simply setting the GC tag as a label on the resource. Kubecfg also implements diff between local config a d deployed state. As for the amount of files: it's up to you. Kubecfg is not opinionated on how you lay out your config. Unlike helm, it doesn't require you to know in advance which values you want to parameterize (and thus out in a values.yaml) since it's trivial to override any value with jsonnet. The k8s API and its data model is the only thing you have to learn. There are some helpers available to help you structure larger configs, e.g. https://github.com/bitnami-labs/kube-libsonnet/blob/master/kube.libsonnet https://github.com/bitnami-labs/kube-libsonnet/blob/master/k... . While this lib also provides some 'macros' to help you build common entities like services and deployments, IMHO the main benefit is the mapping of "foo_: {key: val, ...}" to "foo: [val, ...]". The former is much more friendlier when you have to override values with jsonnet rather than having to depend on array ordering.
  - atombender 8y ago
    Awesome explanation, thanks! (And I'm really glad you like ktail!) This sounds like a much better foundation to build on than Helm. To me, Kubecfg sounds a lot more palatable than both Helm and Ksonnet. As an aside, Kubecfg still isn't something you (or at least I) would expect developers to work directly with. We have devs who currently just do "tool app deploy foo" (using our homegrown CLI) to deploy an app; they don't need to understand much about Kubernetes, though they understand the basics about pods, kubectl and so on. With Helm, they'd need to know to run "helm install --upgrade --values-from k8s/production.yaml ./chart" or some other extremely long command line. In short, none of the existing tools are high-level enough. There are Heroku-like PaaS abstractions on top of Kubernetes that give you a simplified entrypoint into deployment, but I feel like what's needed isn't a whole platform, just an opinionated top layer. Kubernetes deals with discrete objects, what you want is a higher-level tool that deals with atomic groups of objects, i.e. apps. Long story short, are there any rumblings in the community about going in this direction? The lack of such a tool, at least as far as I've found, has lead me to consider maybe creating one, based on the experience we've had with our in-house CLI tooling, and perhaps using Kubecfg as a foundation. Thoughts?

atombender 8y ago

Thanks for the explanation. Is Kubecfg like Helm in that it will apply a "destructive" diff (i.e. delete resources belonging to a deploy that are not part of the new deploy)? I took a look at Ksonnet a while back, and it seemed to have way too many bells and whistles. In particular, I did not particularly like the sheer amount of files, in something like three separate folders, that you have to maintain for each project. I love the idea of declaring a schema that you're generating data for, but Ksonnet seems to have adopted a fairly complex structure to accomplish this. We use Helm right now, though purely as a templating engine and destructive deployer of charts that we store in the same repo as the app itself. The package management isn't useful for our own apps, and really only gets in the way. We're looking at alternatives that don't use text-based templating and come with slightly higher-level concepts of releases. Right now, we wrap Helm with our own little CLI so that we can, for example, automatically build all the template inputs from a set of (defaults, environment-specific stuff, local overrides). Our tool also presents a diff if you want, and records things like git revision info and deployer name in the annotations. Thus, when you ask our tool to deploy something, it can look at what the currently deployed revision is, then do a "git log" to find what's changed, display that history with nice colours on the terminal, etc. before deploying. All things that, in my opinion, a Kubernetes deploy tool should have.

ithkuil 8y ago
(EDIT: the nick rang a bell: thanks for ktail!) kubecfg has the --gc-tag flag, which you explicitly pass so that it can know which of the existing resources in the k8s API server used to be created by this set of kubecfg maintained config (this "system", this "application", it's up to you to decide the grain of your modelling) and thus be able to delete the resources are are no longer output by the evaluation of the current configs. This catches the cases where you delete resources but also when you rename resources, or when you "move" them between namespaces. It's implemented without any in-cluster component (no "tiller") by simply setting the GC tag as a label on the resource. Kubecfg also implements diff between local config a d deployed state. As for the amount of files: it's up to you. Kubecfg is not opinionated on how you lay out your config. Unlike helm, it doesn't require you to know in advance which values you want to parameterize (and thus out in a values.yaml) since it's trivial to override any value with jsonnet. The k8s API and its data model is the only thing you have to learn. There are some helpers available to help you structure larger configs, e.g. https://github.com/bitnami-labs/kube-libsonnet/blob/master/kube.libsonnet https://github.com/bitnami-labs/kube-libsonnet/blob/master/k... . While this lib also provides some 'macros' to help you build common entities like services and deployments, IMHO the main benefit is the mapping of "foo_: {key: val, ...}" to "foo: [val, ...]". The former is much more friendlier when you have to override values with jsonnet rather than having to depend on array ordering.
- atombender 8y ago
  Awesome explanation, thanks! (And I'm really glad you like ktail!) This sounds like a much better foundation to build on than Helm. To me, Kubecfg sounds a lot more palatable than both Helm and Ksonnet. As an aside, Kubecfg still isn't something you (or at least I) would expect developers to work directly with. We have devs who currently just do "tool app deploy foo" (using our homegrown CLI) to deploy an app; they don't need to understand much about Kubernetes, though they understand the basics about pods, kubectl and so on. With Helm, they'd need to know to run "helm install --upgrade --values-from k8s/production.yaml ./chart" or some other extremely long command line. In short, none of the existing tools are high-level enough. There are Heroku-like PaaS abstractions on top of Kubernetes that give you a simplified entrypoint into deployment, but I feel like what's needed isn't a whole platform, just an opinionated top layer. Kubernetes deals with discrete objects, what you want is a higher-level tool that deals with atomic groups of objects, i.e. apps. Long story short, are there any rumblings in the community about going in this direction? The lack of such a tool, at least as far as I've found, has lead me to consider maybe creating one, based on the experience we've had with our in-house CLI tooling, and perhaps using Kubecfg as a foundation. Thoughts?