6 ms·

I just knew this would be about Kubernetes when I saw the title. The Kubernetes API is fairly straightforward, and has a well-defined (JSON) schema, people sho

I just knew this would be about Kubernetes when I saw the title. The Kubernetes API is fairly straightforward, and has a well-defined (JSON) schema, people should be spending a bulk of their time learning k8s understanding how to use the API, but instead they spend it working out how to use a Helm chart. I don't think Jsonnet, Ksonnet, Nu, or CUE ever gained that much traction. I'm convinced most people just use Kustomize, because it's fairly straightforward and built in to kubectl. I'd like a tool that: - Gives definition writers type checking against the k8s schemas - validation, version deprecations, etc. - Gives users a single artefact that can be inspected easily and will fail (ACID) if deployed against a cluster that doesn't support any objects/versions. - Is built into the default toolchain --- I feel like writing a Bun or Deno TypeScript script that exports a function with arguments and returns a list of definitions would work well, esp. with `deno compile`, etc. but that violates the third point.

baq 3y ago
probably doesn't meet the 2nd requirement, most definitely doesn't meet the third, but: https://cdk8s.io/docs/latest/ https://cdk8s.io/docs/latest/
- TheFuzzball 3y ago
  The second requirement is actually probably the most important - if someone that just set up ArgoCD, Flux, or has their own GitOps pipeline, how much of a headache does using a new compile step present? Lots of things are simple in isolation: want to use Cue? Just get your definitions and install the compiler and call it and boom, there are your k8s defs! Ok, but how do I integrate all of that into my existing toolchain? How do I pass config? Etc, etc. The best, fastest tool won't win. The tool that has the most frictionless user story will.
  - shepherdjerred 3y ago
    I was able to get CDK8s working easily by simply committing the built template along with my TypeScript. Then, I just pointed ArgoCD to my repo.
    - how_gauche 3y ago
      We do the same thing but commit to a second git repo that we treat like the "k8s yaml release database".
Havoc 3y ago
How would one use the json api without ending up writing a bunch of custom code?
- TheFuzzball 3y ago
  I think custom code is to be expected, and making it maintainable is what's important. > everything should be made as simple as possible, but no simpler. Helm et al made it simpler than it was, IMO.
  - Havoc 3y ago
    Everyone hand rolling code does not seem like an improvement over tools like helm even if it’s yaml
    - TheFuzzball 3y ago
      No, obviously not, and that's not what I've suggested.
  - rad_gruchalski 3y ago
    Helm is another can of hot garbage. Impossible to vendor without hitting name collisions, can configure only what’s templated. Jsonnet is the way to go with generated helm manifests transformed later. Kustomize with its post-renderer hooks is another can of even hotter garbage.
    - aeyes 3y ago
      > Impossible to vendor without hitting name collisions What problem exactly are you facing? I can change the name of the chart itself in chart.yaml and if the name of the resources collide I change them with nameOverride/fullnameOverride in the values. All charts have these because they are autogenerated by `helm create`. I vendor all charts and never had this problem.
      - rad_gruchalski 3y ago
ithkuil 3y ago
I love the idea of keeping it simple and I do try to use kustomize or even plain yaml as installation method as much as possible. But in practice when managing large systems you inevitably end up benefiting from templating
- worldsayshi 3y ago
  I've begun thinking that if you start thinking about templating you might be better off building an operator. Operators aren't as well understood and documented. But in my mind an operator is just a pod or deployment that creates on demand resources using the k8s api.
  - ryandv 3y ago
    The purpose of an Operator is to realize the resources desired/requested in a (custom) resource manifest, often as YAML or JSON. You give the apiserver a document describing what resources you need. The Operator actually does the work of provisioning those resources in the "real world" and (should) update the status field on the API object to indicate if those resources are ready.
  - ithkuil 3y ago
    oh yeah; operators are great and sometimes they are necessary. On the other hand, most operators I've seen are just k8s manifest templates implemented in Go. I often end up preferring using Jsonnet to deal with that instead of doing the same stuff in Go. Jsonnet is much more close to the underlying datamodel (the k8s manifest Json/Yaml document) and comes with some useful functionality out of the box, such "overlays". It has downsides too! It's untyped, debugging tools are lacking, people are unfamiliar with it and don't care to learn it. So I totally get why one would entertain the possibility of writing your "templates" using a better language. However, an operator is often too much freedom. It's not just using Go or Rust or Typescript to "generate" some Json manifests, but it also contains the code to interact with the API server, setup watches, and reactions etc. I often wish there was a better way to separate those two concerns I'm a fan of metacontroller [1], which is a tool that allows you to write operators without actually writing a lot of imperative code that interacts with the k8s API, but instead just provide a general JSON->JSON transformer, which you could write in any langue (Go, Python, Rust, Javascript, .... and also Jsonnet if you want). I recently implemented something similar but much tailored to just "installing" stuff, called Kubit. An OCI artifact contains some abitrary tarball (generally containing some template sources) and a reference to a docker image containing an "engine" and runs the engine with your provided tarball + some parameters passed in a CRD. The OCI artifact could contain a helm chart and the template engine could contain the helm binary, or the template engine could be kubecfg and the OCI artifact could contain a bunch of jsonnet files. Or you could write your own stuff in python or typescript. The kubit operator then just runs your code, gathers the output and applies with with kubectl apply-set. 1. https://metacontroller.github.io/metacontroller/intro.html https://metacontroller.github.io/metacontroller/intro.html 2. https://github.com/kubecfg/kubit https://github.com/kubecfg/kubit
liveoneggs 3y ago
k8s make me miss xml
fsniper 3y ago
For Helm the value is it is not a configuration managemet solution but a package manager. The rest are just methods of writing json/yaml. I understand the "hate" against yaml, But I don't think it's deserving it that much. Perhaps timoni will take over with it's usage of cue. At least it's a package management solution.
ryandv 3y ago
> The Kubernetes API is fairly straightforward, and has a well-defined (JSON) schema, people should be spending a bulk of their time learning k8s understanding how to use the API, but instead they spend it working out how to use a Helm chart. This is a general pattern in software. Instead of learning the primitives and fundamentals that your system is built on, which would be too hard, instead learn a bunch of abstractions over top of it. Sure, now you are insulated from the lower-level details of the system, but now you have to deal with a massive stack of abstractions that makes diagnosis and debugging difficult once something goes wrong. Now it's much harder to ascertain what exactly is happening in your system, since the details of what is actually going on have been abstracted away from you by design. Further, you are now dependent on that abstraction layer and must support and accommodate whatever updates may be released by the vendor, in addition to whatever else is lurking in your dependency graph.
DrBazza 3y ago
We're using jsonnet for our systems and they have absolutely nothing to do with k8s. I'm not sure it's true to say it has ever gained much traction. It's just a niche case for complex configuration, and isn't the most publicised tool. It does precisely what we need with zero fuss, cross platform and cross _language_ (we've embedded it in C++, .NET, and JVM executables). We can use the resulting json config with a vast array of tools that simply don't exist for the alternatives such toml/yaml/hocon/ini whatever. In fact we tried to get HOCON working for non-JVM languages but there was always some edge case.
sanderjd 3y ago
Yep, I find kustomize and (especially) helm so confusing, while finding kubernetes yaml files very easy to use and understand.

baq 3y ago

probably doesn't meet the 2nd requirement, most definitely doesn't meet the third, but: https://cdk8s.io/docs/latest/ https://cdk8s.io/docs/latest/

TheFuzzball 3y ago
The second requirement is actually probably the most important - if someone that just set up ArgoCD, Flux, or has their own GitOps pipeline, how much of a headache does using a new compile step present? Lots of things are simple in isolation: want to use Cue? Just get your definitions and install the compiler and call it and boom, there are your k8s defs! Ok, but how do I integrate all of that into my existing toolchain? How do I pass config? Etc, etc. The best, fastest tool won't win. The tool that has the most frictionless user story will.
- shepherdjerred 3y ago
  I was able to get CDK8s working easily by simply committing the built template along with my TypeScript. Then, I just pointed ArgoCD to my repo.
  - how_gauche 3y ago
    We do the same thing but commit to a second git repo that we treat like the "k8s yaml release database".

Havoc 3y ago

How would one use the json api without ending up writing a bunch of custom code?

TheFuzzball 3y ago
I think custom code is to be expected, and making it maintainable is what's important. > everything should be made as simple as possible, but no simpler. Helm et al made it simpler than it was, IMO.
- Havoc 3y ago
  Everyone hand rolling code does not seem like an improvement over tools like helm even if it’s yaml
  - TheFuzzball 3y ago
    No, obviously not, and that's not what I've suggested.
- rad_gruchalski 3y ago
  Helm is another can of hot garbage. Impossible to vendor without hitting name collisions, can configure only what’s templated. Jsonnet is the way to go with generated helm manifests transformed later. Kustomize with its post-renderer hooks is another can of even hotter garbage.
  - aeyes 3y ago
    > Impossible to vendor without hitting name collisions What problem exactly are you facing? I can change the name of the chart itself in chart.yaml and if the name of the resources collide I change them with nameOverride/fullnameOverride in the values. All charts have these because they are autogenerated by `helm create`. I vendor all charts and never had this problem.
    - rad_gruchalski 3y ago

ithkuil 3y ago

I love the idea of keeping it simple and I do try to use kustomize or even plain yaml as installation method as much as possible. But in practice when managing large systems you inevitably end up benefiting from templating

worldsayshi 3y ago
I've begun thinking that if you start thinking about templating you might be better off building an operator. Operators aren't as well understood and documented. But in my mind an operator is just a pod or deployment that creates on demand resources using the k8s api.
- ryandv 3y ago
  The purpose of an Operator is to realize the resources desired/requested in a (custom) resource manifest, often as YAML or JSON. You give the apiserver a document describing what resources you need. The Operator actually does the work of provisioning those resources in the "real world" and (should) update the status field on the API object to indicate if those resources are ready.
- ithkuil 3y ago
  oh yeah; operators are great and sometimes they are necessary. On the other hand, most operators I've seen are just k8s manifest templates implemented in Go. I often end up preferring using Jsonnet to deal with that instead of doing the same stuff in Go. Jsonnet is much more close to the underlying datamodel (the k8s manifest Json/Yaml document) and comes with some useful functionality out of the box, such "overlays". It has downsides too! It's untyped, debugging tools are lacking, people are unfamiliar with it and don't care to learn it. So I totally get why one would entertain the possibility of writing your "templates" using a better language. However, an operator is often too much freedom. It's not just using Go or Rust or Typescript to "generate" some Json manifests, but it also contains the code to interact with the API server, setup watches, and reactions etc. I often wish there was a better way to separate those two concerns I'm a fan of metacontroller [1], which is a tool that allows you to write operators without actually writing a lot of imperative code that interacts with the k8s API, but instead just provide a general JSON->JSON transformer, which you could write in any langue (Go, Python, Rust, Javascript, .... and also Jsonnet if you want). I recently implemented something similar but much tailored to just "installing" stuff, called Kubit. An OCI artifact contains some abitrary tarball (generally containing some template sources) and a reference to a docker image containing an "engine" and runs the engine with your provided tarball + some parameters passed in a CRD. The OCI artifact could contain a helm chart and the template engine could contain the helm binary, or the template engine could be kubecfg and the OCI artifact could contain a bunch of jsonnet files. Or you could write your own stuff in python or typescript. The kubit operator then just runs your code, gathers the output and applies with with kubectl apply-set. 1. https://metacontroller.github.io/metacontroller/intro.html https://metacontroller.github.io/metacontroller/intro.html 2. https://github.com/kubecfg/kubit https://github.com/kubecfg/kubit

liveoneggs 3y ago

k8s make me miss xml

fsniper 3y ago

For Helm the value is it is not a configuration managemet solution but a package manager. The rest are just methods of writing json/yaml. I understand the "hate" against yaml, But I don't think it's deserving it that much. Perhaps timoni will take over with it's usage of cue. At least it's a package management solution.

ryandv 3y ago

> The Kubernetes API is fairly straightforward, and has a well-defined (JSON) schema, people should be spending a bulk of their time learning k8s understanding how to use the API, but instead they spend it working out how to use a Helm chart. This is a general pattern in software. Instead of learning the primitives and fundamentals that your system is built on, which would be too hard, instead learn a bunch of abstractions over top of it. Sure, now you are insulated from the lower-level details of the system, but now you have to deal with a massive stack of abstractions that makes diagnosis and debugging difficult once something goes wrong. Now it's much harder to ascertain what exactly is happening in your system, since the details of what is actually going on have been abstracted away from you by design. Further, you are now dependent on that abstraction layer and must support and accommodate whatever updates may be released by the vendor, in addition to whatever else is lurking in your dependency graph.

DrBazza 3y ago

We're using jsonnet for our systems and they have absolutely nothing to do with k8s. I'm not sure it's true to say it has ever gained much traction. It's just a niche case for complex configuration, and isn't the most publicised tool. It does precisely what we need with zero fuss, cross platform and cross _language_ (we've embedded it in C++, .NET, and JVM executables). We can use the resulting json config with a vast array of tools that simply don't exist for the alternatives such toml/yaml/hocon/ini whatever. In fact we tried to get HOCON working for non-JVM languages but there was always some edge case.

sanderjd 3y ago

Yep, I find kustomize and (especially) helm so confusing, while finding kubernetes yaml files very easy to use and understand.