6 ms·

Show HN: Holos – Configure Kubernetes with CUE data structures instead of YAML

77 points by JeffMcCune 2y ago

Hi HN! I’m excited to share Holos, a Go command line tool we wrote to fill the configuration management gap in Kubernetes. Holos uses CUE to configure software distributed with Helm and Kustomize using a well defined, type safe language eliminating the need to template YAML. You probably know (or are) someone who has suffered with the complexity of plain text YAML templates and merging multiple values.yaml files together to configure software running in Kubernetes. We built Holos so we don’t have to template YAML but we can still integrate software distributed with Helm and Kustomize holistically into one unified configuration. At the start of the pandemic I was migrating our platform to Kubernetes from virtual machines managed by Puppet. My primary goal was to build an observability system similar to what we had when we managed Puppet at Twitter prior to the acquisition. I started building the observability system with the official prometheus community charts [1], but quickly ran into issues where the individual charts didn’t work with each other. I was frustrated with how difficult it was to configure these charts. They weren’t well integrated, so I switched to the kube-prometheus-stack [2] umbrella chart which attempts to solve this integration problem. The umbrella chart got us further but we quickly ran into operational challenges. Upgrading the chart introduced breaking changes we couldn’t see until they were applied, causing incidents. We needed to manage secrets securely so we mixed in ExternalSecrets with many of the charts. We decided to handle these customizations by implementing the rendered manifests pattern [3] using scripts in our CI pipeline. These CI scripts got us further, but we found them costly to maintain. We needed to be careful to execute them with the same context they were executed in CI. We realized we were reinventing tools to manage a hierarchy of helm values.yaml files to inject into multiple charts. We saw the value in the rendered manifests pattern but could not find an agreed upon implementation. I’d been thinking about the comments from the Why are we templating YAML? [4][5] posts and wondering what an answer to this question would look like, so I built a Go command line tool to implement the pattern as a data pipeline. We still didn’t have a good way to handle the data values. We were still templating YAML which didn’t catch errors early enough. It was too easy to render invalid resources Kubernetes rejected. I searched for a solution to manage and merge helm values. A few HN comments mentioned CUE [6], and an engineer we worked with at Twitter used CUE to configure Envoy at scale, so I gave it a try. I quickly appreciated how CUE provides both strong type checking and validation of constraints, unifies all configuration data, and provides clarity into where values originate from. Take a look at Holos if you’re looking to implement the rendered manifests pattern or can’t shake that feeling it should be easier to integrate third party software into Kubernetes like we felt. We recently overhauled our docs to be easier to get started and work locally on your device. In the future we’re planning to use Holos much like Debian uses APT, to integrate open source software into a holistic k8s distribution. [1]: <https://github.com/prometheus-community/helm-charts https://github.com/prometheus-community/helm-charts> [2]: <https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack https://github.com/prometheus-community/helm-charts/tree/mai...> [3]: <https://akuity.io/blog/the-rendered-manifests-pattern https://akuity.io/blog/the-rendered-manifests-pattern> [4]: Why are we templating YAML? (2019) - <https://news.ycombinator.com/item?id=19108787 https://news.ycombinator.com/item?id=19108787> [5]: Why are we templating YAML? (2024) - <https://news.ycombinator.com/item?id=39101828 https://news.ycombinator.com/item?id=39101828> [6]: <https://cuelang.org/ https://cuelang.org/>

jtmcn 2y ago
We already have an existing project with a bunch of Helm charts deployed using ArgoCD. What would be the benefit of using Holos now?
- JeffMcCune 2y ago
  Thanks for asking! The teams we've worked with do one of two things when deploying Helm charts with ArgoCD. They either pass values directly to the chart from the Application resource, or they use scripts to merge values.yaml files together and pass them to the helm template command to implement the rendered manifests pattern. In both cases it's tedious to manage the helm values, and they're usually managed without strong type checking or having been integrated into your platform as a whole. For example, you might pass a domain name to one chart and another value derived from the domain name to another chart, but the two charts likely use different field names for the inputs so the values are often inconsistent. Holos uses CUE to unify the configuration into one holistic data structure, so we're able to look up data from well defined structures and pass them into Helm. We have an example of how this helps integrate the prometheus charts together at: https://holos.run/docs/v1alpha5/tutorial/helm-values/ https://holos.run/docs/v1alpha5/tutorial/helm-values/ This unification of configuration into one data structure isn't limited to Helm, you can produce resources for Kustomize from CUE in the same way, something that's otherwise quite difficult because Kustomize doesn't support templating. You can also mix-in resources to your existing Helm charts from CUE without needing to template yaml. This approach works equally well for both in-house Helm charts you may have created to deploy your own software, or third party charts you're using to deploy off the shelf software.
eloip 2y ago
This is wonderful, thank you! A relieve for devops/YAML engineers that need to reason about many key/values coming from many places. Because in the end this is all there is for the user interface of IaC/XaaS, k8s and all cloud apis. There was some effort for "configuration management" but few realizes the complexity, the many layers and aspects there is to "it". YAML ain't mearly enough... But the space of "configuration PLs" (Dhall,Nickel,Pkl,KCL,CUE,Jsonnet,etc.) is still young. Biggest problem I see is usability, CUE focuses on it so people shouldn't be afraid. But it is also little behind the others in term of features, but also have the greatest potential! IMO any new tool in the cloud space that uses code abstractions cannot be serious by not thinking about the language. Transitions may be though but they ough to happen.

77 points by JeffMcCune 2y ago

jtmcn 2y ago

We already have an existing project with a bunch of Helm charts deployed using ArgoCD. What would be the benefit of using Holos now?

JeffMcCune 2y ago
Thanks for asking! The teams we've worked with do one of two things when deploying Helm charts with ArgoCD. They either pass values directly to the chart from the Application resource, or they use scripts to merge values.yaml files together and pass them to the helm template command to implement the rendered manifests pattern. In both cases it's tedious to manage the helm values, and they're usually managed without strong type checking or having been integrated into your platform as a whole. For example, you might pass a domain name to one chart and another value derived from the domain name to another chart, but the two charts likely use different field names for the inputs so the values are often inconsistent. Holos uses CUE to unify the configuration into one holistic data structure, so we're able to look up data from well defined structures and pass them into Helm. We have an example of how this helps integrate the prometheus charts together at: https://holos.run/docs/v1alpha5/tutorial/helm-values/ https://holos.run/docs/v1alpha5/tutorial/helm-values/ This unification of configuration into one data structure isn't limited to Helm, you can produce resources for Kustomize from CUE in the same way, something that's otherwise quite difficult because Kustomize doesn't support templating. You can also mix-in resources to your existing Helm charts from CUE without needing to template yaml. This approach works equally well for both in-house Helm charts you may have created to deploy your own software, or third party charts you're using to deploy off the shelf software.

eloip 2y ago

This is wonderful, thank you! A relieve for devops/YAML engineers that need to reason about many key/values coming from many places. Because in the end this is all there is for the user interface of IaC/XaaS, k8s and all cloud apis. There was some effort for "configuration management" but few realizes the complexity, the many layers and aspects there is to "it". YAML ain't mearly enough... But the space of "configuration PLs" (Dhall,Nickel,Pkl,KCL,CUE,Jsonnet,etc.) is still young. Biggest problem I see is usability, CUE focuses on it so people shouldn't be afraid. But it is also little behind the others in term of features, but also have the greatest potential! IMO any new tool in the cloud space that uses code abstractions cannot be serious by not thinking about the language. Transitions may be though but they ough to happen.