5 ms·

The wide adoption of YAML for devops adjacent tooling was a mistake. I think proper programming language support is the way to go. Ideally a static type syste

The wide adoption of YAML for devops adjacent tooling was a mistake. I think proper programming language support is the way to go. Ideally a static type system that isn't turing complete and guaranteed to terminate. So something like starlark with types.

dventimi 2y ago
> I think proper programming language support is the way to go. Personally, I would prefer a SQLite database. Ok I'll show myself out.
verdverm 2y ago
Have you looked at CUE? (https://cuelang.org/docs/concept/the-logic-of-cue/ https://cuelang.org/docs/concept/the-logic-of-cue/) CUE is also pragmatic in that it has integrations with yaml, json, jsonschema, openapi, protobuf
- dewbrite 2y ago
  I've tried out Pkl which is similar in spirit, and I think it's a real solution for k8s manifests. The only thing holding it back is industry adoption imo. It's leagues better than Helm, and mostly better than Kustomize. See also: KCL, which is very similar and might _actually_ be "the winner". Time will tell.
  - verdverm 2y ago
    I don't expect a winner personally, rather that there will be dozens of alternatives always. Like build systems, deployments are quite bespoke to organizations and legacy has a way of sticking around for a long time Having used CUE, mainly outside of Kubernetes, I cannot see myself switching to KCL. I really like having a configuration language that isn't so tied to a specific system and which I can use with the Go SDK
    - pmig 2y ago
      So whats your take on https://github.com/stripe/skycfg https://github.com/stripe/skycfg do you also have experience with it?
      - verdverm 2y ago
        No, I went with CUE instead of Starlark
dilyevsky 2y ago
> So something like starlark with types. this exists for k8s[0]. there have been other users based on the same library[1], I heard reddit did something similar internally [0] - https://github.com/cruise-automation/isopod https://github.com/cruise-automation/isopod [1] - https://github.com/stripe/skycfg https://github.com/stripe/skycfg
- kminehart 2y ago
  If I was in charge of our infra automation I would have done this. We opted for jsonnet instead which is an absolute nightmare, or at least the way we've set it up is.
davidmdm91 2y ago
I have been developing my own package manager, and my core idea is that proper programming languages are the proper level for describing packages. Programs take inputs and can output arbitrary data such as resources. However they can do so with type safety, and everything else a programming ecosystem can achieve. For asset distribution it uses wasm, and that's it! If you want to check it out its here: github: (https://github.com/davidmdm/yoke https://github.com/davidmdm/yoke) docs: (https://davidmdm.github.io/yoke-website https://davidmdm.github.io/yoke-website) I like that you said: > I think proper programming language support is the way to go. I think we need to stop writing new ways of generating yaml since we already have the perfect way of doing so. Typed languages!
- verdverm 2y ago
  > that isn't turing complete and guaranteed to terminate This means general purpose languages do not qualify, and more generally, no general recursion
  - davidmdm91 2y ago
    Why limit yourself to those types of tools? To protect against somebody writing a non-terminating program? General programming languages come with a lot of general purpose benefits from their ecosystems like package managers npm, cargo, go modules, etc. They have test runners, and control flow. Lots of them already have type definitions for kubernetes and if you are working in Go you have access to almost the entire kubernetes ecosystem. Maybe we are throwing the baby out with the bath water when we disqualify general purpose languages?
    - verdverm 2y ago
      Because people write code that is hard to understand. Configuration doesn't need all that. What it needs is to be provably correct and easy for someone to make predictable changes under high pressure (when prod is down). The non-terminating thing is one of the features of a turing incomplete language, not the goal. You don't want inheritance either, because it becomes hard to know where and when a value gets set (which is what helm overlay via multiple -f uses effectively is) You speak like turning incomplete languages cannot have the control structures, tooling, and ecosystems we enjoy elsewhere, which would be the wrong assessment. I recommend you take a look at CUE to see how this can be true The OpenAPI specs are probably better than the Go language types for k8s. They have more of the validation information and you can get at the CRDs / versions actually running in the cluster.
ants_everywhere 2y ago
The idea of declarative config is that empirically programmatic config was bad at scale. If your config is the source of truth of what your infra should be, then you can use source control tools to roll back to a known good state, or to binary search for when a problem was introduced. If you use programmatic config, then you can't find out the intended state of your system without executing a program. You can't grep through program executions in a meaningful way, especially at scale. So you can't do even simple things like search for a string. Guaranteeing termination is helpful, but it doesn't solve the main problem that programmatic config puts a large complexity barrier between you and the ability to understand your infrastructure. Tools like Helm give up a fair amount of this declarative benefit. And IMO that's one of the reasons why it's almost always a worse experience to use a helm chart than to just render the chart once and for all and forget Helm ever existed.
- nijave 2y ago
  I think codegen/compilation is a middle ground here. A higher level language like starlark can be compiled down to a set of instructions that provide the described guarantees. This is how Pants (build system) works. You have declarative Starlark which supports basic programming semantics and this generates a state the engine reads and tries to produce. I've been meaning to dive into jsonnet for a while but it'd be good to have a higher level representation that didn't rely on sophisticated templating and substitution engines like current k8s. Compare k8s to Terraform where you have modules, composability, variables. These can be achieved in k8s but you need to layer more tooling on (kustomize, helm, etc). There could be a richer config system than "shove it in YAML" Things like explicit ordering and dependencies are hard to represent in pure yaml since they're ",just text fields" without additional tools
- nostrebored 2y ago
  > The idea of declarative config is that empirically programmatic config was bad at scale. Languages can be declarative or imperative. For instance, Pulumi and CDK are declarative. > If you use programmatic config, then you can't find out the intended state of your system without executing a program. You can't grep through program executions in a meaningful way, especially at scale. So you can't do even simple things like search for a string. I don’t understand — nothing stops a language from having an intermediate compilation step that shows the intended state and is searchable. Beyond that, programmatic config means you can add in hooks to make plans or runs more interrogatable. > Guaranteeing termination is helpful, but it doesn't solve the main problem that programmatic config puts a large complexity barrier between you and the ability to understand your infrastructure. It seems like this is untrue — having seen templated IaC that is hundreds of thousands of lines and cdk that defers that complexity to an abstraction that I have to understand once, I’d always take the latter. Agreed that helm use is a giant mistake and liability.
okamiueru 2y ago
My take on this is that the issue is not declarative infrastructure resources, but a tendency to over-complicate the infrastructure. For example: You have a problem that is suitable for some message queue -> Apache Kafka. Now you have 7 new problems and the complexity warrants perhaps 3 other services, and on, and on.
- pmig 2y ago
  Introducing complexity is always something that needs to be introduced carefully. It makes things harder if you introduce it to early, but everything will break in a big bang if you introduce it too late. Nowadays you can also start with a light weight MQ like Rabbit MQ and decouple your service just into a hand full components. This will set you up for scalability without introducing massive overheads. In end it is also always a knowledge game. How experienced are you or how much time are you willing to invest into learning and understanding a technology.
Aeolun 2y ago
I think you can use Pulumi for helm? Or maybe just straight up kube.
oldpersonintx 2y ago
[dead]
shepherdjerred 2y ago
cdk8s + TypeScript is my favorite option. Here's how I use it: https://github.com/shepherdjerred/homelab/tree/main/cdk8s https://github.com/shepherdjerred/homelab/tree/main/cdk8s https://github.com/cdk8s-team/cdk8s https://github.com/cdk8s-team/cdk8s
taspeotis 2y ago
Yelling At My Laptop

dventimi 2y ago

> I think proper programming language support is the way to go. Personally, I would prefer a SQLite database. Ok I'll show myself out.

verdverm 2y ago

Have you looked at CUE? (https://cuelang.org/docs/concept/the-logic-of-cue/ https://cuelang.org/docs/concept/the-logic-of-cue/) CUE is also pragmatic in that it has integrations with yaml, json, jsonschema, openapi, protobuf

dewbrite 2y ago
I've tried out Pkl which is similar in spirit, and I think it's a real solution for k8s manifests. The only thing holding it back is industry adoption imo. It's leagues better than Helm, and mostly better than Kustomize. See also: KCL, which is very similar and might _actually_ be "the winner". Time will tell.
- verdverm 2y ago
  I don't expect a winner personally, rather that there will be dozens of alternatives always. Like build systems, deployments are quite bespoke to organizations and legacy has a way of sticking around for a long time Having used CUE, mainly outside of Kubernetes, I cannot see myself switching to KCL. I really like having a configuration language that isn't so tied to a specific system and which I can use with the Go SDK
  - pmig 2y ago
    So whats your take on https://github.com/stripe/skycfg https://github.com/stripe/skycfg do you also have experience with it?
    - verdverm 2y ago
      No, I went with CUE instead of Starlark

dilyevsky 2y ago

> So something like starlark with types. this exists for k8s[0]. there have been other users based on the same library[1], I heard reddit did something similar internally [0] - https://github.com/cruise-automation/isopod https://github.com/cruise-automation/isopod [1] - https://github.com/stripe/skycfg https://github.com/stripe/skycfg

kminehart 2y ago
If I was in charge of our infra automation I would have done this. We opted for jsonnet instead which is an absolute nightmare, or at least the way we've set it up is.

davidmdm91 2y ago

I have been developing my own package manager, and my core idea is that proper programming languages are the proper level for describing packages. Programs take inputs and can output arbitrary data such as resources. However they can do so with type safety, and everything else a programming ecosystem can achieve. For asset distribution it uses wasm, and that's it! If you want to check it out its here: github: (https://github.com/davidmdm/yoke https://github.com/davidmdm/yoke) docs: (https://davidmdm.github.io/yoke-website https://davidmdm.github.io/yoke-website) I like that you said: > I think proper programming language support is the way to go. I think we need to stop writing new ways of generating yaml since we already have the perfect way of doing so. Typed languages!

verdverm 2y ago
> that isn't turing complete and guaranteed to terminate This means general purpose languages do not qualify, and more generally, no general recursion
- davidmdm91 2y ago
  Why limit yourself to those types of tools? To protect against somebody writing a non-terminating program? General programming languages come with a lot of general purpose benefits from their ecosystems like package managers npm, cargo, go modules, etc. They have test runners, and control flow. Lots of them already have type definitions for kubernetes and if you are working in Go you have access to almost the entire kubernetes ecosystem. Maybe we are throwing the baby out with the bath water when we disqualify general purpose languages?
  - verdverm 2y ago
    Because people write code that is hard to understand. Configuration doesn't need all that. What it needs is to be provably correct and easy for someone to make predictable changes under high pressure (when prod is down). The non-terminating thing is one of the features of a turing incomplete language, not the goal. You don't want inheritance either, because it becomes hard to know where and when a value gets set (which is what helm overlay via multiple -f uses effectively is) You speak like turning incomplete languages cannot have the control structures, tooling, and ecosystems we enjoy elsewhere, which would be the wrong assessment. I recommend you take a look at CUE to see how this can be true The OpenAPI specs are probably better than the Go language types for k8s. They have more of the validation information and you can get at the CRDs / versions actually running in the cluster.

ants_everywhere 2y ago

The idea of declarative config is that empirically programmatic config was bad at scale. If your config is the source of truth of what your infra should be, then you can use source control tools to roll back to a known good state, or to binary search for when a problem was introduced. If you use programmatic config, then you can't find out the intended state of your system without executing a program. You can't grep through program executions in a meaningful way, especially at scale. So you can't do even simple things like search for a string. Guaranteeing termination is helpful, but it doesn't solve the main problem that programmatic config puts a large complexity barrier between you and the ability to understand your infrastructure. Tools like Helm give up a fair amount of this declarative benefit. And IMO that's one of the reasons why it's almost always a worse experience to use a helm chart than to just render the chart once and for all and forget Helm ever existed.

nijave 2y ago
I think codegen/compilation is a middle ground here. A higher level language like starlark can be compiled down to a set of instructions that provide the described guarantees. This is how Pants (build system) works. You have declarative Starlark which supports basic programming semantics and this generates a state the engine reads and tries to produce. I've been meaning to dive into jsonnet for a while but it'd be good to have a higher level representation that didn't rely on sophisticated templating and substitution engines like current k8s. Compare k8s to Terraform where you have modules, composability, variables. These can be achieved in k8s but you need to layer more tooling on (kustomize, helm, etc). There could be a richer config system than "shove it in YAML" Things like explicit ordering and dependencies are hard to represent in pure yaml since they're ",just text fields" without additional tools
nostrebored 2y ago
> The idea of declarative config is that empirically programmatic config was bad at scale. Languages can be declarative or imperative. For instance, Pulumi and CDK are declarative. > If you use programmatic config, then you can't find out the intended state of your system without executing a program. You can't grep through program executions in a meaningful way, especially at scale. So you can't do even simple things like search for a string. I don’t understand — nothing stops a language from having an intermediate compilation step that shows the intended state and is searchable. Beyond that, programmatic config means you can add in hooks to make plans or runs more interrogatable. > Guaranteeing termination is helpful, but it doesn't solve the main problem that programmatic config puts a large complexity barrier between you and the ability to understand your infrastructure. It seems like this is untrue — having seen templated IaC that is hundreds of thousands of lines and cdk that defers that complexity to an abstraction that I have to understand once, I’d always take the latter. Agreed that helm use is a giant mistake and liability.

okamiueru 2y ago

My take on this is that the issue is not declarative infrastructure resources, but a tendency to over-complicate the infrastructure. For example: You have a problem that is suitable for some message queue -> Apache Kafka. Now you have 7 new problems and the complexity warrants perhaps 3 other services, and on, and on.

pmig 2y ago
Introducing complexity is always something that needs to be introduced carefully. It makes things harder if you introduce it to early, but everything will break in a big bang if you introduce it too late. Nowadays you can also start with a light weight MQ like Rabbit MQ and decouple your service just into a hand full components. This will set you up for scalability without introducing massive overheads. In end it is also always a knowledge game. How experienced are you or how much time are you willing to invest into learning and understanding a technology.

Aeolun 2y ago

I think you can use Pulumi for helm? Or maybe just straight up kube.

oldpersonintx 2y ago

[dead]

shepherdjerred 2y ago

cdk8s + TypeScript is my favorite option. Here's how I use it: https://github.com/shepherdjerred/homelab/tree/main/cdk8s https://github.com/shepherdjerred/homelab/tree/main/cdk8s https://github.com/cdk8s-team/cdk8s https://github.com/cdk8s-team/cdk8s

taspeotis 2y ago

Yelling At My Laptop