16 ms·

Validating Kubernetes YAML for best practice and policies

83 points by kiyanwang 6y ago

jasonlotito 6y ago
You are using YAML already. How much do you care about best practices. Only partially kidding here.
- EdwardDiego 6y ago
  You're currently being downvoted, but I agree, YAML is kinda terrible, not sure why anyone thought Python's syntactically relevant whitespace was ideal for a config file. Classic example: - containerPort: 7173 name: http I think that's an object in a list? But it's not overly clear.And if I indented any of those lines wrong...
  - corty 6y ago
    also, keywords of the underlying software mechanisms are decoupled, so uncheckable for plain YAML tooling. Whereas with XML you can at least infer a lot about the desired structure and keywords from the schema. Deeper checks are only possible with "real" programming languages, preferrably statically typed ones. I'm wondering when that wisdom trickles down to configuration languages.
  - dmitriid 6y ago
    Don't forget the plethora of truthy values that put Javascript to shame. - country: NO Of course it's boolean
    - riffraff 6y ago
      that was changed in YAML 1.2[0], which I think is many years old, but people don't specify a version on config files anyway and I'm not even sure most parsers respect it, so it keeps popping up, which is sad. [0] https://yaml.org/spec/1.2/spec.html#id2803629 https://yaml.org/spec/1.2/spec.html#id2803629
      - dmitriid 6y ago
        That's the problem with bad formats: they live forever. They and any number of parsers for any version between version 0 and now.
  - pydry 6y ago
    >not sure why anyone thought Python's syntactically relevant whitespace was ideal for a config file. Because it reduces syntactic noise and increases informational density. Most other serialization formats end up putting in indentation anyway to make them readable (JSON, TOML).
  - aliswe 6y ago
    That's a map as a list element, iirc the terminology. But check out this. When studying the Yaml spec I discovered that a map property (key: value) can have not only a string as its key, but any value. Even a list. (cue screams)
    - shahsyed 6y ago
      Here's another reason why YAML is bad: the fact that I have to read the spec to figure out what's going on. It should have been fairly simple to do this. Why do I have to know the difference between ":" and "=" ...?
    - Already__Taken 6y ago
      Why didn't anyone learn from CSS when we all made pre-processors like stylus, less and sass that added white-space sensitivity. It's not great. I thought HCL was one of the best attempts at a terse config language. Shortening foo { bar { bazz {} }} to foo bar baz{} cleans up plenty.
- corty 6y ago
  YAML is unusable without a sufficiently capable linter. Syntax is very fragile and completely decoupled from function.
- shahsyed 6y ago
  You're partially kidding but you're 100% right. Someone in another thread mentioned using some LISP or LISP-like language to solve this problem. Would something like TOML make things better here? What would have been the solution?
  - pjmlp 6y ago
    XML, but I guess I won't get many friends for asserting that.
    - secondcoming 6y ago
      Nope. Json has schema validation too, and every language has a json parser
      - pjmlp 6y ago
        JSON doesn't do comments.
        secondcoming 6y ago
        Not officially, but it's fakeable
        pjmlp 6y ago
        Which for me is a design smell, why bother with workarounds when XML already offers me all the best tooling out of the box.
        secondcoming 6y ago
        For me, xml is just too much, and when working with C or C++ (not t sure about rust) it's just a pain. Can your json schema file not serve as the documentation?
        6y ago
shimont 6y ago
I think that this is a great approach to test out the files. Mistakes in those files can cause a production outage. I like doing those tests once a PR is open and before it is merged into master and executed on the production cluster. (Disclaimer i am a co-founder of datree.io)
EdwardDiego 6y ago
I find Intellij's K8s plugin really helpful for identifying issues within a single K8s YAML file, but it won't find things like a deployment.yaml without a pdb.yaml but it's a good start.
default-kramer 6y ago
> Config-lint is a promising framework that lets you write custom checks for Kubernetes YAML manifests using a YAML DSL. But what if you want to express more complex logic and checks? Isn't YAML too limiting for that? What if you could express those checks with a real programming language? Having recently worked a little bit with YAML for Kubernetes and HCL for Terraform, I really wish they had both just used "a real programming language" right from the start. I'll choose Racket because I know it best, but there are probably many languages that would work well. You could expose very nearly the same configuration language, but backed by a real programming language. I bet this would make some of the tools the author lists at the end (eg copper, config-lint) much easier to write, or perhaps not necessary at all. And the author didn't mention Helm, but I will. The part of Helm I saw seemed to be a lot of work just to add "functions with parameters" to Kubernetes YAML, something we could have had for free using "a real programming language" from the start: https://helm.sh/docs/chart_template_guide/functions_and_pipelines/ https://helm.sh/docs/chart_template_guide/functions_and_pipe... Why are so few configuration languages not backed by a real language?
- Znafon 6y ago
  > Why are so few configuration languages not backed by a real language? In many cases, not having a full featured language is helpful as you have some additional guarantees that comes with a non Turing complete language like guaranteed completion. In some cases though, you do need a full pledged programming language. For those cases, HashiCorp recently announced CDK support for Terraform: https://www.hashicorp.com/blog/cdk-for-terraform-enabling-python-and-typescript-support/ https://www.hashicorp.com/blog/cdk-for-terraform-enabling-py...
  - default-kramer 6y ago
    Cool, I hadn't seen CDK for Terraform before. But doesn't this support my point? If they had just used Typescript from the start, they wouldn't have to add Typescript support later. I suppose guaranteed completion matters if you are running untrusted code, but wouldn't sandboxing solve that? Are there any other guarantees that sandboxing wouldn't solve?
shahsyed 6y ago
I'm having some arguments with other developers (devs) on whether or not this is important. I'm gonna finally try to implement this for my own pipeline this week, hopefully. I would much rather have devs double check/validate things locally before they edit changes. Modifying config files by using the edit text feature in GitHub (GH), doesn't enable you to do that. & Devs are lazy. I'm lazy. They want things easy. Me too. So let's make it easy. Modify your CI/CD pipeline to validate YAML configs on any file changes (use GH hooks for example) Now devs can do whatever they want - if their pre-deployment checks fail, go back and fix it!
- JamesSwift 6y ago
  This is a good approach because it focuses on the desired outcome ('no invalid configs get deployed'), and doesn't try to use a proxy ('you have to validate locally') to get there.
- Znafon 6y ago
  This is a very sensible approach. One pro of having the checks automated instead of just having the developers check carefully their changes is that onboarding a new developer is easier, you will spend less time on very small and specific details and you won't forget to tell some detail.
- gk1 6y ago
  You're basically describing Sentinel for Terraform (https://www.hashicorp.com/sentinel/ https://www.hashicorp.com/sentinel/) or Datree for Kubernetes (https://www.datree.io https://www.datree.io). There are also a bunch of tools popping up in this space that focus on catching security issues rather than misconfigurations.
pjmlp 6y ago
And so the circle of schema validation does yet another turn, now back to those XML config files.
- juped 6y ago
  Every time I see a JSON config file I miss those kilobytes of XML.
pbiggar 6y ago
I recently had a bug that wouldn't have happened if I'd had these in place: https://dev.to/darklang/a-fun-bug-55cl https://dev.to/darklang/a-fun-bug-55cl. I added similar checks (and kube-score and polaris seem like good tools - I might try adding them).

83 points by kiyanwang 6y ago

jasonlotito 6y ago

You are using YAML already. How much do you care about best practices. Only partially kidding here.

EdwardDiego 6y ago
You're currently being downvoted, but I agree, YAML is kinda terrible, not sure why anyone thought Python's syntactically relevant whitespace was ideal for a config file. Classic example: - containerPort: 7173 name: http I think that's an object in a list? But it's not overly clear.And if I indented any of those lines wrong...
- corty 6y ago
  also, keywords of the underlying software mechanisms are decoupled, so uncheckable for plain YAML tooling. Whereas with XML you can at least infer a lot about the desired structure and keywords from the schema. Deeper checks are only possible with "real" programming languages, preferrably statically typed ones. I'm wondering when that wisdom trickles down to configuration languages.
- dmitriid 6y ago
  Don't forget the plethora of truthy values that put Javascript to shame. - country: NO Of course it's boolean
  - riffraff 6y ago
    that was changed in YAML 1.2[0], which I think is many years old, but people don't specify a version on config files anyway and I'm not even sure most parsers respect it, so it keeps popping up, which is sad. [0] https://yaml.org/spec/1.2/spec.html#id2803629 https://yaml.org/spec/1.2/spec.html#id2803629
    - dmitriid 6y ago
      That's the problem with bad formats: they live forever. They and any number of parsers for any version between version 0 and now.
- pydry 6y ago
  >not sure why anyone thought Python's syntactically relevant whitespace was ideal for a config file. Because it reduces syntactic noise and increases informational density. Most other serialization formats end up putting in indentation anyway to make them readable (JSON, TOML).
- aliswe 6y ago
  That's a map as a list element, iirc the terminology. But check out this. When studying the Yaml spec I discovered that a map property (key: value) can have not only a string as its key, but any value. Even a list. (cue screams)
  - shahsyed 6y ago
    Here's another reason why YAML is bad: the fact that I have to read the spec to figure out what's going on. It should have been fairly simple to do this. Why do I have to know the difference between ":" and "=" ...?
  - Already__Taken 6y ago
    Why didn't anyone learn from CSS when we all made pre-processors like stylus, less and sass that added white-space sensitivity. It's not great. I thought HCL was one of the best attempts at a terse config language. Shortening foo { bar { bazz {} }} to foo bar baz{} cleans up plenty.
corty 6y ago
YAML is unusable without a sufficiently capable linter. Syntax is very fragile and completely decoupled from function.
shahsyed 6y ago
You're partially kidding but you're 100% right. Someone in another thread mentioned using some LISP or LISP-like language to solve this problem. Would something like TOML make things better here? What would have been the solution?
- pjmlp 6y ago
  XML, but I guess I won't get many friends for asserting that.
  - secondcoming 6y ago
    Nope. Json has schema validation too, and every language has a json parser
    - pjmlp 6y ago
      JSON doesn't do comments.
      - secondcoming 6y ago
        Not officially, but it's fakeable
        pjmlp 6y ago
        Which for me is a design smell, why bother with workarounds when XML already offers me all the best tooling out of the box.
        secondcoming 6y ago
        For me, xml is just too much, and when working with C or C++ (not t sure about rust) it's just a pain. Can your json schema file not serve as the documentation?
        6y ago

shimont 6y ago

I think that this is a great approach to test out the files. Mistakes in those files can cause a production outage. I like doing those tests once a PR is open and before it is merged into master and executed on the production cluster. (Disclaimer i am a co-founder of datree.io)

EdwardDiego 6y ago

I find Intellij's K8s plugin really helpful for identifying issues within a single K8s YAML file, but it won't find things like a deployment.yaml without a pdb.yaml but it's a good start.

default-kramer 6y ago

> Config-lint is a promising framework that lets you write custom checks for Kubernetes YAML manifests using a YAML DSL. But what if you want to express more complex logic and checks? Isn't YAML too limiting for that? What if you could express those checks with a real programming language? Having recently worked a little bit with YAML for Kubernetes and HCL for Terraform, I really wish they had both just used "a real programming language" right from the start. I'll choose Racket because I know it best, but there are probably many languages that would work well. You could expose very nearly the same configuration language, but backed by a real programming language. I bet this would make some of the tools the author lists at the end (eg copper, config-lint) much easier to write, or perhaps not necessary at all. And the author didn't mention Helm, but I will. The part of Helm I saw seemed to be a lot of work just to add "functions with parameters" to Kubernetes YAML, something we could have had for free using "a real programming language" from the start: https://helm.sh/docs/chart_template_guide/functions_and_pipelines/ https://helm.sh/docs/chart_template_guide/functions_and_pipe... Why are so few configuration languages not backed by a real language?

Znafon 6y ago
> Why are so few configuration languages not backed by a real language? In many cases, not having a full featured language is helpful as you have some additional guarantees that comes with a non Turing complete language like guaranteed completion. In some cases though, you do need a full pledged programming language. For those cases, HashiCorp recently announced CDK support for Terraform: https://www.hashicorp.com/blog/cdk-for-terraform-enabling-python-and-typescript-support/ https://www.hashicorp.com/blog/cdk-for-terraform-enabling-py...
- default-kramer 6y ago
  Cool, I hadn't seen CDK for Terraform before. But doesn't this support my point? If they had just used Typescript from the start, they wouldn't have to add Typescript support later. I suppose guaranteed completion matters if you are running untrusted code, but wouldn't sandboxing solve that? Are there any other guarantees that sandboxing wouldn't solve?

shahsyed 6y ago

I'm having some arguments with other developers (devs) on whether or not this is important. I'm gonna finally try to implement this for my own pipeline this week, hopefully. I would much rather have devs double check/validate things locally before they edit changes. Modifying config files by using the edit text feature in GitHub (GH), doesn't enable you to do that. & Devs are lazy. I'm lazy. They want things easy. Me too. So let's make it easy. Modify your CI/CD pipeline to validate YAML configs on any file changes (use GH hooks for example) Now devs can do whatever they want - if their pre-deployment checks fail, go back and fix it!

JamesSwift 6y ago
This is a good approach because it focuses on the desired outcome ('no invalid configs get deployed'), and doesn't try to use a proxy ('you have to validate locally') to get there.
Znafon 6y ago
This is a very sensible approach. One pro of having the checks automated instead of just having the developers check carefully their changes is that onboarding a new developer is easier, you will spend less time on very small and specific details and you won't forget to tell some detail.
gk1 6y ago
You're basically describing Sentinel for Terraform (https://www.hashicorp.com/sentinel/ https://www.hashicorp.com/sentinel/) or Datree for Kubernetes (https://www.datree.io https://www.datree.io). There are also a bunch of tools popping up in this space that focus on catching security issues rather than misconfigurations.

pjmlp 6y ago

And so the circle of schema validation does yet another turn, now back to those XML config files.

juped 6y ago
Every time I see a JSON config file I miss those kilobytes of XML.

pbiggar 6y ago

I recently had a bug that wouldn't have happened if I'd had these in place: https://dev.to/darklang/a-fun-bug-55cl https://dev.to/darklang/a-fun-bug-55cl. I added similar checks (and kube-score and polaris seem like good tools - I might try adding them).