5 ms·

My conclusion is that using Ansible with Kubernetes is the WRONG way to go about it. So now instead of a directory of yaml or helm charts we have a bunch of ans

My conclusion is that using Ansible with Kubernetes is the WRONG way to go about it. So now instead of a directory of yaml or helm charts we have a bunch of ansible uselessly wrapping the yaml for application.

Spivak 4y ago
The problem is when you're in a situation where "over here we need YAML document A, and over there we need YAML document B." You either shove a templating system in front of your YAML and fight with that or throw in the towel just use a programming language where constructs in your YAML map to objects.
- hdjjhhvvhga 4y ago
  Helm3 made it a solved problem years ago.
- unxdfa 4y ago
  I think we need to burn this whole thing down and rethink it at this point. A huge amount of the total operational cost of our business is working around weird YAML problems and helm charts. Had one a while back where quoted strings containing padded numbers in were used for something. "01" fine ... "05" fine, "06" fine, "07" fine, "08" kaboom. 01-07 were treated as strings and 08 was suddenly being treated as octal, which it's not and exploded.
  - jpgvm 4y ago
    Helm is a pile of garbage. k8s speaks JSON just fine as expected. Try Tanka + Jsonnet to restore some of your sanity, also allows you to keep using Helm during your transition to a better place.
    - unxdfa 4y ago
      This is just the same problem in JSON, another loose schema’ed mess. Got my fair share of problems with that at the moment as well.
      - jpgvm 4y ago
        If you feel really strongly about getting type safety then there is Pulumi + Typescript though I really hate how it tries to be a "better" kubectl apply and fails miserably. Definitely room for someone to make something better than Helm but I think you should give Jsonnet a shot if you haven't already.
        jen20 4y ago
        > tries to be a "better" kubectl apply and fails How does it try? How does it fail? I don't see any interactive CLI commands that could be seen as a replacement for kubectl.
        jpgvm 4y ago
        It's not about the commands. `pulumi up` is what you are looking for and the k8s resources that accompany it. kubectl apply, especially with server side apply is effective just a mechanism to push the declarative definition of your k8s objects to the server. Pulumi instead tries to manage the lifecycle of each of those objects itself when it really has no business doing so. This results in a lot of roundtrips to the Pulumi state store (which is implemented horrendously inefficiently btw) and generally takes forever and a day compared to kubectl apply. Most of this is down to an impedance mismatch between the models themselves. Pulumi + k8s works but it's just not a great experience. I say this with an absolute -ton- of experience with both options.
        jaxxstorm 4y ago
        This can be turned off by setting an annotation on the resources.
        jpgvm 4y ago
        I'm aware but it's not really ergonomic. For instance Pulumi also generates random names for resources, again you can override these but again, not very ergonomic to work around etc. IMO tools like Tanka + Jsonnet, or Cue are just better fit for k8s because you can stick with server side apply which is a much more native experience.

Spivak 4y ago

The problem is when you're in a situation where "over here we need YAML document A, and over there we need YAML document B." You either shove a templating system in front of your YAML and fight with that or throw in the towel just use a programming language where constructs in your YAML map to objects.

hdjjhhvvhga 4y ago
Helm3 made it a solved problem years ago.
unxdfa 4y ago
I think we need to burn this whole thing down and rethink it at this point. A huge amount of the total operational cost of our business is working around weird YAML problems and helm charts. Had one a while back where quoted strings containing padded numbers in were used for something. "01" fine ... "05" fine, "06" fine, "07" fine, "08" kaboom. 01-07 were treated as strings and 08 was suddenly being treated as octal, which it's not and exploded.
- jpgvm 4y ago
  Helm is a pile of garbage. k8s speaks JSON just fine as expected. Try Tanka + Jsonnet to restore some of your sanity, also allows you to keep using Helm during your transition to a better place.
  - unxdfa 4y ago
    This is just the same problem in JSON, another loose schema’ed mess. Got my fair share of problems with that at the moment as well.
    - jpgvm 4y ago
      If you feel really strongly about getting type safety then there is Pulumi + Typescript though I really hate how it tries to be a "better" kubectl apply and fails miserably. Definitely room for someone to make something better than Helm but I think you should give Jsonnet a shot if you haven't already.
      - jen20 4y ago
        > tries to be a "better" kubectl apply and fails How does it try? How does it fail? I don't see any interactive CLI commands that could be seen as a replacement for kubectl.
        jpgvm 4y ago
        It's not about the commands. `pulumi up` is what you are looking for and the k8s resources that accompany it. kubectl apply, especially with server side apply is effective just a mechanism to push the declarative definition of your k8s objects to the server. Pulumi instead tries to manage the lifecycle of each of those objects itself when it really has no business doing so. This results in a lot of roundtrips to the Pulumi state store (which is implemented horrendously inefficiently btw) and generally takes forever and a day compared to kubectl apply. Most of this is down to an impedance mismatch between the models themselves. Pulumi + k8s works but it's just not a great experience. I say this with an absolute -ton- of experience with both options.
        jaxxstorm 4y ago
        This can be turned off by setting an annotation on the resources.
        jpgvm 4y ago
        I'm aware but it's not really ergonomic. For instance Pulumi also generates random names for resources, again you can override these but again, not very ergonomic to work around etc. IMO tools like Tanka + Jsonnet, or Cue are just better fit for k8s because you can stick with server side apply which is a much more native experience.