5 ms·

I used to be fascinated by the automation power of Kubernetes custom components. The declarative approach and reconciliation loop offers so many possibilities f

I used to be fascinated by the automation power of Kubernetes custom components. The declarative approach and reconciliation loop offers so many possibilities for creating higher level descriptions of domain specific infrastructure. On reflection though, I think this stuff can lead to a lot of complexity layers which don't benefit the product relative to the time investment. You are probably not Google.

fragmede 2y ago
The funny thing about that is that Google doesn't use Kubernetes internally because it doesn't scale to their level. Borg is more advanced than Kubernetes in the ways that Google needs, so really Kubernetes is dumbed down for everyone else, and everyone else isn't Google scale, (except for those that are, eg Meta has Twine). so yeah, you're probably not Google, but people out there are Tinder or Reddit or Pinterest and all shouldn't have to reinvent the wheel.
sofixa 2y ago
> The declarative approach and reconciliation loop offers so many possibilities for creating higher level descriptions of domain specific infrastructure. Terraform running on a schedule gets you 3/4 of the way there for 5% of the complexity though.
- osigurdson 2y ago
  Helm gets you 99% of the way there with less complexity than running Terraform in loop. Terraform is great for bootstrapping Kubernetes of course.
  - sofixa 2y ago
    I'm not sure I'd describe Helm as "less complexity" than Terraform. HCL is much easier to read, write and template than YAML. Helm also cannot tell you exactly what changed/will change (tf plan). It also cannot use dynamic data (e.g. check how many availability zones there are in the region and use that for the number of Pods).
    - godber 2y ago
      Using helmfile (yet another layer of abstraction on top of helm) lets you get diffs between current and new releases. And it’s all templated so could be dynamic.
      - granra 2y ago
        At a previous employer they were using helmfile. I don't remember what exactly I needed to do but I needed a value to be templated by helm _after_ helmfile had done its work (iirc) and ended up having to write along the lines of "{{ `{{` }} .Values.something {{ `}}` }}" but actually had more levels of nesting which required use of `printf` iirc and I don't remember how I actually solved it. My point is though that using string templating for YAML creation is IMO always a bad idea and using helm for anything more complicated than the most basic application always makes me sad in the end. helmfile adds another templating layer and my limited exposure to it made me really dislike it. edit: I remember now that this was for rendering config for vector, which itself has templating support with the famous `{{ .key }}` syntax. So not entirely helmfile's fault, but I still stick to my point as I needed to get through 3 levels of templating.
        godber 2y ago
        Yes, that sucks.
      - sofixa 2y ago
        > Using helmfile (yet another layer of abstraction on top of helm) lets you get diffs between current and new releases Of the config, not of reality. What if someone made a change in the meantime? Or something is in a half or fully broken state? > And it’s all templated so could be dynamic. Not dynamic based on things external to that Helm deployment. You can't have two Helm charts referencing each other dynamically, or anything else external for that matter.
    - osigurdson 2y ago
      I don't find HCL easier to read or write than yaml. I think it is just what you are familiar with. HCL is also only used for one purpose with yaml is used for many things.
  - granra 2y ago
    Íslenskur? :D IMO using string templating for creating structured data in a white-space sensitive configuration language always ends with pain and cursing beyond the most basic application manifests. I'm not saying Terraform or HCL is necessarily the solution either but it certainly wouldn't be Helm in my book. It's a shame language like CUE or nickel didn't take off for this.
    - cogman10 2y ago
      In practice, it simply hasn't mattered all that much. I also somewhat wish that k8s was programmed with something like the CDK where I could get a nice understandable compiler error about what went wrong and why. But for the most part I'm spending very little time writing helm charts and a lot more time writing my applications. A tweak here or there to the chart has to happen, but that's relatively minor and easy to accomplish.
    - osigurdson 2y ago
      Sorry, I unfortunately don't speak Icelandic despite the name. Maybe one day.
  - doctorpangloss 2y ago
    Helm is an ultra low budget operator for anything. Maybe the best thing in the ecosystem, despite what everyone says.
- kmac_ 2y ago
  Terraform's not an orchestrator, it's for something totally different.
osigurdson 2y ago
Not Google but leverage a lot of compute at work and use Kubernetes for that. However, I use it even on small side projects as well because I am on the other side of the learning curve. The control plan is free in some cloud providers or can run locally. It brings a lot of consistency between on-premise and various cloud providers and easy to use once you get the hang of it.
znpy 2y ago
There was an article by one of the original creators of kubernetes on how the mistakes they did when developing the thing to release to the public. I can’t find it again and i suspect the original author has deleted it. One of the point for example was to go with ipv6 from the start, and another was about storage. If anybody has a link, please paste it :) But yes, the “public” kubernetes is a dumbed down version of borg. But frankly i think it’s mostly two things: 1. They probably had to redesign the things that in borg were too specific to the custom google infrastructure 2. They didn’t want to give away the _really_ good things, the competitive advantage.

fragmede 2y ago

The funny thing about that is that Google doesn't use Kubernetes internally because it doesn't scale to their level. Borg is more advanced than Kubernetes in the ways that Google needs, so really Kubernetes is dumbed down for everyone else, and everyone else isn't Google scale, (except for those that are, eg Meta has Twine). so yeah, you're probably not Google, but people out there are Tinder or Reddit or Pinterest and all shouldn't have to reinvent the wheel.

sofixa 2y ago

> The declarative approach and reconciliation loop offers so many possibilities for creating higher level descriptions of domain specific infrastructure. Terraform running on a schedule gets you 3/4 of the way there for 5% of the complexity though.

osigurdson 2y ago
Helm gets you 99% of the way there with less complexity than running Terraform in loop. Terraform is great for bootstrapping Kubernetes of course.
- sofixa 2y ago
  I'm not sure I'd describe Helm as "less complexity" than Terraform. HCL is much easier to read, write and template than YAML. Helm also cannot tell you exactly what changed/will change (tf plan). It also cannot use dynamic data (e.g. check how many availability zones there are in the region and use that for the number of Pods).
  - godber 2y ago
    Using helmfile (yet another layer of abstraction on top of helm) lets you get diffs between current and new releases. And it’s all templated so could be dynamic.
    - granra 2y ago
      At a previous employer they were using helmfile. I don't remember what exactly I needed to do but I needed a value to be templated by helm _after_ helmfile had done its work (iirc) and ended up having to write along the lines of "{{ `{{` }} .Values.something {{ `}}` }}" but actually had more levels of nesting which required use of `printf` iirc and I don't remember how I actually solved it. My point is though that using string templating for YAML creation is IMO always a bad idea and using helm for anything more complicated than the most basic application always makes me sad in the end. helmfile adds another templating layer and my limited exposure to it made me really dislike it. edit: I remember now that this was for rendering config for vector, which itself has templating support with the famous `{{ .key }}` syntax. So not entirely helmfile's fault, but I still stick to my point as I needed to get through 3 levels of templating.
      - godber 2y ago
        Yes, that sucks.
    - sofixa 2y ago
      > Using helmfile (yet another layer of abstraction on top of helm) lets you get diffs between current and new releases Of the config, not of reality. What if someone made a change in the meantime? Or something is in a half or fully broken state? > And it’s all templated so could be dynamic. Not dynamic based on things external to that Helm deployment. You can't have two Helm charts referencing each other dynamically, or anything else external for that matter.
  - osigurdson 2y ago
    I don't find HCL easier to read or write than yaml. I think it is just what you are familiar with. HCL is also only used for one purpose with yaml is used for many things.
- granra 2y ago
  Íslenskur? :D IMO using string templating for creating structured data in a white-space sensitive configuration language always ends with pain and cursing beyond the most basic application manifests. I'm not saying Terraform or HCL is necessarily the solution either but it certainly wouldn't be Helm in my book. It's a shame language like CUE or nickel didn't take off for this.
  - cogman10 2y ago
    In practice, it simply hasn't mattered all that much. I also somewhat wish that k8s was programmed with something like the CDK where I could get a nice understandable compiler error about what went wrong and why. But for the most part I'm spending very little time writing helm charts and a lot more time writing my applications. A tweak here or there to the chart has to happen, but that's relatively minor and easy to accomplish.
  - osigurdson 2y ago
    Sorry, I unfortunately don't speak Icelandic despite the name. Maybe one day.
- doctorpangloss 2y ago
  Helm is an ultra low budget operator for anything. Maybe the best thing in the ecosystem, despite what everyone says.
kmac_ 2y ago
Terraform's not an orchestrator, it's for something totally different.

osigurdson 2y ago

Not Google but leverage a lot of compute at work and use Kubernetes for that. However, I use it even on small side projects as well because I am on the other side of the learning curve. The control plan is free in some cloud providers or can run locally. It brings a lot of consistency between on-premise and various cloud providers and easy to use once you get the hang of it.

znpy 2y ago

There was an article by one of the original creators of kubernetes on how the mistakes they did when developing the thing to release to the public. I can’t find it again and i suspect the original author has deleted it. One of the point for example was to go with ipv6 from the start, and another was about storage. If anybody has a link, please paste it :) But yes, the “public” kubernetes is a dumbed down version of borg. But frankly i think it’s mostly two things: 1. They probably had to redesign the things that in borg were too specific to the custom google infrastructure 2. They didn’t want to give away the _really_ good things, the competitive advantage.