4 ms·

We don’t have a huge system, but already we are seeing bash scripts that sed values in Kubernetes manifests or configuration stored as a dictionary inside a pyt

We don’t have a huge system, but already we are seeing bash scripts that sed values in Kubernetes manifests or configuration stored as a dictionary inside a python script that can generate large yaml files as part of a deploy pipeline. I think it is an inherent error in basically all orchestration tools (Kubernetes, cloud build/formation, etc), that they don’t support scripting.

octopoc 7y ago
For a long time I was frustrated by the lack of scripting tools for orchestration. My ideal situation would be to write the orchestration / deployment config using the same programming language the app is written in. Eventually I found Pulumi, which supports a limited number of programming languages but is basically what I was looking for, except I would like C# to be officially supported. Pulumi has a high-priority issue for deciding how they're going to support arbitrary programming languages: https://github.com/pulumi/pulumi/issues/2430 https://github.com/pulumi/pulumi/issues/2430 I'm watching this with great interest. Edit: looks like it's not high priority anymore :/
- webo 7y ago
  How has your experience been with Pulumi? I tried it back in its alpha days to compare to terraform but I didn’t make much progress due to lack of documentation and limited ecosystem.
  - octopoc 7y ago
    I really like the concept. Here are the highlights: - Helm v2 support didn't use Tiller, instead it rendered the YAML client side and pushed that to the server. Sort of how Helm v3 will be, but it didn't work very well in practice with the charts I wanted to deploy. It felt like Helm suppor t was an afterthought and the best way to deploy was to port all your Helm chart dependencies to Pulumi. - It was really intuitive to create functions that deployed the same thing repeatedly with slightly different settings. I used it for Office365 email server settings. I managed a couple domain names that use Office365 email so I had a single function that deployed the correct settings to the specified domain name, and I just called it twice with slightly different parameters. Way more intuitive than Terraform.
  - ianpurton 7y ago
    We're using Pulumi for a production system and couldn't be happier. The ability to template Yaml in Typescript and create infrastructure is mind blowing, all the time checked by the compiler. Well it's not so much templating as using Typescripts built in JSON syntax. Using VSCode we can refactor our infrastructure code, i.e. create functions for sub-levels of our Yaml. So we effectively combine our Kuberentes Yaml and infrastructure. It's great. Try it.
    - levi_b 7y ago
      Glad to hear you’re enjoying it! We just released a bunch of new k8s content/features this week that you might want to check out. [1] (I maintain the k8s provider at Pulumi) [1] https://www.pulumi.com/blog/crosswalk-kubernetes/ https://www.pulumi.com/blog/crosswalk-kubernetes/
- demosito666 7y ago
  From the readme examples it seems that the code is not provider-agnostic, i.e. you write specifically for e.g. AWS. How is it then better than using AWS' native SDK for your language?
- levi_b 7y ago
  Support for C# (and other .NET languages) just landed last week[1]. (I work on the k8s provider at Pulumi) [1] https://www.pulumi.com/blog/pulumi-dotnet-core/ https://www.pulumi.com/blog/pulumi-dotnet-core/
  - octopoc 7y ago
    Wow that is awesome, thank you! I'll have to check it out.
username90 7y ago
Kubernets doesn't support simple scripting like passing variables and such? I know that Borg supports that via borgcfg, we basically have Borg templates for everything so you rarely have to write a new one from scratch. Not sure why it wasn't ported to the outside world.
- Cedricgc 7y ago
  Author here. The closest thing to borgcfg in Kubernetes would be kubecfg [0]. It is based on jsonnet, the open source equivalent to GCL. The official blessed solution right now would be kustomize [1] since it is directly integrated into kubectl. I categorize both in the article with my evaluation as solutions. [0] https://github.com/bitnami/kubecfg https://github.com/bitnami/kubecfg [1] https://kustomize.io/ https://kustomize.io/
  - wodenokoto 7y ago
    I’ll have to check that out, thank you for mentioning it. Is `-k` very recent?
    - Cedricgc 7y ago
      I am not certain, I think it was adopted mid 2019.
- alexhutcheson 7y ago
  It's funny watching the open-source world gradually converge towards something that looks a lot like borgcfg.
derefr 7y ago
Orchestration tools (which also include chef/puppet/etc.) inherently need to be able to notice when something is just a little out of whack, and respond by just calculating the smallest action that will restore things to the way they “should” be. Unless your scripting language is both deterministic, essentially syntactic sugar for a dependency digraph that the orchestration system knows how to calculate reduced patches of, it’ll break the “convergence” abstraction of the orchestration system: in est, it won’t know for sure what state the system is in after the script runs, so it won’t know what needs to be done to put the system back into any other state later on. Not that such scripting languages don’t exist. Apache BEAM, for example, is a set of SDKs for various languages that let you write code that compiles to dependency+data-flow digraphs. But it’s probably not the type of scripting you’re thinking of. I should also mention, though, that systems like k8s do allow arbitrary programming (not necessarily “scripting”) through Operators (https://coreos.com/blog/introducing-operators.html https://coreos.com/blog/introducing-operators.html). Rather than scripting the orchestration system itself, you write a new type of component for it to orchestrate, where that component is both created/managed by the orchestration system, and, through delegation, is responsible for implementing the method by which the orchestration creates/manages other specified types of resources.
semiotagonal 7y ago
Of course, once you support scripting, why not just do everything in a deployment script? I think a lot of these tools are based on the assumption that declarative configuration should be sufficient. The tools eschew scripting because introducing it would call their basic assumptions into question.
- jacquesm 7y ago
  It's a bit like every spreadsheet ever sooner or later being expanded with a programming language.
rhizome 7y ago
How about running scripts that compile into K configs as the common task? This has been done for a long time in the DNS world where details are held in a database, some of which are then pulled out to create a new current zone file(s).

octopoc 7y ago

For a long time I was frustrated by the lack of scripting tools for orchestration. My ideal situation would be to write the orchestration / deployment config using the same programming language the app is written in. Eventually I found Pulumi, which supports a limited number of programming languages but is basically what I was looking for, except I would like C# to be officially supported. Pulumi has a high-priority issue for deciding how they're going to support arbitrary programming languages: https://github.com/pulumi/pulumi/issues/2430 https://github.com/pulumi/pulumi/issues/2430 I'm watching this with great interest. Edit: looks like it's not high priority anymore :/

webo 7y ago
How has your experience been with Pulumi? I tried it back in its alpha days to compare to terraform but I didn’t make much progress due to lack of documentation and limited ecosystem.
- octopoc 7y ago
  I really like the concept. Here are the highlights: - Helm v2 support didn't use Tiller, instead it rendered the YAML client side and pushed that to the server. Sort of how Helm v3 will be, but it didn't work very well in practice with the charts I wanted to deploy. It felt like Helm suppor t was an afterthought and the best way to deploy was to port all your Helm chart dependencies to Pulumi. - It was really intuitive to create functions that deployed the same thing repeatedly with slightly different settings. I used it for Office365 email server settings. I managed a couple domain names that use Office365 email so I had a single function that deployed the correct settings to the specified domain name, and I just called it twice with slightly different parameters. Way more intuitive than Terraform.
- ianpurton 7y ago
  We're using Pulumi for a production system and couldn't be happier. The ability to template Yaml in Typescript and create infrastructure is mind blowing, all the time checked by the compiler. Well it's not so much templating as using Typescripts built in JSON syntax. Using VSCode we can refactor our infrastructure code, i.e. create functions for sub-levels of our Yaml. So we effectively combine our Kuberentes Yaml and infrastructure. It's great. Try it.
  - levi_b 7y ago
    Glad to hear you’re enjoying it! We just released a bunch of new k8s content/features this week that you might want to check out. [1] (I maintain the k8s provider at Pulumi) [1] https://www.pulumi.com/blog/crosswalk-kubernetes/ https://www.pulumi.com/blog/crosswalk-kubernetes/
demosito666 7y ago
From the readme examples it seems that the code is not provider-agnostic, i.e. you write specifically for e.g. AWS. How is it then better than using AWS' native SDK for your language?
levi_b 7y ago
Support for C# (and other .NET languages) just landed last week[1]. (I work on the k8s provider at Pulumi) [1] https://www.pulumi.com/blog/pulumi-dotnet-core/ https://www.pulumi.com/blog/pulumi-dotnet-core/
- octopoc 7y ago
  Wow that is awesome, thank you! I'll have to check it out.

username90 7y ago

Kubernets doesn't support simple scripting like passing variables and such? I know that Borg supports that via borgcfg, we basically have Borg templates for everything so you rarely have to write a new one from scratch. Not sure why it wasn't ported to the outside world.

Cedricgc 7y ago
Author here. The closest thing to borgcfg in Kubernetes would be kubecfg [0]. It is based on jsonnet, the open source equivalent to GCL. The official blessed solution right now would be kustomize [1] since it is directly integrated into kubectl. I categorize both in the article with my evaluation as solutions. [0] https://github.com/bitnami/kubecfg https://github.com/bitnami/kubecfg [1] https://kustomize.io/ https://kustomize.io/
- wodenokoto 7y ago
  I’ll have to check that out, thank you for mentioning it. Is `-k` very recent?
  - Cedricgc 7y ago
    I am not certain, I think it was adopted mid 2019.
alexhutcheson 7y ago
It's funny watching the open-source world gradually converge towards something that looks a lot like borgcfg.

derefr 7y ago

Orchestration tools (which also include chef/puppet/etc.) inherently need to be able to notice when something is just a little out of whack, and respond by just calculating the smallest action that will restore things to the way they “should” be. Unless your scripting language is both deterministic, essentially syntactic sugar for a dependency digraph that the orchestration system knows how to calculate reduced patches of, it’ll break the “convergence” abstraction of the orchestration system: in est, it won’t know for sure what state the system is in after the script runs, so it won’t know what needs to be done to put the system back into any other state later on. Not that such scripting languages don’t exist. Apache BEAM, for example, is a set of SDKs for various languages that let you write code that compiles to dependency+data-flow digraphs. But it’s probably not the type of scripting you’re thinking of. I should also mention, though, that systems like k8s do allow arbitrary programming (not necessarily “scripting”) through Operators (https://coreos.com/blog/introducing-operators.html https://coreos.com/blog/introducing-operators.html). Rather than scripting the orchestration system itself, you write a new type of component for it to orchestrate, where that component is both created/managed by the orchestration system, and, through delegation, is responsible for implementing the method by which the orchestration creates/manages other specified types of resources.

semiotagonal 7y ago

Of course, once you support scripting, why not just do everything in a deployment script? I think a lot of these tools are based on the assumption that declarative configuration should be sufficient. The tools eschew scripting because introducing it would call their basic assumptions into question.

jacquesm 7y ago
It's a bit like every spreadsheet ever sooner or later being expanded with a programming language.

rhizome 7y ago

How about running scripts that compile into K configs as the common task? This has been done for a long time in the DNS world where details are held in a database, some of which are then pulled out to create a new current zone file(s).