8 ms·

Deploying Kubernetes clusters in increasingly absurd languages

149 points by jaxxstorm 4y ago

protomyth 4y ago
Might as well do MUMPS and COBOL for the next article. I found the Fortran example very interesting. I guess it makes sense modern libraries would be available.
- jaxxstorm 4y ago
  I actually tried to get something working in COBOL, but my lack of familiarity with it made it a struggle. Would happily take a contribution at https://github.com/jaxxstorm/pulumi-examples https://github.com/jaxxstorm/pulumi-examples
throwaway787544 4y ago
I love this article, but I also hate that Pulumi has created yet another bad DSL using YAML. One day, somebody in charge of product design will also be a good engineer, and the crazy shit customers ask for will not end up a terrible feature. Until that day...
- oneplane 4y ago
  It's a bit of an XKCD where they keep re-inventing YAML-based DSLs. We have weird "native" YAMLs for AWS, GCP and Azure, and then YAML DSLs to generate those YAML DSLs. Even SaltStack and Ansible do it... It's almost like someone had a bad idea and put XKCD and Xzibit in a blender...
  - chupasaurus 4y ago
    Ansible even states that it doesn't have DSL while it actually is when you go to the rabbit hole deep enough.
    - deathanatos 4y ago
      … you don't have to but peer slightly into that rabbit hole; your first example Ansible playbook should very clearly demonstrate it has a DSL…
- throwaway894345 4y ago
  All of this IAC crap was sold to barely-tech-literate engineering managers as “declarative makes the complexity go away!” so every IAC vendor wed themselves to YAML or HCL or JSON, but of course the complexity doesn’t go away and so the vendors came up with increasingly absurd ways to DRY up this YAML (text templates a la Helm or encoding a DSL into your config language a la Terraform and CloudFormation). Eventually people realized this was madness and what’s really needed is a programming language that can resolve to these static manifests, so these vendors in their zeal to forever misunderstand the assignment came out with CDKs so we can compile source code to YAML with a whole bunch of pointless inheritance to really make a mess of things.
  - datalopers 4y ago
    It’s okay they’ll all be unemployed by Christmas 2023 and we’ll look back on this tech boom with “maybe we should have solved business problems instead of spending all our time rewriting from one devops tooling to the next?”
    - Jenk 4y ago
      That claim is more absurd than those you mock.
      - datalopers 4y ago
        This tech bubble already popped a leak (Nov 2021) and is deflating rapidly. There’s enough runway to last another year or two. Down-rounds and lay-offs are coming fast. Companies with ample cash will be fine, of course, but all these spurious startups with egregious VC funding, overpaid Pulumi and Kubernetes and <insert all the other unnecessary tech> engineers will soon be out of work and unable to find similar employment comp. People who get shit done and can run lean profit driven businesses will continue to survive just fine.
        mardifoufs 4y ago
        We've been hearing the same exact thing since at least 2013. I remember when 2015 was going to be the next huge tech crash, but nothing happened.
- Jenk 4y ago
  You guys will just love the shared mentality on /r/DevOps. Apparently YAML is the one true declarative language that all others are failing to imitate. I'm literally Judas for preferring to use typescript with pulumi.
  - baq 4y ago
    i hate yaml more than xml and that is saying something. i get looked at strange when i say this loudly, like i'm from mars or whatever. it's like being in a collective hallucination where you know you're right and everybody else is wrong and nothing makes sense anymore. i've tried to fight this by leading by example with https://dhall-lang.org/ https://dhall-lang.org/ but apparently having a configuration compiler is too much... unless the compiler takes yaml as input, then it's fine.
- phillipcarter 4y ago
  why-not-both.jpg? I hate yaml as much as the next developer, but I've met plenty of folks who far prefer it to using a "real" language. Different strokes for different folks? The challenge from a product standpoint is how to balance these different ways to do something without (a) ostracizing one group, and (b) creating a confusing mess on the "get started" path.
  - 0xbadcafebee 4y ago
    Some people prefer some things just because they haven't learned a better way, but if you show them a better way, they'd gladly use it. I think the mission of a product designer should be to see through what the user thinks they need, and provide something that solves their real problem in the best way. In the case of IaC, what the customer really wants is a programmatic way to generate instructions for Pulumi to use to build infrastructure. The dream of "declarative" anything is to explain what you want and have a machine make it reality, right? But in practice that's hard to do. The machine needs to be very, very smart. But it's not that smart. So we cheat and invent non-programming-languages, so that people end up telling the machine in far too much detail what they want. Do I really want to tell the machine to "go in a loop creating resources as long as there are resources in a list" ? Or do I really want to do some other thing, and making these resources should just be an incidental part of that that I shouldn't have to explain at all? Programmers who like YAML for anything other than pure data serialization don't actually like YAML, they just like that they can cheat with it. No need to write a configuration format if you read your configuration from it. No need to write a DSL if you execute logic based on it. The programmers who don't like YAML wanted to do the same things, but were bitten by all the problems that came from using the wrong thing for the wrong purpose. So really, the product person and experienced engineer need to work together to prevent this whole situation from ever occurring. Create a schema, and create libraries that can generate YAML based on the schema, but do it in such a way that no human would ever want to edit it by hand, and too complicated to ever write a generator for it without the schema. This way the customer can't shoot themselves in the foot, or demand changes which would make everything worse.
    - phillipcarter 4y ago
- zozbot234 4y ago
  This article gets one thing right: if you're going to autogenerate something as complex as YAML, you'll really, really want to stick to a sane subset of the underlying language (here, JSON).
- phlakaton 4y ago
  It's a trade-off. In exchange for a hair-tear-outable user experience, you get the ability to (relatively) easily load and transform that data, which gives you lots of automation and integration options. It's not unlike LISP's predilection for sexpr-based DSLs, I suppose.
jaxxstorm 4y ago
Since I wrote this article, my colleagues and friends have chipped in with even more absurd examples. How about some Pascal? https://github.com/jaxxstorm/pulumi-examples/pull/96 https://github.com/jaxxstorm/pulumi-examples/pull/96 Or maybe Emacs Lisp is more your flavour? https://github.com/jaxxstorm/pulumi-examples/pull/95 https://github.com/jaxxstorm/pulumi-examples/pull/95 Or the pièce de résistance: Brainfuck: https://github.com/jaxxstorm/pulumi-examples/pull/97 https://github.com/jaxxstorm/pulumi-examples/pull/97
- lukeramsden 4y ago
  I'm assuming there's some sort of transpiler to brainfuck? I just cannot stomach the idea that that was written by hand somehow....
  - rrdharan 4y ago
    https://en.wikipedia.org/wiki/Brainfuck https://en.wikipedia.org/wiki/Brainfuck > Although Brainfuck programs, especially complicated ones, are difficult to write, it is quite trivial to write an interpreter for Brainfuck in a more typical language such as C due to its simplicity. There even exist Brainfuck interpreters written in the Brainfuck language itself.
    - shpongled 4y ago
      The parent comment was about a transpiler targeting brainfuck, not an interpreter. Writing a brainfuck interpreter (or compiler to x86/etc) is indeed trivial... writing a compiler targeting brainfuck is another beast entirely.
      - rrdharan 4y ago
        You're right of course. Although I guess some do exist: https://github.com/arthaud/c2bf https://github.com/arthaud/c2bf https://github.com/felko/bfpy https://github.com/felko/bfpy
  - retrac 4y ago
dilyevsky 4y ago
https://github.com/cruise-automation/isopod https://github.com/cruise-automation/isopod Starlark -> protos -> k8s api. Jic you wanted more confuse
flurie 4y ago
I know this piece is mostly for the humor of increasingly arcane config generation systems, but given that Pulumi is attempting to displace the most popular application using HCL2, I find it interesting that the HCL example looks ergonomically awful. The CUE support in particular looks fairly interesting, because I think it is going to end up being the universal translator for a bunch of these golang tools due to its ability to pull in types directly from code.
BonoboIO 4y ago
Brainfuck? https://en.wikipedia.org/wiki/Brainfuck https://en.wikipedia.org/wiki/Brainfuck Example: ++++++++[>++++[>++>+++>+++>+<<<<-]>+>+>->>+[<]<-]>>.>---.+++++++..+++.>>.<-.<.+++.------.--------.>>+.>++.
- jaxxstorm 4y ago
  https://github.com/jaxxstorm/pulumi-examples/pull/97 https://github.com/jaxxstorm/pulumi-examples/pull/97
  - BonoboIO 4y ago
    HAHAHHAHAH I LOVE IT
waprin 4y ago
When I worked at Google, I made an App Engine Flexible runtime for the Whitespace language (only valid characters are space, tab, etc). https://en.wikipedia.org/wiki/Whitespace_(programming_language) https://en.wikipedia.org/wiki/Whitespace_(programming_langua... App Engine Flexible was another one of these products that just needed a container that responded on certain ports, it was quite fun to get it working. But I ended up writing a tiny "compiler" that would compile a small subset of Python to whitespace since writing just whitespace itself was quite challenging, obviously. I wanted to release it on April Fools Day but my manager was totally against it because April fools had become its own serious entity in the marketing org and he didn't want me stealing their thunder ( these days Google has dropped April Fools jokes altogether, I guess it got a little played out).
- cecilpl2 4y ago
  > April fools had become its own serious entity in the marketing org This is hilarious and not at all surprising
- dilyevsky 4y ago
  I remember some people being really upset with My Little Pony Borg ui theme
  - ceph_ 4y ago
    I understand where they were coming from, having to look at a pink/rainbow bm ui while debugging a page didn't make it any easier. But it was also one of the greatest April fools jokes of all time. RIP My Little Borgmaster: Production is Magic
timcavel 4y ago
AndyNemmity 4y ago
What is that cat replacement you are using?
- scheme271 4y ago
  It's called bat ( https://github.com/sharkdp/bat https://github.com/sharkdp/bat )
lukehoban 4y ago
There have been some good questions in a few of the threads here about why Pulumi added YAML support in the first place. I thought I’d share a few thoughts on that. Certainly it wasn’t only about being able to compile all sorts of esoteric languages down into a YAML-based data representation - though it’s fun to see the results of that here :-). Our goal at Pulumi has really been to offer the best tools for infrastructure as code, and for defining and managing cloud infrastructure more generally, for any developer working with the cloud. We started with a focus on the high-end - teams managing significant complexity of cloud infrastructure to get the most they can out of the managed services their cloud providers are making available as building blocks. We’ve grown with this part of the market, with great adoption and usage across many of the most advanced cloud engineering teams. In order to grow with these users, we’ve invested in many layers of the Infrastructure as Code stack. Some of those improvements have been related to the software engineering benefits of using traditional general purpose programming languages to manage cloud infrastructure - IDEs, types, abstraction and reuse, test frameworks, packaging and versioning and so much more. But we’ve also been making improvements at many other layers of the stack. Our cloud deployment orchestration engine is now, I believe, the richest option in the market - with multi-cloud support, built-in secrets management, refactoring support with aliases, rich controls over replacement behaviour, built-in support for components, and much more. And our native providers for Azure, Kubernetes, Google Cloud and AWS are the most complete and most up-to-date providers for managing those cloud platforms. We wanted to be able to offer all these benefits to the broadest possible range of developers working in the cloud. Since we already have very rich programming language options, we wanted to add an option at the other end of the spectrum - the simplest possible interface we could to the Pulumi platform. And that is what Pulumi YAML is. It is very simple, and designed for small scale use cases (a few to a dozen resources). It composes with the rest of the Pulumi ecosystem, so it’s easy to push complexity into components built in other Pulumi languages, to reference outputs of stacks deployed by other Pulumi languages, or even to "eject" into another Pulumi language if the complexity gets too high in YAML. So unlike many other IaC ecosystems where YAML or a DSL is the only option, in Pulumi, YAML can be a nice solution for simple use cases, without having to be abused for the complex use cases that are already well served by Pulumi’s existing alternative programming language choices. There’s more details on all of these points at https://www.pulumi.com/blog/pulumi-yaml/ https://www.pulumi.com/blog/pulumi-yaml/ for those interested in learning more!

149 points by jaxxstorm 4y ago

Might as well do MUMPS and COBOL for the next article. I found the Fortran example very interesting. I guess it makes sense modern libraries would be available.

jaxxstorm 4y ago
I actually tried to get something working in COBOL, but my lack of familiarity with it made it a struggle. Would happily take a contribution at https://github.com/jaxxstorm/pulumi-examples https://github.com/jaxxstorm/pulumi-examples

throwaway787544 4y ago

I love this article, but I also hate that Pulumi has created yet another bad DSL using YAML. One day, somebody in charge of product design will also be a good engineer, and the crazy shit customers ask for will not end up a terrible feature. Until that day...

oneplane 4y ago
It's a bit of an XKCD where they keep re-inventing YAML-based DSLs. We have weird "native" YAMLs for AWS, GCP and Azure, and then YAML DSLs to generate those YAML DSLs. Even SaltStack and Ansible do it... It's almost like someone had a bad idea and put XKCD and Xzibit in a blender...
- chupasaurus 4y ago
  Ansible even states that it doesn't have DSL while it actually is when you go to the rabbit hole deep enough.
  - deathanatos 4y ago
    … you don't have to but peer slightly into that rabbit hole; your first example Ansible playbook should very clearly demonstrate it has a DSL…
throwaway894345 4y ago
All of this IAC crap was sold to barely-tech-literate engineering managers as “declarative makes the complexity go away!” so every IAC vendor wed themselves to YAML or HCL or JSON, but of course the complexity doesn’t go away and so the vendors came up with increasingly absurd ways to DRY up this YAML (text templates a la Helm or encoding a DSL into your config language a la Terraform and CloudFormation). Eventually people realized this was madness and what’s really needed is a programming language that can resolve to these static manifests, so these vendors in their zeal to forever misunderstand the assignment came out with CDKs so we can compile source code to YAML with a whole bunch of pointless inheritance to really make a mess of things.
- datalopers 4y ago
  It’s okay they’ll all be unemployed by Christmas 2023 and we’ll look back on this tech boom with “maybe we should have solved business problems instead of spending all our time rewriting from one devops tooling to the next?”
  - Jenk 4y ago
    That claim is more absurd than those you mock.
    - datalopers 4y ago
      This tech bubble already popped a leak (Nov 2021) and is deflating rapidly. There’s enough runway to last another year or two. Down-rounds and lay-offs are coming fast. Companies with ample cash will be fine, of course, but all these spurious startups with egregious VC funding, overpaid Pulumi and Kubernetes and <insert all the other unnecessary tech> engineers will soon be out of work and unable to find similar employment comp. People who get shit done and can run lean profit driven businesses will continue to survive just fine.
      - mardifoufs 4y ago
        We've been hearing the same exact thing since at least 2013. I remember when 2015 was going to be the next huge tech crash, but nothing happened.
Jenk 4y ago
You guys will just love the shared mentality on /r/DevOps. Apparently YAML is the one true declarative language that all others are failing to imitate. I'm literally Judas for preferring to use typescript with pulumi.
- baq 4y ago
  i hate yaml more than xml and that is saying something. i get looked at strange when i say this loudly, like i'm from mars or whatever. it's like being in a collective hallucination where you know you're right and everybody else is wrong and nothing makes sense anymore. i've tried to fight this by leading by example with https://dhall-lang.org/ https://dhall-lang.org/ but apparently having a configuration compiler is too much... unless the compiler takes yaml as input, then it's fine.
phillipcarter 4y ago
why-not-both.jpg? I hate yaml as much as the next developer, but I've met plenty of folks who far prefer it to using a "real" language. Different strokes for different folks? The challenge from a product standpoint is how to balance these different ways to do something without (a) ostracizing one group, and (b) creating a confusing mess on the "get started" path.
- 0xbadcafebee 4y ago
  Some people prefer some things just because they haven't learned a better way, but if you show them a better way, they'd gladly use it. I think the mission of a product designer should be to see through what the user thinks they need, and provide something that solves their real problem in the best way. In the case of IaC, what the customer really wants is a programmatic way to generate instructions for Pulumi to use to build infrastructure. The dream of "declarative" anything is to explain what you want and have a machine make it reality, right? But in practice that's hard to do. The machine needs to be very, very smart. But it's not that smart. So we cheat and invent non-programming-languages, so that people end up telling the machine in far too much detail what they want. Do I really want to tell the machine to "go in a loop creating resources as long as there are resources in a list" ? Or do I really want to do some other thing, and making these resources should just be an incidental part of that that I shouldn't have to explain at all? Programmers who like YAML for anything other than pure data serialization don't actually like YAML, they just like that they can cheat with it. No need to write a configuration format if you read your configuration from it. No need to write a DSL if you execute logic based on it. The programmers who don't like YAML wanted to do the same things, but were bitten by all the problems that came from using the wrong thing for the wrong purpose. So really, the product person and experienced engineer need to work together to prevent this whole situation from ever occurring. Create a schema, and create libraries that can generate YAML based on the schema, but do it in such a way that no human would ever want to edit it by hand, and too complicated to ever write a generator for it without the schema. This way the customer can't shoot themselves in the foot, or demand changes which would make everything worse.
  - phillipcarter 4y ago
zozbot234 4y ago
This article gets one thing right: if you're going to autogenerate something as complex as YAML, you'll really, really want to stick to a sane subset of the underlying language (here, JSON).
phlakaton 4y ago
It's a trade-off. In exchange for a hair-tear-outable user experience, you get the ability to (relatively) easily load and transform that data, which gives you lots of automation and integration options. It's not unlike LISP's predilection for sexpr-based DSLs, I suppose.

jaxxstorm 4y ago

Since I wrote this article, my colleagues and friends have chipped in with even more absurd examples. How about some Pascal? https://github.com/jaxxstorm/pulumi-examples/pull/96 https://github.com/jaxxstorm/pulumi-examples/pull/96 Or maybe Emacs Lisp is more your flavour? https://github.com/jaxxstorm/pulumi-examples/pull/95 https://github.com/jaxxstorm/pulumi-examples/pull/95 Or the pièce de résistance: Brainfuck: https://github.com/jaxxstorm/pulumi-examples/pull/97 https://github.com/jaxxstorm/pulumi-examples/pull/97

lukeramsden 4y ago
I'm assuming there's some sort of transpiler to brainfuck? I just cannot stomach the idea that that was written by hand somehow....
- rrdharan 4y ago
  https://en.wikipedia.org/wiki/Brainfuck https://en.wikipedia.org/wiki/Brainfuck > Although Brainfuck programs, especially complicated ones, are difficult to write, it is quite trivial to write an interpreter for Brainfuck in a more typical language such as C due to its simplicity. There even exist Brainfuck interpreters written in the Brainfuck language itself.
  - shpongled 4y ago
    The parent comment was about a transpiler targeting brainfuck, not an interpreter. Writing a brainfuck interpreter (or compiler to x86/etc) is indeed trivial... writing a compiler targeting brainfuck is another beast entirely.
    - rrdharan 4y ago
      You're right of course. Although I guess some do exist: https://github.com/arthaud/c2bf https://github.com/arthaud/c2bf https://github.com/felko/bfpy https://github.com/felko/bfpy
- retrac 4y ago

dilyevsky 4y ago

https://github.com/cruise-automation/isopod https://github.com/cruise-automation/isopod Starlark -> protos -> k8s api. Jic you wanted more confuse

flurie 4y ago

I know this piece is mostly for the humor of increasingly arcane config generation systems, but given that Pulumi is attempting to displace the most popular application using HCL2, I find it interesting that the HCL example looks ergonomically awful. The CUE support in particular looks fairly interesting, because I think it is going to end up being the universal translator for a bunch of these golang tools due to its ability to pull in types directly from code.

BonoboIO 4y ago

Brainfuck? https://en.wikipedia.org/wiki/Brainfuck https://en.wikipedia.org/wiki/Brainfuck Example: ++++++++[>++++[>++>+++>+++>+<<<<-]>+>+>->>+[<]<-]>>.>---.+++++++..+++.>>.<-.<.+++.------.--------.>>+.>++.

jaxxstorm 4y ago
https://github.com/jaxxstorm/pulumi-examples/pull/97 https://github.com/jaxxstorm/pulumi-examples/pull/97
- BonoboIO 4y ago
  HAHAHHAHAH I LOVE IT

waprin 4y ago

When I worked at Google, I made an App Engine Flexible runtime for the Whitespace language (only valid characters are space, tab, etc). https://en.wikipedia.org/wiki/Whitespace_(programming_language) https://en.wikipedia.org/wiki/Whitespace_(programming_langua... App Engine Flexible was another one of these products that just needed a container that responded on certain ports, it was quite fun to get it working. But I ended up writing a tiny "compiler" that would compile a small subset of Python to whitespace since writing just whitespace itself was quite challenging, obviously. I wanted to release it on April Fools Day but my manager was totally against it because April fools had become its own serious entity in the marketing org and he didn't want me stealing their thunder ( these days Google has dropped April Fools jokes altogether, I guess it got a little played out).

cecilpl2 4y ago
> April fools had become its own serious entity in the marketing org This is hilarious and not at all surprising
dilyevsky 4y ago
I remember some people being really upset with My Little Pony Borg ui theme
- ceph_ 4y ago
  I understand where they were coming from, having to look at a pink/rainbow bm ui while debugging a page didn't make it any easier. But it was also one of the greatest April fools jokes of all time. RIP My Little Borgmaster: Production is Magic

timcavel 4y ago

AndyNemmity 4y ago

What is that cat replacement you are using?

scheme271 4y ago
It's called bat ( https://github.com/sharkdp/bat https://github.com/sharkdp/bat )

lukehoban 4y ago

There have been some good questions in a few of the threads here about why Pulumi added YAML support in the first place. I thought I’d share a few thoughts on that. Certainly it wasn’t only about being able to compile all sorts of esoteric languages down into a YAML-based data representation - though it’s fun to see the results of that here :-). Our goal at Pulumi has really been to offer the best tools for infrastructure as code, and for defining and managing cloud infrastructure more generally, for any developer working with the cloud. We started with a focus on the high-end - teams managing significant complexity of cloud infrastructure to get the most they can out of the managed services their cloud providers are making available as building blocks. We’ve grown with this part of the market, with great adoption and usage across many of the most advanced cloud engineering teams. In order to grow with these users, we’ve invested in many layers of the Infrastructure as Code stack. Some of those improvements have been related to the software engineering benefits of using traditional general purpose programming languages to manage cloud infrastructure - IDEs, types, abstraction and reuse, test frameworks, packaging and versioning and so much more. But we’ve also been making improvements at many other layers of the stack. Our cloud deployment orchestration engine is now, I believe, the richest option in the market - with multi-cloud support, built-in secrets management, refactoring support with aliases, rich controls over replacement behaviour, built-in support for components, and much more. And our native providers for Azure, Kubernetes, Google Cloud and AWS are the most complete and most up-to-date providers for managing those cloud platforms. We wanted to be able to offer all these benefits to the broadest possible range of developers working in the cloud. Since we already have very rich programming language options, we wanted to add an option at the other end of the spectrum - the simplest possible interface we could to the Pulumi platform. And that is what Pulumi YAML is. It is very simple, and designed for small scale use cases (a few to a dozen resources). It composes with the rest of the Pulumi ecosystem, so it’s easy to push complexity into components built in other Pulumi languages, to reference outputs of stacks deployed by other Pulumi languages, or even to "eject" into another Pulumi language if the complexity gets too high in YAML. So unlike many other IaC ecosystems where YAML or a DSL is the only option, in Pulumi, YAML can be a nice solution for simple use cases, without having to be abused for the complex use cases that are already well served by Pulumi’s existing alternative programming language choices. There’s more details on all of these points at https://www.pulumi.com/blog/pulumi-yaml/ https://www.pulumi.com/blog/pulumi-yaml/ for those interested in learning more!