23 ms·

Pulumi Insights – AI generated IaC programs

85 points by kaotown 3y ago

awinter-py 3y ago
ohhhh this is really smart; every terraform user has at some point run up against the limits of it not being a language and flirted with pulumi. natural excuse now to get started, esp if the auto import works targeting translation between declarative-ish languages is a natural day-1 application for LLMs declarative languages generally good, bc 1) it's relatively easy to check them, 2) they are low-boilerplate (less to get wrong + better support for small token memories) wonder if this backfires bc pulumi is more powerful, giving the LLM more chances to get in trouble? like there are cases where in order to check a pulumi program you would need something like TLA+ with a good model of the system you are targeting
fdgsdfogijq 3y ago
CDK/Pulumi are going to be used by LLMs to spin up website infra exactly to spec. Devs will fill in the business logic interfaces. Going to be wild
- smt88 3y ago
  Why would you need LLMs for a boilerplate web application?
  - verdverm 3y ago
    There is a "need" vs "want" aspect here, but often the boilerplates that we reach for become out of date. In theory, an LLM producing these could remain up to date without maintenance, key phrase "in theory"
    - trufas 3y ago
      There's no guarantee the boilerplate an LLM spits out will be up to date. It'll almost definitely have some outdated code in it's dataset that it can reference.
      - verdverm 3y ago
        yes, a basic LLM is susceptible to this. The other major issue is that they will generate different output, even with the same input. There is work going into giving the LLMs access to external data & systems. This is my basis for saying they will be able to stay up to date. I have a very different approach I'm working on: https://docs.hofstadter.io/getting-started/creators https://docs.hofstadter.io/getting-started/creators (human-made blueprints that can be started from and later updated to bring in the upstream changes)
- JohnMakin 3y ago
  but how will it be maintained? "IAC" seems like a bit of a misnomer here, if I'm understanding correctly. Now, if Pulumi generated terraform for me to maintain, that'd be a different matter, but this article seems to just produce cloud resources based on LLM prompts, which are not deterministic at all.
  - twalla 3y ago
    It produces pulumi programs which are the equivalent of a terraform module or main.tf file. Whether or not said programs are deterministic is another matter.
  - dmattia 3y ago
    Pulumi is an IAC tool similar to Terraform (it actually usually calls terraform providers over gRPC under the hood), just written in languages like Typescript/Python. I think the intention would be to use this AI prompt thing to generate pulumi and then to insert it into your codebase, just like you'd do with terraform, and then it becomes deterministic. I've used ChatGPT and other tools to generate Pulumi before, so I'm not totally sure how this specific tool from Pulumi is different, but I'd guess they've somehow trained it more specifically on their sdks/docs or something
    - JohnMakin 3y ago
      thanks for the clarification
- shadycuz 3y ago
  Yeah but ChatGPT can already write Terraform and Cloudformation really well. So this doesn't seem that special besides it's already baked into the pulumi eco system.
- gtirloni 3y ago
  > exactly to spec And that spec is?
  - fdgsdfogijq 3y ago
    Natural language :)
    - diarrhea 3y ago
      Reminds me of UML. Instead of laboriously hand-writing all this nasty code, why not use a tool that can autogenerate it for us? After all, all that’s needed is a spec (UML)! Of course, the spec is the work. LLMs are then just very potent at translating it to be machine-readable.
cedws 3y ago
Sorry to be offtopic, but I've been using Pulumi at work for the past 6 months and I'm really not impressed. It's basically just Terraform but worse, with a million ways to declare your infrastructure instead of just one. Infrastructure people tend not to write the best code and from my observation the extra freedom of an imperative language just makes stuff even more complex and harder to maintain. It's also much harder to automate than Terraform, I am not aware of any equivalent to Atlantis. Also, Pulumi previews (equivalent to plans) are complete bullshit. If you don't write your code carefully, resources can be created and removed and you won't know it's going to happen until you start applying... it's an engineer's worst nightmare when a tool lies.
- verdverm 3y ago
  Agreement, I do not understand this backwards movement in the DevOps world. My hypothesis is that they are catering to a different group, i.e. enabling developers to do Ops, who don't want to learn TF and want to use their preferred language. DevOps first practitioners are in short supply, so it makes sense there is a market for this.
  - re-thc 3y ago
    Why is it backwards? Is making it more accessible and inclusive backwards? So these "DevOps practitioners" who are so different according to you never used Python or any programming language? We should enable everyone to at least aware of Ops and be able to contribute. Why does it need to be gated behind a special language i.e. HCL? A lot of times things go rogue exactly because developers don't understand and claim to not have a need to understand because it's not their job. Ultimately the code runs on the infrastructure provisioned just like how we live on Earth altogether. Just like moving to recycling and clean energy the only way is to go at it together and not create more divide.
    - verdverm 3y ago
      It's backwards because we used to do it that way and then upgraded to declarative IaC which gave us better reliability and confidence. It's backwards because tools like Pulumi use techniques from before we learned better. It adds more complexity and makes understanding harder. I still don't see how this is a win. The only exception is for developers who don't want to learn the best tools & techniques for IaC, pandering to their preferences over providing better systems. Sure there is a market, but that doesn't make it a good product, especially at scale. The process of writing, reading, and understanding how the infrastructure comes to be is important. If you have never been on call for a production outage, you won't know how hard it can be to make the correct fix in a stressful situation. Being able to do that is more important than how easy it is for anyone to write the initial version. Take your time writing good IaC during development, make high-stress situations easier. It's not gated behind a single language, there are multiple tools that provide declarative IaC. You would be surprised at how many folks in the Ops space have not written code beyond simple scripts to glue various tools together. It's something I require for our devops hires, but it is not necessary for all orgs. Most of the time you are writing TF, CF, or Yaml anyway. It is interesting that Pulumi now supports a Yaml interface, but at that point why use that over TF directly? In the end, Pulumi is just a wrapper around TF. Personally, I use CUE -> tf.json for IaC. It's a much better wrapper with provable correctness.
jaxxstorm 3y ago
Pulumi employee here! Worth noting that while lots of insights are leveraging LLMs/AI, it's only one pillar of the launch today. The cross cloud search and analytics are really valuable additions to the product. On day 1, being able to export all your resource definitions to BI intelligence tools and drill down into specific areas of interest is something that has traditionally been very difficult. On a personal note, being able to simply search for "who spun up this EC2 instance" without traversing through cloudtrail is a godsend, too.
cube2222 3y ago
Hey, I'm curious about the pulumi-ai cli. Specifically, did you solve the problem of stale API information? What I mean is that using gpt-4 to generate code is generally very straightforward, but due to the knowledge cutoff it won't know about i.e. new AWS APIs like Lambda URLs. Is this something you've managed to solve? Or is it just the "even with that knowledge cutoff there's enough value" situation?
- verdverm 3y ago
  There are attempts at solving this issue more generally, by giving the GPTs access to external sources of information, and ofc the right prompts & chaining
- AaronFriel 3y ago
  Pulumian here - this is something I'm working on and hopefully we'll have more to share soon. We're in a good position here in that our providers have rich schemas: https://raw.githubusercontent.com/pulumi/pulumi-kubernetes/master/provider/cmd/pulumi-resource-kubernetes/schema.json https://raw.githubusercontent.com/pulumi/pulumi-kubernetes/m... However our larger providers, primarily cloud platforms, have schemas much larger than the context length of the model. So the trick is scoping that down to the necessary & sufficient amount of data into a prompt, whether via plugin (not yet available via API), preprocessing the prompt, or using a langchain-esque approach. As Károly Zsolnai-Fehér[1] says, "what a time to be alive!" [1] of Two Minute Papers fame: https://www.patreon.com/TwoMinutePapers https://www.patreon.com/TwoMinutePapers
  - cube2222 3y ago
    Thanks for the explanation and good luck, then! Very curious to see what you come up with.
rcarr 3y ago
I had a really bad experience with Pulumi in February/March that probably cost me around 4 weeks of lost dev time, possibly more. Context: Not a DevOps guy, created a serverless site using Typescript, created the infrastructure for it on AWS console and then decided to try and replicate it using Pulumi to get some IaC skills. Issues: - The majority of the documentation/examples assume you are going to be writing all your code in one big long index.js file rather than the micro stacks approach. No idea why this is considering no-one organises their code this way in any other part of a coding project so don't know why micro stack approach would not be the default approach. - Major issues getting it to work with typescript/ts-node/tsx correctly. Could only manage to do this if I used a Pulumi Automation Runtime program rather than a Pulumi Automation Local program which was super awkward as it meant I lost access to using the CLI. - No way of testing any serverless function that called another serverless function. Would have to use something like LocalStack if you wanted to do this (and that seemed like a nightmare) or use a npm module that didn't seem like a safe long term bet. Not entirely pulumi's fault but if you want to do this it can be done with SST or with Amazon SAM. In the end I ended up giving up on Pulumi and will be rewriting the entire infrastructure code using SST. If I had an app that wasn't serverless I'd consider using it again but definitely wouldn't give it another shot for a serverless app until it either has similar testing functionality for serverless functions as SST or if there's an official way of interfacing with the SST framework. I'd also probably hold out until monorepo and Typescript support was better. My frustration levels were elevated going into the month as I built the site using a new framework that went from 1.0 to 2.0 mid development and had to learn the idiosyncrasies of that. However the month from hell with Pulumi was enough to send me over the edge and I'm now taking a few weeks off from coding to get my patience back again. Oh and I've no idea what this AI thing is like but when I tried to get ChatGPT to answer questions on Pulumi it was useless. It would flat out lie and make up classes and functions that didn't exist.

85 points by kaotown 3y ago

ohhhh this is really smart; every terraform user has at some point run up against the limits of it not being a language and flirted with pulumi. natural excuse now to get started, esp if the auto import works targeting translation between declarative-ish languages is a natural day-1 application for LLMs declarative languages generally good, bc 1) it's relatively easy to check them, 2) they are low-boilerplate (less to get wrong + better support for small token memories) wonder if this backfires bc pulumi is more powerful, giving the LLM more chances to get in trouble? like there are cases where in order to check a pulumi program you would need something like TLA+ with a good model of the system you are targeting

fdgsdfogijq 3y ago

CDK/Pulumi are going to be used by LLMs to spin up website infra exactly to spec. Devs will fill in the business logic interfaces. Going to be wild

smt88 3y ago
Why would you need LLMs for a boilerplate web application?
- verdverm 3y ago
  There is a "need" vs "want" aspect here, but often the boilerplates that we reach for become out of date. In theory, an LLM producing these could remain up to date without maintenance, key phrase "in theory"
  - trufas 3y ago
    There's no guarantee the boilerplate an LLM spits out will be up to date. It'll almost definitely have some outdated code in it's dataset that it can reference.
    - verdverm 3y ago
      yes, a basic LLM is susceptible to this. The other major issue is that they will generate different output, even with the same input. There is work going into giving the LLMs access to external data & systems. This is my basis for saying they will be able to stay up to date. I have a very different approach I'm working on: https://docs.hofstadter.io/getting-started/creators https://docs.hofstadter.io/getting-started/creators (human-made blueprints that can be started from and later updated to bring in the upstream changes)
JohnMakin 3y ago
but how will it be maintained? "IAC" seems like a bit of a misnomer here, if I'm understanding correctly. Now, if Pulumi generated terraform for me to maintain, that'd be a different matter, but this article seems to just produce cloud resources based on LLM prompts, which are not deterministic at all.
- twalla 3y ago
  It produces pulumi programs which are the equivalent of a terraform module or main.tf file. Whether or not said programs are deterministic is another matter.
- dmattia 3y ago
  Pulumi is an IAC tool similar to Terraform (it actually usually calls terraform providers over gRPC under the hood), just written in languages like Typescript/Python. I think the intention would be to use this AI prompt thing to generate pulumi and then to insert it into your codebase, just like you'd do with terraform, and then it becomes deterministic. I've used ChatGPT and other tools to generate Pulumi before, so I'm not totally sure how this specific tool from Pulumi is different, but I'd guess they've somehow trained it more specifically on their sdks/docs or something
  - JohnMakin 3y ago
    thanks for the clarification
shadycuz 3y ago
Yeah but ChatGPT can already write Terraform and Cloudformation really well. So this doesn't seem that special besides it's already baked into the pulumi eco system.
gtirloni 3y ago
> exactly to spec And that spec is?
- fdgsdfogijq 3y ago
  Natural language :)
  - diarrhea 3y ago
    Reminds me of UML. Instead of laboriously hand-writing all this nasty code, why not use a tool that can autogenerate it for us? After all, all that’s needed is a spec (UML)! Of course, the spec is the work. LLMs are then just very potent at translating it to be machine-readable.

cedws 3y ago

Sorry to be offtopic, but I've been using Pulumi at work for the past 6 months and I'm really not impressed. It's basically just Terraform but worse, with a million ways to declare your infrastructure instead of just one. Infrastructure people tend not to write the best code and from my observation the extra freedom of an imperative language just makes stuff even more complex and harder to maintain. It's also much harder to automate than Terraform, I am not aware of any equivalent to Atlantis. Also, Pulumi previews (equivalent to plans) are complete bullshit. If you don't write your code carefully, resources can be created and removed and you won't know it's going to happen until you start applying... it's an engineer's worst nightmare when a tool lies.

verdverm 3y ago
Agreement, I do not understand this backwards movement in the DevOps world. My hypothesis is that they are catering to a different group, i.e. enabling developers to do Ops, who don't want to learn TF and want to use their preferred language. DevOps first practitioners are in short supply, so it makes sense there is a market for this.
- re-thc 3y ago
  Why is it backwards? Is making it more accessible and inclusive backwards? So these "DevOps practitioners" who are so different according to you never used Python or any programming language? We should enable everyone to at least aware of Ops and be able to contribute. Why does it need to be gated behind a special language i.e. HCL? A lot of times things go rogue exactly because developers don't understand and claim to not have a need to understand because it's not their job. Ultimately the code runs on the infrastructure provisioned just like how we live on Earth altogether. Just like moving to recycling and clean energy the only way is to go at it together and not create more divide.
  - verdverm 3y ago
    It's backwards because we used to do it that way and then upgraded to declarative IaC which gave us better reliability and confidence. It's backwards because tools like Pulumi use techniques from before we learned better. It adds more complexity and makes understanding harder. I still don't see how this is a win. The only exception is for developers who don't want to learn the best tools & techniques for IaC, pandering to their preferences over providing better systems. Sure there is a market, but that doesn't make it a good product, especially at scale. The process of writing, reading, and understanding how the infrastructure comes to be is important. If you have never been on call for a production outage, you won't know how hard it can be to make the correct fix in a stressful situation. Being able to do that is more important than how easy it is for anyone to write the initial version. Take your time writing good IaC during development, make high-stress situations easier. It's not gated behind a single language, there are multiple tools that provide declarative IaC. You would be surprised at how many folks in the Ops space have not written code beyond simple scripts to glue various tools together. It's something I require for our devops hires, but it is not necessary for all orgs. Most of the time you are writing TF, CF, or Yaml anyway. It is interesting that Pulumi now supports a Yaml interface, but at that point why use that over TF directly? In the end, Pulumi is just a wrapper around TF. Personally, I use CUE -> tf.json for IaC. It's a much better wrapper with provable correctness.

jaxxstorm 3y ago

Pulumi employee here! Worth noting that while lots of insights are leveraging LLMs/AI, it's only one pillar of the launch today. The cross cloud search and analytics are really valuable additions to the product. On day 1, being able to export all your resource definitions to BI intelligence tools and drill down into specific areas of interest is something that has traditionally been very difficult. On a personal note, being able to simply search for "who spun up this EC2 instance" without traversing through cloudtrail is a godsend, too.

cube2222 3y ago

Hey, I'm curious about the pulumi-ai cli. Specifically, did you solve the problem of stale API information? What I mean is that using gpt-4 to generate code is generally very straightforward, but due to the knowledge cutoff it won't know about i.e. new AWS APIs like Lambda URLs. Is this something you've managed to solve? Or is it just the "even with that knowledge cutoff there's enough value" situation?

verdverm 3y ago
There are attempts at solving this issue more generally, by giving the GPTs access to external sources of information, and ofc the right prompts & chaining
AaronFriel 3y ago
Pulumian here - this is something I'm working on and hopefully we'll have more to share soon. We're in a good position here in that our providers have rich schemas: https://raw.githubusercontent.com/pulumi/pulumi-kubernetes/master/provider/cmd/pulumi-resource-kubernetes/schema.json https://raw.githubusercontent.com/pulumi/pulumi-kubernetes/m... However our larger providers, primarily cloud platforms, have schemas much larger than the context length of the model. So the trick is scoping that down to the necessary & sufficient amount of data into a prompt, whether via plugin (not yet available via API), preprocessing the prompt, or using a langchain-esque approach. As Károly Zsolnai-Fehér[1] says, "what a time to be alive!" [1] of Two Minute Papers fame: https://www.patreon.com/TwoMinutePapers https://www.patreon.com/TwoMinutePapers
- cube2222 3y ago
  Thanks for the explanation and good luck, then! Very curious to see what you come up with.

rcarr 3y ago

I had a really bad experience with Pulumi in February/March that probably cost me around 4 weeks of lost dev time, possibly more. Context: Not a DevOps guy, created a serverless site using Typescript, created the infrastructure for it on AWS console and then decided to try and replicate it using Pulumi to get some IaC skills. Issues: - The majority of the documentation/examples assume you are going to be writing all your code in one big long index.js file rather than the micro stacks approach. No idea why this is considering no-one organises their code this way in any other part of a coding project so don't know why micro stack approach would not be the default approach. - Major issues getting it to work with typescript/ts-node/tsx correctly. Could only manage to do this if I used a Pulumi Automation Runtime program rather than a Pulumi Automation Local program which was super awkward as it meant I lost access to using the CLI. - No way of testing any serverless function that called another serverless function. Would have to use something like LocalStack if you wanted to do this (and that seemed like a nightmare) or use a npm module that didn't seem like a safe long term bet. Not entirely pulumi's fault but if you want to do this it can be done with SST or with Amazon SAM. In the end I ended up giving up on Pulumi and will be rewriting the entire infrastructure code using SST. If I had an app that wasn't serverless I'd consider using it again but definitely wouldn't give it another shot for a serverless app until it either has similar testing functionality for serverless functions as SST or if there's an official way of interfacing with the SST framework. I'd also probably hold out until monorepo and Typescript support was better. My frustration levels were elevated going into the month as I built the site using a new framework that went from 1.0 to 2.0 mid development and had to learn the idiosyncrasies of that. However the month from hell with Pulumi was enough to send me over the edge and I'm now taking a few weeks off from coding to get my patience back again. Oh and I've no idea what this AI thing is like but when I tried to get ChatGPT to answer questions on Pulumi it was useless. It would flat out lie and make up classes and functions that didn't exist.