12 ms·

Pulumi 3.0

321 points by bovermyer 6y ago

kjjjjjjjjjjjjjj 6y ago
I just don't get why someone would use Pulumi
bloopernova 6y ago
Can someone who has used both Pulumi and AWS CDK describe the differences between the two? I am using Terraform 100% now, but sometimes wish I had more than the HCL (hashicorp configuration language) syntax available to use in my code.
- aequitas 6y ago
  You could try Terraform CDK[0] which supports TypeScript, Python, Java, and C#. [0] https://github.com/hashicorp/terraform-cdk https://github.com/hashicorp/terraform-cdk
- k__ 6y ago
  The CDK synthesizes CloudFormation templates (for AWS) and Pulumi seems to use the AWS APIs to provision and doesn't touch CloudFormation.
  - bloopernova 6y ago
    Right - I was wondering about the developer "experience" between the two.
- satya71 6y ago
  CDK maintains the state with AWS itself, Pulumi operates their own state service (with option of self management). I have never used CDK, but I've heard that it's hard to recover from state corruption in CDK.
  - jen20 6y ago
    While this is true, the reason is not called out: CDK simply generates a CloudFormation Template, and thus uses CloudFormation for state management, rather than using the cloud APIs directly.
    - bogomipz 6y ago
      Perhaps I'm misunderstanding your point but doesn't Cloudformation use the cloud APIs directly? I'm curious why the extra level of indirection is important.
      - debaserab2 6y ago
        CloudFormation does quite a bit of work to manage and calculate state differences. It's not CDK itself managing it (to my knowledge). This is noteworthy because many times CDK's constraints match CloudFormation's constraints. I've noticed a few issues in the CDK github repo that a feature is incomplete because they are waiting for a change or feature to land in CloudFormation first.
    - watermelon0 6y ago
      IME this is one of the best features of CDK, coming from someone who uses both Terraform and Cloudformation for development/production. I never have to worry about losing the state or ending with the incomplete state, and no matter what, I can always just delete the whole CF stack (in rare cases, you might have to retry deletion, but it never loses resources.)
      - jen20 6y ago
        I just saw (5 mins ago) a CloudFormation stack lose resources - it happens all the time if one is not careful to prevent all modification outside of the tool (the same as other infrastructure provisioning tools). DELETE_FAILED is not a fun state to be in, nor is it one that can easily be reasoned about. That said, I don’t think that “generates CloudFormation JSON” is a bad attribute of CDK, but it is a major difference between Pulumi and CDK. In particular it means arbitrary values cannot be used for control flow at run time with CDK, since the template has a static generation step. However, it also does not require an additional tool at runtime, which has value in some cases.
- nhoughto 6y ago
  CDK is just a different way of writing cloudformation essentially, its code that is transformed into cloudformation templates, so inherits all its problems. Pulumi is more like Terraform but without HCL. So imagine instead of having to find-the-HCL-way-to-do-things and work within those constraints to make terraform work, you can use one of a few languages to build up resources (similar to you would in HCL) and have pulumi provision/act on them for you including managing their state. Quite different to CDK/CloudFormation and much more like Terraform without the handcuffs of HCL. (sometimes handcuffs are useful tho..)
- mdeeks 6y ago
  In addition to what others have said, it is important to point out that the CDK is AWS specific. Your "infrastructure" is rarely just "AWS" even if you're not multi-cloud. Terraform and Pulumi can operate on things outside of AWS like Github, Okta, Cloudflare, Datadog, Hashicorp Vault, etc. It is nice to have one system or even repo for managing all of that. https://www.pulumi.com/docs/intro/vs/cloud_template_transpilers/ https://www.pulumi.com/docs/intro/vs/cloud_template_transpil...
  - cmclaughlin 6y ago
    > In addition to what others have said, it is important to point out that the CDK is AWS specific. As mentioned in other comments here, CDK for Terraform is another option and it's not AWS specific - it should work with any Terraform provider and there are heaps available. https://github.com/hashicorp/terraform-cdk https://github.com/hashicorp/terraform-cdk
    - mdeeks 6y ago
      Oh interesting (and kind of weird). I didn't know this was an option. Though it does say this at the top: > This experimental repository contains software which is still being developed and in the alpha testing stage. It is not ready for production use.
      - simtel20 6y ago
        It also seems to make using modules awkward or impossible, making non trivial use of the terraform cdk impractical.
        redeux 6y ago
        Interesting. I used the Terraform CDK to deploy Typhoon from a module on terraform registry without an issue.
- rossmohax 6y ago
  I used both in depth. CDK as it is now is a wrapper around CloudFormation and therefore suffers from same limitations: AWS only, 'static' templates. Pulimi on the other hand is closer to terraform, except you can write in a language which actually doesn't stand in your way and allows you to programmatically access values available during execution only. With CDK you can only reference them in CF template (lets say instance ID which you just created), but these are never "lifted" to your code. Also CDK misses native 'datasources' and offers limited mechanism for lookups. Pulumi also has a killer Automatioin featre, where you can code infrastructure migrations, not unlike you'd do it for SQL. Neither TF nor CDK allows you to do that, and you'd need to code infrastructure state transitions in error prone bash wrappers. As a developer, I quite liked Pulumi resource model, documentation and team responsiveness on Github and slack channel. I am working with CDK now, because of costs and I'd prefer Pulumi if there was a choice.
  - tasssko 6y ago
    When i looked the SDK was only available in certain languages which was a issue for me at the time. I just checked and they have python and csharp. Pulumi has potential to reduce the verbosity of infrastructure configuration and enable engineers to tailor it more to the application on demand. Adding a new queue can be done as part of a app release and not handled in a separate pipeline. It isn’t fun to wrangle infrastructure pipelines and when necessary orchestrate changes.
  - fulafel 6y ago
    The migration style changes sound really interesting. CDK indeed is just another layer of abstraction on top of CloudFormation, itself already a leaky complex abstraction, and it bites you in a lot of ways.
    - nickcox 6y ago
      Same could be sad for Pulumi though, couldn't it? It's a leaky abstraction on top of Terraform which is a leaky abstraction on top of $YourProvider's API. I'm still much happier using CDK vs CF or Pulumi vs HCL though.
- carlosf 6y ago
  I migrated from Terraform to AWS CDK and I'm very happy. The good: - Tooling for any major language (Typescript, Python)… is lightyears ahead of anything you will find for Terraform / HCL. - You can write your code as declaratively as possible (like you would do using TF), but you always have the escape hatch of using all libs available to your language of choice. - AWS CDK uses CloudFormation under the hood, so you get cool stuff like automatic rollbacks in case of failure. - You have access to much more mature testing frameworks, compared to what is available to TF. Because AWS CDK synthetize CF templates, you can also snapshot those for regression testing. Applying good software engineering practices is overall much easier. Most languages are much easier to extend than HCL. - AWS is constantly releasing higher level libraries so you don't need to fiddle with low level API details. When using Terraform, you generally must understand your cloud provider API at a very fine level to implement IaC. The bad: - AWS CDK is written in Typescript and automatically translated to other languages. Even though you can use Python, C#, etc... Most examples and tutorials exist in Typescript. Also, you always must install npm + cdk + the libraries in the language you are using, so using any language other than Typescript means supporting two toolchains, which is a pain in the ass. I started with AWS CDK in Python and now I'm migrating to Typescript. - Some modules only exist for Typescript. - Since CF is used under the hood, it only supports resources supported by CF. Sometimes CF support takes a while. Since TF is just a wrapper for a Cloud Provider API, it generally implements new resources much quicker. - It has its own vocabulary and and ways to do stuff (L1/L2 Construct? Stack? Retention Policy?)... even if you've used a lot of TF and know AWS very well, there's a learning curve. Some cool videos: - An AWS dev shows how to create your own construct and test it (that's the less "toy example" tutorial I have found): https://www.youtube.com/watch?v=cTsSXYOYQPw https://www.youtube.com/watch?v=cTsSXYOYQPw - Same guy shows how to contribute to AWS CDK. I've found that looking at their source code is a great way to learn about good practices and patterns: https://www.youtube.com/watch?v=OXQSSibrt-A https://www.youtube.com/watch?v=OXQSSibrt-A
- manigandham 6y ago
  All those frameworks have their own DSL/config languages that lack expressiveness and inevitably become a really bad scripting language to support everything people want to do. Pulumi started with actual programming languages instead which gives you full expressiveness, IDE integration, and more, without learning yet another new language. And since infrastructure is just represented as objects in the language, creating and combining them in complex ways is much more natural than wiring up configs.
methyl 6y ago
Native Provider means it no longer uses Terraform under the hood and it excites me a lot. We were having hard time with some more complex setups and it seems it can be remedy for issues that we faced. Stoked to check it out!
- leg100 6y ago
  According to [1], "Native providers are built directly and automatically from the cloud provider’s API...", and are not hand-coded, unlike Terraform's providers. I'm surprised this is possible. If it was, why didn't Terraform follow this approach a long time ago? The cloud providers' APIs provide the endpoints for creating, updating, reading and deleting resources. Terraform and Pulumi's value is to provide an idempotent abstraction on top of that. And that abstraction is not straightforward to write, because it has to handle numerous nuances and anachronisms in the underlying APIs. For example, in the event of updating an existing storage bucket, the abstraction has to determine whether the bucket can be simply updated, or if it needs to be re-created (say if you were changing the name or location). And the underlying API will not necessarily reveal this kind of information. Hence one would think completely avoiding handwritten code is incredibly difficult if not an insurmountable problem. [1]: https://www.pulumi.com/blog/pulumiup-native-providers/ https://www.pulumi.com/blog/pulumiup-native-providers/
  - k__ 6y ago
    Wouldn't this mean that Pulumi has better coverage than CloudFormation? Some things in the AWS API aren't (yet) available in CFN.
    - leg100 6y ago
      It would mean Pulumi has better coverage than CFN and terraform. Indeed as their blurb states, Pulumi would have 100% coverage. But I'm skeptical.
    - chucky_z 6y ago
      afaict the aws terraform module tends to have 100% coverage. i've never found a resource that didn't work, or didn't already have some way to use. i was able to deploy eks when it was still in beta through terraform, while i think the cloudformation stuff for eks lagged behind like almost by a year.
octopoc 6y ago
The Pulumi Automation API lets you have an API to update infrastructure from your own process instead of via CLI commands that call your executable. I hope this makes it easy to implement multi-tenancy by spinning up separate infrastructure for different customers. This would make it easier for software for dynamic schema built on top of a relational database. Dynamic schema is one area where most libraries don't help you out much, but it's an increasingly important feature, especially for businesses.
- deleted 6y ago
  [deleted]
- giovannibonetti 6y ago
  Some relational databases like Postgres and MySQL support JSON columns. This is useful if, for example, you want to create an ecommerce application with multiple product variations stored in a single table, but with different fields - that is called Single Table Inheritance (STI). You can have some regular columns for the common fields, and a JSON column for the specific ones. If your app is built with Rails, you can use this library to help you on that (I'm not affiliated with it): https://github.com/jrochkind/attr_json https://github.com/jrochkind/attr_json
  - vdfs 6y ago
    Is search in large tables slow in Postgres JSON columns? can it be index?
    - paulryanrogers 6y ago
      PostgreSQL JSON can be indexed and searched. They also have a JSONB format which is more efficient. Though I believe Mysql's JSON operators are more standards compliant.
mssundaram 6y ago
I am having trouble understand what Pulumi does, since I rarely work in web architecture. Could someone explain it?
- adflux 6y ago
  Define your cloud infrastructure in code (python, typescript, other languages) and deploy it with ease. I think it beats other formats.
- tomca32 6y ago
  It's a library around Terraform that lets you write infra stuff in Python, JavaScript, Go, and maybe some other langs. For me the biggest benefit is the fact that I can write my infra with Types in TypeScript.
  - spooneybarger 6y ago
    It's no longer built around terraform. It uses the cloud provider APIs directly now. See native provider.
    - tomca32 6y ago
      oh I didn't realize this. Thanks for pointing it out
- qbasic_forever 6y ago
  It's infrastructure-as-code to the logical conclusion that you write real programming language code (Python, Javascript, Go, etc.) to build your infrastructure. Something like Terraform lets you declare infrastructure with HCL/JSON but doesn't give you a lot of flexibility to reason about or dynamically change things. If you want to spin up a frontend server for every backend for example, you have to write out the configs and tweak them to match the desired state. Pulumi and similar tools go the next step and let you write code to build your infrastructure--if you want your frontends and backends to be in sync, write some Python to make it happen instead of spelunking in a pile of configs. It's super useful and compelling at scale, but there are some potential downsides. For one, it's a real program and IMHO you should treat it as such with the same rigor of documentation, testing, etc. as the production code you're deploying. This might actually make your processes a bit slower or more risky. Because if you don't do that then you just risk building unmaintainable, untrustworthy deployment code that nobody wants to use. The second downside is you're taking a big bet on Pulumi to stick around. At this point, it's probably a safe bet. But the more you use their SDK and system the more tightly coupled your production deployment is to the continued existence of Pulumi. If they go bust and can't maintain the SDK you might have a production system that can't be deployed anymore.
andrewSC 6y ago
"Self-hosting is only available with the Pulumi Enterprise Edition. Contact us if you would like to evaluate the Self-Hosted Enterprise Edition." [0] [0] https://www.pulumi.com/docs/guides/self-hosted/ https://www.pulumi.com/docs/guides/self-hosted/
- Jgrubb 6y ago
  Just curious, why'd you post that bit?
  - all2 6y ago
    A big appeal of "open source" is control. Not allowing self-hosting for non-paid tiers is cutting control off from the user and moving it to the producer.
    - xyzzy123 6y ago
      This is just the state storage backend. You can use Pulumi cloud or keep your state in a bucket, or write your own backend. Terraform cloud is a similar deal.
  - nine_k 6y ago
    Because it says: «the next major version of the Pulumi open source project». Apparently, either not very complete, or not very open-source, if self-hosting requires a license.
    - jacques_chester 6y ago
      I suspect the key words are "Enterprise Edition". There will be features that matter to larger customers that matter less to others.
- spooneybarger 6y ago
  You don't need to use the service, although it solves a lot of problems. You can use only the Pulumi SDK and then solve the various state management issues that their service provides in another fashion. I think its a reasonable business model. The SDK is completely open and free. The service is how they make money .
fock 6y ago
just because I don't get it from their marketing: their API is a cloud-service wrapping other cloudservices. Or is it a classical library with wrappers?
- spooneybarger 6y ago
  The Pulumi cloud offering provides state management. With it, you can bring up some infrastructure on machine A and tear it down on another machine. Without the service, you can only tear down on the same machine you brought it up. Ditto for adding to existing infrastructure. You don't need their service but you have to solve problems like knowing what is running, replicating the needed state etc. The service is very nice and makes it easy to know what is running, how long it has been running, who created it etc- across clouds. But, it is not strictly needed. As an individual user, you could get away without using. For a team, it fills a very vital gap.
  - bogomipz 6y ago
    Is the difference the same as using Terraform Cloud vs just keeping your state in your own S3 bucket then?
  - pm90 6y ago
    This seems like a huge con. Terraform at least allows you to store state yourself if you want to on multiple backends. With Pulumi you have to pay to store state, and you have to store it with Pulumi Inc. ? Why?
    - Chickenosaurus 6y ago
      Pulumi also allows storing state yourself in different backends [1]. The SaaS is a convenience. [1] https://www.pulumi.com/docs/intro/concepts/state/ https://www.pulumi.com/docs/intro/concepts/state/
- aChrisSmith 6y ago
  It’s a bit more involved than that… Suppose you wrote a program to go and create your cloud infrastructure. Using AWS’ APIs you write an app to spin up a VM, setup a load balancer, and provision a database. You run your program, it works like a charm. Now you have all the infrastructure setup just the way you want it! The problem is what happens when you want to _update_ that infrastructure? The program you wrote, using the aws.CreateVM(..) and aws.CreateLoadBalancer(..) API calls is no longer applicable. Instead, you probably want to use a different set of APIs, like aws.UpdateVM(…), to update your existing cloud resources. For example, change some port settings on your load balancer. So your app needs to be smart enough to check if the resources already exist, and if so, update them. Otherwise, create them fresh. And it gets even worse. What if you want to create some new resources, such as attach an SSL certificate to your load balancer… but still keep all of your existing infrastructure as-is. Or what if you want to update an IAM usage policy that is already in-use by several other resources… Somehow your app needs to know the impact of that change, and how it will ripple out across other cloud resources. Does that start to make sense? You don’t really want a “wrapper” for cloud APIs. You really want something that allows you to effectively describe your cloud infrastructure, and “make it happen”. And leave the specifics of “how” as an implementation detail… accomplished by a cloud provider’s APIs. That is what Pulumi does — and other Infrastructure as Code tools, like Terraform. It provides you a way to describe your cloud infrastructure in a programming language, so that every time you run your app it will make the cloud reflect that target state. It will: - Create resources if they don’t exist. - Update existing resources if they do. - Delete any resources that you no longer need. I work at Pulumi and am happy to go into details about the joys of not dealing with cloud APIs directly, and just using Pulumi :)
wpietri 6y ago
Do folks have opinions on how Pulumi compares to Terraform CDK? I'm the only ops-y person in my group. I've done things so far with Terraform HCL, but a) it's not a great language, and b) I can't really ask other folks to learn it. But they all know Python, so my theory is that if I can wrap our ops stuff up in familiar-looking code, they'll be able to work with it effectively. Is one of these toolkits better than the other? I'm inclined to go with Terraform CDK just because they company's further along, but I'd love other people's takes.
- cmclaughlin 6y ago
  > Do folks have opinions on how Pulumi compares to Terraform CDK? I last tried Pulumi around the 2.0 release and it flat-out didn't work for me. After working through a few bugs with their tech sales I gave up on it. I'm guessing their TypeScript support is better than their Python - for me it just wasn't ready for production. On the other hand I've been using Terraform CDK in production since the early alpha releases. Had to work around a couple of bugs here and there, but no blockers at all. It's been a game changer for my team and we're really enjoying it!
  - wpietri 6y ago
    Thanks! Very helpful.
- nsonha 6y ago
  just looking from far away I don't see any point of Pulumi while Terraform now has a CDK, for now Pulumi has play to catch up with Terraform on providers
chrischen 6y ago
Any thoughts on Pulumi vs Terraform?
- whydoineedthis 6y ago
  imho, if you think typecasting the AWS api in your local code repository is useful, than Pulumi is the way to go. Maybe one day someone will find an api that AWS didn't properly typecast before releasing and then the hundreds of hours of typecasting in your local repo will be worth it. Other than that, it's really annoying way to keep infrastructure. I prefer the Terraform DSL myself, but I'm certain there is an audience that disagrees with me.
  - cies 6y ago
    I dont understand your 2 opening statements: > if you think typecasting the AWS api in your local code repository is useful, then Pulumi is the way to go. What do you mean by typecasting here? Isn't providing the types what Pulumi (or TF CDK for that sake) does by the libs it provides? I dot see how I, the consumer of those libs, needs to "type cast" anything. > someone will find an api that AWS didn't properly typecast before releasing and then the hundreds of hours of typecasting in your local repo will be worth it. ?
- cmclaughlin 6y ago
  > Any thoughts on Pulumi vs Terraform? If you're just getting started with Terraform be sure to check out CDK for Terraform. It allows you to drive Terraform from a variety of high-level programming languages instead of it's DSL. https://github.com/hashicorp/terraform-cdk/ https://github.com/hashicorp/terraform-cdk/
  - StavrosK 6y ago
    > Getting Started with Python. Prerequisites: Terraform >= v0.12, Node.js >= v12.16 What? :(
    - flurie 6y ago
ncmncm 6y ago
Imagine, a release announcement containing a sentence that bothers to say what the thing being announced is. They will surely lose points for reduced brevity.
dith3r 6y ago
Using Pulumi for about 1 year to manage gcp and AWS environments (5+) - each deployment is around 500 resources. At the begining it was buggy but when they started to add typing i see a lot of improvment.
daralthus 6y ago
<3 Pulumi. It's really nice defining what you want to exist at a cloud provider or as a k8s service, and it will make sure that it is there and nothing else. All in typescript.
catern 6y ago
I hope Pulumi eventually develops a "bare-metal" mode. I'm very much in favor of using a real programming language to set up systems[0], but I don't like to use cloud services beyond basic hosting. It would be nicer if Pulumi provided, as a library, code to launch and maintain self-hosted databases, proxies, etc, rather than just providing clients for cloud APIs. [0] http://catern.com/progsys.html http://catern.com/progsys.html
- octopoc 6y ago
  IIUC it already does this. Many of the libraries Pulumi exposes are not specific to any cloud platform. Examples: - Docker: https://www.pulumi.com/docs/reference/pkg/docker/ https://www.pulumi.com/docs/reference/pkg/docker/ - Postgresql: https://www.pulumi.com/docs/intro/cloud-providers/postgresql/ https://www.pulumi.com/docs/intro/cloud-providers/postgresql... - Keycloak: https://www.pulumi.com/docs/reference/pkg/keycloak/ https://www.pulumi.com/docs/reference/pkg/keycloak/
  - catern 6y ago
    I believe those are libraries for using these services (e.g., connecting to them and allocating resources inside them), not launching them and setting them up.
    - httgp 6y ago
      Folks generally use Ansible to launch and configure stuff; it is better suited for that kind of work.
      - nickcox 6y ago
        How so?
- adevx 6y ago
  A 100 times this. I would drop Ansible in heartbeat for Pulumi and have a strictly typed IaC base. Hope this use case gets more traction.
throwaway823882 6y ago
Dear Pulumi: Do you have an HCL to Pulumi converter thingy? The first big barrier to using Pulumi in my org is that they basically banned anything other than Terraform "because standardization". I would love to get rid of TF, but we need a migration path. The next big stumbling block is your license...
- jaxxstorm 6y ago
  Yes we do! https://www.pulumi.com/tf2pulumi/ https://www.pulumi.com/tf2pulumi/ https://github.com/pulumi/tf2pulumi https://github.com/pulumi/tf2pulumi
kodeninja 6y ago
Pulumi looks super interesting! Is there a tutorial out there that demonstrates using _just_ the Pulumi SDK, perhaps in a CI/CD setting, to deploy a simple service to Kubernetes? Many thanks!
haolez 6y ago
I've been really interested in the code inlining feature[0], even though it's just available for JavaScript/TypeScript. What's holding me back right now is that supported for Azure Functions is still in its infancy for code inlining. It's a shame, since I'm stuck in Azure for now. [0] https://www.pulumi.com/docs/intro/concepts/function-serialization/ https://www.pulumi.com/docs/intro/concepts/function-serializ...
TruthWillHurt 6y ago
I'm sorry, I really liked Pulumi, but it became too complex after a short while. Since all infra resources are defined as code you end up defining them in a procedural manner (Python), and struggle to arrange/resolve dependencies.
- happyrock 6y ago
  I agree, I enjoyed using it and found it very powerful, but it was tricky to organize the code in a way that kept things simple, and there weren't many great examples to draw from in the documentation or elsewhere. I'd love to see what a large multi-provider Pulumi deployment looks like when it's done the right way.
ynouri 6y ago
Glad to see all the innovation happening in the IaC space. I like the multi-language premise of Pulumi but I have the feeling that over time one of the languages will gain a significant advantage over the others (better supported, more features, native...) and thus will become the primary language for Pulumi package authors and users (similar to HCL for Terraform and TypeScript for CDK). Meaning that eventually all the "serious" Pulumi users will end up converging on one single language (similar to this comment of a CDK user migrating from Python to TypeScript). Feedback loops will only help amplify this effect (e.g. more Pulumi users on language A -> more examples online -> more package authors working in this language -> better established best practices -> more submitted issue tickets for language A -> language better supported by Pulumi -> more users...) Curious about how Pulumi multi-language components [0] work under the hood. Isn't writing a "Pulumi schema" describing resources the same as writing declarative HCL at the end of the day? [0] https://www.pulumi.com/blog/pulumiup-pulumi-packages-multi-language-components/ https://www.pulumi.com/blog/pulumiup-pulumi-packages-multi-l...
- joeduffy 6y ago
  Excellent topic. [caveat, Pulumi co-founder here] Indeed what you say is true of many other "multi-language" platforms. I was an early engineer on .NET at Microsoft, and although it was multi-language from the outset (COBAL.NET was a thing!), the reality is most folks write in C# these days. And yet, you still see a lot of excitement for PowerShell, Visual Basic, and F#, each of which has a rich community, but uses that same common core. A similar phenomenon has happened in the JVM ecosystem with Java dominating most usage until the late 2000s, at which point my impression is that Groovy, Scala, and now Kotlin won significant mindshare. I have reasons to be optimistic the infrastructure language domain will play out similarly. Especially as we are fundamentally serving multiple audiences -- we see that infrastructure teams often elect Python, while developers often go with TypeScript or Go, because they are more familiar with it. For those scenarios, the new multi-language package support is essential, since many companies have both sorts of engineers working together. A "default language for IaC" may emerge with time, but I suspect that is more likely to be Python than, say, HCL. (And even then, I'm not so sure it will happen.) One of the things I'm ridiculously excited about, by the way, is bringing IaC to new audiences -- many folks learn Python at school, not so much for other infrastructure languages. Again, I'm biased. But, even if a default emerges, I guarantee there will be reasons for the others to exist. I for one am a big functional language fan and especially for simple serverless apps, I love seeing that written in F#. And we've had a ton of interest in PowerShell support since many folks working with infrastructure for the Microsoft stack know it. And Ruby due to the Chef and Puppet journeys. I also won't discount the idea of us introducing a cloud infrastructure-specific language ;-). But it would be more general purpose than not. I worked a lot on parallel computing in the mid-2000s and that temptation was always there, but I'm glad we resisted it and instead just added tasks/promises and await to existing languages. As to the Pulumi schema, you're right, that's a step we aim to remove soon. For TypeScript, we'll generate it off the d.ts files; for Go we'll use struct tags; and so on. Now that the basic runtime is in place, we are now going to focus there. This issue tracks it: https://github.com/pulumi/pulumi/issues/6804 https://github.com/pulumi/pulumi/issues/6804. Our goal is to make this as ridiculously easy as just writing a type/class in your language of choice.
mbStavola 6y ago
I'm in love with Pulumi. Simply one of the best tools I have in my stack. After about a year of use, I simply cannot go back to editing YAML files or clicking things in a web UI. It's ruined me completely.
deleted 6y ago
[deleted]
Kunigaikstis 6y ago
I've been using Pulumi for the past few months as a person who's never touched infrastructure-as-code before. I love it. It has its downsides in the documentation. Mostly around limitations in resource name lengths. Apart from that, it's an amazing tool to work with. The company I'm working at right now uses it extensively. All of our microservices have Pulumi in the CI/CD pipeline. It's an extremely powerful tool and I'd even use it for personal projects over provisioning resources through a vendor like AWS.
- kjuulh 6y ago
  Yeah, it has been a while since I used it, but I remember having to fall back on Terraform docs more than once. But it really is nice being able to ditch Terraform DSL with your language of choice. And when I used it they also had quite permissive pricing for personal use, where you could store your tfstate or equivalent on their servers which is nice as well for solo devs, who doesn't want to build or shell our for a storage platform or CI for terraform. The only thing I had some problems with was errors, which wasn't the easiest to decode. But it wasn't too bad.
  - Kunigaikstis 6y ago
    Agree with the errors but I believe they've gotten better/you get used to their true meaning after a while. It's a complex piece of technology so I don't bash them for it too much
- lloydatkinson 6y ago
  How do you add it to CICD? Does it pull down and create infra every new push to the master branch? What does the workflow look like?
muxator 6y ago
Last time I checked (admittedly, not so hard) I discovered that Pulumi only works by calling in to the mothership: i.e.: you cannot build an airgapped infrastructure with it. This was very off-putting for me, and I had to give up on Pulumi. Is something changed on this regard?
- hzia 6y ago
  You totally can. They support storing pulumi state in any storage platform (like S3 or self hosted minio service) The cloud offering is just there to streamline the storage + fancy dashboard. Identical to how terraform works.
  - muxator 6y ago
    Thank you. So this is a case of me not looking well enough while exploring many different solutions. I have to say that reading this passage on one of the getting started guides (https://www.pulumi.com/docs/get-started/azure/create-project/ https://www.pulumi.com/docs/get-started/azure/create-project...): > If this is your first time running pulumi new or most other pulumi commands, you will be prompted to log in to the Pulumi service. > The Pulumi CLI works in tandem with the Pulumi service in order to deliver a reliable experience. It is free for individual use, with features available for teams. did not help. At least mentioning that the Pulumi service could be self hosted would have helped. Thanks again.
    - happyrock 6y ago
      I agree – their marketing site doesn't exactly go out of its way to help you find out whether there's a free option.
lloydatkinson 6y ago
I’m just getting into infrastructure stuff like this, how does Pulumi work with CI/CD? Does it check what infrastructure it needs to create every deploy? What assurances are there that it won’t delete the whole production database every push?
- Kunigaikstis 6y ago
  We use it exclusively in CI/CD to provision/delete resources. It keeps track of everything created via Pulumi (or imported from outside of it) in a config on the cloud. Haven't experienced random resource deletion apart from genuine user error.
PostThisTooFast 6y ago
Which is? Post a description in your title, not just a name. Come on.
rswail 6y ago
A couple of years ago I started down the Pulumi path, but what was unclear at the time is that there are two phases of a pulumi run. The first is on constructing the DAG of resources, the second is using that DAG to orchestrate the changes from the previous state. The problem I had is that values that are only available in the second phase (eg a subnet ID) can't be accessed in the first phase directly. The subnet's ID is accessed via "my_subnet.id" but it's actually effectively a future. You can't write something like "if my_subnet.id == "1234" (this is an arbitrary example), but effectively a reference to a resource's attributes is only available on the "right hand side" of an expression, and pretty much as a simple assignment only. To use the value of an attribute, you essentially need to access the value in the second phase of the pulumi process, and to do that (at the time) involved essentially pushing through source code to be "eval"ed on the "other side". I'll be interested to see what's changed :)
bionhoward 6y ago
Twice I've tried to love Pulumi but wind up going back to Terraform because the Terraform provider docs put clear usage examples front and center
- supernovae 6y ago
  I do believe the constraints of TF and similar DSLs are what makes them great.

321 points by bovermyer 6y ago

I just don't get why someone would use Pulumi

Can someone who has used both Pulumi and AWS CDK describe the differences between the two? I am using Terraform 100% now, but sometimes wish I had more than the HCL (hashicorp configuration language) syntax available to use in my code.

aequitas 6y ago
You could try Terraform CDK[0] which supports TypeScript, Python, Java, and C#. [0] https://github.com/hashicorp/terraform-cdk https://github.com/hashicorp/terraform-cdk
k__ 6y ago
The CDK synthesizes CloudFormation templates (for AWS) and Pulumi seems to use the AWS APIs to provision and doesn't touch CloudFormation.
- bloopernova 6y ago
  Right - I was wondering about the developer "experience" between the two.
satya71 6y ago
CDK maintains the state with AWS itself, Pulumi operates their own state service (with option of self management). I have never used CDK, but I've heard that it's hard to recover from state corruption in CDK.
- jen20 6y ago
  While this is true, the reason is not called out: CDK simply generates a CloudFormation Template, and thus uses CloudFormation for state management, rather than using the cloud APIs directly.
  - bogomipz 6y ago
    Perhaps I'm misunderstanding your point but doesn't Cloudformation use the cloud APIs directly? I'm curious why the extra level of indirection is important.
    - debaserab2 6y ago
      CloudFormation does quite a bit of work to manage and calculate state differences. It's not CDK itself managing it (to my knowledge). This is noteworthy because many times CDK's constraints match CloudFormation's constraints. I've noticed a few issues in the CDK github repo that a feature is incomplete because they are waiting for a change or feature to land in CloudFormation first.
  - watermelon0 6y ago
    IME this is one of the best features of CDK, coming from someone who uses both Terraform and Cloudformation for development/production. I never have to worry about losing the state or ending with the incomplete state, and no matter what, I can always just delete the whole CF stack (in rare cases, you might have to retry deletion, but it never loses resources.)
    - jen20 6y ago
      I just saw (5 mins ago) a CloudFormation stack lose resources - it happens all the time if one is not careful to prevent all modification outside of the tool (the same as other infrastructure provisioning tools). DELETE_FAILED is not a fun state to be in, nor is it one that can easily be reasoned about. That said, I don’t think that “generates CloudFormation JSON” is a bad attribute of CDK, but it is a major difference between Pulumi and CDK. In particular it means arbitrary values cannot be used for control flow at run time with CDK, since the template has a static generation step. However, it also does not require an additional tool at runtime, which has value in some cases.
nhoughto 6y ago
CDK is just a different way of writing cloudformation essentially, its code that is transformed into cloudformation templates, so inherits all its problems. Pulumi is more like Terraform but without HCL. So imagine instead of having to find-the-HCL-way-to-do-things and work within those constraints to make terraform work, you can use one of a few languages to build up resources (similar to you would in HCL) and have pulumi provision/act on them for you including managing their state. Quite different to CDK/CloudFormation and much more like Terraform without the handcuffs of HCL. (sometimes handcuffs are useful tho..)
mdeeks 6y ago
In addition to what others have said, it is important to point out that the CDK is AWS specific. Your "infrastructure" is rarely just "AWS" even if you're not multi-cloud. Terraform and Pulumi can operate on things outside of AWS like Github, Okta, Cloudflare, Datadog, Hashicorp Vault, etc. It is nice to have one system or even repo for managing all of that. https://www.pulumi.com/docs/intro/vs/cloud_template_transpilers/ https://www.pulumi.com/docs/intro/vs/cloud_template_transpil...
- cmclaughlin 6y ago
  > In addition to what others have said, it is important to point out that the CDK is AWS specific. As mentioned in other comments here, CDK for Terraform is another option and it's not AWS specific - it should work with any Terraform provider and there are heaps available. https://github.com/hashicorp/terraform-cdk https://github.com/hashicorp/terraform-cdk
  - mdeeks 6y ago
    Oh interesting (and kind of weird). I didn't know this was an option. Though it does say this at the top: > This experimental repository contains software which is still being developed and in the alpha testing stage. It is not ready for production use.
    - simtel20 6y ago
      It also seems to make using modules awkward or impossible, making non trivial use of the terraform cdk impractical.
      - redeux 6y ago
        Interesting. I used the Terraform CDK to deploy Typhoon from a module on terraform registry without an issue.
rossmohax 6y ago
I used both in depth. CDK as it is now is a wrapper around CloudFormation and therefore suffers from same limitations: AWS only, 'static' templates. Pulimi on the other hand is closer to terraform, except you can write in a language which actually doesn't stand in your way and allows you to programmatically access values available during execution only. With CDK you can only reference them in CF template (lets say instance ID which you just created), but these are never "lifted" to your code. Also CDK misses native 'datasources' and offers limited mechanism for lookups. Pulumi also has a killer Automatioin featre, where you can code infrastructure migrations, not unlike you'd do it for SQL. Neither TF nor CDK allows you to do that, and you'd need to code infrastructure state transitions in error prone bash wrappers. As a developer, I quite liked Pulumi resource model, documentation and team responsiveness on Github and slack channel. I am working with CDK now, because of costs and I'd prefer Pulumi if there was a choice.
- tasssko 6y ago
  When i looked the SDK was only available in certain languages which was a issue for me at the time. I just checked and they have python and csharp. Pulumi has potential to reduce the verbosity of infrastructure configuration and enable engineers to tailor it more to the application on demand. Adding a new queue can be done as part of a app release and not handled in a separate pipeline. It isn’t fun to wrangle infrastructure pipelines and when necessary orchestrate changes.
- fulafel 6y ago
  The migration style changes sound really interesting. CDK indeed is just another layer of abstraction on top of CloudFormation, itself already a leaky complex abstraction, and it bites you in a lot of ways.
  - nickcox 6y ago
    Same could be sad for Pulumi though, couldn't it? It's a leaky abstraction on top of Terraform which is a leaky abstraction on top of $YourProvider's API. I'm still much happier using CDK vs CF or Pulumi vs HCL though.
carlosf 6y ago
I migrated from Terraform to AWS CDK and I'm very happy. The good: - Tooling for any major language (Typescript, Python)… is lightyears ahead of anything you will find for Terraform / HCL. - You can write your code as declaratively as possible (like you would do using TF), but you always have the escape hatch of using all libs available to your language of choice. - AWS CDK uses CloudFormation under the hood, so you get cool stuff like automatic rollbacks in case of failure. - You have access to much more mature testing frameworks, compared to what is available to TF. Because AWS CDK synthetize CF templates, you can also snapshot those for regression testing. Applying good software engineering practices is overall much easier. Most languages are much easier to extend than HCL. - AWS is constantly releasing higher level libraries so you don't need to fiddle with low level API details. When using Terraform, you generally must understand your cloud provider API at a very fine level to implement IaC. The bad: - AWS CDK is written in Typescript and automatically translated to other languages. Even though you can use Python, C#, etc... Most examples and tutorials exist in Typescript. Also, you always must install npm + cdk + the libraries in the language you are using, so using any language other than Typescript means supporting two toolchains, which is a pain in the ass. I started with AWS CDK in Python and now I'm migrating to Typescript. - Some modules only exist for Typescript. - Since CF is used under the hood, it only supports resources supported by CF. Sometimes CF support takes a while. Since TF is just a wrapper for a Cloud Provider API, it generally implements new resources much quicker. - It has its own vocabulary and and ways to do stuff (L1/L2 Construct? Stack? Retention Policy?)... even if you've used a lot of TF and know AWS very well, there's a learning curve. Some cool videos: - An AWS dev shows how to create your own construct and test it (that's the less "toy example" tutorial I have found): https://www.youtube.com/watch?v=cTsSXYOYQPw https://www.youtube.com/watch?v=cTsSXYOYQPw - Same guy shows how to contribute to AWS CDK. I've found that looking at their source code is a great way to learn about good practices and patterns: https://www.youtube.com/watch?v=OXQSSibrt-A https://www.youtube.com/watch?v=OXQSSibrt-A
manigandham 6y ago
All those frameworks have their own DSL/config languages that lack expressiveness and inevitably become a really bad scripting language to support everything people want to do. Pulumi started with actual programming languages instead which gives you full expressiveness, IDE integration, and more, without learning yet another new language. And since infrastructure is just represented as objects in the language, creating and combining them in complex ways is much more natural than wiring up configs.

methyl 6y ago

Native Provider means it no longer uses Terraform under the hood and it excites me a lot. We were having hard time with some more complex setups and it seems it can be remedy for issues that we faced. Stoked to check it out!

leg100 6y ago
According to [1], "Native providers are built directly and automatically from the cloud provider’s API...", and are not hand-coded, unlike Terraform's providers. I'm surprised this is possible. If it was, why didn't Terraform follow this approach a long time ago? The cloud providers' APIs provide the endpoints for creating, updating, reading and deleting resources. Terraform and Pulumi's value is to provide an idempotent abstraction on top of that. And that abstraction is not straightforward to write, because it has to handle numerous nuances and anachronisms in the underlying APIs. For example, in the event of updating an existing storage bucket, the abstraction has to determine whether the bucket can be simply updated, or if it needs to be re-created (say if you were changing the name or location). And the underlying API will not necessarily reveal this kind of information. Hence one would think completely avoiding handwritten code is incredibly difficult if not an insurmountable problem. [1]: https://www.pulumi.com/blog/pulumiup-native-providers/ https://www.pulumi.com/blog/pulumiup-native-providers/
- k__ 6y ago
  Wouldn't this mean that Pulumi has better coverage than CloudFormation? Some things in the AWS API aren't (yet) available in CFN.
  - leg100 6y ago
    It would mean Pulumi has better coverage than CFN and terraform. Indeed as their blurb states, Pulumi would have 100% coverage. But I'm skeptical.
  - chucky_z 6y ago
    afaict the aws terraform module tends to have 100% coverage. i've never found a resource that didn't work, or didn't already have some way to use. i was able to deploy eks when it was still in beta through terraform, while i think the cloudformation stuff for eks lagged behind like almost by a year.

octopoc 6y ago

The Pulumi Automation API lets you have an API to update infrastructure from your own process instead of via CLI commands that call your executable. I hope this makes it easy to implement multi-tenancy by spinning up separate infrastructure for different customers. This would make it easier for software for dynamic schema built on top of a relational database. Dynamic schema is one area where most libraries don't help you out much, but it's an increasingly important feature, especially for businesses.

deleted 6y ago
[deleted]
giovannibonetti 6y ago
Some relational databases like Postgres and MySQL support JSON columns. This is useful if, for example, you want to create an ecommerce application with multiple product variations stored in a single table, but with different fields - that is called Single Table Inheritance (STI). You can have some regular columns for the common fields, and a JSON column for the specific ones. If your app is built with Rails, you can use this library to help you on that (I'm not affiliated with it): https://github.com/jrochkind/attr_json https://github.com/jrochkind/attr_json
- vdfs 6y ago
  Is search in large tables slow in Postgres JSON columns? can it be index?
  - paulryanrogers 6y ago
    PostgreSQL JSON can be indexed and searched. They also have a JSONB format which is more efficient. Though I believe Mysql's JSON operators are more standards compliant.

mssundaram 6y ago

I am having trouble understand what Pulumi does, since I rarely work in web architecture. Could someone explain it?

adflux 6y ago
Define your cloud infrastructure in code (python, typescript, other languages) and deploy it with ease. I think it beats other formats.
tomca32 6y ago
It's a library around Terraform that lets you write infra stuff in Python, JavaScript, Go, and maybe some other langs. For me the biggest benefit is the fact that I can write my infra with Types in TypeScript.
- spooneybarger 6y ago
  It's no longer built around terraform. It uses the cloud provider APIs directly now. See native provider.
  - tomca32 6y ago
    oh I didn't realize this. Thanks for pointing it out
qbasic_forever 6y ago
It's infrastructure-as-code to the logical conclusion that you write real programming language code (Python, Javascript, Go, etc.) to build your infrastructure. Something like Terraform lets you declare infrastructure with HCL/JSON but doesn't give you a lot of flexibility to reason about or dynamically change things. If you want to spin up a frontend server for every backend for example, you have to write out the configs and tweak them to match the desired state. Pulumi and similar tools go the next step and let you write code to build your infrastructure--if you want your frontends and backends to be in sync, write some Python to make it happen instead of spelunking in a pile of configs. It's super useful and compelling at scale, but there are some potential downsides. For one, it's a real program and IMHO you should treat it as such with the same rigor of documentation, testing, etc. as the production code you're deploying. This might actually make your processes a bit slower or more risky. Because if you don't do that then you just risk building unmaintainable, untrustworthy deployment code that nobody wants to use. The second downside is you're taking a big bet on Pulumi to stick around. At this point, it's probably a safe bet. But the more you use their SDK and system the more tightly coupled your production deployment is to the continued existence of Pulumi. If they go bust and can't maintain the SDK you might have a production system that can't be deployed anymore.

andrewSC 6y ago

"Self-hosting is only available with the Pulumi Enterprise Edition. Contact us if you would like to evaluate the Self-Hosted Enterprise Edition." [0] [0] https://www.pulumi.com/docs/guides/self-hosted/ https://www.pulumi.com/docs/guides/self-hosted/

Jgrubb 6y ago
Just curious, why'd you post that bit?
- all2 6y ago
  A big appeal of "open source" is control. Not allowing self-hosting for non-paid tiers is cutting control off from the user and moving it to the producer.
  - xyzzy123 6y ago
    This is just the state storage backend. You can use Pulumi cloud or keep your state in a bucket, or write your own backend. Terraform cloud is a similar deal.
- nine_k 6y ago
  Because it says: «the next major version of the Pulumi open source project». Apparently, either not very complete, or not very open-source, if self-hosting requires a license.
  - jacques_chester 6y ago
    I suspect the key words are "Enterprise Edition". There will be features that matter to larger customers that matter less to others.
spooneybarger 6y ago
You don't need to use the service, although it solves a lot of problems. You can use only the Pulumi SDK and then solve the various state management issues that their service provides in another fashion. I think its a reasonable business model. The SDK is completely open and free. The service is how they make money .

fock 6y ago

just because I don't get it from their marketing: their API is a cloud-service wrapping other cloudservices. Or is it a classical library with wrappers?

spooneybarger 6y ago
The Pulumi cloud offering provides state management. With it, you can bring up some infrastructure on machine A and tear it down on another machine. Without the service, you can only tear down on the same machine you brought it up. Ditto for adding to existing infrastructure. You don't need their service but you have to solve problems like knowing what is running, replicating the needed state etc. The service is very nice and makes it easy to know what is running, how long it has been running, who created it etc- across clouds. But, it is not strictly needed. As an individual user, you could get away without using. For a team, it fills a very vital gap.
- bogomipz 6y ago
  Is the difference the same as using Terraform Cloud vs just keeping your state in your own S3 bucket then?
- pm90 6y ago
  This seems like a huge con. Terraform at least allows you to store state yourself if you want to on multiple backends. With Pulumi you have to pay to store state, and you have to store it with Pulumi Inc. ? Why?
  - Chickenosaurus 6y ago
    Pulumi also allows storing state yourself in different backends [1]. The SaaS is a convenience. [1] https://www.pulumi.com/docs/intro/concepts/state/ https://www.pulumi.com/docs/intro/concepts/state/
aChrisSmith 6y ago
It’s a bit more involved than that… Suppose you wrote a program to go and create your cloud infrastructure. Using AWS’ APIs you write an app to spin up a VM, setup a load balancer, and provision a database. You run your program, it works like a charm. Now you have all the infrastructure setup just the way you want it! The problem is what happens when you want to _update_ that infrastructure? The program you wrote, using the aws.CreateVM(..) and aws.CreateLoadBalancer(..) API calls is no longer applicable. Instead, you probably want to use a different set of APIs, like aws.UpdateVM(…), to update your existing cloud resources. For example, change some port settings on your load balancer. So your app needs to be smart enough to check if the resources already exist, and if so, update them. Otherwise, create them fresh. And it gets even worse. What if you want to create some new resources, such as attach an SSL certificate to your load balancer… but still keep all of your existing infrastructure as-is. Or what if you want to update an IAM usage policy that is already in-use by several other resources… Somehow your app needs to know the impact of that change, and how it will ripple out across other cloud resources. Does that start to make sense? You don’t really want a “wrapper” for cloud APIs. You really want something that allows you to effectively describe your cloud infrastructure, and “make it happen”. And leave the specifics of “how” as an implementation detail… accomplished by a cloud provider’s APIs. That is what Pulumi does — and other Infrastructure as Code tools, like Terraform. It provides you a way to describe your cloud infrastructure in a programming language, so that every time you run your app it will make the cloud reflect that target state. It will: - Create resources if they don’t exist. - Update existing resources if they do. - Delete any resources that you no longer need. I work at Pulumi and am happy to go into details about the joys of not dealing with cloud APIs directly, and just using Pulumi :)

wpietri 6y ago

Do folks have opinions on how Pulumi compares to Terraform CDK? I'm the only ops-y person in my group. I've done things so far with Terraform HCL, but a) it's not a great language, and b) I can't really ask other folks to learn it. But they all know Python, so my theory is that if I can wrap our ops stuff up in familiar-looking code, they'll be able to work with it effectively. Is one of these toolkits better than the other? I'm inclined to go with Terraform CDK just because they company's further along, but I'd love other people's takes.

cmclaughlin 6y ago
> Do folks have opinions on how Pulumi compares to Terraform CDK? I last tried Pulumi around the 2.0 release and it flat-out didn't work for me. After working through a few bugs with their tech sales I gave up on it. I'm guessing their TypeScript support is better than their Python - for me it just wasn't ready for production. On the other hand I've been using Terraform CDK in production since the early alpha releases. Had to work around a couple of bugs here and there, but no blockers at all. It's been a game changer for my team and we're really enjoying it!
- wpietri 6y ago
  Thanks! Very helpful.
nsonha 6y ago
just looking from far away I don't see any point of Pulumi while Terraform now has a CDK, for now Pulumi has play to catch up with Terraform on providers

chrischen 6y ago

Any thoughts on Pulumi vs Terraform?

whydoineedthis 6y ago
imho, if you think typecasting the AWS api in your local code repository is useful, than Pulumi is the way to go. Maybe one day someone will find an api that AWS didn't properly typecast before releasing and then the hundreds of hours of typecasting in your local repo will be worth it. Other than that, it's really annoying way to keep infrastructure. I prefer the Terraform DSL myself, but I'm certain there is an audience that disagrees with me.
- cies 6y ago
  I dont understand your 2 opening statements: > if you think typecasting the AWS api in your local code repository is useful, then Pulumi is the way to go. What do you mean by typecasting here? Isn't providing the types what Pulumi (or TF CDK for that sake) does by the libs it provides? I dot see how I, the consumer of those libs, needs to "type cast" anything. > someone will find an api that AWS didn't properly typecast before releasing and then the hundreds of hours of typecasting in your local repo will be worth it. ?
cmclaughlin 6y ago
> Any thoughts on Pulumi vs Terraform? If you're just getting started with Terraform be sure to check out CDK for Terraform. It allows you to drive Terraform from a variety of high-level programming languages instead of it's DSL. https://github.com/hashicorp/terraform-cdk/ https://github.com/hashicorp/terraform-cdk/
- StavrosK 6y ago
  > Getting Started with Python. Prerequisites: Terraform >= v0.12, Node.js >= v12.16 What? :(
  - flurie 6y ago

ncmncm 6y ago

Imagine, a release announcement containing a sentence that bothers to say what the thing being announced is. They will surely lose points for reduced brevity.

dith3r 6y ago

Using Pulumi for about 1 year to manage gcp and AWS environments (5+) - each deployment is around 500 resources. At the begining it was buggy but when they started to add typing i see a lot of improvment.

daralthus 6y ago

<3 Pulumi. It's really nice defining what you want to exist at a cloud provider or as a k8s service, and it will make sure that it is there and nothing else. All in typescript.

catern 6y ago

I hope Pulumi eventually develops a "bare-metal" mode. I'm very much in favor of using a real programming language to set up systems[0], but I don't like to use cloud services beyond basic hosting. It would be nicer if Pulumi provided, as a library, code to launch and maintain self-hosted databases, proxies, etc, rather than just providing clients for cloud APIs. [0] http://catern.com/progsys.html http://catern.com/progsys.html

octopoc 6y ago
IIUC it already does this. Many of the libraries Pulumi exposes are not specific to any cloud platform. Examples: - Docker: https://www.pulumi.com/docs/reference/pkg/docker/ https://www.pulumi.com/docs/reference/pkg/docker/ - Postgresql: https://www.pulumi.com/docs/intro/cloud-providers/postgresql/ https://www.pulumi.com/docs/intro/cloud-providers/postgresql... - Keycloak: https://www.pulumi.com/docs/reference/pkg/keycloak/ https://www.pulumi.com/docs/reference/pkg/keycloak/
- catern 6y ago
  I believe those are libraries for using these services (e.g., connecting to them and allocating resources inside them), not launching them and setting them up.
  - httgp 6y ago
    Folks generally use Ansible to launch and configure stuff; it is better suited for that kind of work.
    - nickcox 6y ago
      How so?
adevx 6y ago
A 100 times this. I would drop Ansible in heartbeat for Pulumi and have a strictly typed IaC base. Hope this use case gets more traction.

throwaway823882 6y ago

Dear Pulumi: Do you have an HCL to Pulumi converter thingy? The first big barrier to using Pulumi in my org is that they basically banned anything other than Terraform "because standardization". I would love to get rid of TF, but we need a migration path. The next big stumbling block is your license...

jaxxstorm 6y ago
Yes we do! https://www.pulumi.com/tf2pulumi/ https://www.pulumi.com/tf2pulumi/ https://github.com/pulumi/tf2pulumi https://github.com/pulumi/tf2pulumi

kodeninja 6y ago

Pulumi looks super interesting! Is there a tutorial out there that demonstrates using _just_ the Pulumi SDK, perhaps in a CI/CD setting, to deploy a simple service to Kubernetes? Many thanks!

haolez 6y ago

I've been really interested in the code inlining feature[0], even though it's just available for JavaScript/TypeScript. What's holding me back right now is that supported for Azure Functions is still in its infancy for code inlining. It's a shame, since I'm stuck in Azure for now. [0] https://www.pulumi.com/docs/intro/concepts/function-serialization/ https://www.pulumi.com/docs/intro/concepts/function-serializ...

TruthWillHurt 6y ago

I'm sorry, I really liked Pulumi, but it became too complex after a short while. Since all infra resources are defined as code you end up defining them in a procedural manner (Python), and struggle to arrange/resolve dependencies.

happyrock 6y ago
I agree, I enjoyed using it and found it very powerful, but it was tricky to organize the code in a way that kept things simple, and there weren't many great examples to draw from in the documentation or elsewhere. I'd love to see what a large multi-provider Pulumi deployment looks like when it's done the right way.

ynouri 6y ago

Glad to see all the innovation happening in the IaC space. I like the multi-language premise of Pulumi but I have the feeling that over time one of the languages will gain a significant advantage over the others (better supported, more features, native...) and thus will become the primary language for Pulumi package authors and users (similar to HCL for Terraform and TypeScript for CDK). Meaning that eventually all the "serious" Pulumi users will end up converging on one single language (similar to this comment of a CDK user migrating from Python to TypeScript). Feedback loops will only help amplify this effect (e.g. more Pulumi users on language A -> more examples online -> more package authors working in this language -> better established best practices -> more submitted issue tickets for language A -> language better supported by Pulumi -> more users...) Curious about how Pulumi multi-language components [0] work under the hood. Isn't writing a "Pulumi schema" describing resources the same as writing declarative HCL at the end of the day? [0] https://www.pulumi.com/blog/pulumiup-pulumi-packages-multi-language-components/ https://www.pulumi.com/blog/pulumiup-pulumi-packages-multi-l...

joeduffy 6y ago
Excellent topic. [caveat, Pulumi co-founder here] Indeed what you say is true of many other "multi-language" platforms. I was an early engineer on .NET at Microsoft, and although it was multi-language from the outset (COBAL.NET was a thing!), the reality is most folks write in C# these days. And yet, you still see a lot of excitement for PowerShell, Visual Basic, and F#, each of which has a rich community, but uses that same common core. A similar phenomenon has happened in the JVM ecosystem with Java dominating most usage until the late 2000s, at which point my impression is that Groovy, Scala, and now Kotlin won significant mindshare. I have reasons to be optimistic the infrastructure language domain will play out similarly. Especially as we are fundamentally serving multiple audiences -- we see that infrastructure teams often elect Python, while developers often go with TypeScript or Go, because they are more familiar with it. For those scenarios, the new multi-language package support is essential, since many companies have both sorts of engineers working together. A "default language for IaC" may emerge with time, but I suspect that is more likely to be Python than, say, HCL. (And even then, I'm not so sure it will happen.) One of the things I'm ridiculously excited about, by the way, is bringing IaC to new audiences -- many folks learn Python at school, not so much for other infrastructure languages. Again, I'm biased. But, even if a default emerges, I guarantee there will be reasons for the others to exist. I for one am a big functional language fan and especially for simple serverless apps, I love seeing that written in F#. And we've had a ton of interest in PowerShell support since many folks working with infrastructure for the Microsoft stack know it. And Ruby due to the Chef and Puppet journeys. I also won't discount the idea of us introducing a cloud infrastructure-specific language ;-). But it would be more general purpose than not. I worked a lot on parallel computing in the mid-2000s and that temptation was always there, but I'm glad we resisted it and instead just added tasks/promises and await to existing languages. As to the Pulumi schema, you're right, that's a step we aim to remove soon. For TypeScript, we'll generate it off the d.ts files; for Go we'll use struct tags; and so on. Now that the basic runtime is in place, we are now going to focus there. This issue tracks it: https://github.com/pulumi/pulumi/issues/6804 https://github.com/pulumi/pulumi/issues/6804. Our goal is to make this as ridiculously easy as just writing a type/class in your language of choice.

mbStavola 6y ago

I'm in love with Pulumi. Simply one of the best tools I have in my stack. After about a year of use, I simply cannot go back to editing YAML files or clicking things in a web UI. It's ruined me completely.

deleted 6y ago

[deleted]

Kunigaikstis 6y ago

I've been using Pulumi for the past few months as a person who's never touched infrastructure-as-code before. I love it. It has its downsides in the documentation. Mostly around limitations in resource name lengths. Apart from that, it's an amazing tool to work with. The company I'm working at right now uses it extensively. All of our microservices have Pulumi in the CI/CD pipeline. It's an extremely powerful tool and I'd even use it for personal projects over provisioning resources through a vendor like AWS.

kjuulh 6y ago
Yeah, it has been a while since I used it, but I remember having to fall back on Terraform docs more than once. But it really is nice being able to ditch Terraform DSL with your language of choice. And when I used it they also had quite permissive pricing for personal use, where you could store your tfstate or equivalent on their servers which is nice as well for solo devs, who doesn't want to build or shell our for a storage platform or CI for terraform. The only thing I had some problems with was errors, which wasn't the easiest to decode. But it wasn't too bad.
- Kunigaikstis 6y ago
  Agree with the errors but I believe they've gotten better/you get used to their true meaning after a while. It's a complex piece of technology so I don't bash them for it too much
lloydatkinson 6y ago
How do you add it to CICD? Does it pull down and create infra every new push to the master branch? What does the workflow look like?

muxator 6y ago

Last time I checked (admittedly, not so hard) I discovered that Pulumi only works by calling in to the mothership: i.e.: you cannot build an airgapped infrastructure with it. This was very off-putting for me, and I had to give up on Pulumi. Is something changed on this regard?

hzia 6y ago
You totally can. They support storing pulumi state in any storage platform (like S3 or self hosted minio service) The cloud offering is just there to streamline the storage + fancy dashboard. Identical to how terraform works.
- muxator 6y ago
  Thank you. So this is a case of me not looking well enough while exploring many different solutions. I have to say that reading this passage on one of the getting started guides (https://www.pulumi.com/docs/get-started/azure/create-project/ https://www.pulumi.com/docs/get-started/azure/create-project...): > If this is your first time running pulumi new or most other pulumi commands, you will be prompted to log in to the Pulumi service. > The Pulumi CLI works in tandem with the Pulumi service in order to deliver a reliable experience. It is free for individual use, with features available for teams. did not help. At least mentioning that the Pulumi service could be self hosted would have helped. Thanks again.
  - happyrock 6y ago
    I agree – their marketing site doesn't exactly go out of its way to help you find out whether there's a free option.

lloydatkinson 6y ago

I’m just getting into infrastructure stuff like this, how does Pulumi work with CI/CD? Does it check what infrastructure it needs to create every deploy? What assurances are there that it won’t delete the whole production database every push?

Kunigaikstis 6y ago
We use it exclusively in CI/CD to provision/delete resources. It keeps track of everything created via Pulumi (or imported from outside of it) in a config on the cloud. Haven't experienced random resource deletion apart from genuine user error.

PostThisTooFast 6y ago

Which is? Post a description in your title, not just a name. Come on.

rswail 6y ago

A couple of years ago I started down the Pulumi path, but what was unclear at the time is that there are two phases of a pulumi run. The first is on constructing the DAG of resources, the second is using that DAG to orchestrate the changes from the previous state. The problem I had is that values that are only available in the second phase (eg a subnet ID) can't be accessed in the first phase directly. The subnet's ID is accessed via "my_subnet.id" but it's actually effectively a future. You can't write something like "if my_subnet.id == "1234" (this is an arbitrary example), but effectively a reference to a resource's attributes is only available on the "right hand side" of an expression, and pretty much as a simple assignment only. To use the value of an attribute, you essentially need to access the value in the second phase of the pulumi process, and to do that (at the time) involved essentially pushing through source code to be "eval"ed on the "other side". I'll be interested to see what's changed :)

bionhoward 6y ago

Twice I've tried to love Pulumi but wind up going back to Terraform because the Terraform provider docs put clear usage examples front and center

supernovae 6y ago
I do believe the constraints of TF and similar DSLs are what makes them great.