10 ms·

Pulumi – A new open-source cloud development platform

317 points by MikusR 8y ago

spooneybarger 8y ago
We've had a chance to try out Pulumi when it was in private beta. We are very happy with it so far. Having used a variety of tools in the space, this is the first time I ever had an engineer say to me: "I really like it, I hope we can keep using it."
polskibus 8y ago
It's a pity they didn't offer .net support from the start. I wonder what were the reasons, besides popularity contest.
- ellism 8y ago
  > I wonder what were the reasons, besides popularity contest. (Disclosure: I'm a member of the Pulumi team). Strictly a timing issue. We brought TypeScript/JavaScript online first and most of our focus was there while building up the supporting tooling. Fortunately, the design of the system makes it pretty easy to add new languages (we've since then done Python and Go). Most of the team is ex-.NET folks and I've been itching to build out the .NET support. Now that we've launched, it will at least become /my/ spare time project. I'll be really excited when we have it well. Do keep your eye out!
  - thrww20180618 8y ago
    Are you worried that adding "non-hip" languages from the start will alienate the kind of Microsoft-avoiding crowd you as a corporation are trying so hard to win? Is Pulumi some sort of open-ended research project like Midori? If I want to offer it to an enterprise client shall I say "don't worry one of their guys might add C#/Java in his spare time if he has a chance"?
- pjmlp 8y ago
  Yep, lacking Java and .NET support makes it a bit un-appealing to me. On the other hand, it looks quite interesting and is good to finally be able to know what Pulumi is all about. Looking forward how it evolves.
  - ellism 8y ago
    You'all spoke and we've listened. I've dusted off my old very barebones prototype, pushed it to a branch at https://github.com/pulumi/pulumi/tree/ellismg/add-dotnet https://github.com/pulumi/pulumi/tree/ellismg/add-dotnet, and will be hacking on this over the course of the next week. Collaborators are welcome (please do join the #contribute channel in our slack at https://slack.pulumi.io https://slack.pulumi.io). And watch the branch, I expect we'll have something usable in a few days and it might help folks who are interested in supporting other languages with Pulumi. In the meantime, as someone who had to learn TypeScript to start using Pulumi when I started here, I do encourage you to look at some of the examples we have and play around with them. It's actually a pretty nice language and with VSCode it's pretty easy to get started. (Disclosure: I still work at Pulumi)
    - thrww20180618 8y ago
      Problem with TypeScript is that it's a temporary bridging language until we have proper WASM toolchains and WASM DOM/GC integration for real languages. Just look at Blazor go! So betting on TS on _server side_ of all places is quite weird.
meritt 8y ago
> How do we know? We don’t need to Pulumi looks like clever technology but the complete detachment from the underlying services and associated costs seems like the perfect storm for an extremely expensive disaster. How does Pulumi keep track of which services are launched, especially during testing/development, and how does it ensure those are shut down once they are no longer needed? How does it determine the optimal size of instances/volumes/etc to launch?
- jen20 8y ago
  That detachment certainly has not harmed Terraform (some of the providers for which are used by Pulumi) - so I'm not sure it's as big an issue as you make out?
- aChrisSmith 8y ago
  Hi! I work at Pulumi and have been using it to standup and manage all of our service infrastructure. > How does Pulumi keep track of which services are launched, especially during testing/development Each Pulumi program is ran within the context of "a stack". The stack is essentially a collection of cloud resources. So when the Pulumi program runs, it will create resources that aren't in the stack, or update existing ones. So if you create any resources during dev/testing, you just need to `pulumi destroy` those stacks and all of the cloud resources will be reclaimed. This, IMHO, is one of Pulumi's best features. In that it makes it super-easy to create your own instance of a cloud application. For example, I have my own development instance of app.pulumi.com by just creating my own Pulumi stack and rerunning the same application. > How does it determine the optimal size of instances/volumes/etc to launch? It doesn't. The Pulumi program ran determines what resources to create. So you are left to configure, tune or tweak that as makes sense.
  - atombender 8y ago
    From the examples it looks like Pulumi programs declare their infrastructure, causing it to be created. Doesn't that mean that the program will need privileged credentials? How do you make sure the app only has, say, read access to an S3 bucket it needs to listen to, and can't accidentally delete it? And how does that then allow it to declare the bucket?
antics 8y ago
I was a part of the alpha testing and I liked it so much I added Kubernetes support in my free time. A thing I notice about the space is that existing tooling tends to be very focused on a specific persona (e.g., ops), and when you want to do anything outside those use cases, it starts to feel like you're jamming a square peg into a round hole. So, e.g., Go templates/sed/awk work great if all you want to do is template a couple YAML files, but if your use cases is even slightly more demanding (sharing, dependencies, transforming), it gets painful quick. Using a "real language" makes all of this much simpler. You can write tooling that captures the needs you actually have. If you have an ops team that wants a mostly-declarative or mostly-hermitic API, you can write that, and you don't have to map it to CloudFormation, ARM templates, Terraform, etc. Huge win. You can write JS/TS/Python libraries that provide secure defaults for your core infrastructure components. You can use standard dev tools to audit and refactor code you've written. In Terraform, modules have proven to be very useful. For Pulumi, I'm interested to see what the tooling ecosystem will look like as the product becomes more mature.
garganzol 8y ago
[deleted per personal request]
- sidcool 8y ago
  What are you talking about?
  - garganzol 8y ago
    .NET, future, responsibility. In that order. Tangentially touching human ego and how it can spoil the intentions. Unless one finds the way to work with it. Sorry if that's too intense.
    - maslam 8y ago
      Joe's a really nice guy. Have you ever worked with him or been in a meeting with him? Sorry, this type of ad-hominem attack really does not belong here.
      - garganzol 8y ago
        Sorry. I hear you. I'm deleting the original message.
        maslam 8y ago
        Thank you. Kudos for doing the right thing :)
fiatjaf 8y ago
What is an open-source cloud development platform? I've read the first 5 paragraphs of the blog post and skimmed through the landing page, I've also clicked on the "38 lines of code" example, and I still don't understand where does this fits. I imagined it was something like Heroku, but I guess I was wrong. I'll keep trying to understand it, anyway.
- lindydonna 8y ago
  (I'm a product manager at Pulumi.) Pulumi lets you describe cloud resources using code instead of a config language. It's not like Heroku, it's more like a deployment tool (e.g. Serverless Framework, Terraform, Claudia.js, Chalice, etc). The difference compared to other deployment tools is that you use regular code, but it's turned into a declarative plan when you run `pulumi update`. So, you get the benefits of a regular programming language, while still following best practices of immutable infrastructure. Does that help?
  - akavel 8y ago
    Then is this something like NixOps [1]? Or how does it differ? [1] https://nixos.org/nixops/ https://nixos.org/nixops/
    - deleted 8y ago
      [deleted]
    - cwp 8y ago
      Joe mentions working with "existing language package managers". That probably doesn't include nixpkgs, but perhaps it could.
andyidsinga 8y ago
this looks really interesting - keep going!
startupdiscuss 8y ago
Reading through it, it sounds very exciting and productive for people who already know what it is and can use it. But for a complete newbie. Say someone who can write an app and push it to Heroku, what is this and why might it be worth trying out?
- antics 8y ago
  Pulumi is meant to replace CloudFormation/ARM Templates/Kubernetes YAML. So if you write those things to provision infrastructure, this could be for you. If you are writing (e.g.) Rails code that you `git push` to a PaaS like Heroku, then they're provisioning this infrastructure on your behalf. So this is a bit lower level than that.
avip 8y ago
When I read a blog about "cloud development platform", I expect to understand how it fairs vs. Terraform. I'm already using Terraform. All the cloud "consultants" I know of, use Terraform. In order to not use it, I need a reason - I need something that Pulumi can do, and Terraform can't.
- lindydonna 8y ago
  (I'm a product manager on Pulumi.) If Terraform works for you, then definitely continue using it. Pulumi is just another option and works well for building libraries and components. I did a rundown of different tools for serverless apps at Velocity SF last week. Slides are here: https://cdn.oreillystatic.com/en/assets/1/event/270/Tooling%20in%20the%20age%20of%20serverless%20computing%20Presentation.pdf https://cdn.oreillystatic.com/en/assets/1/event/270/Tooling%... and GitHub samples are here: https://github.com/lindydonna/velocity-examples https://github.com/lindydonna/velocity-examples There are a ton of tools in this space, all with overlapping functionality, so there tends not to be a clear comparison of tool X vs tool Y.
  - avip 8y ago
    Hi, thank you for the slides and repo - lots of good material there. Just in case I've failed to clarify my intention: I'd really like to know the upsides of "Pulumi". I'm sure it was built for a reason (i.e dissatisfaction wrt other tools). I just don't have the bandwidth to blindly invest myself in tech without some promise that something will end for the better for me or my company. Thanks!
    - lindydonna 8y ago
      The main upside is that it's just regular code. This is more natural for some folks, but mainly the advantage is that you can build abstractions. So, you can build a component that's built up from lower-level resources in AWS. For example, directly setting up API Gateway requires a specific setup (and can be error-prone), so there's a Pulumi library for that. My colleague wrote a blog post with more of the technical motivation: https://medium.com/@lukeh/programming-the-cloud-e795cafffc2b https://medium.com/@lukeh/programming-the-cloud-e795cafffc2b, and there's also this Twitch live-coding video: https://youtu.be/DM8Wd4f1MNA https://youtu.be/DM8Wd4f1MNA
ethanwillis 8y ago
I wish this worked on bare metal. The options for running on premise cloud are slim to none and the CM tools for managing that have shaky support. It's such a shame. I don't want to just be another customer for DO, AWS, Azure, etc.
bthornbury 8y ago
Been looking forward to the launch, congrats to all involved on pulumi team!
rcarmo 8y ago
Python 2 support only, which is sad. I fully intend to try out the Azure provider, but without access to Python libraries there is not much I can create off the cuff. Also, this reminds me a bit of metaparticle, which I’m quite fond of.
- slsinsea 8y ago
  I was able to play with the Python support in the Beta. The team intends to stick with 2/3 compatible code and support Python 3 next milestone, so you should be able to try it out soon.
catern 8y ago
Hmm, I was excited to see a former Midori developer working on a "cloud development platform". And I was even more excited to read: >Using Pulumi, you author cloud programs using your favorite language, spanning low-level infrastructure-as-code to highly productive and modern container- and serverless-powered applications. I'm enthusiastic about the prospect of using a real programming language to deploy and control a distributed system. It would be fantastic to have a library that gives me a nice, clean API for performing low-level distributed system tasks, manipulating and controlling hosts at the level of individual processes and sockets. But I was disappointed to see that Pulumi doesn't seem to be designed to support deployment onto bare metal. In fact it seems to be just another layer of abstraction over various SaaS providers, or a huge dependency like Kubernetes. Why add another layer? Today's systems are already heavily abstracted. For me, the value of using a really expressive programming language to implement a system, is that you can keep more of the abstraction in the language, and thereby avoid complexity in the underlying architecture. To me, it is not very interesting to layer yet another library on top of the Kubernetes API. I'd be much more interested to see an attempt to replace the Kubernetes API with a library. This seems to be the former, and not oriented towards the latter, which is a bit of a disappointment.
- joeduffy 8y ago
  > Pulumi doesn't seem to be designed to support deployment onto bare metal. In principle, there is no reason we couldn't support bare metal, provided we have an orchestrator. Pulumi bottoms out on raw CRUD operations that can do anything. It turns out that, when targeting existing cloud providers, that means invoking their APIs. As you say, this is just a layer of abstraction, but the key aspect at the center here is diffable immutable infrastructure. I was, in fact, inspired to pursue Pulumi in part because of the experience of building a distributed OS that ran on bare metal. Many of the same challenges Pulumi solves around configuration, composition, and the application model, were inspired by this experience, and we've taken some of those lessons and applied them to cloud programs. > you can keep more of the abstraction in the language, and thereby avoid complexity in the underlying architecture. I look at other ways we've integrated with systems as inspiration here. For instance, we can productively code using tasks, and async await, these days, and reap the benefits of concurrency, despite there being massive differences between threading, I/O, and scheduling behavior across different systems. There was abstraction in the language, but it bottoms out on a set of primitives. In a sense, I view the current APIs we're using as the syscall layer for a given cloud, Kubernetes included. (Also note, we don't require depending on Kubernetes.) That's not to say everybody will want to eschew the syscalls -- there's always a time and a place -- but we can be more productive by using the abstractions. I believe that frameworks are languages, in that they give you a set of nouns and verbs you can use to express yourself. I do think there's opportunity for language-level innovation here, but if there's one thing I've taken away from our industry experiences building PaaS's, it's that it's a losing battle to bet against the incredible momentum of the public cloud vendors. > I'd be much more interested to see an attempt to replace the Kubernetes API with a library. We are just getting started, so expect to see a lot more progress in this direction in the weeks to come. The foundation of languages is a huge enabler for us, and we already have several projects in the works that I think will make you happier than this initial release. YMMV, this is just my opinion, and I really love hearing the feedback. Thank you for sharing. This is a vibrant space and time and I'm enjoying the diversity of perspective.
digitalsanctum 8y ago
Looks very interesting. I have a couple of questions: 1. One of my pet peeves with CloudFormation is the disconnect between the state of CF and the "actual" state if someone decides to update resources by not using CF. Does Pulumi do checks to verify that the actual state matches its assumed state? 2. Is there a guide for contributing a new language and/or provider? Also, I tried getting an invite for the Pulumi Slack community and was greeted with "invite_limit_reached".
- lindydonna 8y ago
  > Also, I tried getting an invite for the Pulumi Slack community and was greeted with "invite_limit_reached". We're hitting Slack's rate limiting on invites. Please DM your email @PulumiCorp on Twitter and we'll get you added!
- elcritch 8y ago
  I’d like to know the answer to that as well... Error handling semantics across languages vary greatly. Does the user code need to handle errors or mismatched state or does Pulumi? If Pulumi does how do you get/ report errors?
  - swgillespie 8y ago
    [Disclosure - I'm a Pulumi employee] There is a top-level CLI verb called `refresh` that explicitly checks Pulumi's idea of the state of the world against the actual state of the world. If there's a diff, Pulumi reconciles the diff by updating its own state. `pulumi update` does not do this by default, but you can run `pulumi refresh` at any time to verify that your state has not drifted from reality.
amanzi 8y ago
This looks interesting but I really don't like how you're prompted to download and run a shell script from their website without any indication of what it does. I'm all for making it easy to get started, but at least explain what the script does.
- ellism 8y ago
  Thanks for the feedback! The script will download the latest version of Pulumi, extract it to ~/.pulumi/bin and then try to add that folder to your .bashrc or .bash_profile. You should be able to read the source of the script directly at https://get.pulumi.com/ https://get.pulumi.com/ or even download it yourself, inspect it and then decide if you want to run it or not. You can also follow the manual install instructions which are listed a little further down on our install page at https://pulumi.io/install/ https://pulumi.io/install/, if you want total control over where it is installed. I also just opened a pull request to our docs website: https://github.com/pulumi/docs/pull/430 https://github.com/pulumi/docs/pull/430 to explain in a little more detail what we're doing with the scripts.
  - amanzi 8y ago
    Thanks for responding. :-)
  - monch1962 8y ago
    I'd appreciate it if I could install Pulumi via e.g. $ brew install pulumi That way I'd have a convenient way of keeping it updated to the latest version
teyc 8y ago
Hi Joe, congrats on the launch. What is difference in the use case for Pulumi compared to scripting infrastructure using vendor provided libraries e.g. New-AzureRmVm et c
- joeduffy 8y ago
  > congrats on the launch. Thanks -- we're very excited to be out and talking to the community! > What is difference in the use case for Pulumi compared to scripting infrastructure using vendor provided libraries The major difference is that Pulumi does immutable infrastructure. The code describes a goal state and our system manages your environment to ensure that it matches the goal state. This means you can preview changes before making them, and that, if you do make them, you've got a full audit trail of who changed what and when. Rollbacks are trivial because you just point us at an old goal state worked and we can chew on it until your live environment matches. We can even detect drift, so if someone manually updates in the cloud console, we can tell you about it. As a result, we always have a full object graph of your cloud resources, and can tie it back to the source code which created it, opening up some unique diagnostics capabilities. The difference with scripting libraries, like AWS's Boto, or the Azure PowerShell SDK, is that they mutate cloud resources directly, and in an ad-hoc manner. So, you don't know what they are going to do before you run them. And in the event of failure, you're more likely to be in a corrupt state and unable to recover. Rollbacks are difficult. There's also no audit trail beyond the cloud access logs, and information like dependencies are lost, so resources end up disassociated from the code and workflow that created or updated them. Many people encounter these problems and need to build complex systems on top to address them. Or they end up using a solution like CloudFormation or Azure Resource Manager, which eschews code in favor of JSON/YAML/templates. The hope here is that we've landed on a sweet spot between immutable infrastructure and fully programmable languages.
Edmond 8y ago
I am coming at this from a competing perspective so take it with a grain of salt...isn't the whole idea of the declarative approach that users/Ops can focus only on specifying what they want and let whatever target execution run-time handle the actual semantics? An approach that goes back to relying on procedural programming languages as a primary means of configuration seems like it defeats the purpose. I am working in this space, mainly focused on configurations: https://youtu.be/4yepPOznakk https://youtu.be/4yepPOznakk The approach we've adopted is to maintain the declarative presentation while providing a way to plug in procedural language programmability without using templates or other similar concoctions (ksonnet..etc).
deleted 8y ago
[deleted]
atombender 8y ago
I may be wrong, but from what I can tell this is really a souped-up framework for working with clouds. Most of it looks like primitive for immutable infrastructure, so Terraform but as a library. As far as can tell, there's no deployment system, no orchestration etc. -- a "Pulumi app" doesn't know how to deploy itself. Is that accurate? So if, for example, you're running an app on GKE, then you can tell the Kubernetes API about the resources you want, but there's no system for building your app (Docker build + push to GCR), no way to do supervised rolling updates, and so on. If you're doing CI/CD, then presumably you have to build, push, then let the Pulumi code inject the right image and tag. I like the idea of declaring the infrastructure with extensible code (as opposed to a DSL like Terraform where you're completely dependent on upstream support for all the objects you want to manage), but I'm a bit skeptical about crossing the bridge from ops to applications. Typically applications aren't self-contained enough that they can, or should, declare all their resources. So presumably then you have to centralize common stuff (e.g. your central Postgres server) in a shared module that all your apps import. Now you get into versioning hell as your app depends on an old version of the "common" module and "pulumi update" tears down your Postgres 10 install and creates a 9.6 install instead. (Presumably it asks first. But still. Versioning has got to be a challenge here.)
- lindydonna 8y ago
  (Disclosure: I work at Pulumi) > I may be wrong, but from what I can tell this is really a souped-up framework for working with clouds. Most of it looks like primitive for immutable infrastructure, so Terraform but as a library. As far as can tell, there's no deployment system, no orchestration etc. -- a "Pulumi app" doesn't know how to deploy itself. Is that accurate? Not quite. A Pulumi app is always deployed through the Pulumi CLI, which deploys and manages Pulumi programs. Note that most Pulumi code runs at deployment time, not runtime. Instead of specifying resources in a configuration language, you write them in code. The Pulumi CLI turns this code into a declarative plan, and updates your infrastructure when you run `pulumi update`. You may find this doc page helpful: https://pulumi.io/reference/how.html https://pulumi.io/reference/how.html Docker builds, plus provisioning of a container registry instance (and deploying to it), are handled automatically. This blog post walks through the e2e container scenario (but on AWS, rather than GCP): http://blog.pulumi.com/deploying-production-ready-containers-with-pulumi http://blog.pulumi.com/deploying-production-ready-containers... > Typically applications aren't self-contained enough that they can, or should, declare all their resources. So presumably then you have to centralize common stuff (e.g. your central Postgres server) in a shared module that all your apps import. Now you get into versioning hell as your app depends on an old version of the "common" module and "pulumi update" tears down your Postgres 10 install and creates a 9.6 install instead. (Presumably it asks first. But still. Versioning has got to be a challenge here.) This is just one way to architect an app, and it's unlikely to work well for a database, as you say. In this case, you'd likely use a `database.get` call, where you reference an existing database, that may be managed in a different Pulumi program or stack, or even outside of Pulumi.
mbrumlow 8y ago
All I know is the fonts on HN now look weak and puny compared to the the fonts used in the blog. I don't mean this as a joke, I am seriously having issues looking at HN fonts after reading the blog. Even my emacs fonts are broken to me now. How can a site just steal my ignorant bliss away and ruin my entire system with something as simple as fonts!
catchmeifyoucan 8y ago
This is beautiful. Looked at the serverless url shortener example. It abstracts over all the configuration garbage and combines all the AWS infrastructure elements as one unified working body. I'm worried that I might run into a case where it might provision something and I wouldn't know what it did to my AWS account necessarily. I'll 10/10 give this a go for my next AWS project. Also, no YAML woo!
- lindydonna 8y ago
  (Disclosure: I work at Pulumi.) The `pulumi update` command first shows you a preview of the exact resources it will create. Once you've deployed, you can view all your resources on the pulumi.com console, so you know exactly what has happened.
  - catchmeifyoucan 8y ago
    Perfect thanks for letting me know :)
ofrzeta 8y ago
From the "tour" on the homepage it seems that you need an account with Pulumi ("Most commands will prompt you to login to the Pulumi service when you run them"). Is this mandatory or can I use the open source code on its own?

317 points by MikusR 8y ago

We've had a chance to try out Pulumi when it was in private beta. We are very happy with it so far. Having used a variety of tools in the space, this is the first time I ever had an engineer say to me: "I really like it, I hope we can keep using it."

polskibus 8y ago

It's a pity they didn't offer .net support from the start. I wonder what were the reasons, besides popularity contest.

ellism 8y ago
> I wonder what were the reasons, besides popularity contest. (Disclosure: I'm a member of the Pulumi team). Strictly a timing issue. We brought TypeScript/JavaScript online first and most of our focus was there while building up the supporting tooling. Fortunately, the design of the system makes it pretty easy to add new languages (we've since then done Python and Go). Most of the team is ex-.NET folks and I've been itching to build out the .NET support. Now that we've launched, it will at least become /my/ spare time project. I'll be really excited when we have it well. Do keep your eye out!
- thrww20180618 8y ago
  Are you worried that adding "non-hip" languages from the start will alienate the kind of Microsoft-avoiding crowd you as a corporation are trying so hard to win? Is Pulumi some sort of open-ended research project like Midori? If I want to offer it to an enterprise client shall I say "don't worry one of their guys might add C#/Java in his spare time if he has a chance"?
pjmlp 8y ago
Yep, lacking Java and .NET support makes it a bit un-appealing to me. On the other hand, it looks quite interesting and is good to finally be able to know what Pulumi is all about. Looking forward how it evolves.
- ellism 8y ago
  You'all spoke and we've listened. I've dusted off my old very barebones prototype, pushed it to a branch at https://github.com/pulumi/pulumi/tree/ellismg/add-dotnet https://github.com/pulumi/pulumi/tree/ellismg/add-dotnet, and will be hacking on this over the course of the next week. Collaborators are welcome (please do join the #contribute channel in our slack at https://slack.pulumi.io https://slack.pulumi.io). And watch the branch, I expect we'll have something usable in a few days and it might help folks who are interested in supporting other languages with Pulumi. In the meantime, as someone who had to learn TypeScript to start using Pulumi when I started here, I do encourage you to look at some of the examples we have and play around with them. It's actually a pretty nice language and with VSCode it's pretty easy to get started. (Disclosure: I still work at Pulumi)
  - thrww20180618 8y ago
    Problem with TypeScript is that it's a temporary bridging language until we have proper WASM toolchains and WASM DOM/GC integration for real languages. Just look at Blazor go! So betting on TS on _server side_ of all places is quite weird.

meritt 8y ago

> How do we know? We don’t need to Pulumi looks like clever technology but the complete detachment from the underlying services and associated costs seems like the perfect storm for an extremely expensive disaster. How does Pulumi keep track of which services are launched, especially during testing/development, and how does it ensure those are shut down once they are no longer needed? How does it determine the optimal size of instances/volumes/etc to launch?

jen20 8y ago
That detachment certainly has not harmed Terraform (some of the providers for which are used by Pulumi) - so I'm not sure it's as big an issue as you make out?
aChrisSmith 8y ago
Hi! I work at Pulumi and have been using it to standup and manage all of our service infrastructure. > How does Pulumi keep track of which services are launched, especially during testing/development Each Pulumi program is ran within the context of "a stack". The stack is essentially a collection of cloud resources. So when the Pulumi program runs, it will create resources that aren't in the stack, or update existing ones. So if you create any resources during dev/testing, you just need to `pulumi destroy` those stacks and all of the cloud resources will be reclaimed. This, IMHO, is one of Pulumi's best features. In that it makes it super-easy to create your own instance of a cloud application. For example, I have my own development instance of app.pulumi.com by just creating my own Pulumi stack and rerunning the same application. > How does it determine the optimal size of instances/volumes/etc to launch? It doesn't. The Pulumi program ran determines what resources to create. So you are left to configure, tune or tweak that as makes sense.
- atombender 8y ago
  From the examples it looks like Pulumi programs declare their infrastructure, causing it to be created. Doesn't that mean that the program will need privileged credentials? How do you make sure the app only has, say, read access to an S3 bucket it needs to listen to, and can't accidentally delete it? And how does that then allow it to declare the bucket?

antics 8y ago

I was a part of the alpha testing and I liked it so much I added Kubernetes support in my free time. A thing I notice about the space is that existing tooling tends to be very focused on a specific persona (e.g., ops), and when you want to do anything outside those use cases, it starts to feel like you're jamming a square peg into a round hole. So, e.g., Go templates/sed/awk work great if all you want to do is template a couple YAML files, but if your use cases is even slightly more demanding (sharing, dependencies, transforming), it gets painful quick. Using a "real language" makes all of this much simpler. You can write tooling that captures the needs you actually have. If you have an ops team that wants a mostly-declarative or mostly-hermitic API, you can write that, and you don't have to map it to CloudFormation, ARM templates, Terraform, etc. Huge win. You can write JS/TS/Python libraries that provide secure defaults for your core infrastructure components. You can use standard dev tools to audit and refactor code you've written. In Terraform, modules have proven to be very useful. For Pulumi, I'm interested to see what the tooling ecosystem will look like as the product becomes more mature.

garganzol 8y ago

[deleted per personal request]

sidcool 8y ago
What are you talking about?
- garganzol 8y ago
  .NET, future, responsibility. In that order. Tangentially touching human ego and how it can spoil the intentions. Unless one finds the way to work with it. Sorry if that's too intense.
  - maslam 8y ago
    Joe's a really nice guy. Have you ever worked with him or been in a meeting with him? Sorry, this type of ad-hominem attack really does not belong here.
    - garganzol 8y ago
      Sorry. I hear you. I'm deleting the original message.
      - maslam 8y ago
        Thank you. Kudos for doing the right thing :)

fiatjaf 8y ago

What is an open-source cloud development platform? I've read the first 5 paragraphs of the blog post and skimmed through the landing page, I've also clicked on the "38 lines of code" example, and I still don't understand where does this fits. I imagined it was something like Heroku, but I guess I was wrong. I'll keep trying to understand it, anyway.

lindydonna 8y ago
(I'm a product manager at Pulumi.) Pulumi lets you describe cloud resources using code instead of a config language. It's not like Heroku, it's more like a deployment tool (e.g. Serverless Framework, Terraform, Claudia.js, Chalice, etc). The difference compared to other deployment tools is that you use regular code, but it's turned into a declarative plan when you run `pulumi update`. So, you get the benefits of a regular programming language, while still following best practices of immutable infrastructure. Does that help?
- akavel 8y ago
  Then is this something like NixOps [1]? Or how does it differ? [1] https://nixos.org/nixops/ https://nixos.org/nixops/
  - deleted 8y ago
    [deleted]
  - cwp 8y ago
    Joe mentions working with "existing language package managers". That probably doesn't include nixpkgs, but perhaps it could.

andyidsinga 8y ago

this looks really interesting - keep going!

startupdiscuss 8y ago

Reading through it, it sounds very exciting and productive for people who already know what it is and can use it. But for a complete newbie. Say someone who can write an app and push it to Heroku, what is this and why might it be worth trying out?

antics 8y ago
Pulumi is meant to replace CloudFormation/ARM Templates/Kubernetes YAML. So if you write those things to provision infrastructure, this could be for you. If you are writing (e.g.) Rails code that you `git push` to a PaaS like Heroku, then they're provisioning this infrastructure on your behalf. So this is a bit lower level than that.

avip 8y ago

When I read a blog about "cloud development platform", I expect to understand how it fairs vs. Terraform. I'm already using Terraform. All the cloud "consultants" I know of, use Terraform. In order to not use it, I need a reason - I need something that Pulumi can do, and Terraform can't.

lindydonna 8y ago
(I'm a product manager on Pulumi.) If Terraform works for you, then definitely continue using it. Pulumi is just another option and works well for building libraries and components. I did a rundown of different tools for serverless apps at Velocity SF last week. Slides are here: https://cdn.oreillystatic.com/en/assets/1/event/270/Tooling%20in%20the%20age%20of%20serverless%20computing%20Presentation.pdf https://cdn.oreillystatic.com/en/assets/1/event/270/Tooling%... and GitHub samples are here: https://github.com/lindydonna/velocity-examples https://github.com/lindydonna/velocity-examples There are a ton of tools in this space, all with overlapping functionality, so there tends not to be a clear comparison of tool X vs tool Y.
- avip 8y ago
  Hi, thank you for the slides and repo - lots of good material there. Just in case I've failed to clarify my intention: I'd really like to know the upsides of "Pulumi". I'm sure it was built for a reason (i.e dissatisfaction wrt other tools). I just don't have the bandwidth to blindly invest myself in tech without some promise that something will end for the better for me or my company. Thanks!
  - lindydonna 8y ago
    The main upside is that it's just regular code. This is more natural for some folks, but mainly the advantage is that you can build abstractions. So, you can build a component that's built up from lower-level resources in AWS. For example, directly setting up API Gateway requires a specific setup (and can be error-prone), so there's a Pulumi library for that. My colleague wrote a blog post with more of the technical motivation: https://medium.com/@lukeh/programming-the-cloud-e795cafffc2b https://medium.com/@lukeh/programming-the-cloud-e795cafffc2b, and there's also this Twitch live-coding video: https://youtu.be/DM8Wd4f1MNA https://youtu.be/DM8Wd4f1MNA

ethanwillis 8y ago

I wish this worked on bare metal. The options for running on premise cloud are slim to none and the CM tools for managing that have shaky support. It's such a shame. I don't want to just be another customer for DO, AWS, Azure, etc.

bthornbury 8y ago

Been looking forward to the launch, congrats to all involved on pulumi team!

rcarmo 8y ago

Python 2 support only, which is sad. I fully intend to try out the Azure provider, but without access to Python libraries there is not much I can create off the cuff. Also, this reminds me a bit of metaparticle, which I’m quite fond of.

slsinsea 8y ago
I was able to play with the Python support in the Beta. The team intends to stick with 2/3 compatible code and support Python 3 next milestone, so you should be able to try it out soon.

catern 8y ago

Hmm, I was excited to see a former Midori developer working on a "cloud development platform". And I was even more excited to read: >Using Pulumi, you author cloud programs using your favorite language, spanning low-level infrastructure-as-code to highly productive and modern container- and serverless-powered applications. I'm enthusiastic about the prospect of using a real programming language to deploy and control a distributed system. It would be fantastic to have a library that gives me a nice, clean API for performing low-level distributed system tasks, manipulating and controlling hosts at the level of individual processes and sockets. But I was disappointed to see that Pulumi doesn't seem to be designed to support deployment onto bare metal. In fact it seems to be just another layer of abstraction over various SaaS providers, or a huge dependency like Kubernetes. Why add another layer? Today's systems are already heavily abstracted. For me, the value of using a really expressive programming language to implement a system, is that you can keep more of the abstraction in the language, and thereby avoid complexity in the underlying architecture. To me, it is not very interesting to layer yet another library on top of the Kubernetes API. I'd be much more interested to see an attempt to replace the Kubernetes API with a library. This seems to be the former, and not oriented towards the latter, which is a bit of a disappointment.

joeduffy 8y ago
> Pulumi doesn't seem to be designed to support deployment onto bare metal. In principle, there is no reason we couldn't support bare metal, provided we have an orchestrator. Pulumi bottoms out on raw CRUD operations that can do anything. It turns out that, when targeting existing cloud providers, that means invoking their APIs. As you say, this is just a layer of abstraction, but the key aspect at the center here is diffable immutable infrastructure. I was, in fact, inspired to pursue Pulumi in part because of the experience of building a distributed OS that ran on bare metal. Many of the same challenges Pulumi solves around configuration, composition, and the application model, were inspired by this experience, and we've taken some of those lessons and applied them to cloud programs. > you can keep more of the abstraction in the language, and thereby avoid complexity in the underlying architecture. I look at other ways we've integrated with systems as inspiration here. For instance, we can productively code using tasks, and async await, these days, and reap the benefits of concurrency, despite there being massive differences between threading, I/O, and scheduling behavior across different systems. There was abstraction in the language, but it bottoms out on a set of primitives. In a sense, I view the current APIs we're using as the syscall layer for a given cloud, Kubernetes included. (Also note, we don't require depending on Kubernetes.) That's not to say everybody will want to eschew the syscalls -- there's always a time and a place -- but we can be more productive by using the abstractions. I believe that frameworks are languages, in that they give you a set of nouns and verbs you can use to express yourself. I do think there's opportunity for language-level innovation here, but if there's one thing I've taken away from our industry experiences building PaaS's, it's that it's a losing battle to bet against the incredible momentum of the public cloud vendors. > I'd be much more interested to see an attempt to replace the Kubernetes API with a library. We are just getting started, so expect to see a lot more progress in this direction in the weeks to come. The foundation of languages is a huge enabler for us, and we already have several projects in the works that I think will make you happier than this initial release. YMMV, this is just my opinion, and I really love hearing the feedback. Thank you for sharing. This is a vibrant space and time and I'm enjoying the diversity of perspective.

digitalsanctum 8y ago

Looks very interesting. I have a couple of questions: 1. One of my pet peeves with CloudFormation is the disconnect between the state of CF and the "actual" state if someone decides to update resources by not using CF. Does Pulumi do checks to verify that the actual state matches its assumed state? 2. Is there a guide for contributing a new language and/or provider? Also, I tried getting an invite for the Pulumi Slack community and was greeted with "invite_limit_reached".

lindydonna 8y ago
> Also, I tried getting an invite for the Pulumi Slack community and was greeted with "invite_limit_reached". We're hitting Slack's rate limiting on invites. Please DM your email @PulumiCorp on Twitter and we'll get you added!
elcritch 8y ago
I’d like to know the answer to that as well... Error handling semantics across languages vary greatly. Does the user code need to handle errors or mismatched state or does Pulumi? If Pulumi does how do you get/ report errors?
- swgillespie 8y ago
  [Disclosure - I'm a Pulumi employee] There is a top-level CLI verb called `refresh` that explicitly checks Pulumi's idea of the state of the world against the actual state of the world. If there's a diff, Pulumi reconciles the diff by updating its own state. `pulumi update` does not do this by default, but you can run `pulumi refresh` at any time to verify that your state has not drifted from reality.

amanzi 8y ago

This looks interesting but I really don't like how you're prompted to download and run a shell script from their website without any indication of what it does. I'm all for making it easy to get started, but at least explain what the script does.

ellism 8y ago
Thanks for the feedback! The script will download the latest version of Pulumi, extract it to ~/.pulumi/bin and then try to add that folder to your .bashrc or .bash_profile. You should be able to read the source of the script directly at https://get.pulumi.com/ https://get.pulumi.com/ or even download it yourself, inspect it and then decide if you want to run it or not. You can also follow the manual install instructions which are listed a little further down on our install page at https://pulumi.io/install/ https://pulumi.io/install/, if you want total control over where it is installed. I also just opened a pull request to our docs website: https://github.com/pulumi/docs/pull/430 https://github.com/pulumi/docs/pull/430 to explain in a little more detail what we're doing with the scripts.
- amanzi 8y ago
  Thanks for responding. :-)
- monch1962 8y ago
  I'd appreciate it if I could install Pulumi via e.g. $ brew install pulumi That way I'd have a convenient way of keeping it updated to the latest version

teyc 8y ago

Hi Joe, congrats on the launch. What is difference in the use case for Pulumi compared to scripting infrastructure using vendor provided libraries e.g. New-AzureRmVm et c

joeduffy 8y ago
> congrats on the launch. Thanks -- we're very excited to be out and talking to the community! > What is difference in the use case for Pulumi compared to scripting infrastructure using vendor provided libraries The major difference is that Pulumi does immutable infrastructure. The code describes a goal state and our system manages your environment to ensure that it matches the goal state. This means you can preview changes before making them, and that, if you do make them, you've got a full audit trail of who changed what and when. Rollbacks are trivial because you just point us at an old goal state worked and we can chew on it until your live environment matches. We can even detect drift, so if someone manually updates in the cloud console, we can tell you about it. As a result, we always have a full object graph of your cloud resources, and can tie it back to the source code which created it, opening up some unique diagnostics capabilities. The difference with scripting libraries, like AWS's Boto, or the Azure PowerShell SDK, is that they mutate cloud resources directly, and in an ad-hoc manner. So, you don't know what they are going to do before you run them. And in the event of failure, you're more likely to be in a corrupt state and unable to recover. Rollbacks are difficult. There's also no audit trail beyond the cloud access logs, and information like dependencies are lost, so resources end up disassociated from the code and workflow that created or updated them. Many people encounter these problems and need to build complex systems on top to address them. Or they end up using a solution like CloudFormation or Azure Resource Manager, which eschews code in favor of JSON/YAML/templates. The hope here is that we've landed on a sweet spot between immutable infrastructure and fully programmable languages.

Edmond 8y ago

I am coming at this from a competing perspective so take it with a grain of salt...isn't the whole idea of the declarative approach that users/Ops can focus only on specifying what they want and let whatever target execution run-time handle the actual semantics? An approach that goes back to relying on procedural programming languages as a primary means of configuration seems like it defeats the purpose. I am working in this space, mainly focused on configurations: https://youtu.be/4yepPOznakk https://youtu.be/4yepPOznakk The approach we've adopted is to maintain the declarative presentation while providing a way to plug in procedural language programmability without using templates or other similar concoctions (ksonnet..etc).

deleted 8y ago

[deleted]

atombender 8y ago

I may be wrong, but from what I can tell this is really a souped-up framework for working with clouds. Most of it looks like primitive for immutable infrastructure, so Terraform but as a library. As far as can tell, there's no deployment system, no orchestration etc. -- a "Pulumi app" doesn't know how to deploy itself. Is that accurate? So if, for example, you're running an app on GKE, then you can tell the Kubernetes API about the resources you want, but there's no system for building your app (Docker build + push to GCR), no way to do supervised rolling updates, and so on. If you're doing CI/CD, then presumably you have to build, push, then let the Pulumi code inject the right image and tag. I like the idea of declaring the infrastructure with extensible code (as opposed to a DSL like Terraform where you're completely dependent on upstream support for all the objects you want to manage), but I'm a bit skeptical about crossing the bridge from ops to applications. Typically applications aren't self-contained enough that they can, or should, declare all their resources. So presumably then you have to centralize common stuff (e.g. your central Postgres server) in a shared module that all your apps import. Now you get into versioning hell as your app depends on an old version of the "common" module and "pulumi update" tears down your Postgres 10 install and creates a 9.6 install instead. (Presumably it asks first. But still. Versioning has got to be a challenge here.)

lindydonna 8y ago
(Disclosure: I work at Pulumi) > I may be wrong, but from what I can tell this is really a souped-up framework for working with clouds. Most of it looks like primitive for immutable infrastructure, so Terraform but as a library. As far as can tell, there's no deployment system, no orchestration etc. -- a "Pulumi app" doesn't know how to deploy itself. Is that accurate? Not quite. A Pulumi app is always deployed through the Pulumi CLI, which deploys and manages Pulumi programs. Note that most Pulumi code runs at deployment time, not runtime. Instead of specifying resources in a configuration language, you write them in code. The Pulumi CLI turns this code into a declarative plan, and updates your infrastructure when you run `pulumi update`. You may find this doc page helpful: https://pulumi.io/reference/how.html https://pulumi.io/reference/how.html Docker builds, plus provisioning of a container registry instance (and deploying to it), are handled automatically. This blog post walks through the e2e container scenario (but on AWS, rather than GCP): http://blog.pulumi.com/deploying-production-ready-containers-with-pulumi http://blog.pulumi.com/deploying-production-ready-containers... > Typically applications aren't self-contained enough that they can, or should, declare all their resources. So presumably then you have to centralize common stuff (e.g. your central Postgres server) in a shared module that all your apps import. Now you get into versioning hell as your app depends on an old version of the "common" module and "pulumi update" tears down your Postgres 10 install and creates a 9.6 install instead. (Presumably it asks first. But still. Versioning has got to be a challenge here.) This is just one way to architect an app, and it's unlikely to work well for a database, as you say. In this case, you'd likely use a `database.get` call, where you reference an existing database, that may be managed in a different Pulumi program or stack, or even outside of Pulumi.

mbrumlow 8y ago

All I know is the fonts on HN now look weak and puny compared to the the fonts used in the blog. I don't mean this as a joke, I am seriously having issues looking at HN fonts after reading the blog. Even my emacs fonts are broken to me now. How can a site just steal my ignorant bliss away and ruin my entire system with something as simple as fonts!

catchmeifyoucan 8y ago

This is beautiful. Looked at the serverless url shortener example. It abstracts over all the configuration garbage and combines all the AWS infrastructure elements as one unified working body. I'm worried that I might run into a case where it might provision something and I wouldn't know what it did to my AWS account necessarily. I'll 10/10 give this a go for my next AWS project. Also, no YAML woo!

lindydonna 8y ago
(Disclosure: I work at Pulumi.) The `pulumi update` command first shows you a preview of the exact resources it will create. Once you've deployed, you can view all your resources on the pulumi.com console, so you know exactly what has happened.
- catchmeifyoucan 8y ago
  Perfect thanks for letting me know :)

ofrzeta 8y ago

From the "tour" on the homepage it seems that you need an account with Pulumi ("Most commands will prompt you to login to the Pulumi service when you run them"). Is this mandatory or can I use the open source code on its own?