22 ms·

The moment you need control flow to define your resources, I'd argue that you're verging away from the realm of declarative infrastructure. I'm using Terraform

The moment you need control flow to define your resources, I'd argue that you're verging away from the realm of declarative infrastructure. I'm using Terraform to manage 10^4 machines in combination with sane CI/CD, Bash/JQ (for dealing with Terraform outputs), Packer and Ansible. Everytime I see somebody reaching to a full programming language to define their infrastructure, they seem to be doing too much with one tool. Terraform should merely provision things and in that role I find it fine as is. Preferred, even.

hardwaresofton 6y ago
> The moment you need control flow to define your resources, I'd argue that you're verging away from the realm of declarative infrastructure. Declarative infrastructure shouldn't be pursued for it's own sake -- what I want is efficient, and simple to manage infrastructure automation. The declarative nature is awesome, but once you start doing plumbing of variables and complexity from one static script from another, the cognitive load of keeping this all in line is better managed with a programming language in my opinion, you're just choosing bash/jq/awk/etc instead of a different language. I think "the way the declarations are made must be static files" is dogmatic or at least limiting for me. Yes it is absolutely the simplest way to view what's present, but the problem is when someone goes into change any of this they will be dealing with your bolted-together complexity (even if it's not very complex). > I'm using Terraform to manage 10^4 machines in combination with sane CI/CD, Bash/JQ (for dealing with Terraform outputs), Packer and Ansible. Everytime I see somebody reaching to a full programming language to define their infrastructure, they seem to be doing too much with one tool. > Terraform should merely provision things and in that role I find it fine as is. Preferred, even. I can't argue with the efficiency and efficacy of your setup, but I don't think much of this has to do with what we were discussing -- Pulumi does not seek to do the jobs of those other tools -- it's not going to build your VM images or do provisioning (unless you use it that way like with terraform[0]). Here's a concrete example of a benefit I got form using Pulumi over terraform recently, in some code working with SES: import * as fs from "fs"; // ... more imports and other lines // Email Access key const emailAccessKey = new aws.iam.AccessKey( `${stack}-ses-access-key`, {user: emailUser.name} ); export const emailUserSMTPPassword = emailAccessKey.sesSmtpPasswordV4; export const emailUserSecret = emailAccessKey.encryptedSecret; // Write the smtp username and password out to a local secret file const apiSecretsDir = path.join(__dirname, "secrets", "api", stack); const smtpUsernameFilePath = path.resolve(path.join(apiSecretsDir, "SES_USERNAME.secret")); const smtpPasswordFilePath = path.resolve(path.join(apiSecretsDir, "SES_PASSWORD.secret")); emailAccessKey.sesSmtpPasswordV4.apply(password => { console.log(`Writing SES SMTP username to [${smtpUsernameFilePath}]`); fs.writeFileSync(smtpUsernameFilePath, emailUsername); console.log(`Writing SES SMTP password to [${smtpPasswordFilePath}]`); fs.writeFileSync(smtpPasswordFilePath, password); }); I wanted to write information out to a file... So I just did, and that was it. No need to reach for the stack output later and pipe it anywhere -- any time pulumi runs it will update that variable if/when it changes, and the next tool (which requires the file at that path to be present) will continue on without knowing a thing. I can't say that this is perfect Pulumi code (ex. I could have defined a custom Resource to do this for me), but I have saved myself having to do the plumbing with bash scripts and terraform output awk-ing, and the information goes just where I want it (NOTE: the secrets folder is encrypted with git-crypt[1]). When someone comes to this file (ses.ts), they're going to be able to easily trace where these values where generated -- similar with bash scripts, but now they don't have to be a bash/awk/jq master to manipulate information. There are definitely some gotchas to using Pulumi (like the `.apply` there), but in the end, I'd prefer to make changes like this in a consistent language I like (Typescript). My toolkit looks very similar to you, except I basically only use make + kubectl + pulumi + ansible (rarely, because of the kind of servers I rent). [0]: https://www.terraform.io/docs/provisioners/ https://www.terraform.io/docs/provisioners/ [1]: https://www.agwa.name/projects/git-crypt/ https://www.agwa.name/projects/git-crypt/
- busterarm 6y ago
  You can just write information out to files in Terraform with no stress. In terraform resources this is what that looks like: resource aws_iam_user email { name = "email" } resource aws_iam_access_key email { user = aws_iam_user.email.name } resource local_file smtp_password { content = aws_iam_access_key.email.ses_smtp_password_v4 filename = "SES_PASSWORD.secret" } So what's the plumbing that you would have to do? Under the hood, Pulumi is using the Terraform providers... (I left out username, because I don't see where you're setting emailUsername) In my pipelines I don't bother writing out to files things that are in terraform state. I just create an output for that state (potentially set to sensitive) and then use that output in my CI/CD. Remote state stays encrypted and without wide access and I don't have to worry about secrets being in files anywhere. That's where the bash scripts do things with outputs. It could by python or whatever, it doesn't matter really. But with bash I can easily just set variables to `terraform output -json | jq <select output &/|| do stuff>`. Mainly all I do is write terraform outputs to vault (i have simple bash automation to do all of this) and then I can use the Vault secrets in other CI/CD pipelines.
  - hardwaresofton 6y ago
    > You can just write information out to files in Terraform with no stress. This wasn't the point -- it was that I wanted to do something that I know how to do in a fully featured programming language, and I can "just do it". Writing local files is a very simple example -- unless you're arguing that terraform's capabilities amount to an entire language's ecosystem, this is just the tip of the iceberg. > So what's the plumbing that you would have to do? Under the hood, Pulumi is using the Terraform providers... I think I didn't explain it well enough. Pulumi and Terraform are almost the same tool, but the difference is how the pieces are plumbed together. I prefer plumbing with a programming language more than shell scripts, utilities and/or some other programming languages. Also, Pulumi's system of custom resources which are just pieces of code sitting in your codebase is fantastic and novel (terraform has custom providers but this feels significantly more heavy weight). > In my pipelines I don't bother writing out to files things that are in terraform state. I just create an output for that state (potentially set to sensitive) and then use that output in my CI/CD. Remote state stays encrypted and without wide access and I don't have to worry about secrets being in files anywhere. If you're really adjusted to Terraform, and it works great for you, then awesome -- I'm not out to change how you do things. It sounds like you've fully bought in to the terraform way of doing things, and it's working for you, and that's great. > That's where the bash scripts do things with outputs. It could by python or whatever, it doesn't matter really. But with bash I can easily just set variables to `terraform output -json | jq <select output &/|| do stuff>`. Here it is again... My point is that the plumbing matters, and a fully baked programming language offers the possibility of better plumbing. I didn't touch on it much, but just having access to custom resources with Pulumi might be able to cut down the external plumbing to zero, and enable creating more reusable pieces.
throwaway894345 6y ago
It’s not flow control so much as sane, expression-based generation of resources.Terraform has been evolving toward the dynamic with features like for_each, but these features have awful ergonomics compared to something like list comprehensions. Similarly, sometimes you want to reuse some pattern, but the unit for reuse in terraform is the modules which involves a lot of ceremony, so you don’t reach for it as often as you might if the unit of reuse was a simple function definition. I don’t especially care if Terraform is a simple static language for generating resources and the DRY-ness comes from an exterior Python/etc script that generates the Terraform files or whether the requisite dynamism is built into the terraform language itself, but make no mistake, the dynamism is absolutely essential for maintainable terraform.
0xbadcafebee 6y ago
I really dislike the idea of declarative infrastructure. It's literally a program that is designed to do one thing, but will change a million things in order to do that one thing. It's Configuration Management for Infrastructure. Yet so many people have this idea that it's something else, like it's supposedly simpler or more effective. Saying I want an S3 bucket named "foo" is the same as aws s3 list-buckets | grep "foo" || aws s3 create-bucket --bucket "foo" Did I need a big fat declarative infrastructure system to make that? No. But people want more complexity and features, and they want to make it look simple. So they write big fat applications and libraries to do that. The idea that there's some inherently superior difference of "declarative programming" over "regular programming" is giving people the idea that wrapping everything up in interfaces somehow removes the complexity, or somehow ends up in a better program.
- busterarm 6y ago
  Yes but AWS CLI commands change over time and don't have a native way of maintaining which version of the CLI you use. Also, you have to maintain that knowledge for however many things you have to do across however many providers. The point of Terraform isn't to add complexity, it's to have a general way of interacting with a vast number of APIs that's effectively the same and to abstract away the tribal knowledge of knowing how each individual API works. On the same provider version, you generally can expect Terraform to work the same over time (okay this is less true for say Google provider...) as the CLI keeps evolving. It's still helpful to understand the providers and their CLIs, but Terraform is a substantial force multipler because of how generic it is across the absurdly long list of APIs that it talks to. That is what its value is.
  - 0xbadcafebee 6y ago
    But it's not generic. I have to track the provider version, the Terraform version, my module version, and any sub-module versions, long-term. Each internal team has to jump through hoop after hoop just to run terraform apply reliably. I've never yet had to rewrite a shell script that used a new version of AWS CLI. It's very possible that that's only because I've not been using it enough. But even that would be just one level of complexity to manage, rather than four. And in fact, even within a single provider, interfaces aren't generic at all. You have to write every single resource of every single provider to be specific to its definition. It would be the same amount of work if you were writing a shell script with curl to plug into each API call. I know, because it was actually faster for me to write a Bash implementation of NewRelic's APIs than to figure out its ridiculous Terraform provider and resources with no documentation. The only benefit of Terraform is that I don't have to write the API implementation [for most providers]

hardwaresofton 6y ago

> The moment you need control flow to define your resources, I'd argue that you're verging away from the realm of declarative infrastructure. Declarative infrastructure shouldn't be pursued for it's own sake -- what I want is efficient, and simple to manage infrastructure automation. The declarative nature is awesome, but once you start doing plumbing of variables and complexity from one static script from another, the cognitive load of keeping this all in line is better managed with a programming language in my opinion, you're just choosing bash/jq/awk/etc instead of a different language. I think "the way the declarations are made must be static files" is dogmatic or at least limiting for me. Yes it is absolutely the simplest way to view what's present, but the problem is when someone goes into change any of this they will be dealing with your bolted-together complexity (even if it's not very complex). > I'm using Terraform to manage 10^4 machines in combination with sane CI/CD, Bash/JQ (for dealing with Terraform outputs), Packer and Ansible. Everytime I see somebody reaching to a full programming language to define their infrastructure, they seem to be doing too much with one tool. > Terraform should merely provision things and in that role I find it fine as is. Preferred, even. I can't argue with the efficiency and efficacy of your setup, but I don't think much of this has to do with what we were discussing -- Pulumi does not seek to do the jobs of those other tools -- it's not going to build your VM images or do provisioning (unless you use it that way like with terraform[0]). Here's a concrete example of a benefit I got form using Pulumi over terraform recently, in some code working with SES: import * as fs from "fs"; // ... more imports and other lines // Email Access key const emailAccessKey = new aws.iam.AccessKey( `${stack}-ses-access-key`, {user: emailUser.name} ); export const emailUserSMTPPassword = emailAccessKey.sesSmtpPasswordV4; export const emailUserSecret = emailAccessKey.encryptedSecret; // Write the smtp username and password out to a local secret file const apiSecretsDir = path.join(__dirname, "secrets", "api", stack); const smtpUsernameFilePath = path.resolve(path.join(apiSecretsDir, "SES_USERNAME.secret")); const smtpPasswordFilePath = path.resolve(path.join(apiSecretsDir, "SES_PASSWORD.secret")); emailAccessKey.sesSmtpPasswordV4.apply(password => { console.log(`Writing SES SMTP username to [${smtpUsernameFilePath}]`); fs.writeFileSync(smtpUsernameFilePath, emailUsername); console.log(`Writing SES SMTP password to [${smtpPasswordFilePath}]`); fs.writeFileSync(smtpPasswordFilePath, password); }); I wanted to write information out to a file... So I just did, and that was it. No need to reach for the stack output later and pipe it anywhere -- any time pulumi runs it will update that variable if/when it changes, and the next tool (which requires the file at that path to be present) will continue on without knowing a thing. I can't say that this is perfect Pulumi code (ex. I could have defined a custom Resource to do this for me), but I have saved myself having to do the plumbing with bash scripts and terraform output awk-ing, and the information goes just where I want it (NOTE: the secrets folder is encrypted with git-crypt[1]). When someone comes to this file (ses.ts), they're going to be able to easily trace where these values where generated -- similar with bash scripts, but now they don't have to be a bash/awk/jq master to manipulate information. There are definitely some gotchas to using Pulumi (like the `.apply` there), but in the end, I'd prefer to make changes like this in a consistent language I like (Typescript). My toolkit looks very similar to you, except I basically only use make + kubectl + pulumi + ansible (rarely, because of the kind of servers I rent). [0]: https://www.terraform.io/docs/provisioners/ https://www.terraform.io/docs/provisioners/ [1]: https://www.agwa.name/projects/git-crypt/ https://www.agwa.name/projects/git-crypt/

busterarm 6y ago
You can just write information out to files in Terraform with no stress. In terraform resources this is what that looks like: resource aws_iam_user email { name = "email" } resource aws_iam_access_key email { user = aws_iam_user.email.name } resource local_file smtp_password { content = aws_iam_access_key.email.ses_smtp_password_v4 filename = "SES_PASSWORD.secret" } So what's the plumbing that you would have to do? Under the hood, Pulumi is using the Terraform providers... (I left out username, because I don't see where you're setting emailUsername) In my pipelines I don't bother writing out to files things that are in terraform state. I just create an output for that state (potentially set to sensitive) and then use that output in my CI/CD. Remote state stays encrypted and without wide access and I don't have to worry about secrets being in files anywhere. That's where the bash scripts do things with outputs. It could by python or whatever, it doesn't matter really. But with bash I can easily just set variables to `terraform output -json | jq <select output &/|| do stuff>`. Mainly all I do is write terraform outputs to vault (i have simple bash automation to do all of this) and then I can use the Vault secrets in other CI/CD pipelines.
- hardwaresofton 6y ago
  > You can just write information out to files in Terraform with no stress. This wasn't the point -- it was that I wanted to do something that I know how to do in a fully featured programming language, and I can "just do it". Writing local files is a very simple example -- unless you're arguing that terraform's capabilities amount to an entire language's ecosystem, this is just the tip of the iceberg. > So what's the plumbing that you would have to do? Under the hood, Pulumi is using the Terraform providers... I think I didn't explain it well enough. Pulumi and Terraform are almost the same tool, but the difference is how the pieces are plumbed together. I prefer plumbing with a programming language more than shell scripts, utilities and/or some other programming languages. Also, Pulumi's system of custom resources which are just pieces of code sitting in your codebase is fantastic and novel (terraform has custom providers but this feels significantly more heavy weight). > In my pipelines I don't bother writing out to files things that are in terraform state. I just create an output for that state (potentially set to sensitive) and then use that output in my CI/CD. Remote state stays encrypted and without wide access and I don't have to worry about secrets being in files anywhere. If you're really adjusted to Terraform, and it works great for you, then awesome -- I'm not out to change how you do things. It sounds like you've fully bought in to the terraform way of doing things, and it's working for you, and that's great. > That's where the bash scripts do things with outputs. It could by python or whatever, it doesn't matter really. But with bash I can easily just set variables to `terraform output -json | jq <select output &/|| do stuff>`. Here it is again... My point is that the plumbing matters, and a fully baked programming language offers the possibility of better plumbing. I didn't touch on it much, but just having access to custom resources with Pulumi might be able to cut down the external plumbing to zero, and enable creating more reusable pieces.

throwaway894345 6y ago

It’s not flow control so much as sane, expression-based generation of resources.Terraform has been evolving toward the dynamic with features like for_each, but these features have awful ergonomics compared to something like list comprehensions. Similarly, sometimes you want to reuse some pattern, but the unit for reuse in terraform is the modules which involves a lot of ceremony, so you don’t reach for it as often as you might if the unit of reuse was a simple function definition. I don’t especially care if Terraform is a simple static language for generating resources and the DRY-ness comes from an exterior Python/etc script that generates the Terraform files or whether the requisite dynamism is built into the terraform language itself, but make no mistake, the dynamism is absolutely essential for maintainable terraform.

0xbadcafebee 6y ago

I really dislike the idea of declarative infrastructure. It's literally a program that is designed to do one thing, but will change a million things in order to do that one thing. It's Configuration Management for Infrastructure. Yet so many people have this idea that it's something else, like it's supposedly simpler or more effective. Saying I want an S3 bucket named "foo" is the same as aws s3 list-buckets | grep "foo" || aws s3 create-bucket --bucket "foo" Did I need a big fat declarative infrastructure system to make that? No. But people want more complexity and features, and they want to make it look simple. So they write big fat applications and libraries to do that. The idea that there's some inherently superior difference of "declarative programming" over "regular programming" is giving people the idea that wrapping everything up in interfaces somehow removes the complexity, or somehow ends up in a better program.

busterarm 6y ago
Yes but AWS CLI commands change over time and don't have a native way of maintaining which version of the CLI you use. Also, you have to maintain that knowledge for however many things you have to do across however many providers. The point of Terraform isn't to add complexity, it's to have a general way of interacting with a vast number of APIs that's effectively the same and to abstract away the tribal knowledge of knowing how each individual API works. On the same provider version, you generally can expect Terraform to work the same over time (okay this is less true for say Google provider...) as the CLI keeps evolving. It's still helpful to understand the providers and their CLIs, but Terraform is a substantial force multipler because of how generic it is across the absurdly long list of APIs that it talks to. That is what its value is.
- 0xbadcafebee 6y ago
  But it's not generic. I have to track the provider version, the Terraform version, my module version, and any sub-module versions, long-term. Each internal team has to jump through hoop after hoop just to run terraform apply reliably. I've never yet had to rewrite a shell script that used a new version of AWS CLI. It's very possible that that's only because I've not been using it enough. But even that would be just one level of complexity to manage, rather than four. And in fact, even within a single provider, interfaces aren't generic at all. You have to write every single resource of every single provider to be specific to its definition. It would be the same amount of work if you were writing a shell script with curl to plug into each API call. I know, because it was actually faster for me to write a Bash implementation of NewRelic's APIs than to figure out its ridiculous Terraform provider and resources with no documentation. The only benefit of Terraform is that I don't have to write the API implementation [for most providers]