3 ms·

Vault is great. But if I were to start learning Terraform now, I'd probably ditch it in favor of Pulumi. Terraform was supposed to make writing DRY easy. It's

Vault is great. But if I were to start learning Terraform now, I'd probably ditch it in favor of Pulumi. Terraform was supposed to make writing DRY easy. It's often more reasonable to copy-paste parts of configuration than to try make it DRY. HashiCorp seems to have the approach of "if something can be hacked around, even in a very convoluted way, we can ignore any feature requests for that thing to be doable in a normal fashion". Instead of fixing Terraform, Hashi is creating more and more products that likely will be increasingly unpleasant to use and won't get reasonable fixes, just as Terraform. The way HashiCorp approaches creating an ecosystem, makes me want to stay away.

raffraffraff 4y ago
I found a way to make Terraform code completely dry by using Hiera as a data source and writing every stack and module to be entirely data-driven (ie: only ever use "for_each" to iterate over a data map, never hard code anything, use strict naming conventions). However there are some major obstacles to using Terraform this way, that emerge from it being a clunky DSL with hidden "special rules", instead of behaving in a consistent way like a proper language. Like these: https://github.com/hashicorp/terraform/issues/31072 https://github.com/hashicorp/terraform/issues/31072 https://github.com/hashicorp/terraform/issues/24476 https://github.com/hashicorp/terraform/issues/24476
sshine 4y ago
I find that HCL (Terraform's declarative resource specification format) is an excellent target for a compiler for a DRY description of my resources. :-D
- erik_seaberg 4y ago
  HCL doesn’t interop with anything. In a moment of weakness I once generated HCL using M4 (because passing a DynamoDB key schema into a module used to be impossible), but now I think it’s better to generate the JSON equivalents they also parse. Their gRPC API for plugins looks pretty tough, and I keep hoping for an SDK for custom resources that doesn’t require writing Go.
mountainriver 4y ago
100% - HCL just isn’t expressive enough to handle complex environments and you end up needing to do very strange things that would have been simple in a general purpose language. I’ve been using Pulumi over the last year and it’s the best time I’ve ever had provisioning infrastructure. I actually enjoy it now. The automation API is also incredibly powerful, I don’t think the ecosystem has wrapped their heads around it yet.
EduardoRT 4y ago
I agree that Hashi's approach to Terraform is “if there's a workaround, we won't fix it” I don't see Terraform as hard to use for DRY stuff, we have fully adopted Terraform, we now have 100+ modules that are used in more than 20+ AWS accounts with no major issues, these modules are opinionated and are mostly just plug and play for the teams that use them. Pulumi is great, but you need to really trust your people to write stuff with Pulumi, in our case most teams can just grab a module, submit a pull-request with the values they want and it gets reviewed by a member of the infrastructure team, if everything looks ok then it's merged and the infra gets deployed/changed. Even with that simple workflow we have issues with devs misunderstanding some concepts, even devs that have been using Terraform for months, it's not a simple tool to use, I would be worried about them having to use Pulumi for the same things. It might just be an issue of scale though, we have a gigantic dev team and are riddled with regulations, so Terraform is the perfect fit for us, and again, we don't have DRYness issues, nothing obvious at least.
- hericium 4y ago
  AWS provider seems to be covering for some of Terraform's shortcomings. Until I was using just AWS, Cloudflare and other widely-used providers the problems weren't that much annoying. After switching to some niche services with not so extensively developed (yet not buggy to my knowledge) provider modules, the whole Terraform experience went sideways. Those "let's just use silly count or for_each to enable/disable the block" suddenly not always worked for randomish weird reasons requiring digging into TF_LOG=trace logs. Other simple things that you are used to with AWS were not so simple anymore. DRY writing wasn't an issue with AWS provider but I find it to be an issue with Terraform itself when provider module isn't covering for tf's shortcomings.
  - EduardoRT 4y ago
    Agree, smaller providers are more difficult to work with, hard to draw the line between an issue with the tool or a developer introduced issue though, I do agree that smaller providers tend to be weird, so I think Terraform/Hashi could do a better job at making it easier for developers to create better/more reliable providers. It baffles me that there's still no way of just disabling a block without having to introduce a count parameter, that should have been fixed ages ago.

raffraffraff 4y ago

I found a way to make Terraform code completely dry by using Hiera as a data source and writing every stack and module to be entirely data-driven (ie: only ever use "for_each" to iterate over a data map, never hard code anything, use strict naming conventions). However there are some major obstacles to using Terraform this way, that emerge from it being a clunky DSL with hidden "special rules", instead of behaving in a consistent way like a proper language. Like these: https://github.com/hashicorp/terraform/issues/31072 https://github.com/hashicorp/terraform/issues/31072 https://github.com/hashicorp/terraform/issues/24476 https://github.com/hashicorp/terraform/issues/24476

sshine 4y ago

I find that HCL (Terraform's declarative resource specification format) is an excellent target for a compiler for a DRY description of my resources. :-D

erik_seaberg 4y ago
HCL doesn’t interop with anything. In a moment of weakness I once generated HCL using M4 (because passing a DynamoDB key schema into a module used to be impossible), but now I think it’s better to generate the JSON equivalents they also parse. Their gRPC API for plugins looks pretty tough, and I keep hoping for an SDK for custom resources that doesn’t require writing Go.

mountainriver 4y ago

100% - HCL just isn’t expressive enough to handle complex environments and you end up needing to do very strange things that would have been simple in a general purpose language. I’ve been using Pulumi over the last year and it’s the best time I’ve ever had provisioning infrastructure. I actually enjoy it now. The automation API is also incredibly powerful, I don’t think the ecosystem has wrapped their heads around it yet.

EduardoRT 4y ago

I agree that Hashi's approach to Terraform is “if there's a workaround, we won't fix it” I don't see Terraform as hard to use for DRY stuff, we have fully adopted Terraform, we now have 100+ modules that are used in more than 20+ AWS accounts with no major issues, these modules are opinionated and are mostly just plug and play for the teams that use them. Pulumi is great, but you need to really trust your people to write stuff with Pulumi, in our case most teams can just grab a module, submit a pull-request with the values they want and it gets reviewed by a member of the infrastructure team, if everything looks ok then it's merged and the infra gets deployed/changed. Even with that simple workflow we have issues with devs misunderstanding some concepts, even devs that have been using Terraform for months, it's not a simple tool to use, I would be worried about them having to use Pulumi for the same things. It might just be an issue of scale though, we have a gigantic dev team and are riddled with regulations, so Terraform is the perfect fit for us, and again, we don't have DRYness issues, nothing obvious at least.

hericium 4y ago
AWS provider seems to be covering for some of Terraform's shortcomings. Until I was using just AWS, Cloudflare and other widely-used providers the problems weren't that much annoying. After switching to some niche services with not so extensively developed (yet not buggy to my knowledge) provider modules, the whole Terraform experience went sideways. Those "let's just use silly count or for_each to enable/disable the block" suddenly not always worked for randomish weird reasons requiring digging into TF_LOG=trace logs. Other simple things that you are used to with AWS were not so simple anymore. DRY writing wasn't an issue with AWS provider but I find it to be an issue with Terraform itself when provider module isn't covering for tf's shortcomings.
- EduardoRT 4y ago
  Agree, smaller providers are more difficult to work with, hard to draw the line between an issue with the tool or a developer introduced issue though, I do agree that smaller providers tend to be weird, so I think Terraform/Hashi could do a better job at making it easier for developers to create better/more reliable providers. It baffles me that there's still no way of just disabling a block without having to introduce a count parameter, that should have been fixed ages ago.