8 ms·

Create Terraform files using Python scripts

60 points by sharjeelsayed 6y ago

fishnchips 6y ago
Not sure about that. It’s a clever trick alright but it adds a possibly unnecessary layer of leaky abstraction and makes code reuse more difficult when working with “vanilla” Terraform. If you really want to use a full programming language with infra as code, I’d suggest looking into Pulumi. Or, if you want to (or need to) stick to Terraform, try the official CDK.
TabTwo 6y ago
Now we need some Perl scripts to generate the Python scripts.
- sunshinerag 6y ago
  and then we need a DSL to get rid of all the imperative Perl.
- erik_seaberg 6y ago
  I feel I should take this opportunity to confess that I once used M4 to generate HCL, before I figured out Terraform JSON.
pram 6y ago
Terraform added a lot of these procedural functions natively in the time since this was created. At any rate if you want to stick with Python, I'd say Pulumi would be a safer choice these days.
- jpalomaki 6y ago
  Yes, Pulumi uses the Terraform providers, and allows you script your stuff in Typescript/Python/Java/C#. https://www.pulumi.com/ https://www.pulumi.com/
  - verdverm 6y ago
    How do Pulumi models in different languages interact or interop?
    - 7sigma 6y ago
      Not an expert but it is possible to mix and match with the new automation api. The example below mixes typescript with go. https://github.com/pulumi/automation-api-examples/tree/main/nodejs/crossLanguage-tsnode https://github.com/pulumi/automation-api-examples/tree/main/...
      - verdverm 6y ago
        Looks like you get a runtime explosion mad then some. Your CI will need all of the runtimes used plus the extra code to handle multiple language runtimes in Pulumi via the automation api. Anyhow, thanks for the link.
        jpalomaki 6y ago
        I guess typically you would pick one language and use that throughout the Pulumi project.
        verdverm 6y ago
        As would I. I see this becoming a pain for the operator of the build systems in a mutli-team environment.
dkdk8283 6y ago
What I don’t get about “modern” development is lack of complexity management. It feels like 8 degrees of work to write HCL that any component developer can do. What maddened me was the lack of support for the count parameter for modules. Made me rage. But not enough to switch to tools like this or troposphere. Not to mention the supply chain implications and security risk that comes with it.
- erik_seaberg 6y ago
  I don't know about the author's issues, but I wrote something like this because early versions of HCL were very limited. E.g., you couldn't write a module that cranks out autoscaled DynamoDB tables per region because you could only pass scalars to modules, not entire key schemata or tag sets. I really wish Terraform itself were written in Python or something similarly hackable without rebuilding the binary for everyone. I've glanced at CDK but it looks like building blocks for reinventing our own Terraform.
  - deleted 6y ago
    [deleted]
- busterarm 6y ago
  > the lack of support for the count parameter for modules. Made me rage. Supported as of Terraform 0.13. It was on the roadmap for a really long time, but because of early design decisions took a while to get there.
programmerslave 6y ago
CDK from AWS usurps cloud formation
- antonvs 6y ago
  CDK still uses CloudFormation to provision resources, so "usurps" is not quite right.
deleted 6y ago
[deleted]
mirekrusin 6y ago
I find cue lang [1] very practical/useful/terse, their idea of lattice based type system is very intuitive and expressive with very little code. [1] https://cuelang.org/ https://cuelang.org/
- verdverm 6y ago
  With Cue, there is the possibility to import TFs definitions into Cue, write your IaC, and output to JSON. Like parent says, Cue feels like the right way to approach this IaC problem with a purpose built language. It was designed to manage this complexity.
- codewithcheese 6y ago
  So happy (because cue is a great solution) and surprised (it's niche atm) to see cue mentioned! I invite readers to try the cue/Kubernetes tutorial, its inspired: https://github.com/cuelang/cue/blob/v0.2.2/doc/tutorial/kubernetes/README.md https://github.com/cuelang/cue/blob/v0.2.2/doc/tutorial/kube... Here the language creator explains the inspiration and the concept, then he works through the above tutorial https://www.youtube.com/watch?v=b3fhA12KS48 https://www.youtube.com/watch?v=b3fhA12KS48
ramaro 6y ago
Kapitan allows you to achieve the same using (python) Kadet to compile JSON tf files https://kapitan.dev/compile/#kadet https://kapitan.dev/compile/#kadet
birdyrooster 6y ago
Did someone just put me in a time machine to 10 years ago?
milkshakes 6y ago
we've been doing this a while with streamalert. nice to see it generalized though.
vr46 6y ago
Code to generate DSL to call APIs? This is somewhat outdated now, although I wouldn’t have even used it five years ago TBH. Abstraction pile-up.
- busterarm 6y ago
  I feel like a lot of people are missing the fact that Terraform's value is in being a generic way to interact with a vast number of APIs. But then when I look at things like Pulumi, I'm reminded that software engineers have their hammer and tend to see all problems as nails. That's not a knock on the profession, seeing as I am one, but writing Terraform serves business needs that writing and deploying software does not. It takes a certain amount of maturity across the industry to not tunnel vision and write code to solve every problem.
  - vr46 6y ago
    Terraform is simply fantastic and it looks like it’s finally stabilized.
- erik_seaberg 6y ago
  Terraform also knows how to orchestrate those API calls in dependency order, update in place when possible or else recreate (reusing the cloud platform’s unique ID from creating each resource), and check for drift (because sometimes people alter the actual cloud resource but not the template file). The problem is that some cloud APIs require a lot of boilerplate params to set up correctly (looking at you, aws_appautoscaling_policy), and HCL’s unit of reuse makes it fairly hard to factor out what our resources have in common and end up with template files clear enough for a reviewer to catch mistakes.
  - vr46 6y ago
    Yaar, not everything available in the resource is supported in the cloud API (hello Elasticbeanstalk) but Terraform is still infinitely better to use and work with directly.
beastman82 6y ago
This exists because terraform DSL, yaml, and Json are not programming languages and are thus tragic choices for making declarative statements
- BaronSamedi 6y ago
  After many years working with these tools I agree. Declarative and template DSLs are OK for basic things but you quickly bump into their limitations when you want to do advanced configurations. At this point administrators should know how to code so being able to use a normal programming language with IaC modules should be the standard. Amazon's CDK for AWS is a step in the right direction. Microsoft's new Bicep for Azure misses the mark in my view because it is yet another DSL and not a real programming language.
- hardwaresofton 6y ago
  I have been saying this for a while now -- as much as I respect Hashicorp and admire their engineering, HCL is the biggest problem with the tools they have that it's used in -- biggest of which being Terraform. Don't use a DSL where a full programming language is what you need. I can see that they probably wanted to be able to skip writing 2/3 language CDKs out of the box, and DSLs probably hit on the need for simplicity really early, but infrastructure code is somewhere you really want to be able to drop into the full expressive power of a programming language. I use Pulumi[0] for my projects, and while I doubt people will be able to get it as much adoption as I think it should have in corporate environments, Terraform has introduced the CDK[1] which is a similar approach. Have a talk with your engineers about complexity (and how not to strangle the rest of the team with it) and stop using DSLs in places you need more expressive power. I think a good indication of when you need to stop using DSLs or when you should maybe realize a DSL wasn't the right way to go is when you start doing things like writing control flow. You could can get away with DSLs if you were using s-expressions, but that is an exception that proves the rule (because there happens to a family of languages that treat s-expressions as syntax). [0]: https://www.pulumi.com https://www.pulumi.com [1]: https://www.hashicorp.com/blog/cdk-for-terraform-enabling-python-and-typescript-support https://www.hashicorp.com/blog/cdk-for-terraform-enabling-py...
kesor 6y ago
Why not just use Python to create the "things" in whenever you need to create them in the first place? Just write the code taking idempotency in mind, and no reason to use Terraform at all.
- vbernat 6y ago
  You will hand up recreating a lot of what Terraform is doing. It is not a matter of just creating resources if they don't exist. Notably, modifying existing resources in the right order is not trivial.
  - fennecfoxen 6y ago
    You’d think it would be possible to make like Python bindings or something though.
- mixedCase 6y ago
  > Just write the code taking idempotency in mind In the real world, this is akin to "just don't write bugs".
ohuhu 6y ago
Planning any support for Azure?
0xbadcafebee 6y ago
Does anyone know if there's OSS projects we could use to replace Terraform with Python + boto3? I specifically do not want to use Pulumi. The dependency validation wouldn't be too hard to implement, but all the inter-resource and feature-specific logic would need to be duplicated. The reason I won't use Pulumi is 1) licensing and 2) corporate ownership. I'd rather use CloudFormation.

60 points by sharjeelsayed 6y ago

Not sure about that. It’s a clever trick alright but it adds a possibly unnecessary layer of leaky abstraction and makes code reuse more difficult when working with “vanilla” Terraform. If you really want to use a full programming language with infra as code, I’d suggest looking into Pulumi. Or, if you want to (or need to) stick to Terraform, try the official CDK.

TabTwo 6y ago

Now we need some Perl scripts to generate the Python scripts.

sunshinerag 6y ago
and then we need a DSL to get rid of all the imperative Perl.
erik_seaberg 6y ago
I feel I should take this opportunity to confess that I once used M4 to generate HCL, before I figured out Terraform JSON.

pram 6y ago

Terraform added a lot of these procedural functions natively in the time since this was created. At any rate if you want to stick with Python, I'd say Pulumi would be a safer choice these days.

jpalomaki 6y ago
Yes, Pulumi uses the Terraform providers, and allows you script your stuff in Typescript/Python/Java/C#. https://www.pulumi.com/ https://www.pulumi.com/
- verdverm 6y ago
  How do Pulumi models in different languages interact or interop?
  - 7sigma 6y ago
    Not an expert but it is possible to mix and match with the new automation api. The example below mixes typescript with go. https://github.com/pulumi/automation-api-examples/tree/main/nodejs/crossLanguage-tsnode https://github.com/pulumi/automation-api-examples/tree/main/...
    - verdverm 6y ago
      Looks like you get a runtime explosion mad then some. Your CI will need all of the runtimes used plus the extra code to handle multiple language runtimes in Pulumi via the automation api. Anyhow, thanks for the link.
      - jpalomaki 6y ago
        I guess typically you would pick one language and use that throughout the Pulumi project.
        verdverm 6y ago
        As would I. I see this becoming a pain for the operator of the build systems in a mutli-team environment.

dkdk8283 6y ago

What I don’t get about “modern” development is lack of complexity management. It feels like 8 degrees of work to write HCL that any component developer can do. What maddened me was the lack of support for the count parameter for modules. Made me rage. But not enough to switch to tools like this or troposphere. Not to mention the supply chain implications and security risk that comes with it.

erik_seaberg 6y ago
I don't know about the author's issues, but I wrote something like this because early versions of HCL were very limited. E.g., you couldn't write a module that cranks out autoscaled DynamoDB tables per region because you could only pass scalars to modules, not entire key schemata or tag sets. I really wish Terraform itself were written in Python or something similarly hackable without rebuilding the binary for everyone. I've glanced at CDK but it looks like building blocks for reinventing our own Terraform.
- deleted 6y ago
  [deleted]
busterarm 6y ago
> the lack of support for the count parameter for modules. Made me rage. Supported as of Terraform 0.13. It was on the roadmap for a really long time, but because of early design decisions took a while to get there.

programmerslave 6y ago

CDK from AWS usurps cloud formation

antonvs 6y ago
CDK still uses CloudFormation to provision resources, so "usurps" is not quite right.

deleted 6y ago

[deleted]

mirekrusin 6y ago

I find cue lang [1] very practical/useful/terse, their idea of lattice based type system is very intuitive and expressive with very little code. [1] https://cuelang.org/ https://cuelang.org/

verdverm 6y ago
With Cue, there is the possibility to import TFs definitions into Cue, write your IaC, and output to JSON. Like parent says, Cue feels like the right way to approach this IaC problem with a purpose built language. It was designed to manage this complexity.
codewithcheese 6y ago
So happy (because cue is a great solution) and surprised (it's niche atm) to see cue mentioned! I invite readers to try the cue/Kubernetes tutorial, its inspired: https://github.com/cuelang/cue/blob/v0.2.2/doc/tutorial/kubernetes/README.md https://github.com/cuelang/cue/blob/v0.2.2/doc/tutorial/kube... Here the language creator explains the inspiration and the concept, then he works through the above tutorial https://www.youtube.com/watch?v=b3fhA12KS48 https://www.youtube.com/watch?v=b3fhA12KS48

ramaro 6y ago

Kapitan allows you to achieve the same using (python) Kadet to compile JSON tf files https://kapitan.dev/compile/#kadet https://kapitan.dev/compile/#kadet

birdyrooster 6y ago

Did someone just put me in a time machine to 10 years ago?

milkshakes 6y ago

we've been doing this a while with streamalert. nice to see it generalized though.

vr46 6y ago

Code to generate DSL to call APIs? This is somewhat outdated now, although I wouldn’t have even used it five years ago TBH. Abstraction pile-up.

busterarm 6y ago
I feel like a lot of people are missing the fact that Terraform's value is in being a generic way to interact with a vast number of APIs. But then when I look at things like Pulumi, I'm reminded that software engineers have their hammer and tend to see all problems as nails. That's not a knock on the profession, seeing as I am one, but writing Terraform serves business needs that writing and deploying software does not. It takes a certain amount of maturity across the industry to not tunnel vision and write code to solve every problem.
- vr46 6y ago
  Terraform is simply fantastic and it looks like it’s finally stabilized.
erik_seaberg 6y ago
Terraform also knows how to orchestrate those API calls in dependency order, update in place when possible or else recreate (reusing the cloud platform’s unique ID from creating each resource), and check for drift (because sometimes people alter the actual cloud resource but not the template file). The problem is that some cloud APIs require a lot of boilerplate params to set up correctly (looking at you, aws_appautoscaling_policy), and HCL’s unit of reuse makes it fairly hard to factor out what our resources have in common and end up with template files clear enough for a reviewer to catch mistakes.
- vr46 6y ago
  Yaar, not everything available in the resource is supported in the cloud API (hello Elasticbeanstalk) but Terraform is still infinitely better to use and work with directly.

beastman82 6y ago

This exists because terraform DSL, yaml, and Json are not programming languages and are thus tragic choices for making declarative statements

BaronSamedi 6y ago
After many years working with these tools I agree. Declarative and template DSLs are OK for basic things but you quickly bump into their limitations when you want to do advanced configurations. At this point administrators should know how to code so being able to use a normal programming language with IaC modules should be the standard. Amazon's CDK for AWS is a step in the right direction. Microsoft's new Bicep for Azure misses the mark in my view because it is yet another DSL and not a real programming language.
hardwaresofton 6y ago
I have been saying this for a while now -- as much as I respect Hashicorp and admire their engineering, HCL is the biggest problem with the tools they have that it's used in -- biggest of which being Terraform. Don't use a DSL where a full programming language is what you need. I can see that they probably wanted to be able to skip writing 2/3 language CDKs out of the box, and DSLs probably hit on the need for simplicity really early, but infrastructure code is somewhere you really want to be able to drop into the full expressive power of a programming language. I use Pulumi[0] for my projects, and while I doubt people will be able to get it as much adoption as I think it should have in corporate environments, Terraform has introduced the CDK[1] which is a similar approach. Have a talk with your engineers about complexity (and how not to strangle the rest of the team with it) and stop using DSLs in places you need more expressive power. I think a good indication of when you need to stop using DSLs or when you should maybe realize a DSL wasn't the right way to go is when you start doing things like writing control flow. You could can get away with DSLs if you were using s-expressions, but that is an exception that proves the rule (because there happens to a family of languages that treat s-expressions as syntax). [0]: https://www.pulumi.com https://www.pulumi.com [1]: https://www.hashicorp.com/blog/cdk-for-terraform-enabling-python-and-typescript-support https://www.hashicorp.com/blog/cdk-for-terraform-enabling-py...

kesor 6y ago

Why not just use Python to create the "things" in whenever you need to create them in the first place? Just write the code taking idempotency in mind, and no reason to use Terraform at all.

vbernat 6y ago
You will hand up recreating a lot of what Terraform is doing. It is not a matter of just creating resources if they don't exist. Notably, modifying existing resources in the right order is not trivial.
- fennecfoxen 6y ago
  You’d think it would be possible to make like Python bindings or something though.
mixedCase 6y ago
> Just write the code taking idempotency in mind In the real world, this is akin to "just don't write bugs".

ohuhu 6y ago

Planning any support for Azure?

0xbadcafebee 6y ago

Does anyone know if there's OSS projects we could use to replace Terraform with Python + boto3? I specifically do not want to use Pulumi. The dependency validation wouldn't be too hard to implement, but all the inter-resource and feature-specific logic would need to be duplicated. The reason I won't use Pulumi is 1) licensing and 2) corporate ownership. I'd rather use CloudFormation.