12 ms·

I hate Terraform with a passion but it is probably the best tool out there for managing cloud infrastructure so I use it at work with no plans to replace it. T

I hate Terraform with a passion but it is probably the best tool out there for managing cloud infrastructure so I use it at work with no plans to replace it. The biggest downsides are the awful half-baked language and the awkwardness of modules and passing values throughout your config. Also the staticness of providers are a serious pain, for example you can't create a kubernetes cluster then add a resource to it. The work around is to use two separate Terraform stacks which brings a lot of pain for passing values across the boundary. Furthermore you can no longer effectively plan any change that affects the boundary between the two stacks. "Luckily" Terraform's performance is so bad that you need to split the stacks anyways. The biggest feature I would like to see is the ability to dump a pure representation of your evaluated configuration. This would allow reasonable diffs in CI. There are of course complications, especially if you use `data` resources but technically it is possible to do a very good job here which would make it so much easier to make changes.

k__ 5y ago
I never used Terraform, I started with Vagrant, then CloudFormation, CDK, and now Pulumi. I like Pulumi the most right now. It integrates with services like Cloudflare and Auth0 and I can use TypeScript to write my code.
mwarkentin 5y ago
The module passing got a lot better in 0.12 when you could pass full modules or resources as outputs and vars.
sandGorgon 5y ago
have you tried pulumi ? what's your opinion ?
- kevincox 5y ago
  I have not.
Hikikomori 5y ago
Maybe try out https://www.pulumi.com/ https://www.pulumi.com/
cube2222 5y ago
> The biggest feature I would like to see is the ability to dump a pure representation of your evaluated configuration. This would allow reasonable diffs in CI. There are of course complications, especially if you use `data` resources but technically it is possible to do a very good job here which would make it so much easier to make changes. The planned state, current state, and diff of them are all available as separate fields in the Terraform plan file, is that not what you're looking for?
- kevincox 5y ago
  The key word is "pure" here. These things all depend on the current state of the infrastructure. The "planned state" is close to what I want, but it can be very confusing if someone has deployed a new change since you forked off.
  - lowercase1 5y ago
    Yeah. I have a poor view of terraform since my first interaction was trying to a few one line changes to avoid repetition but couldn't find why it didn't work without setting up connection to the AWS S3 bucket.
ptomato 5y ago
> for example you can't create a kubernetes cluster then add a resource to it I have no love for HCL, but you can do this by creating a kubernetes provider with the auth token pointing at the resource output for the auth token you generated for the cluster.
- nrmitchi 5y ago
  Yes, however this will work (typically) if the cluster already exists (a previous run), but typically not if you creating the cluster, and kubernetes provider, as part of the same run. IIRC you'll end up with a kubernetes provider without auth (typically pointing at your local machine), which is 1, not helpful, and 2) can be actively bad. I believe the core issue here is that providers don't have the ability to specify a `depends_on` relation: https://github.com/hashicorp/terraform/issues/2430 https://github.com/hashicorp/terraform/issues/2430
  - marenkay 5y ago
    This works even without the depends_on property. All you need to is have the module you use for creating the cluster have an output that is guaranteed to be a computed property. Then use that computed property as input variable for whatever you want to deploy into Kubernetes. We're using this with multiple providers and it works. Of course, an actual dependency that's visible would be better.
    - nrmitchi 5y ago
      I'd love to see an example of this actually working, because I have had the opposite experience (explicitly with the Kubernetes and Helm providers); I've had to do applies in multiple steps.
      - gouggoug 5y ago
        This should work (as in, it will create the cluster and only then add the k8s resource to it, in the same plan/apply). Here the module creates an EKS cluster, but this would work for any module that creates a k8s cluster. module "my_cluster" { source = "terraform-aws-modules/eks/aws" version = "17.0.2" cluster_name = "my-cluster" cluster_version = "1.18" } # Queries for Kubernetes authentication # this data query depends on the module my_cluster data "aws_eks_cluster" "my_cluster" { name = module.my_cluster.cluster_id } # this data query depends on the module my_cluster data "aws_eks_cluster_auth" "my_cluster" { name = module.my_cluster.cluster_id } # this provider depends on the data query above, which depends on the module my_cluster provider "kubernetes" { host = data.aws_eks_cluster.my_cluster.endpoint cluster_ca_certificate = base64decode(data.aws_eks_cluster.my_cluster.certificate_authority.0.data) token = data.aws_eks_cluster_auth.my_cluster.token load_config_file = false } # this provider depends on the data query above, which depends on the module my_cluster provider "helm" { kubernetes { host = data.aws_eks_cluster.my_cluster.endpoint cluster_ca_certificate = base64decode(data.aws_eks_cluster.my_cluster.certificate_authority.0.data) token = data.aws_eks_cluster_auth.my_cluster.token load_config_file = false } } # this resource depends on the k8s provider, which depends on the data query above, which depends on the module my_cluster resource "kubernetes_namespace" "namespaces" { metadata { name = "my-namespace" } }
thanatos519 5y ago
Amen! I found it excruciating that the language was always a few simple steps away from being homomorphic to JSON. I desperately needed to be able to manipulate it as data structures, not as strings. All of the ways I found to work around its limitations made me wish for something else entirely.
- Znafon 5y ago
  You can use JSON for your configuration: https://www.terraform.io/docs/language/syntax/json.html https://www.terraform.io/docs/language/syntax/json.html !
- cube2222 5y ago
  You can write Terraform using JSON if you want to: https://www.terraform.io/docs/language/syntax/json.html https://www.terraform.io/docs/language/syntax/json.html Or do you mean something deeper?
  - thanatos519 5y ago
    I mean being able to reliably convert HCL↔JSON.
- rad_gruchalski 5y ago
  It's a language for a reason, there's a grammar, parser, lexer, ast. What's the problem?
- garettmd 5y ago
  Have you used it since they introduced HCL2? It supports other data structures much better than it used to. Maps, lists, sets, etc. are much easier to work with.
  - throwaway894345 5y ago
    Still a far cry away from a proper programming language, which is what we need. For example, if you want to loop over some config and generate a resource for each config, but the resources need different providers (e.g., different AWS accounts) then you just can't do it. Further, if you just want a little function, you have to build a fully fledged module. Then there are the crazy namespaces (`var`, `local`, `resource`, `module`, etc).
marenkay 5y ago
Add an secrets provider to avoid having secrets in state files to that list.
granra 5y ago
People mention pulumi but hashicorp are creating something similar with https://github.com/hashicorp/terraform-cdk https://github.com/hashicorp/terraform-cdk. But all the existing terraform providers work with it afaik.
- k__ 5y ago
  Right, tfcdk and k8scdk are a thing. Pulumi is also integrating with TF.
- tonyhb 5y ago
  I don't know if people have even tried Pulumi before recommending it. I've tried it, and it has buggy defaults, diff generation, etc. Each time I applied the same code, it would generate a diff based off of some internal defaults and... recreate the exact same infrastructure by _tearing it down_ and making it fresh. Not ideal. Would advise using the TF CDK specifically.
  - ManWith2Plans 5y ago
    The token system is broken in TF CDK still and it's not ready for adoption. I've built two stacks with it but I'm back at terraform for now. I intend to explore pulumi though when the opportunity presents itself. I think using a Turing-complete language like typescript with mature tooling to define cloud infrastructure feels very natural and makes things much more manageable than using HCL. One thing I absolutely can't do without is the state management api terraform provides with its CLI. This is absent from terraform-cdk and aws's CDK, although many of the same APIs seem to exist for pulumi.
    - throwaway894345 5y ago
      > I think using a Turing-complete language like typescript with mature tooling to define cloud infrastructure feels very natural and makes things much more manageable than using HCL. Fully agree. Not sure if any of the CDKs (or Pulumi) get the ergonomics right though. The ergonomics should feel like we're just generating YAML/JSON/etc, but the CDKs I've seen require inheritance, mutable state, etc. > One thing I absolutely can't do without is the state management api terraform provides with its CLI. This is absent from terraform-cdk and aws's CDK, although many of the same APIs seem to exist for pulumi. AWS's CDK is built on CloudFormation, so I don't think it has analogs for Terraform's state APIs. As for TF CDK, I would think you would just use Terraform's CLI state management directly? Maybe I'm confused about what you're trying to do?
gdubya 5y ago
I like Terraform for infrastructure, up to the point of creating the K8s cluster, then ArgoCD for keeping K8s in sync.
- stuff4ben 5y ago
  That's an interesting combo. What are you keeping in sync in K8s with Argo?
  - gdubya 5y ago
    The operators we offer in our clusters (e.g. ECK, Prometheus, etc... the ArgoCD ApplicationSet generators make it easy to configure which features are installed on each cluster), as well as the applications developed by the development teams. Our work isn't complete yet (still working on sync for secrets and RBAC), but it's working nicely so far.
throwaway894345 5y ago
I strongly agree both with respect for the half-baked-ness of the language and with the "it's probably the best out there". Ultimately, these tools should have a static/yaml-like "assembly language" that describes the state of your infrastructure without any of the DRY. There would be a diffing engine which would figure out what changes need to be applied and apply them accordingly. Users could use some vanilla programming language to generate that yaml in a DRY way; then the Terraform folks don't need to badly reinvent a programming language. I know they also have a CDK, but I can't tell if it properly solves that problem or if it still forces us into Terraform idiosyncrasies (i.e., if I rename something in Terraform, it will try to delete the corresponding resource and recreate it, and I think that absurd behavior remains with the CDK).
- kevincox 5y ago
  100%. Terraform is half-way between a tool for generating the configuration and applying it. I think Terraform's application engine is actually quite good, but I would like to use a much better tool to generate the config. (And be able to diff that config) You can feed JSON to Terraform however this falls over if you need dependencies for output values. This usually isn't an issue because most Cloud provider resources have predictable IDs but as soon as you have one that doesn't you are up for a lot of pain and suffering.
  - mwarkentin 5y ago
    There's CDK for Terraform: https://github.com/hashicorp/terraform-cdk https://github.com/hashicorp/terraform-cdk
    - cormacrelf 5y ago
      Someone should make a Clojure demo of those Java bindings, or even cljs. I hope Clojure has good type based completions these days, because it would be a fantastic language for this.
  - lvncelot 5y ago
    > You can feed JSON to Terraform however this falls over if you need dependencies for output values This is what I've started doing with Jsonnet for generation, and also exactly why I've stopped doing it.
unethical_ban 5y ago
Yep, the documentation is sometimes lacking, and the concept of moving variables in and out of modules is not intuitive, to say the least.
syntaxstic 5y ago
How does terraform compare to ansible?
- zimbatm 5y ago
  Both tools can be used to create cloud resources and configure machines but fundamentally they are very different. Ansible is a list of actions that you apply linearly. Each action might be a noop if it already exists. Terraform is a tree of resources that are applied by order of dependency. Terraform also records the previous run and deletes resources that are no longer in the code. Generally, Ansible is great at performing actions on a lot of hosts. A sort of multi-ssh. And Terraform is best adapted to manage cloud resources.
- kevincox 5y ago
  Ansible focuses on provisioning machines whereas Terraform focuses on creating Cloud infrastructure. A common combo is using Terraform to provision VMs and networking settings then using Ansible to configure those VMs. I find few if any reasons to use Ansible over a shell script. IMHO Ansible is just a weird YAML syntax to generate a "shell" script with some utilities to ship that script to nodes over the network. I find it super awkward not to mention slow and inconsistent. For deployments I much prefer using Nix and for imperative actions I just use actual shell/python.
  - gerbilly 5y ago
    You can totally provision using ansible too, on most cloud vendors. The reason to use ansible over a shell script is that the ansible playbook will be idempotent. That is to say you can run/rerun the playbook from any point without having to wipe any previous work, or worry about double applying your config changes.
    - kevincox 5y ago
      > is that the ansible playbook will be idempotent This isn't really true. I think you are correct that most of the built-in operations are idempotent but you can also do this with a small library of functions in a shell/python script or whatever you prefer. Most things you want to do on provision are idempotent anyways (install this package, download this file) or are trivial to make so (create this directory). I would take a real programming language any day for the minor cost of having to handle idempotency myself. It would take a couple of hours to reimplement idempotent primitives to replace the Ansible standard library in just about any language. In my mind the main value of Ansible is playbooks that others have made for you, but many people avoid these anyways to have full control.
himinlomax 5y ago
The tool is ok, but developing plugins for it shows how inadequate Golang is for the job. There's so much repetition and boilerplate required. I wrote a FreeIPA plugin a few years back, it handled just registering a host and the executable weighed over 100 MB! WTF? Haven't looked at that side of things lately, I wonder if it's different nowadays.
- cube2222 5y ago
  We have a big amount of resources available inside of our Spacelift provider[0] and it weights ~20 MB. It'll probably mostly depend on the libraries you use. [0]:https://github.com/spacelift-io/terraform-provider-spacelift/ https://github.com/spacelift-io/terraform-provider-spacelift...
- tornato7 5y ago
  Is it a Go problem or a new-to-Go problem? I haven't written terraform plugins specifically but I have been writing Go for years and never find myself needing to write an excessive amount of boilerplate. There can definitely be some frustrations in dealing with dynamic JSON though. JSON-to-Go converters are your friend.
  - himinlomax 5y ago
    I was not using anything special, I had implemented my own client for IPA. Te equivalent functionality in Python (ended up using Ansible to do my thing) uses just a few kB ...
- StopHammoTime 5y ago
  Definitely agree with this, Go is so verbose for the application. When I wrote a provider, I had the same problem. What made it even more worse is that I was connecting into an API that made use of dynamic json generation. So many interfaces and other hacks to get the json documents to parse correctly.
eKIK 5y ago
+1 from me on the "awful half-baked language" (HCL). I just recently wrote an article about my experience, including issues and workarounds, when migrating from Terraform to Pulumi: https://blog.ekik.org/my-experience-migrating-my-infrastructure-from-terraform-to-pulumi https://blog.ekik.org/my-experience-migrating-my-infrastruct... Hope it's OK that I'm sharing it here. I think it's relevant because there seems to be quite a lot of interest around Pulumi, and how one would go about moving from Terraform to Pulumi.
- mwcampbell 5y ago
  I'm actually thinking of going the other way. I've been using Pulumi for several months now, and I'm thinking of moving to Terraform, because it has a so much larger third-party ecosystem, including more providers, and tools that can analyze HCL, like Infracost and security scanners. When will I learn to see the bigger picture and value popularity over quality?
  - eKIK 5y ago
    It's a very interesting point. I've been part of managing rather large Terraform infrastructures (1000+ resources) for a couple of years, but I'm a Pulumi n00b with only about a month of experience. The infrastructure I'm managing right now with Pulumi is much smaller, only around 130-140 different resources. For me it ultimately came down to developer productivity. I'm much better at convincing Pulumi to do what I want compared to how it was with Terraform. This also makes me a much happier and less frustrated developer :). My priorities might very well be different if I were to manage much larger infrastructures (infra cost would be more important for example).
    - mwcampbell 5y ago
      The stack I manage with Pulumi is currently around 300 resources. (I think that count is inflated by all the secrets in AWS Secrets Manager, because each secret has two resources: the secret and the current version.) I currently manage it by myself, but I'm hoping that won't be the case for very long. Maybe the ending of my previous comment was too cynical. But I think I've repeatedly made the mistake of valuing my productivity and happiness as a currently solo developer over what will let my company take full advantage of a big third-party ecosystem (including a large talent pool).
time0ut 5y ago
Have you tried Terragrunt [0]? It helps a lot with managing a set of related stacks. Still feels like a bandaid on a broken model, but it is what we have. [0] https://terragrunt.gruntwork.io/ https://terragrunt.gruntwork.io/ Regarding performance, last time I looked, Hashicorp's documentation implied there was no limit to the size of a Terraform stack. I think they meant theoretically in a science fiction universe where humanity had captured all of the sun's output to perform terraform plan and apply...
tdumitrescu 5y ago
Yeah, these days I try to avoid writing any HCL and instead feed Terraform with JSON generated via jsonnet (which we were already using to generate k8s YAML). Much better templating and language features while still remaining declarative, and it helps on a team to have a single source language for such configs.
etxm 5y ago
> Also the staticness of providers are a serious pain, for example you can't create a kubernetes cluster then add a resource to it. TF def has some rough edges, but you can certainly create a cluster and add resources in a single root module (I don’t think it’s a great practice). In this example the EKS cluster is in a module, but it can be a ref to a resource in the same module as well. data "aws_eks_cluster_auth" "current" { name = module.eks.cluster_id } provider "kubernetes" { load_config_file = false host = module.eks.cluster_endpoint cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data) token = data.aws_eks_cluster_auth.current.token }
xorcist 5y ago
Why not use something like Ansible instead? It too is declarative. It too can be easily extended. It's also something a lot of people already know. I used to use Ansible or Puppet for these things before Terraform was all the rage. It was a lot more stable than trying to distributing those state files, which is a strange design to pick. There are plenty of existing modules but it's also dead simple to write your own.
- akvadrako 5y ago
  Ansible is not really made for managing cloud resources and it shows - the modules are not production ready.
  - throwdbaaway 5y ago
    For GCP, both ansible modules and terraform modules are actually generated from https://github.com/GoogleCloudPlatform/magic-modules https://github.com/GoogleCloudPlatform/magic-modules, so their "production readiness" are the same. I understand that mitchellh himself personally created a bunch of cloud modules for terraform at the beginning, and those were likely of higher quality than whatever created by some internal developers assigned by Google/Microsoft, and might be slightly better than the AWS modules maintained by community. Anyway, when it comes to ansible versus terraform, we shall move the discourse to states management instead. With ansible, you don't have to deal with states, but will need to clean up the cloud resources separately. With terraform, you can use the tool to clean up the cloud resources easily, but then you also have the headache of managing states. Plus, whenever you change something, there is always the nagging feeling that it will do a destroy/recreate instead of an in-place update.
- dtech 5y ago
  I have limited experience with Ansible, but afaik calling it declarative when compared to Terraform is a stretch [1] [1] https://blog.gruntwork.io/why-we-use-terraform-and-not-chef-puppet-ansible-saltstack-or-cloudformation-7989dad2865c#f26f https://blog.gruntwork.io/why-we-use-terraform-and-not-chef-...
oxplot 5y ago
> for example you can't create a kubernetes cluster then add a resource to it Of course you can! DM me if you want details.
- gouggoug 5y ago
  Yes you can, see https://news.ycombinator.com/item?id=27435269 https://news.ycombinator.com/item?id=27435269
  - hnjst 5y ago
    I can confirm, we're using a similar approach at it mostly works. There are still issues though, if you try to remove your cluster the k8s provider can't be configured (no module.my_cluster.cluster_id anymore) and the refresh phase of plan will fail. You can find workarounds but those I know are quite manual / ugly.
AcerbicZero 5y ago
I’ve had many similar frustrations about terraform, and the overall lack of visibility into what’s happening drives me mad at times. A proper repl, with the ability to actually manage a config would be a huge step forward - I spend more time trying to figure out what vars get populated and how I can get a value into another resource than anything else. It’s like I’m constantly fighting with the HCL syntax to get what I want to happen.
- clipradiowallet 5y ago
  If you want visibility(spoiler: it's just API calls), try using `TF_LOG=DEBUG terraform <foo>`. You might also want to set `-parallelism=1` or you'll be treated to statements printing in an order you are not expecting.
Nilithus 5y ago
> "Luckily" Terraform's performance is so bad that you need to split the stacks anyways Not sure what about terraforms performance is so bad. Seems hard to blame a tool who's main execution path is potentially 100's of network IO requests with 3rd party API's. Most of the "split stacks" I've seen is more for code organization and security reasons rather than performance. Seems safer to know 100% that deploying infra for my app isn't going to mess with my VPC settings and can be executed with a lower privileged role. > Furthermore you can no longer effectively plan any change that affects the boundary between the two stacks. That's fair -- you do end up with these "foundational" modules a lot of the time. Like an 'aws-account basics' module or something that other modules expect the account to be setup with that base for being able to query data objects for subnets ect... planning changes if that changes be difficult but not impossible. Good versioning is critical. Feels in the same vein as apps that need to manage framework updates and things like that. (though can be made more difficult or easier based on how you've broken up using your cloud provider -- multiple accounts by buisness unit or all in one).
- jmccarthy 5y ago
  Our experience of building a provider: performance is fast with fast APIs, and slow with slow APIs. Haven't observed any of the core diffing, DAG, or apply scheduling to be problematic (but also haven't tried an apply at extremely high - 10^4? 10^5? - resource count)
more_corn 5y ago
Same (hate it, love it, use it every day). I can't believe you left out the the stilted looping syntax.
- kevincox 5y ago
  I didn't figure it was worth starting... - Lack of functions (the only real functions are modules which are basically unusable for quick computations such as "slugify this string"). - Very primitive loops. - Lack of temporary variables. I often end up looping over a list multiple times and storing intermediates. - No panic or log functions.
clipradiowallet 5y ago
> The biggest feature I would like to see is the ability to dump a pure representation of your evaluated configuration. Are you asking for a dump of existing state or desired state? For existing state, see `terraform state pull`. For delta between desired+existing, see `terraform plan -out`. My apologies in advance if I completely misunderstood what you were asking for.
- kevincox 5y ago
  I am asking to dump the desired. So that I can diff the desired against the commit vs the desired of the last commit. I don't want to include production at all.

k__ 5y ago

I never used Terraform, I started with Vagrant, then CloudFormation, CDK, and now Pulumi. I like Pulumi the most right now. It integrates with services like Cloudflare and Auth0 and I can use TypeScript to write my code.

mwarkentin 5y ago

The module passing got a lot better in 0.12 when you could pass full modules or resources as outputs and vars.

sandGorgon 5y ago

have you tried pulumi ? what's your opinion ?

kevincox 5y ago
I have not.

Hikikomori 5y ago

Maybe try out https://www.pulumi.com/ https://www.pulumi.com/

cube2222 5y ago

> The biggest feature I would like to see is the ability to dump a pure representation of your evaluated configuration. This would allow reasonable diffs in CI. There are of course complications, especially if you use `data` resources but technically it is possible to do a very good job here which would make it so much easier to make changes. The planned state, current state, and diff of them are all available as separate fields in the Terraform plan file, is that not what you're looking for?

kevincox 5y ago
The key word is "pure" here. These things all depend on the current state of the infrastructure. The "planned state" is close to what I want, but it can be very confusing if someone has deployed a new change since you forked off.
- lowercase1 5y ago
  Yeah. I have a poor view of terraform since my first interaction was trying to a few one line changes to avoid repetition but couldn't find why it didn't work without setting up connection to the AWS S3 bucket.

ptomato 5y ago

> for example you can't create a kubernetes cluster then add a resource to it I have no love for HCL, but you can do this by creating a kubernetes provider with the auth token pointing at the resource output for the auth token you generated for the cluster.

nrmitchi 5y ago
Yes, however this will work (typically) if the cluster already exists (a previous run), but typically not if you creating the cluster, and kubernetes provider, as part of the same run. IIRC you'll end up with a kubernetes provider without auth (typically pointing at your local machine), which is 1, not helpful, and 2) can be actively bad. I believe the core issue here is that providers don't have the ability to specify a `depends_on` relation: https://github.com/hashicorp/terraform/issues/2430 https://github.com/hashicorp/terraform/issues/2430
- marenkay 5y ago
  This works even without the depends_on property. All you need to is have the module you use for creating the cluster have an output that is guaranteed to be a computed property. Then use that computed property as input variable for whatever you want to deploy into Kubernetes. We're using this with multiple providers and it works. Of course, an actual dependency that's visible would be better.
  - nrmitchi 5y ago
    I'd love to see an example of this actually working, because I have had the opposite experience (explicitly with the Kubernetes and Helm providers); I've had to do applies in multiple steps.
    - gouggoug 5y ago
      This should work (as in, it will create the cluster and only then add the k8s resource to it, in the same plan/apply). Here the module creates an EKS cluster, but this would work for any module that creates a k8s cluster. module "my_cluster" { source = "terraform-aws-modules/eks/aws" version = "17.0.2" cluster_name = "my-cluster" cluster_version = "1.18" } # Queries for Kubernetes authentication # this data query depends on the module my_cluster data "aws_eks_cluster" "my_cluster" { name = module.my_cluster.cluster_id } # this data query depends on the module my_cluster data "aws_eks_cluster_auth" "my_cluster" { name = module.my_cluster.cluster_id } # this provider depends on the data query above, which depends on the module my_cluster provider "kubernetes" { host = data.aws_eks_cluster.my_cluster.endpoint cluster_ca_certificate = base64decode(data.aws_eks_cluster.my_cluster.certificate_authority.0.data) token = data.aws_eks_cluster_auth.my_cluster.token load_config_file = false } # this provider depends on the data query above, which depends on the module my_cluster provider "helm" { kubernetes { host = data.aws_eks_cluster.my_cluster.endpoint cluster_ca_certificate = base64decode(data.aws_eks_cluster.my_cluster.certificate_authority.0.data) token = data.aws_eks_cluster_auth.my_cluster.token load_config_file = false } } # this resource depends on the k8s provider, which depends on the data query above, which depends on the module my_cluster resource "kubernetes_namespace" "namespaces" { metadata { name = "my-namespace" } }

thanatos519 5y ago

Amen! I found it excruciating that the language was always a few simple steps away from being homomorphic to JSON. I desperately needed to be able to manipulate it as data structures, not as strings. All of the ways I found to work around its limitations made me wish for something else entirely.

Znafon 5y ago
You can use JSON for your configuration: https://www.terraform.io/docs/language/syntax/json.html https://www.terraform.io/docs/language/syntax/json.html !
cube2222 5y ago
You can write Terraform using JSON if you want to: https://www.terraform.io/docs/language/syntax/json.html https://www.terraform.io/docs/language/syntax/json.html Or do you mean something deeper?
- thanatos519 5y ago
  I mean being able to reliably convert HCL↔JSON.
rad_gruchalski 5y ago
It's a language for a reason, there's a grammar, parser, lexer, ast. What's the problem?
garettmd 5y ago
Have you used it since they introduced HCL2? It supports other data structures much better than it used to. Maps, lists, sets, etc. are much easier to work with.
- throwaway894345 5y ago
  Still a far cry away from a proper programming language, which is what we need. For example, if you want to loop over some config and generate a resource for each config, but the resources need different providers (e.g., different AWS accounts) then you just can't do it. Further, if you just want a little function, you have to build a fully fledged module. Then there are the crazy namespaces (`var`, `local`, `resource`, `module`, etc).

marenkay 5y ago

Add an secrets provider to avoid having secrets in state files to that list.

granra 5y ago

People mention pulumi but hashicorp are creating something similar with https://github.com/hashicorp/terraform-cdk https://github.com/hashicorp/terraform-cdk. But all the existing terraform providers work with it afaik.

k__ 5y ago
Right, tfcdk and k8scdk are a thing. Pulumi is also integrating with TF.
tonyhb 5y ago
I don't know if people have even tried Pulumi before recommending it. I've tried it, and it has buggy defaults, diff generation, etc. Each time I applied the same code, it would generate a diff based off of some internal defaults and... recreate the exact same infrastructure by _tearing it down_ and making it fresh. Not ideal. Would advise using the TF CDK specifically.
- ManWith2Plans 5y ago
  The token system is broken in TF CDK still and it's not ready for adoption. I've built two stacks with it but I'm back at terraform for now. I intend to explore pulumi though when the opportunity presents itself. I think using a Turing-complete language like typescript with mature tooling to define cloud infrastructure feels very natural and makes things much more manageable than using HCL. One thing I absolutely can't do without is the state management api terraform provides with its CLI. This is absent from terraform-cdk and aws's CDK, although many of the same APIs seem to exist for pulumi.
  - throwaway894345 5y ago
    > I think using a Turing-complete language like typescript with mature tooling to define cloud infrastructure feels very natural and makes things much more manageable than using HCL. Fully agree. Not sure if any of the CDKs (or Pulumi) get the ergonomics right though. The ergonomics should feel like we're just generating YAML/JSON/etc, but the CDKs I've seen require inheritance, mutable state, etc. > One thing I absolutely can't do without is the state management api terraform provides with its CLI. This is absent from terraform-cdk and aws's CDK, although many of the same APIs seem to exist for pulumi. AWS's CDK is built on CloudFormation, so I don't think it has analogs for Terraform's state APIs. As for TF CDK, I would think you would just use Terraform's CLI state management directly? Maybe I'm confused about what you're trying to do?

gdubya 5y ago

I like Terraform for infrastructure, up to the point of creating the K8s cluster, then ArgoCD for keeping K8s in sync.

stuff4ben 5y ago
That's an interesting combo. What are you keeping in sync in K8s with Argo?
- gdubya 5y ago
  The operators we offer in our clusters (e.g. ECK, Prometheus, etc... the ArgoCD ApplicationSet generators make it easy to configure which features are installed on each cluster), as well as the applications developed by the development teams. Our work isn't complete yet (still working on sync for secrets and RBAC), but it's working nicely so far.

throwaway894345 5y ago

I strongly agree both with respect for the half-baked-ness of the language and with the "it's probably the best out there". Ultimately, these tools should have a static/yaml-like "assembly language" that describes the state of your infrastructure without any of the DRY. There would be a diffing engine which would figure out what changes need to be applied and apply them accordingly. Users could use some vanilla programming language to generate that yaml in a DRY way; then the Terraform folks don't need to badly reinvent a programming language. I know they also have a CDK, but I can't tell if it properly solves that problem or if it still forces us into Terraform idiosyncrasies (i.e., if I rename something in Terraform, it will try to delete the corresponding resource and recreate it, and I think that absurd behavior remains with the CDK).

kevincox 5y ago
100%. Terraform is half-way between a tool for generating the configuration and applying it. I think Terraform's application engine is actually quite good, but I would like to use a much better tool to generate the config. (And be able to diff that config) You can feed JSON to Terraform however this falls over if you need dependencies for output values. This usually isn't an issue because most Cloud provider resources have predictable IDs but as soon as you have one that doesn't you are up for a lot of pain and suffering.
- mwarkentin 5y ago
  There's CDK for Terraform: https://github.com/hashicorp/terraform-cdk https://github.com/hashicorp/terraform-cdk
  - cormacrelf 5y ago
    Someone should make a Clojure demo of those Java bindings, or even cljs. I hope Clojure has good type based completions these days, because it would be a fantastic language for this.
- lvncelot 5y ago
  > You can feed JSON to Terraform however this falls over if you need dependencies for output values This is what I've started doing with Jsonnet for generation, and also exactly why I've stopped doing it.

unethical_ban 5y ago

Yep, the documentation is sometimes lacking, and the concept of moving variables in and out of modules is not intuitive, to say the least.

syntaxstic 5y ago

How does terraform compare to ansible?

zimbatm 5y ago
Both tools can be used to create cloud resources and configure machines but fundamentally they are very different. Ansible is a list of actions that you apply linearly. Each action might be a noop if it already exists. Terraform is a tree of resources that are applied by order of dependency. Terraform also records the previous run and deletes resources that are no longer in the code. Generally, Ansible is great at performing actions on a lot of hosts. A sort of multi-ssh. And Terraform is best adapted to manage cloud resources.
kevincox 5y ago
Ansible focuses on provisioning machines whereas Terraform focuses on creating Cloud infrastructure. A common combo is using Terraform to provision VMs and networking settings then using Ansible to configure those VMs. I find few if any reasons to use Ansible over a shell script. IMHO Ansible is just a weird YAML syntax to generate a "shell" script with some utilities to ship that script to nodes over the network. I find it super awkward not to mention slow and inconsistent. For deployments I much prefer using Nix and for imperative actions I just use actual shell/python.
- gerbilly 5y ago
  You can totally provision using ansible too, on most cloud vendors. The reason to use ansible over a shell script is that the ansible playbook will be idempotent. That is to say you can run/rerun the playbook from any point without having to wipe any previous work, or worry about double applying your config changes.
  - kevincox 5y ago
    > is that the ansible playbook will be idempotent This isn't really true. I think you are correct that most of the built-in operations are idempotent but you can also do this with a small library of functions in a shell/python script or whatever you prefer. Most things you want to do on provision are idempotent anyways (install this package, download this file) or are trivial to make so (create this directory). I would take a real programming language any day for the minor cost of having to handle idempotency myself. It would take a couple of hours to reimplement idempotent primitives to replace the Ansible standard library in just about any language. In my mind the main value of Ansible is playbooks that others have made for you, but many people avoid these anyways to have full control.

himinlomax 5y ago

The tool is ok, but developing plugins for it shows how inadequate Golang is for the job. There's so much repetition and boilerplate required. I wrote a FreeIPA plugin a few years back, it handled just registering a host and the executable weighed over 100 MB! WTF? Haven't looked at that side of things lately, I wonder if it's different nowadays.

cube2222 5y ago
We have a big amount of resources available inside of our Spacelift provider[0] and it weights ~20 MB. It'll probably mostly depend on the libraries you use. [0]:https://github.com/spacelift-io/terraform-provider-spacelift/ https://github.com/spacelift-io/terraform-provider-spacelift...
tornato7 5y ago
Is it a Go problem or a new-to-Go problem? I haven't written terraform plugins specifically but I have been writing Go for years and never find myself needing to write an excessive amount of boilerplate. There can definitely be some frustrations in dealing with dynamic JSON though. JSON-to-Go converters are your friend.
- himinlomax 5y ago
  I was not using anything special, I had implemented my own client for IPA. Te equivalent functionality in Python (ended up using Ansible to do my thing) uses just a few kB ...
StopHammoTime 5y ago
Definitely agree with this, Go is so verbose for the application. When I wrote a provider, I had the same problem. What made it even more worse is that I was connecting into an API that made use of dynamic json generation. So many interfaces and other hacks to get the json documents to parse correctly.

eKIK 5y ago

+1 from me on the "awful half-baked language" (HCL). I just recently wrote an article about my experience, including issues and workarounds, when migrating from Terraform to Pulumi: https://blog.ekik.org/my-experience-migrating-my-infrastructure-from-terraform-to-pulumi https://blog.ekik.org/my-experience-migrating-my-infrastruct... Hope it's OK that I'm sharing it here. I think it's relevant because there seems to be quite a lot of interest around Pulumi, and how one would go about moving from Terraform to Pulumi.

mwcampbell 5y ago
I'm actually thinking of going the other way. I've been using Pulumi for several months now, and I'm thinking of moving to Terraform, because it has a so much larger third-party ecosystem, including more providers, and tools that can analyze HCL, like Infracost and security scanners. When will I learn to see the bigger picture and value popularity over quality?
- eKIK 5y ago
  It's a very interesting point. I've been part of managing rather large Terraform infrastructures (1000+ resources) for a couple of years, but I'm a Pulumi n00b with only about a month of experience. The infrastructure I'm managing right now with Pulumi is much smaller, only around 130-140 different resources. For me it ultimately came down to developer productivity. I'm much better at convincing Pulumi to do what I want compared to how it was with Terraform. This also makes me a much happier and less frustrated developer :). My priorities might very well be different if I were to manage much larger infrastructures (infra cost would be more important for example).
  - mwcampbell 5y ago
    The stack I manage with Pulumi is currently around 300 resources. (I think that count is inflated by all the secrets in AWS Secrets Manager, because each secret has two resources: the secret and the current version.) I currently manage it by myself, but I'm hoping that won't be the case for very long. Maybe the ending of my previous comment was too cynical. But I think I've repeatedly made the mistake of valuing my productivity and happiness as a currently solo developer over what will let my company take full advantage of a big third-party ecosystem (including a large talent pool).

time0ut 5y ago

Have you tried Terragrunt [0]? It helps a lot with managing a set of related stacks. Still feels like a bandaid on a broken model, but it is what we have. [0] https://terragrunt.gruntwork.io/ https://terragrunt.gruntwork.io/ Regarding performance, last time I looked, Hashicorp's documentation implied there was no limit to the size of a Terraform stack. I think they meant theoretically in a science fiction universe where humanity had captured all of the sun's output to perform terraform plan and apply...

tdumitrescu 5y ago

Yeah, these days I try to avoid writing any HCL and instead feed Terraform with JSON generated via jsonnet (which we were already using to generate k8s YAML). Much better templating and language features while still remaining declarative, and it helps on a team to have a single source language for such configs.

etxm 5y ago

> Also the staticness of providers are a serious pain, for example you can't create a kubernetes cluster then add a resource to it. TF def has some rough edges, but you can certainly create a cluster and add resources in a single root module (I don’t think it’s a great practice). In this example the EKS cluster is in a module, but it can be a ref to a resource in the same module as well. data "aws_eks_cluster_auth" "current" { name = module.eks.cluster_id } provider "kubernetes" { load_config_file = false host = module.eks.cluster_endpoint cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data) token = data.aws_eks_cluster_auth.current.token }

xorcist 5y ago

Why not use something like Ansible instead? It too is declarative. It too can be easily extended. It's also something a lot of people already know. I used to use Ansible or Puppet for these things before Terraform was all the rage. It was a lot more stable than trying to distributing those state files, which is a strange design to pick. There are plenty of existing modules but it's also dead simple to write your own.

akvadrako 5y ago
Ansible is not really made for managing cloud resources and it shows - the modules are not production ready.
- throwdbaaway 5y ago
  For GCP, both ansible modules and terraform modules are actually generated from https://github.com/GoogleCloudPlatform/magic-modules https://github.com/GoogleCloudPlatform/magic-modules, so their "production readiness" are the same. I understand that mitchellh himself personally created a bunch of cloud modules for terraform at the beginning, and those were likely of higher quality than whatever created by some internal developers assigned by Google/Microsoft, and might be slightly better than the AWS modules maintained by community. Anyway, when it comes to ansible versus terraform, we shall move the discourse to states management instead. With ansible, you don't have to deal with states, but will need to clean up the cloud resources separately. With terraform, you can use the tool to clean up the cloud resources easily, but then you also have the headache of managing states. Plus, whenever you change something, there is always the nagging feeling that it will do a destroy/recreate instead of an in-place update.
dtech 5y ago
I have limited experience with Ansible, but afaik calling it declarative when compared to Terraform is a stretch [1] [1] https://blog.gruntwork.io/why-we-use-terraform-and-not-chef-puppet-ansible-saltstack-or-cloudformation-7989dad2865c#f26f https://blog.gruntwork.io/why-we-use-terraform-and-not-chef-...

oxplot 5y ago

> for example you can't create a kubernetes cluster then add a resource to it Of course you can! DM me if you want details.

gouggoug 5y ago
Yes you can, see https://news.ycombinator.com/item?id=27435269 https://news.ycombinator.com/item?id=27435269
- hnjst 5y ago
  I can confirm, we're using a similar approach at it mostly works. There are still issues though, if you try to remove your cluster the k8s provider can't be configured (no module.my_cluster.cluster_id anymore) and the refresh phase of plan will fail. You can find workarounds but those I know are quite manual / ugly.

AcerbicZero 5y ago

I’ve had many similar frustrations about terraform, and the overall lack of visibility into what’s happening drives me mad at times. A proper repl, with the ability to actually manage a config would be a huge step forward - I spend more time trying to figure out what vars get populated and how I can get a value into another resource than anything else. It’s like I’m constantly fighting with the HCL syntax to get what I want to happen.

clipradiowallet 5y ago
If you want visibility(spoiler: it's just API calls), try using `TF_LOG=DEBUG terraform <foo>`. You might also want to set `-parallelism=1` or you'll be treated to statements printing in an order you are not expecting.

Nilithus 5y ago

> "Luckily" Terraform's performance is so bad that you need to split the stacks anyways Not sure what about terraforms performance is so bad. Seems hard to blame a tool who's main execution path is potentially 100's of network IO requests with 3rd party API's. Most of the "split stacks" I've seen is more for code organization and security reasons rather than performance. Seems safer to know 100% that deploying infra for my app isn't going to mess with my VPC settings and can be executed with a lower privileged role. > Furthermore you can no longer effectively plan any change that affects the boundary between the two stacks. That's fair -- you do end up with these "foundational" modules a lot of the time. Like an 'aws-account basics' module or something that other modules expect the account to be setup with that base for being able to query data objects for subnets ect... planning changes if that changes be difficult but not impossible. Good versioning is critical. Feels in the same vein as apps that need to manage framework updates and things like that. (though can be made more difficult or easier based on how you've broken up using your cloud provider -- multiple accounts by buisness unit or all in one).

jmccarthy 5y ago
Our experience of building a provider: performance is fast with fast APIs, and slow with slow APIs. Haven't observed any of the core diffing, DAG, or apply scheduling to be problematic (but also haven't tried an apply at extremely high - 10^4? 10^5? - resource count)

more_corn 5y ago

Same (hate it, love it, use it every day). I can't believe you left out the the stilted looping syntax.

kevincox 5y ago
I didn't figure it was worth starting... - Lack of functions (the only real functions are modules which are basically unusable for quick computations such as "slugify this string"). - Very primitive loops. - Lack of temporary variables. I often end up looping over a list multiple times and storing intermediates. - No panic or log functions.

clipradiowallet 5y ago

> The biggest feature I would like to see is the ability to dump a pure representation of your evaluated configuration. Are you asking for a dump of existing state or desired state? For existing state, see `terraform state pull`. For delta between desired+existing, see `terraform plan -out`. My apologies in advance if I completely misunderstood what you were asking for.

kevincox 5y ago
I am asking to dump the desired. So that I can diff the desired against the commit vs the desired of the last commit. I don't want to include production at all.