11 ms·

Terraform is not the golden hammer

147 points by kiyanwang 5y ago

pojzon 5y ago
I don’t have issue with terraform as it has a very clear defined usecases. I have issues with all the providers that make no sense like application configuration providers or all the flavors of kubectl providers.. Those are often very low quality and have various issues dedicated solutions don’t have. An example could be helm and helm_provider. The former just works, with the latter Im constantly running into weird bugs that break terraform state..
picardo 5y ago
Terraform can be frustratingly slow at times. You have to realize that at the end of the day, it's an abstraction layer on top of the public APIs of a cloud service. If all your services are hosted on a single cloud, you don't need Terraform. We saw a huge improvement in our build times after we started using AWS CDK directly.
- amarshall 5y ago
  What does the AWA CDK do differently than use public APIs?
  - scrollaway 5y ago
    CDK generates cloudformation stacks. Those stacks are deployed as units, within AWS itself. AWS treats all the resources as part of that stack etc; it's a concept entirely proper to AWS. Terraform can create cloudformation stacks as well, you just have to write the resources for it. It doesn't really make sense to do that. I also don't know that it's … "faster" in any way; cfn is really slow.
    - orf 5y ago
      It’s just tags on the resources and a managed statefile. There isn’t anything different between a bucket created via CDK and a bucket created via terraform, the resources are the same and the API calls to create them are also the same.
- kenerwin88 5y ago
  Hmm, as a former AWS employee who has used both heavily, my experience has been the opposite. Terraform’s AWS provider calls the APIs directly, whereas CDK generates Cloudformation, an abstraction on top of the AWS APIs. For me, using Terraform was significantly faster than applying the same stack via CDK. Or do you mean you’re able to iterate faster writing CDK vs TF?
  - picardo 5y ago
    Thinking back on it, we always used Terraform with Pulumi, which creates its own abstraction layer for a CF stack. It's hard to pinpoint where the root cause of the slowness was.. but in principle having fewer abstractions allowed us to iterate faster, and fix the bugs more quickly.
- qaq 5y ago
  CDK synth is reasonably fast but CF is really slow
- ulzeraj 5y ago
  I'd rather devote my time learning an agnostic tool like terraform. I can be part of your team right now working on AWS but tomorrow I might be working for an Azure shop.
- snom380 5y ago
  From my experience, terraform is almost always slow because it's making API calls out to the cloud providers, and a lot of that in turn is slow because many providers offer "eventually consistent" which terraform needs to compensate for by doing roundtrips to validate that changes are have become visible (applys failing because of that was a common problem in the early days of terraform).
NKosmatos 5y ago
Came here to read about planet terraforming but instead I learned about (yet another) cloud deployment tool :-)
- scrollaway 5y ago
  Ah, congrats on being part of today's ten thousand. I very highly recommend investigating it more and trying it a bit. Terraform isn't mere cloud deployment. As a small project you can start by deploying an EC2, RDS and some cloudflare records to go with them, all linked together with terraform. This will give you an initial idea of its capabilities.
- rad_gruchalski 5y ago
  > learned about (yet another) cloud deployment tool Please don't read it as an attack, not the intent. Amazing that a HN regular can "learn about Terraform" only in the latter part of 2021!
  - jhgb 5y ago
    I made the same mistake. First hearing about this. It turns out that different people may have different backgrounds! ;)
  - johannes1234321 5y ago
    https://xkcd.com/1053/ https://xkcd.com/1053/ I assume there is some selective reading and most articles referring to Terraform have a cloud or hashicorp reference in the title. If you don't care about either, you don't read the Terraform things on HN.
scrollaway 5y ago
It's very difficult to communicate what Terraform's strengths and weaknesses are to someone who's never used it or IAC in general. Spend enough time playing with it and understanding it, you'll end up like me thinking about all the shit you configure left and right such as hooking up Stripe's secret keys, Google Analytics and the webmaster console, and just about everything else we configure via web interfaces, and you'll think: Why can't we use Terraform for this as well? Manage these SaaS products the same way we manage the rest of our cloud, test and audit changes, automatically roll secrets and update anything that needs updating the moment you change a setting. Ah well. Not enough APIs out there. And it's difficult to write and maintain terraform plugins for these throwaway cases especially if they are going to use private APIs. Anyone know if Pulumi plugins are easier to write?
- satya71 5y ago
  Pulumi has dynamic providers. You define the crud operations and Pulumi manage the state.
  - reilly3000 5y ago
    After a little grokking it’s a surprisingly easy way to manage arbitrary API resources. They are just functions in your language of choice and can accept parameters from any dependency. It’s also possible to roll your own provider, but dynamic providers cover all kinds of use cases.
- deleted 5y ago
  [deleted]
- crabmusket 5y ago
  I've used https://github.com/Mastercard/terraform-provider-restapi https://github.com/Mastercard/terraform-provider-restapi successfully with a cloud provider which provides a suitable HTTP API. There was a bit of fiddling with JSON formatting and their API docs, but it wasn't too hard all in all. But like you say - now I've done that, I want to do it for every UI that I'm forced to log in to!
phendrenad2 5y ago
Terraform basically gives you what cloud providers should have. AWS/Azure are these overcomplicated web interfaces, or undocumented REST APIs, and Terraform gives you a simpler way to configure stuff.
kleinsch 5y ago
Surprising that the author is writing a tool for managing your servers, so writes a post about how Terraform isn't great at managing your servers... It seems like the root of the problem is that the author wants to use Terraform to manage their AWS state, but also wants to use the web console to directly change things, so Terraform gets out of sync. Terraform has a command to handle this - https://www.terraform.io/docs/cli/commands/refresh.html https://www.terraform.io/docs/cli/commands/refresh.html
- orf 5y ago
  There’s also a refresh flag on plan. It’s always worth using before your apply CI step.
- throwaway894345 5y ago
  As the sibling commenter notes, Terraform has a refresh flag, but I wonder if Kubernetes’ model is better here. Rather than a one-off process that tries to update everything, Kubernetes has many small controllers which are essentially processes on the cluster that just run a control loop. Each controller corresponds to one resource type, so it will just loop over incoming events that pertain to the resource type in question and attempt to reconcile the state of the target resource instance with the desired state. If it fails initially, it will retry with back off. If something doesn’t stabilize after several minutes, an alert can notify a human. The key differences between the controller approach and the IaC approach are, I think, lots of little processes continuously reconciling state for all resources of a given type (many small loops that touch all resources of a given type on the entire cluster) versus a one-off process that tries to touch just the resources it cares about and if it fails it just gives up. One thing Kubernetes definitely improves upon Terraform is that Kubernetes uses a YAML “assembly language” for its infra as code, but that YAML could be generated by a real programming language. Terraform expects you to write HCL, which is an accidentally re-invented programming language (every IaC tool provider thought static configs like YAML would suffice as a human interface, but as they gradually realized the need for more dynamism, Terraform and others would bolt on one dynamic feature after another until they had a slow, unfamiliar, and counterintuitive programming language). Terraform has a CDK that allows writing in other languages, but I’m skeptical that it liberated you from Terraform’s model of the world (e.g., if I rename a variable in CDK, does it try to destroy and recreate the underlying resource as with Terraform?). I’m also concerned that rather than allowing us to generate YAML in the obvious way, it will require bizarre inheritance patterns like the AWS CDK. I would be curious to hear from folks who have used the CDK.
marcinzm 5y ago
We use Terraform for cloud infrastructure and, basically, helm deploys of external apps. In other words things that don't change too often and the update of which has to be managed carefully anyway. The internal apps which get deployed at a much faster cadence use helm directly.
mvanaltvorst 5y ago
I wish Terraform were less opinionated. It has a very clear set of rules you have to adhere to, and if you try to do anything remotely complex you will encounter barriers left and right. An example is the fact that `for_each` is not supported on providers [1], an issue with 230 likes which has not been solved since January 2019. This had me resort to a Python script which generates a `.tf.json` file, definitely not ideal. Infrastructure as code sounds great, but in practice it's closer to "infrastructure as a non-standard markup language". [1]: https://github.com/hashicorp/terraform/issues/19932 https://github.com/hashicorp/terraform/issues/19932
- kevincox 5y ago
  I think this is less that it is opinionated but more that the HCL evaluation feels like a pile of hacks. There are unclear rules on what can be evaluated when and what dependencies are possible. Part of this is so that `plan` can work as it does, but it seems like there are just major gaps in general. For example providers can't depend on resources. This makes it very difficult to for example set up EKS then use the kubernetes provider to manage the resources in the cluster. The solution is obviously separate stacks but that brings in a whole bunch of other problems. I think Terraform is quite possibly the best tool available, but there are clear flaws with both the model and the implementation. I think if I were to make a Terraform v2 I would make `plan` completely pure. This would avoid the provider issues, make validation and testing in CI easier and a whole bunch of other benefits. Of course there are downsides. For example EC2 instance IDs are random so you can't just include them in your pure plan. You would need some type of placeholder that is used for evaluation. This does cause some issues as it limits the operations that you can do with that value (so you can't pick the instance size based on the random instance ID) but overall I don't think it would be a major issue if the final substitution was handled well by the framework.
- throwaway894345 5y ago
  You have to understand that when IaC was new, the marketing was “it’s so simple you can just write YAML/JSON/etc” because frankly the industry was too dumb to understand that “using a real programming language to generate a description of the desired resource state” and “using a real programming language to imperatively reconcile the current and desired states oneself” are different things. So Terraform began with something that resembled YAML in its static-ness, and over time, more power was required so they would bolt on a dynamic feature but were reluctant to give the impression that they were building a programming language so the feature would be as obscure as possible. But that wouldn’t be enough either so they would add still more dynamic features, each comparably obscure until in time they’d built a complete, obscure programming language. But this wasn’t just Terraform! The entire industry did this too. CloudFormation began as simple JSON, but over time they allowed you to encode the abstract syntax tree of a shitty programming language in your YAML, and CloudFormation would interpret it. However stupid that may sound, in the Kubernetes world, we have Helm which lets you generate YAML with text templates which is honestly the dumbest idea in the world (imagine a compiler that generates syntactically invalid machine code if the input program has an extra white space character). Of course in all of these cases the answer is staring us in the face: use a static language (YAML, JSON, etc) to describe the desired state, and use a higher level language (like Python or Starlark or Dhall or etc) to generate that static desired state description. The only thing Terraform (or any IaC tool) should care about is the YAML description. That it is generated from Starlark or TypeScript is just an implementation detail. Instead of that, though, we get CDKs which are so close, but admittedly I haven’t used them in anger yet.
wayoutthere 5y ago
The new use case for Terraform is IT departments using it as an IaC policy control system for self-service. Rather than push teams through a web interface, you just expose Terraform via Terragrunt to the dev teams and run their files through a policy-driven linter before executing. And make it so that Terraform is the only way you can push to prod. I think people get into trouble with Terraform when they try to use it to do more than provision infrastructure. Things that should probably be part of build scripts, CI/CD pipelines or config management. Terraform isn’t good at those things; but it is very good at provisioning cloud infra in a cloud-agnostic fashion.
- pvtmert 5y ago
  I agree most of your comment but the cloud-agnostic part. Heck, I never get how terraform can be cloud-agnostic... If everybody thinks having a same language (HCL) is equalivent to cloud agnostic, YAML exists... It is literrally impossible to create a simple VM in 2 different cloud providers without defining them twice with their own specific parameters. If you use AWS provider, resources start with aws_, if it's GCP it starts with gcp_ and so on. It is not possible to have a "resource vm { name = ... provider = aws }"
  - jen20 5y ago
    Terraform provides the same _workflow_ across clouds, not the same resource model (which would be dumb, since it would necessarily provide only a lowest common denominator representation). > It is not possible to have a "resource vm { name = ... provider = aws }" It actually is via modules. It’s a lot of work with basically no benefit though, so in practice people don’t do this. Pulumi is better at specifically this kind of thing however, since you can implement a common interface which can be specialised for each available cloud.
jdub 5y ago
My biggest issue with Terraform is the impedance mismatch between HCL and the rest of the known universe. When you write a provider, you spend half your time converting data structures from the HCL submitted to your provider into the JSON your target service inevitably expects, then you spend half your time converting the JSON your target service inevitably returns into HCL for Terraform to consume, and then you spend another half of your time fixing bugs and polishing. It's okay when you're building simple providers, but anything reasonably complex becomes unwieldy. I had a go at building some providers for AWS services that were not supported by Terraform or CloudFormation... and I just retreated to cheesy Lambda custom resources for CloudFormation.
- gtirloni 5y ago
  The upside though is that you're adapting that known universe to a way of working that makes sense for terraform users.
  - jdub 5y ago
    But I wanted to be a Terraform user. So an alternative interpretation is that, as designed, Terraform is slowing down adaptation of the known universe for Terraform users.
  - throwaway894345 5y ago
    Unfortunately HCL is also an unnecessary learning curve for users. I still don’t have my head around a list versus a bunch of blocks with the same name, for example. I originally thought it was syntax sugar for a single mechanism, but I’ve had errors for trying to use one instead of the other before.
  - ClumsyPilot 5y ago
    Good tools are fit for the world we live in, bad tools require you to "adapt the the known universe" to the tool
- kubanczyk 5y ago
  I read you. The half of the Terraform's value today is "let's have common aesthetics - especially the examples and snake_case_naming_convention - and convert any REST API in the world to that". The typical example is to start from there: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_Vpc.html https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_V... and arrive here: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc https://registry.terraform.io/providers/hashicorp/aws/latest... If you review it carefully, it is apparent how much coding effort and many moving parts were used to perform a transformation which seems disproportionately primitive.
dharmab 5y ago
An alternative opinion: When I worked at a large tech company, my team made a conscious decision to not use terraform. This gave us some key advantages- we are able to adopt new cloud features immediately, months before they were available in tf, and our direct cloud access let us build features that would surprise the teams using tf within the company. If your core competency isn't dependent on your cloud platform tf is a great tool. But using cloud APIs directly was great for us.
- CSDude 5y ago
  What new cloud features was not available in Terraform for months?
  - pvtmert 5y ago
    eg. AWS ChatBot not available in TF yet. TBH AWS haven't even added it to their Go SDK. So, I cannot blame TF. But anyway that's one of the inheret problems of TF plugin system. Compare to kubectl. Where you can write plugins in bash/shell and mark with execute bit, put it in somewhere in your $PATH as kubectl-blabla and use it as "kubectl blabla".
    - CSDude 5y ago
      It's not fair to compare imperative simple shell scripts with the things Terraform does. It has schema validation, state comparison, retries, failure handlers etc. Also, just as you can write extensions to kubectl, you can write your own provider in Terraform if it does not exists. See https://registry.terraform.io/modules/waveaccounting/chatbot-slack-configuration/aws/latest https://registry.terraform.io/modules/waveaccounting/chatbot... Also, Chatbot does not have a public API, that's why, it's only configured via Cloudformation. So the expectation is not fair either. I've seen Cloudformation getting features years later. i.e 2021 - https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-dynamodb-global-tables-now-support-aws-cloudformation/ https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-dy... 2015 - https://aws.amazon.com/about-aws/whats-new/2015/07/amazon-dynamodb-available-now-cross-region-replication-triggers-and-streams/ https://aws.amazon.com/about-aws/whats-new/2015/07/amazon-dy...
bovermyer 5y ago
From the problems the author is having, it would appear that perhaps Pulumi would be the better choice in this case.
- rgoulter 5y ago
  In the article, the problems the author discusses are: 1. Inconsistent behaviour between providers. e.g. if a resource has been destroyed since the last `terraform apply`, then some resources/providers would recreate the resource, others wouldn't. (Similarly, there's not a guarantee that the state after running `terraform apply` matches up with what's there, if the provider is happy with its state file). 2. The dependencies of already-applied resources can block `terraform apply` if the upstream API for these resources suffers an outage. 3. If a `terraform apply` applies some resources before failing, this can result in an inconsistent state. Either the resources need to be deleted, or imported. I'm not familiar with Pulumi; what aspects of these would Pulumi help with?
  - kall 5y ago
    3 happens regularly and I don‘t see how the other two would really be different, since some pulumi providers are using terraform providers under the hood.
gjhr 5y ago
> When you run Terraform against AWS on the subnets part, it will create (anytime you deploy) the missing subnets That is one of the core features of Terraform? Detecting and fixing drift is useful.
poisonta 5y ago
I had mostly similar feelings four years ago. As I understood the reasons behind them, I started respecting more the people at HashiCorp. They are really smart.
cube2222 5y ago
Terraform definitely has it's warts, though, as other commenters wrote, not everything in the article is true (the reconciliation part): dependency resolution blows up in time as your number of resources grows, so you need to split up your statefiles; it can't passively listen for drift happening in a dataflow-like way (that would be awesome); it's not transactional like CloudFormation (which is more of a tradeoff, than a cons), and more. It is however a great improvement over the previous ways of doing things, and probably the best out of the current similar alternatives out there (you might mention Pulumi as a strong contender, especially for AWS glue writing). And - though as per the disclaimer, I may be biased - until a better tool comes up, I'd advise looking for specialized IaC CI/CD tools to ease your path with Terraform, like Spacelift[0]. It can help you with orchestrating dependencies among multiple state files; take care of scheduling regular drift detection/reconciliation without going into your way and locking your state; gives you a policy system for making sure preventable mistakes don't happen (i.e. recreating a resource you definitely never want to recreate); manages your credentials depending on whether you just want to run a plan, or apply your changes, and much more. I can't imagine doing Terraform again without a tool like it. Disclaimer: Software Engineer at Spacelift. If you want me to expand on the "and much more" part, you can find a demo-scheduling link in my bio! [0]: https://spacelift.io https://spacelift.io
p2t2p 5y ago
My experience is that terraform sucks. A lot. Yet everything else seem to suck even more
d0gsg0w00f 5y ago
IME, Terraform is great for "fire and forget" deployments. However, if you're trying to use it to continously bring older existing deployed infra up to date it can get tricky. I strongly suggest self versioning your TF files and following strict pretested upgrade paths.
- nanis 5y ago
  > Terraform is great for "fire and forget" deployments. Indeed, terraform is great for exactly the opposite: For making sure both the initial infrastructure and subsequent modifications to it are first proposed, discussed, reviewed in code and applied afterwards instead of anyone with privileges tinkering with settings on an "as needed" basis in whatever console thereby ending up with an infrastructure where you have no idea who turned on the frobnicator, why it was set to 11, and what might be the consequences of changing the setting.
  - d0gsg0w00f 5y ago
    We experienced a lot of problems with when trying to manage standard configurations of hundreds of AWS VPC's ranging in age from 3 days old to 6 years. Accounts built years ago using older terraform would have to be handled completely differently because inevitable TF template drift and TF versions made each upgrade path unique. Not insurmountable by any means but also not trivial. Just sharing my experience.
- robertlagrant 5y ago
  If you just use a cloud provider's UI you will have no separation between desired and actual states. Then, whatever people change it to is the only truth.
digianarchist 5y ago
You can run terraform apply against a particular resource which will only provision that resource and its dependencies. tfstate files can be painful to manage, we had a lot of trouble with them at Capital One but mostly because: 1. People would modify state outside of TF which you should avoid. 2. People didn't architect their apps well which led to long lived infra. TF works best with cattle like infra. terraform import feels very much like an afterthought which is why projects like terraformer exist.
snom380 5y ago
> On terraform it's different, because of the tfstate. All the deployed elements are stored in the tfstate, re-running terraform won't update resources that are supposed to be in a specific state but are not. This is incorrect, and makes me wonder how the author has used terraform. Terraform will certainly detect differences between managed and current state for the resources it manages whenever you do a plan/apply. The major challenge is that terraform _can only reconcile resources or configuration values it knows about_, and that depends very much on how a particular cloud vendor or terraform provider has modelled resources. I believe the Helm provider is one example where it (at least in the past) haven't had a good way to reconcile state.
- deleted 5y ago
  [deleted]
- dead10ck 5y ago
  This stuck out to me too. Terraform absolutely does check the current reality of the state and applies changes to do what the HCL tells is to. They are either using terraform in a really weird way, or this article was written by someone that doesn't actually run terraform themselves.
  - jrochkind1 5y ago
    Phew! I thought I was misunderstanding terraform when I saw that. Perhaps they are working with certain poorly implemented or buggy providers, and not realizing those providers were doing something different than terraform's properly working behavior that most providers implemented. Bugs happen, but the first step is agreeing on intended behavior so we know what's a bug!
- shadycuz 5y ago
  I also agree, but I don't have experience with the providers he is using. When working with AWS it will always reapply the terraform configuration. A great use of this is for account hardening. You can run it daily to make sure it's configured correctly.
jrochkind1 5y ago
> On terraform it's different, because of the tfstate. All the deployed elements are stored in the tfstate, re-running terraform won't update resources that are supposed to be in a specific state but are not. Huh, is that true? I'm just getting started with terraform but I assumed that was the idea of terraform (where it didn't happen would be a bug), and I think I had seen it happening for the few basic resources I have started out with (S3, cloudfront). If the state doesn't match the actual configuration of S3, terraform notices, and the plan is to make it so. No? Am I confused and it hasn't been doing this? Or is this is inconsistent, true of some resources and not of others? That seems surprising. What's the idea?
- shatteredspace 5y ago
  If Terraform was used for the deployment of the infrastructure, then state IS the actual configuration of the system. All that a plan does is evaluate what is going to change in the current Terraform state by performing a dry-run of the Terraform code that you have supplied. If you would actually like to make changes to the Terraform state based on what the Terraform code evaluated then you run a Terraform apply - which will, for the resources deployed via Terraform, update the configurations themselves and update the Terraform state by using the Terraform code as the instruction set. You can actually see this in action with plan and apply as the output will show you +,-, and ~ where ~ is settings that are going to change but are not new configurations or configuration to be removed. Edit: Learned from some other comments that Terraform has a 'refresh' command that will take deploy+n time configurations done outside of Terraform and sync those configurations with the state. This might be what you ideally are looking for after deployments?
  - jrochkind1 5y ago
    Right. I guess I'm asking about what happens if state changed outside of terraform. I thought I had seen terraform correcting it (to match what terraform thinks it should be) in some cases. OP seems to suggest that in some cases it does and in other cases it doens't. I am surprised if that is inconsistent and unpredictable, and would have expected terraform to (modulo bugs) either always or never do that. And am wondering what terraform's intent is with that.
3np 5y ago
The author seems to have some misunderstandings on how Terraform is supposed to work - you should get the "automatic reconciliation" they're saying is missing. Also, > I run once again the “terraform apply” command. But for some reason, Cloudflare API doesn't answer and I got completely stuck there without the possibility to update with Terraform this field because of linked dependencies. You should be able to circumvent this with a `-target`. That being said, I know exactly what they're talking about with helm. IME the helm provider was/is a complete mess and gets inconsistent state a lot. Helm specifically I would also keep out of TF until that is fixed, if ever. I haven't had that happen with other providers, though. Perhaps OP was just really unlucky ending up with the odd half-broken AWS module.
danielovichdk 5y ago
I run only on Azure. Nearly all my IAC is written in PowerShell uilizing the Azure Cli. Terraform and yaml as well are so verbose and you have no clue whats going on from the local side of things. How do debug your terraform markup locally. My guess is you can't.
- awslattery 5y ago
  The console command [1], mixed with local variables, is quite handy for debugging locally. 1. https://www.terraform.io/docs/cli/commands/console.html https://www.terraform.io/docs/cli/commands/console.html
  - x86_64Ubuntu 5y ago
    Seems as if people have very strong feelings about Terraform, but little actual experience using it.
    - toyg 5y ago
      It's not the easiest tool to grok, writing configuration files can be very verbose, and its opinions about a number of things can be very off-putting. I tried to "get into" TF many times and never actually fell in love. I like the idea, I like the cross-cloud approach, I just don't particularly like what the experience ends up being. This shit should be easier.
    - httgp 5y ago
      Bingo. While I see some valid complaints about Terraform in the comments, most of them seem to come from a place of not fully understanding how it works or the features it offers.
nojvek 5y ago
Yeah the tfstate was a big gotcha. Pulumi has the same problems. What we really need is automatic reconciliation. I.e ask the provider what they have and then diff against that. Or periodically auto-importing. Are there any good solutions to auto-importing?
- arpinum 5y ago
  The hard part with auto-importing is the resource id. This is usually generated server-side and is not included in the hcl. Often resources define a user-supplied identifier as well, such as a name property, and this could be used for auto-importing if the property has unique constraints applied against it. However, not all resources have this feature, so its not a universal solution.
- kall 5y ago
  Hm, maybe what we really need is a new entrant in hyperscale clouds that is built from the ground up for IaC and just does away with the split between state and reality. I would love to see one, anyway.
- robertlagrant 5y ago
  What's the point of reimporting? Aren't you then no longer separating desired state from actual?
airocker 5y ago
We use this exact stack, but we generally would not rely on tfstate. We will remove everything and regenerate it. It’s not an operation done too frequently that we have a big problem. Also, helm we use as a separate layer that is applied after terraform and can be repeated many times. This changes often.
jounker 5y ago
What countries have fewer deaths per 100K than Sweden? All of Sweden’s neighbors.
- duskwuff 5y ago
  Wrong thread?
nrvn 5y ago
One of the biggest reasons that have kept me away from terraform apart from the esoteric language is that terraform modules are always a few steps being from the upstream public cloud offerings. In the sense that whenever there’s a new API or service available in any of public clouds and their official SDKs there will always be a delay before this new service/feature/API will become available in terraform. First time I encountered it with GKE private clusters 3 or 4 years ago. Now it is AWS Keyspaces. The second biggest reason is whenever you have a requirement for a hybrid or multicloud then well you are left with rigidity if HCL. It is probably doable but for what sake? Solution: get a real language, write a STATELESS configuration management(IaC) system for your own needs and maintain it. The majority of public and private cloud providers ship SDKs in most popular languages that will help you build your own software solution and reduce your dependence on a third party which I would put under progressing operational risk category. Yaml/json/cue/toml for end user configs would suffice. Example: for one of my previous projects were built a tool for a hybrid AWS-openstack setup, and were managing a dozen of busy environments.
- nrvn 5y ago
  The emphasis on stateless here is that your desired state us described in code that resides in your repository. Actual state is what you have in your cloud. No need to spend time on state format, storage and related logic and complexity
- xyzzy_plugh 5y ago
  This is my preference as well. I've done everything from makefiles and bash scripts to a monolith Go program that statelessly provisions/tears down resources. Even makefiles are pretty straightforward, though you really want operations to only trigger when checksums differ -- timestamps result in a lot of redundant operations. As long as everything is idempotent, it's pretty straightforward.
- oceanplexian 5y ago
  > Terraform modules are always a few steps being from the upstream public cloud offerings. My experience has been the exact opposite. Usually Terraform offers support for cloud services long before the vendor provides an SDK or supports it with their own offering (e.g. Cloudformation). There are still dozens of AWS services, for example that have no CF support offered by AWS.
0xbadcafebee 5y ago
Terraform is actually kind of a nightmare. It's deceptively simple yet requires a massive amount of real world expertise to use it properly. It's a configuration management tools, but more difficult to use and extend. I'm thinking of designing a series of tools to replace Terraform. The idea would be to break down how modern cloud environments are managed into a couple concepts, and then make a variety of tools that work within those concepts together, so that it's easy to expand and modify the way you use them for your use case. This would enable things like tailoring the use of the tools to a particular deployment strategy, or adding custom business logic, or replacing individual functionality, without being tied to one tool, language, etc.
danw1979 5y ago
This article is not entirely incorrect, but there’s some glaring falsehoods as others have pointed out. There’s a plug for managing Helm related resources with the author’s own SaaS product at the end, so I’ll file this under “half hearted hit job”.
gattacamovie 5y ago
If you have kuternetes in the stack, I consider CNCF's https://www.crossplane.io https://www.crossplane.io the best path. For the cases where the native integration with the cloud provider does not provide the exact parameters you look for, it provides the alternative to integrate with terraform in the background, while making it transparrent to you.
htrp 5y ago
It's not the golden hammer, but it is a hammer in a world full of nails.

147 points by kiyanwang 5y ago

I don’t have issue with terraform as it has a very clear defined usecases. I have issues with all the providers that make no sense like application configuration providers or all the flavors of kubectl providers.. Those are often very low quality and have various issues dedicated solutions don’t have. An example could be helm and helm_provider. The former just works, with the latter Im constantly running into weird bugs that break terraform state..

picardo 5y ago

Terraform can be frustratingly slow at times. You have to realize that at the end of the day, it's an abstraction layer on top of the public APIs of a cloud service. If all your services are hosted on a single cloud, you don't need Terraform. We saw a huge improvement in our build times after we started using AWS CDK directly.

amarshall 5y ago
What does the AWA CDK do differently than use public APIs?
- scrollaway 5y ago
  CDK generates cloudformation stacks. Those stacks are deployed as units, within AWS itself. AWS treats all the resources as part of that stack etc; it's a concept entirely proper to AWS. Terraform can create cloudformation stacks as well, you just have to write the resources for it. It doesn't really make sense to do that. I also don't know that it's … "faster" in any way; cfn is really slow.
  - orf 5y ago
    It’s just tags on the resources and a managed statefile. There isn’t anything different between a bucket created via CDK and a bucket created via terraform, the resources are the same and the API calls to create them are also the same.
kenerwin88 5y ago
Hmm, as a former AWS employee who has used both heavily, my experience has been the opposite. Terraform’s AWS provider calls the APIs directly, whereas CDK generates Cloudformation, an abstraction on top of the AWS APIs. For me, using Terraform was significantly faster than applying the same stack via CDK. Or do you mean you’re able to iterate faster writing CDK vs TF?
- picardo 5y ago
  Thinking back on it, we always used Terraform with Pulumi, which creates its own abstraction layer for a CF stack. It's hard to pinpoint where the root cause of the slowness was.. but in principle having fewer abstractions allowed us to iterate faster, and fix the bugs more quickly.
qaq 5y ago
CDK synth is reasonably fast but CF is really slow
ulzeraj 5y ago
I'd rather devote my time learning an agnostic tool like terraform. I can be part of your team right now working on AWS but tomorrow I might be working for an Azure shop.
snom380 5y ago
From my experience, terraform is almost always slow because it's making API calls out to the cloud providers, and a lot of that in turn is slow because many providers offer "eventually consistent" which terraform needs to compensate for by doing roundtrips to validate that changes are have become visible (applys failing because of that was a common problem in the early days of terraform).

NKosmatos 5y ago

Came here to read about planet terraforming but instead I learned about (yet another) cloud deployment tool :-)

scrollaway 5y ago
Ah, congrats on being part of today's ten thousand. I very highly recommend investigating it more and trying it a bit. Terraform isn't mere cloud deployment. As a small project you can start by deploying an EC2, RDS and some cloudflare records to go with them, all linked together with terraform. This will give you an initial idea of its capabilities.
rad_gruchalski 5y ago
> learned about (yet another) cloud deployment tool Please don't read it as an attack, not the intent. Amazing that a HN regular can "learn about Terraform" only in the latter part of 2021!
- jhgb 5y ago
  I made the same mistake. First hearing about this. It turns out that different people may have different backgrounds! ;)
- johannes1234321 5y ago
  https://xkcd.com/1053/ https://xkcd.com/1053/ I assume there is some selective reading and most articles referring to Terraform have a cloud or hashicorp reference in the title. If you don't care about either, you don't read the Terraform things on HN.

scrollaway 5y ago

It's very difficult to communicate what Terraform's strengths and weaknesses are to someone who's never used it or IAC in general. Spend enough time playing with it and understanding it, you'll end up like me thinking about all the shit you configure left and right such as hooking up Stripe's secret keys, Google Analytics and the webmaster console, and just about everything else we configure via web interfaces, and you'll think: Why can't we use Terraform for this as well? Manage these SaaS products the same way we manage the rest of our cloud, test and audit changes, automatically roll secrets and update anything that needs updating the moment you change a setting. Ah well. Not enough APIs out there. And it's difficult to write and maintain terraform plugins for these throwaway cases especially if they are going to use private APIs. Anyone know if Pulumi plugins are easier to write?

satya71 5y ago
Pulumi has dynamic providers. You define the crud operations and Pulumi manage the state.
- reilly3000 5y ago
  After a little grokking it’s a surprisingly easy way to manage arbitrary API resources. They are just functions in your language of choice and can accept parameters from any dependency. It’s also possible to roll your own provider, but dynamic providers cover all kinds of use cases.
deleted 5y ago
[deleted]
crabmusket 5y ago
I've used https://github.com/Mastercard/terraform-provider-restapi https://github.com/Mastercard/terraform-provider-restapi successfully with a cloud provider which provides a suitable HTTP API. There was a bit of fiddling with JSON formatting and their API docs, but it wasn't too hard all in all. But like you say - now I've done that, I want to do it for every UI that I'm forced to log in to!

phendrenad2 5y ago

Terraform basically gives you what cloud providers should have. AWS/Azure are these overcomplicated web interfaces, or undocumented REST APIs, and Terraform gives you a simpler way to configure stuff.

kleinsch 5y ago

Surprising that the author is writing a tool for managing your servers, so writes a post about how Terraform isn't great at managing your servers... It seems like the root of the problem is that the author wants to use Terraform to manage their AWS state, but also wants to use the web console to directly change things, so Terraform gets out of sync. Terraform has a command to handle this - https://www.terraform.io/docs/cli/commands/refresh.html https://www.terraform.io/docs/cli/commands/refresh.html

orf 5y ago
There’s also a refresh flag on plan. It’s always worth using before your apply CI step.
throwaway894345 5y ago
As the sibling commenter notes, Terraform has a refresh flag, but I wonder if Kubernetes’ model is better here. Rather than a one-off process that tries to update everything, Kubernetes has many small controllers which are essentially processes on the cluster that just run a control loop. Each controller corresponds to one resource type, so it will just loop over incoming events that pertain to the resource type in question and attempt to reconcile the state of the target resource instance with the desired state. If it fails initially, it will retry with back off. If something doesn’t stabilize after several minutes, an alert can notify a human. The key differences between the controller approach and the IaC approach are, I think, lots of little processes continuously reconciling state for all resources of a given type (many small loops that touch all resources of a given type on the entire cluster) versus a one-off process that tries to touch just the resources it cares about and if it fails it just gives up. One thing Kubernetes definitely improves upon Terraform is that Kubernetes uses a YAML “assembly language” for its infra as code, but that YAML could be generated by a real programming language. Terraform expects you to write HCL, which is an accidentally re-invented programming language (every IaC tool provider thought static configs like YAML would suffice as a human interface, but as they gradually realized the need for more dynamism, Terraform and others would bolt on one dynamic feature after another until they had a slow, unfamiliar, and counterintuitive programming language). Terraform has a CDK that allows writing in other languages, but I’m skeptical that it liberated you from Terraform’s model of the world (e.g., if I rename a variable in CDK, does it try to destroy and recreate the underlying resource as with Terraform?). I’m also concerned that rather than allowing us to generate YAML in the obvious way, it will require bizarre inheritance patterns like the AWS CDK. I would be curious to hear from folks who have used the CDK.

marcinzm 5y ago

We use Terraform for cloud infrastructure and, basically, helm deploys of external apps. In other words things that don't change too often and the update of which has to be managed carefully anyway. The internal apps which get deployed at a much faster cadence use helm directly.

mvanaltvorst 5y ago

I wish Terraform were less opinionated. It has a very clear set of rules you have to adhere to, and if you try to do anything remotely complex you will encounter barriers left and right. An example is the fact that `for_each` is not supported on providers [1], an issue with 230 likes which has not been solved since January 2019. This had me resort to a Python script which generates a `.tf.json` file, definitely not ideal. Infrastructure as code sounds great, but in practice it's closer to "infrastructure as a non-standard markup language". [1]: https://github.com/hashicorp/terraform/issues/19932 https://github.com/hashicorp/terraform/issues/19932

kevincox 5y ago
I think this is less that it is opinionated but more that the HCL evaluation feels like a pile of hacks. There are unclear rules on what can be evaluated when and what dependencies are possible. Part of this is so that `plan` can work as it does, but it seems like there are just major gaps in general. For example providers can't depend on resources. This makes it very difficult to for example set up EKS then use the kubernetes provider to manage the resources in the cluster. The solution is obviously separate stacks but that brings in a whole bunch of other problems. I think Terraform is quite possibly the best tool available, but there are clear flaws with both the model and the implementation. I think if I were to make a Terraform v2 I would make `plan` completely pure. This would avoid the provider issues, make validation and testing in CI easier and a whole bunch of other benefits. Of course there are downsides. For example EC2 instance IDs are random so you can't just include them in your pure plan. You would need some type of placeholder that is used for evaluation. This does cause some issues as it limits the operations that you can do with that value (so you can't pick the instance size based on the random instance ID) but overall I don't think it would be a major issue if the final substitution was handled well by the framework.
throwaway894345 5y ago
You have to understand that when IaC was new, the marketing was “it’s so simple you can just write YAML/JSON/etc” because frankly the industry was too dumb to understand that “using a real programming language to generate a description of the desired resource state” and “using a real programming language to imperatively reconcile the current and desired states oneself” are different things. So Terraform began with something that resembled YAML in its static-ness, and over time, more power was required so they would bolt on a dynamic feature but were reluctant to give the impression that they were building a programming language so the feature would be as obscure as possible. But that wouldn’t be enough either so they would add still more dynamic features, each comparably obscure until in time they’d built a complete, obscure programming language. But this wasn’t just Terraform! The entire industry did this too. CloudFormation began as simple JSON, but over time they allowed you to encode the abstract syntax tree of a shitty programming language in your YAML, and CloudFormation would interpret it. However stupid that may sound, in the Kubernetes world, we have Helm which lets you generate YAML with text templates which is honestly the dumbest idea in the world (imagine a compiler that generates syntactically invalid machine code if the input program has an extra white space character). Of course in all of these cases the answer is staring us in the face: use a static language (YAML, JSON, etc) to describe the desired state, and use a higher level language (like Python or Starlark or Dhall or etc) to generate that static desired state description. The only thing Terraform (or any IaC tool) should care about is the YAML description. That it is generated from Starlark or TypeScript is just an implementation detail. Instead of that, though, we get CDKs which are so close, but admittedly I haven’t used them in anger yet.

wayoutthere 5y ago

The new use case for Terraform is IT departments using it as an IaC policy control system for self-service. Rather than push teams through a web interface, you just expose Terraform via Terragrunt to the dev teams and run their files through a policy-driven linter before executing. And make it so that Terraform is the only way you can push to prod. I think people get into trouble with Terraform when they try to use it to do more than provision infrastructure. Things that should probably be part of build scripts, CI/CD pipelines or config management. Terraform isn’t good at those things; but it is very good at provisioning cloud infra in a cloud-agnostic fashion.

pvtmert 5y ago
I agree most of your comment but the cloud-agnostic part. Heck, I never get how terraform can be cloud-agnostic... If everybody thinks having a same language (HCL) is equalivent to cloud agnostic, YAML exists... It is literrally impossible to create a simple VM in 2 different cloud providers without defining them twice with their own specific parameters. If you use AWS provider, resources start with aws_, if it's GCP it starts with gcp_ and so on. It is not possible to have a "resource vm { name = ... provider = aws }"
- jen20 5y ago
  Terraform provides the same _workflow_ across clouds, not the same resource model (which would be dumb, since it would necessarily provide only a lowest common denominator representation). > It is not possible to have a "resource vm { name = ... provider = aws }" It actually is via modules. It’s a lot of work with basically no benefit though, so in practice people don’t do this. Pulumi is better at specifically this kind of thing however, since you can implement a common interface which can be specialised for each available cloud.

jdub 5y ago

My biggest issue with Terraform is the impedance mismatch between HCL and the rest of the known universe. When you write a provider, you spend half your time converting data structures from the HCL submitted to your provider into the JSON your target service inevitably expects, then you spend half your time converting the JSON your target service inevitably returns into HCL for Terraform to consume, and then you spend another half of your time fixing bugs and polishing. It's okay when you're building simple providers, but anything reasonably complex becomes unwieldy. I had a go at building some providers for AWS services that were not supported by Terraform or CloudFormation... and I just retreated to cheesy Lambda custom resources for CloudFormation.

gtirloni 5y ago
The upside though is that you're adapting that known universe to a way of working that makes sense for terraform users.
- jdub 5y ago
  But I wanted to be a Terraform user. So an alternative interpretation is that, as designed, Terraform is slowing down adaptation of the known universe for Terraform users.
- throwaway894345 5y ago
  Unfortunately HCL is also an unnecessary learning curve for users. I still don’t have my head around a list versus a bunch of blocks with the same name, for example. I originally thought it was syntax sugar for a single mechanism, but I’ve had errors for trying to use one instead of the other before.
- ClumsyPilot 5y ago
  Good tools are fit for the world we live in, bad tools require you to "adapt the the known universe" to the tool
kubanczyk 5y ago
I read you. The half of the Terraform's value today is "let's have common aesthetics - especially the examples and snake_case_naming_convention - and convert any REST API in the world to that". The typical example is to start from there: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_Vpc.html https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_V... and arrive here: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc https://registry.terraform.io/providers/hashicorp/aws/latest... If you review it carefully, it is apparent how much coding effort and many moving parts were used to perform a transformation which seems disproportionately primitive.

dharmab 5y ago

An alternative opinion: When I worked at a large tech company, my team made a conscious decision to not use terraform. This gave us some key advantages- we are able to adopt new cloud features immediately, months before they were available in tf, and our direct cloud access let us build features that would surprise the teams using tf within the company. If your core competency isn't dependent on your cloud platform tf is a great tool. But using cloud APIs directly was great for us.

CSDude 5y ago
What new cloud features was not available in Terraform for months?
- pvtmert 5y ago
  eg. AWS ChatBot not available in TF yet. TBH AWS haven't even added it to their Go SDK. So, I cannot blame TF. But anyway that's one of the inheret problems of TF plugin system. Compare to kubectl. Where you can write plugins in bash/shell and mark with execute bit, put it in somewhere in your $PATH as kubectl-blabla and use it as "kubectl blabla".
  - CSDude 5y ago
    It's not fair to compare imperative simple shell scripts with the things Terraform does. It has schema validation, state comparison, retries, failure handlers etc. Also, just as you can write extensions to kubectl, you can write your own provider in Terraform if it does not exists. See https://registry.terraform.io/modules/waveaccounting/chatbot-slack-configuration/aws/latest https://registry.terraform.io/modules/waveaccounting/chatbot... Also, Chatbot does not have a public API, that's why, it's only configured via Cloudformation. So the expectation is not fair either. I've seen Cloudformation getting features years later. i.e 2021 - https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-dynamodb-global-tables-now-support-aws-cloudformation/ https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-dy... 2015 - https://aws.amazon.com/about-aws/whats-new/2015/07/amazon-dynamodb-available-now-cross-region-replication-triggers-and-streams/ https://aws.amazon.com/about-aws/whats-new/2015/07/amazon-dy...

bovermyer 5y ago

From the problems the author is having, it would appear that perhaps Pulumi would be the better choice in this case.

rgoulter 5y ago
In the article, the problems the author discusses are: 1. Inconsistent behaviour between providers. e.g. if a resource has been destroyed since the last `terraform apply`, then some resources/providers would recreate the resource, others wouldn't. (Similarly, there's not a guarantee that the state after running `terraform apply` matches up with what's there, if the provider is happy with its state file). 2. The dependencies of already-applied resources can block `terraform apply` if the upstream API for these resources suffers an outage. 3. If a `terraform apply` applies some resources before failing, this can result in an inconsistent state. Either the resources need to be deleted, or imported. I'm not familiar with Pulumi; what aspects of these would Pulumi help with?
- kall 5y ago
  3 happens regularly and I don‘t see how the other two would really be different, since some pulumi providers are using terraform providers under the hood.

gjhr 5y ago

> When you run Terraform against AWS on the subnets part, it will create (anytime you deploy) the missing subnets That is one of the core features of Terraform? Detecting and fixing drift is useful.

poisonta 5y ago

I had mostly similar feelings four years ago. As I understood the reasons behind them, I started respecting more the people at HashiCorp. They are really smart.

cube2222 5y ago

Terraform definitely has it's warts, though, as other commenters wrote, not everything in the article is true (the reconciliation part): dependency resolution blows up in time as your number of resources grows, so you need to split up your statefiles; it can't passively listen for drift happening in a dataflow-like way (that would be awesome); it's not transactional like CloudFormation (which is more of a tradeoff, than a cons), and more. It is however a great improvement over the previous ways of doing things, and probably the best out of the current similar alternatives out there (you might mention Pulumi as a strong contender, especially for AWS glue writing). And - though as per the disclaimer, I may be biased - until a better tool comes up, I'd advise looking for specialized IaC CI/CD tools to ease your path with Terraform, like Spacelift[0]. It can help you with orchestrating dependencies among multiple state files; take care of scheduling regular drift detection/reconciliation without going into your way and locking your state; gives you a policy system for making sure preventable mistakes don't happen (i.e. recreating a resource you definitely never want to recreate); manages your credentials depending on whether you just want to run a plan, or apply your changes, and much more. I can't imagine doing Terraform again without a tool like it. Disclaimer: Software Engineer at Spacelift. If you want me to expand on the "and much more" part, you can find a demo-scheduling link in my bio! [0]: https://spacelift.io https://spacelift.io

p2t2p 5y ago

My experience is that terraform sucks. A lot. Yet everything else seem to suck even more

d0gsg0w00f 5y ago

IME, Terraform is great for "fire and forget" deployments. However, if you're trying to use it to continously bring older existing deployed infra up to date it can get tricky. I strongly suggest self versioning your TF files and following strict pretested upgrade paths.

nanis 5y ago
> Terraform is great for "fire and forget" deployments. Indeed, terraform is great for exactly the opposite: For making sure both the initial infrastructure and subsequent modifications to it are first proposed, discussed, reviewed in code and applied afterwards instead of anyone with privileges tinkering with settings on an "as needed" basis in whatever console thereby ending up with an infrastructure where you have no idea who turned on the frobnicator, why it was set to 11, and what might be the consequences of changing the setting.
- d0gsg0w00f 5y ago
  We experienced a lot of problems with when trying to manage standard configurations of hundreds of AWS VPC's ranging in age from 3 days old to 6 years. Accounts built years ago using older terraform would have to be handled completely differently because inevitable TF template drift and TF versions made each upgrade path unique. Not insurmountable by any means but also not trivial. Just sharing my experience.
robertlagrant 5y ago
If you just use a cloud provider's UI you will have no separation between desired and actual states. Then, whatever people change it to is the only truth.

digianarchist 5y ago

You can run terraform apply against a particular resource which will only provision that resource and its dependencies. tfstate files can be painful to manage, we had a lot of trouble with them at Capital One but mostly because: 1. People would modify state outside of TF which you should avoid. 2. People didn't architect their apps well which led to long lived infra. TF works best with cattle like infra. terraform import feels very much like an afterthought which is why projects like terraformer exist.

snom380 5y ago

> On terraform it's different, because of the tfstate. All the deployed elements are stored in the tfstate, re-running terraform won't update resources that are supposed to be in a specific state but are not. This is incorrect, and makes me wonder how the author has used terraform. Terraform will certainly detect differences between managed and current state for the resources it manages whenever you do a plan/apply. The major challenge is that terraform _can only reconcile resources or configuration values it knows about_, and that depends very much on how a particular cloud vendor or terraform provider has modelled resources. I believe the Helm provider is one example where it (at least in the past) haven't had a good way to reconcile state.

deleted 5y ago
[deleted]
dead10ck 5y ago
This stuck out to me too. Terraform absolutely does check the current reality of the state and applies changes to do what the HCL tells is to. They are either using terraform in a really weird way, or this article was written by someone that doesn't actually run terraform themselves.
- jrochkind1 5y ago
  Phew! I thought I was misunderstanding terraform when I saw that. Perhaps they are working with certain poorly implemented or buggy providers, and not realizing those providers were doing something different than terraform's properly working behavior that most providers implemented. Bugs happen, but the first step is agreeing on intended behavior so we know what's a bug!
shadycuz 5y ago
I also agree, but I don't have experience with the providers he is using. When working with AWS it will always reapply the terraform configuration. A great use of this is for account hardening. You can run it daily to make sure it's configured correctly.

jrochkind1 5y ago

> On terraform it's different, because of the tfstate. All the deployed elements are stored in the tfstate, re-running terraform won't update resources that are supposed to be in a specific state but are not. Huh, is that true? I'm just getting started with terraform but I assumed that was the idea of terraform (where it didn't happen would be a bug), and I think I had seen it happening for the few basic resources I have started out with (S3, cloudfront). If the state doesn't match the actual configuration of S3, terraform notices, and the plan is to make it so. No? Am I confused and it hasn't been doing this? Or is this is inconsistent, true of some resources and not of others? That seems surprising. What's the idea?

shatteredspace 5y ago
If Terraform was used for the deployment of the infrastructure, then state IS the actual configuration of the system. All that a plan does is evaluate what is going to change in the current Terraform state by performing a dry-run of the Terraform code that you have supplied. If you would actually like to make changes to the Terraform state based on what the Terraform code evaluated then you run a Terraform apply - which will, for the resources deployed via Terraform, update the configurations themselves and update the Terraform state by using the Terraform code as the instruction set. You can actually see this in action with plan and apply as the output will show you +,-, and ~ where ~ is settings that are going to change but are not new configurations or configuration to be removed. Edit: Learned from some other comments that Terraform has a 'refresh' command that will take deploy+n time configurations done outside of Terraform and sync those configurations with the state. This might be what you ideally are looking for after deployments?
- jrochkind1 5y ago
  Right. I guess I'm asking about what happens if state changed outside of terraform. I thought I had seen terraform correcting it (to match what terraform thinks it should be) in some cases. OP seems to suggest that in some cases it does and in other cases it doens't. I am surprised if that is inconsistent and unpredictable, and would have expected terraform to (modulo bugs) either always or never do that. And am wondering what terraform's intent is with that.

3np 5y ago

The author seems to have some misunderstandings on how Terraform is supposed to work - you should get the "automatic reconciliation" they're saying is missing. Also, > I run once again the “terraform apply” command. But for some reason, Cloudflare API doesn't answer and I got completely stuck there without the possibility to update with Terraform this field because of linked dependencies. You should be able to circumvent this with a `-target`. That being said, I know exactly what they're talking about with helm. IME the helm provider was/is a complete mess and gets inconsistent state a lot. Helm specifically I would also keep out of TF until that is fixed, if ever. I haven't had that happen with other providers, though. Perhaps OP was just really unlucky ending up with the odd half-broken AWS module.

danielovichdk 5y ago

I run only on Azure. Nearly all my IAC is written in PowerShell uilizing the Azure Cli. Terraform and yaml as well are so verbose and you have no clue whats going on from the local side of things. How do debug your terraform markup locally. My guess is you can't.

awslattery 5y ago
The console command [1], mixed with local variables, is quite handy for debugging locally. 1. https://www.terraform.io/docs/cli/commands/console.html https://www.terraform.io/docs/cli/commands/console.html
- x86_64Ubuntu 5y ago
  Seems as if people have very strong feelings about Terraform, but little actual experience using it.
  - toyg 5y ago
    It's not the easiest tool to grok, writing configuration files can be very verbose, and its opinions about a number of things can be very off-putting. I tried to "get into" TF many times and never actually fell in love. I like the idea, I like the cross-cloud approach, I just don't particularly like what the experience ends up being. This shit should be easier.
  - httgp 5y ago
    Bingo. While I see some valid complaints about Terraform in the comments, most of them seem to come from a place of not fully understanding how it works or the features it offers.

nojvek 5y ago

Yeah the tfstate was a big gotcha. Pulumi has the same problems. What we really need is automatic reconciliation. I.e ask the provider what they have and then diff against that. Or periodically auto-importing. Are there any good solutions to auto-importing?

arpinum 5y ago
The hard part with auto-importing is the resource id. This is usually generated server-side and is not included in the hcl. Often resources define a user-supplied identifier as well, such as a name property, and this could be used for auto-importing if the property has unique constraints applied against it. However, not all resources have this feature, so its not a universal solution.
kall 5y ago
Hm, maybe what we really need is a new entrant in hyperscale clouds that is built from the ground up for IaC and just does away with the split between state and reality. I would love to see one, anyway.
robertlagrant 5y ago
What's the point of reimporting? Aren't you then no longer separating desired state from actual?

airocker 5y ago

We use this exact stack, but we generally would not rely on tfstate. We will remove everything and regenerate it. It’s not an operation done too frequently that we have a big problem. Also, helm we use as a separate layer that is applied after terraform and can be repeated many times. This changes often.

jounker 5y ago

What countries have fewer deaths per 100K than Sweden? All of Sweden’s neighbors.

duskwuff 5y ago
Wrong thread?

nrvn 5y ago

One of the biggest reasons that have kept me away from terraform apart from the esoteric language is that terraform modules are always a few steps being from the upstream public cloud offerings. In the sense that whenever there’s a new API or service available in any of public clouds and their official SDKs there will always be a delay before this new service/feature/API will become available in terraform. First time I encountered it with GKE private clusters 3 or 4 years ago. Now it is AWS Keyspaces. The second biggest reason is whenever you have a requirement for a hybrid or multicloud then well you are left with rigidity if HCL. It is probably doable but for what sake? Solution: get a real language, write a STATELESS configuration management(IaC) system for your own needs and maintain it. The majority of public and private cloud providers ship SDKs in most popular languages that will help you build your own software solution and reduce your dependence on a third party which I would put under progressing operational risk category. Yaml/json/cue/toml for end user configs would suffice. Example: for one of my previous projects were built a tool for a hybrid AWS-openstack setup, and were managing a dozen of busy environments.

nrvn 5y ago
The emphasis on stateless here is that your desired state us described in code that resides in your repository. Actual state is what you have in your cloud. No need to spend time on state format, storage and related logic and complexity
xyzzy_plugh 5y ago
This is my preference as well. I've done everything from makefiles and bash scripts to a monolith Go program that statelessly provisions/tears down resources. Even makefiles are pretty straightforward, though you really want operations to only trigger when checksums differ -- timestamps result in a lot of redundant operations. As long as everything is idempotent, it's pretty straightforward.
oceanplexian 5y ago
> Terraform modules are always a few steps being from the upstream public cloud offerings. My experience has been the exact opposite. Usually Terraform offers support for cloud services long before the vendor provides an SDK or supports it with their own offering (e.g. Cloudformation). There are still dozens of AWS services, for example that have no CF support offered by AWS.

0xbadcafebee 5y ago

Terraform is actually kind of a nightmare. It's deceptively simple yet requires a massive amount of real world expertise to use it properly. It's a configuration management tools, but more difficult to use and extend. I'm thinking of designing a series of tools to replace Terraform. The idea would be to break down how modern cloud environments are managed into a couple concepts, and then make a variety of tools that work within those concepts together, so that it's easy to expand and modify the way you use them for your use case. This would enable things like tailoring the use of the tools to a particular deployment strategy, or adding custom business logic, or replacing individual functionality, without being tied to one tool, language, etc.

danw1979 5y ago

This article is not entirely incorrect, but there’s some glaring falsehoods as others have pointed out. There’s a plug for managing Helm related resources with the author’s own SaaS product at the end, so I’ll file this under “half hearted hit job”.

gattacamovie 5y ago

If you have kuternetes in the stack, I consider CNCF's https://www.crossplane.io https://www.crossplane.io the best path. For the cases where the native integration with the cloud provider does not provide the exact parameters you look for, it provides the alternative to integrate with terraform in the background, while making it transparrent to you.

htrp 5y ago

It's not the golden hammer, but it is a hammer in a world full of nails.