21 ms·

Terraform should have remained stateless

• weitzj 4y ago
  You could work in some way without a state but actually the state is a feature which helps you. Imagine you go down the ansible route and you write idempotent ansible code than you could argue: “See. I don’t need state. My code is idempotent. I just use this playbook to apply” Now think of deleting resources. You could have a delete playbook maybe. Then how would choose whether to create or delete stuff? Maybe a colleague gives you a ticket: “please delete” Now there are various scenarios: You do your thing (on your laptop) and tell everybody else in your team to not run the CreatePlaybook as you have to run the DeletePlaybook first for this one machine. Afterwards you delete the actual machine/resources from your ansible repository and tell the team: “please use the newest main branch”. So: Here is your equivalent to terraform state, the coordination effort on your side since you are the only person who currently “knows” what’s going on with deletion/applying. So your next idea is: “no problem, I make a Pipelines which runs the playbook on your behalf”. And the pipeline will update maybe the Git repo accordingly in some way - after the run (since ansible needs to know in the run, what to delete). Everybody can see the pipeline . The pipeline will ensure you sequentially apply the playbooks to coordinate with your colleagues. Next problem: how does the pipeline know which playbook and resources to run on? You create a selection box for your hosts and for the playbook yaml to trigger the pipeline. => there is your state. Your ticket information are transferred to your manual labor to fill in the correct items in your selection box. To reestablish want went on you now have to look in the ansible code and the pipeline Paramus and pipeline logs. More examples are: you only want to update some stuff with your ansible playbook. Therefore you might introduce tags on the resources and the playbook knows how to handle those tags. The extreme case might be: TTag:state:present, tag:state:absent. Then you can run a single playbook which can call the deletion and installation playbook for you and everybody is happy that you have everything visible in Git. Problem here: your 2 step process to decommission things from git. First a commit which sets state:absent. Then run the pipeline and then another git commit to delete the code. So what I am saying is: You can do all of this with ansible for sure. But you will have state somewhere: In a Ticket, a Pipeline log, in git, in a wiki I am not saying absible should not be used. It makes sense to configure things. (I personally would wrap a terraform hull around my absinble code and call it, just to have terraform handle the locking for my playbooks) But just watch out for the hidden state in your workflows and better make it explicit. This is why people love GitOps for traceability. You can all do this by hand and document your process in a Wiki for your colleagues so that they known what the “tag:state:absent” means for them. Or you can rely on somebody who has done this for you already and maintains documentation and what not.
• squiffsquiff 4y ago
  As TFA article mentions, we already have Ansible and friends. You can use Ansible with AWS https://www.ansible.com/integrations/cloud/amazon-web-services https://www.ansible.com/integrations/cloud/amazon-web-servic... if you want to
• scanr 4y ago
  Terraform having state is a trade-off: Pros: - it can manage resources where not all information can be queried - reading state is almost always significantly faster - it can identify deleted resources Cons: - state can go out of sync - every resource type has to implement state well while tracking changing features, so it’s fragile - fixing broken state can be a PITA Without making these trade-offs, terraform would only be able to support a smaller subset of the providers it currently does and perform poorly.
• linhka 4y ago
  dfdsfs
• linhka 4y ago
  sdsdfdsf
• boesboes 4y ago
  Personally I think the custom dsl’s are a bigger issue. I spend a lot of time wrangling tf to have reusable, configurable modules. The more i use tf, the more i think it would be better to remove _all_ dynamic features and use a real language to generate tf configs as flat, static files.
  • hackandtrip 4y ago
    Did you give a try to https://www.terraform.io/cdktf https://www.terraform.io/cdktf ? Unfortunately, it is not stable yet... but we all used Terraform <1.0 for years, and I'd say the experience was never that bad.
  • pst 4y ago
    Yes, conditionals and loops in TF are limited and have various annoying and surprising edge cases. But if something is hard to do with the Terraform DSL it's usually a good idea to reconsider if it is really something that one should be doing. We want infrastructure automation to be boring and just work. The risk with general purpose programming languages is that people will always find a way to outsmart themselves. Yes, sure, you can use the testing tool chain of the language of your choosing. But it's not like we have figured out to write software without bugs, despite all the awesomeness of modern languages.
    • boesboes 4y ago
      Yeah there is definitly something to be said for having a limited DSL. I feel though the language as-is, has a bit to much dynamic features that kinda half work and then give you completly useless error messages. But tbf, it has gotten alot beter with recent versions. But to explain where I'm coming fromm, this is something I ran into recently: this breaks and returns null if http is not set: lookup(each.value.http, "port", 8080) So I need to do this to make it work: coalesce(lookup(each.value.http, "port"), 8080) Not a huge deal, but just one of these many things over the years where I think don't reinvent the wheel.
    • erik_seaberg 4y ago
      I want both config and code to be concise and clear. “Boring” means it’s unfinished and contains small mistakes that we miss when our eyes glaze over.
• xorcist 4y ago
  Trends come and go. I too use terraform now because that is what my customers use. Previously we have used Ansible for the same thing, in a largeish environment, in production. It had the obvious benefit of declarative and stateless. The other comments here seem to focus on how that works badly together with manual state changes made externally from the system. The answer is that it requires another way of working where the state is the git repo. The question is not what to do when someone spawns extra test nodes, but why they did not do it in a version controlled manner. Perhaps as a reaction to the discipline required, something about Kubernetes attracted a lot of people used to manipulating state by manual interaction. Every installation I have seen has a web interface in use, whereas not as many have in the Ansible world. I fully expect this pendulum to swing back and become more declarative and version controlled again.
• AtNightWeCode 4y ago
  Would say that half of the point with Terraform is that you have a canonical expected view of the infrastructure and the state as a log to assert that in each environment. It forces you to do all changes in a code first manner. Terraform only solves the provisioning part and is easier to work with than Ansible. One need something like Ansible too for configuration though.
• pharmakom 4y ago
  Here’s an idea: a new cloud provider that can only be interacted with in a declarative way. It forces you to work like Terraform.
• dangus 4y ago
  I think the author of this article misunderstands what Terraform is for and what it does. The author compares Terraform to Ansible and Puppet, but these are not analogous tools. If you compare to Pulumi, you'll notice it also has state. Keeping track of what is live and what is expected to be live is much of the utility of Terraform. If the author is getting some Terraform state pain (e.g., from drift caused by a team making changes to infrastructure and forgetting to represent those changes in code), they should try using Terraform Cloud or building a CI pipeline for their infrastructure.
• master_crab 4y ago
  I like the idea of stateless (or at least not having to manage the state as in the case of CF) but I’d hate to see the performance times for API calls if one needed to do a search for tags to conduct a CRUD. Particularly if this were a multi-account setup.
• moondev 4y ago
  Assuming every resource has a name field, auto prefix it with "tf-" If the resource is prefixed with "tf-" but missing in the terraform config, delete it.
  • ntoskrnl 4y ago
    A perfect solution. We'll just have to move our home page to tf-www.example.com
    • moondev 4y ago
      The simple solution is to have tf launch an EKS cluster with route 53 external-dns, then the ingresses can be defined as k8s resources which tf also manages :D
  • zeckalpha 4y ago
    What happens when you have multiple terraform repositories interacting with the same account?
    • moondev 4y ago
      another prefix for terraform workspace name, similar to how netflix uses the name field for metadata https://gist.github.com/johnrengelman/12e8d975344c7945ab112715a62516de https://gist.github.com/johnrengelman/12e8d975344c7945ab1127...
• deleted 4y ago
  [deleted]
• agounaris 4y ago
  Terraform could have been stateless if the target provider was stateless as well. Having said that I don't mind state at all! There is always a storage layer and a schema somewhere storing some kind of information.
• oneplane 4y ago
  Well, this would ruin the point of terraform and turn it into Ansible... You really need all three to get any use out of terraform: - The desired state in your .tf files - The actual state in your provider (what you describe) - The expected state in the state file This essentially gives you drift detection and delta updates. A simple "terraform refresh" and "terraform plan" read your actual state and does a diff between desired state and actual state. If all you had was the real world state, then the absence of resources gives you zero information. The other way around: planning a change without comparing it against a stored state gives you no way to actually purge the real world state. You could technically argue that everyone should keep their .tf files in VCS/SCM and then have terraform first check the real world against the previous commit before creating a delta based on the current changes, but then you're just moving state to Git which is already a state backend... The triangle this creates is why terraform generally is better than most other IaC systems which either don't have all three legs (and thus collapses them into a 2-dimensional also-ran tool) or they do but only for one special system (i.e. only AWS/GCP/Azure and no integration with anything else). Next thing you know someone is coming to advocate against locking and hash comparison... Edit: the best 'simple' explanation I could come up with is: you can't remove or update what you don't know shouldn't exist anymore. And you can't realistically 'download' the configuration of an entire cloud to 'check' all the tags for state information.
• lloydatkinson 4y ago
  I am a big fan of Azure ARM/Bicep. I can’t imagine needing to deal with stateful IaC. Perhaps Azure doesn’t need to do that because it’s able to provide guarantees about their own platform or something. The actual infrastructure and its configuration is the state. It does a diff to see what needs changing.
  • giaour 4y ago
    Azure has some pretty strict internal requirements around resource creation that make this possible. Every service must provide a "Resource Provider"[0] API that supports idempotent creation operations and standardized query patterns for read operations. Whether you create a resource through an ARM template, in the portal, or via the CLI, it's going through the same unified API surface. So, there is definitely still state, it's just stored centrally. [0]: https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/resource-providers-and-types https://docs.microsoft.com/en-us/azure/azure-resource-manage...
• bobobob420 4y ago
  Terraform is amazing and better than its competitors because of its statefulness
• sly010 4y ago
  Re: handling unmanaged resources, you have 2 options: 1. Everything not declared is unmanaged. Deleting is done by a "deleted" attribute of some sort. The resource is declared as deleted. 2. Everything unmanaged is declared as unmanaged. The resource is declared as unmanaged and TF would ignore it. Either of those would work without extra state.
• andrelaszlo 4y ago
  Simple counterexample from this week. I added a Heroku app webhook to Terraform. They have some flags, and an ID. Without storing the state, there would be no way, using only the Heroku API, to know which webhook is supposed to be managed by Terraform. I don't understand where the 99% comes from. Maybe I'm just using the wrong services, but it seems more like 50% to me, anecdotally. Maybe 90% if you include hacks like storing "tags" in other fields like the description or resource name. If it's truly 99% for you, it seems like it wouldn't be too hard to make a tool that generates an ephemeral TF state file from the services on the fly. Win-win? Or you'd run into the next problem, collecting all the state all the time would probably be really slow in big projects.
• Androider 4y ago
  You couldn't do everything that Terraform does today without a local state, but perhaps that would be a good thing? Call it "Terraform strict mode". As every application developer knows, duplication of state is a primary source of bugs. To combat this, React/Flux type of architectures became extremely popular where state flows in one direction only. They dictate that, no you can't just cheat a little and use jQuery to modify some element, it _will_ get bulldozed on the next render. And a lot of Terraform headaches do come from this analogous reconciliation of what really is (our cloud env), what we want (our TF code) and this intermediate state of what TF thinks the cloud state is. So, by saying that you cannot have resources outside those defined by TF there is actually a massive simplification with far reaching consequences possible. How I imagine the experience would be: - You could say that your Dev env is a shitshow and always will be, of manually created resources and only partially TFed. But your Production and Staging envs are opted in to "strict mode". This means that if there is a conflict you do only have two options: import the offender or destroy it, and the critical mindset change is that this is a good thing and will save us a lot of tears later on. - Caching is an orthogonal concern. Terraform mixes these two together to its detriment, but the nature of a cache is such that you can safely blow it away and perhaps the next reconciliation will be slow, but it will be accurate. I also don't believe it would actually be that slow, tools like Cloudcraft map the entire metadata of your account in seconds. - I find the excuse that some resources don't support tags intellectually lazy. Of the top of my head thinking about it for a minute, could you tag a parent resource with the child metadata you need? E.g. individual DNS records don't have tags, OK tag the Zone with childA=value. Same thing with tag length limits, you can work around it, concatenate values or whatever. However, in a truly strict mode you wouldn't even need metadata in tags because the TF code describes the entire target environment. I hope Terraform would entertain such a strict stateless mode. Unfortunately it will probably take another tool, because the problem is not so much technical as it is an entire mindset change.
  • mountainriver 4y ago
    Yeah it wouldn’t have all the capabilities but would be a good tradeoff for lots of use cases
• KasianFranks 4y ago
  I would not argue against maintaining state. Immutable state is key to security.
• rad_gruchalski 4y ago
  Interesting read. Seems to be focusing on the neck of the woods the author is exposed to. Terraform isn’t about cloud apis only. Terraform allows me managing keycloack realms, ssh keys, cloud resources, postgres databases, git repos, imap accounts, and so on. The state is there to be treated as the source of truth. It gives an answer to „do I have what I want to have”. With that state, it is possible to cross reference various resource types without having to load the curent state on every run. I’m surprised that the author did not see that as a performance issue. Imagine that you have to load all route53 state, all buckets, ec2 instances, iam roles, … on every execution, and imagine you have 200+ machines… That’s what we used to do with puppet and chef, no? Turned out that always fetching the view of the world from an api is pretty expensive and quickly exhausts api rate limits. It’s a pretty weak article without any effort to suggest how could it work without a state.
• dagss 4y ago
  A huge issue with the statefile IMO is storing secrets and API keys. We try to make sure EVERYTHING is done through service identities (Azure). No secrets in deploy scripts or resources. Terraform will happily generate and store passwords and API keys for resources even if we don't want to use them and save the keys to everything in a file on a "random" disk and suddenly we need to lock this file down. With a statefile we have to worry that secrets gets accidentally stored -- I am sure it id possible to configure it away case for case, but that is a more fragile approach. With tools without a statefile the whole problem just goes away. For us this was enough reason to disqualify Terraform.
• breakalot 4y ago
  The resource IS the state!
• PhineasRex 4y ago
  If you do any non-trivial devops works on cloud providers it's immediately obvious why this is nonsensical. Let's take the most basic example: auto-generated ids. Many resources in AWS, GCS, etc have auto generated ids (just use tags you say, but many don't have tags or tags are used as part of some other system). Now, when terraform creates that resource you have to modify the config to contain the id. But if you have any sense terraform runs as part of a CI system that lets others review your code before merging, deploy to staging, etc. So now does the terraform process need to make an automatic git push? What if there's a conflict? Does it make a PR that has to be manually merged? All of this is much more complicated than just having one JSON file in S3. I have actually managed resources with Ansible where you have this problem and it's worse. And this is just _one_ thing. Is Terraform's state story perfect? No. There are definitely annoyances, and one thing I'd love to see is a way to declaratively handle imports, renames, etc. when you need to, but it's better than the alternative.
• rwmcfa1 4y ago
  Not having to deal with external state was one of the foundational design goals of octoDNS. The other being specifying a record once and pushing it to multiple providers. Those two in combination were the main reasons we didn't end up using Terraform to manage DNS and started octoDNS. That did require making the initial decision that octoDNS would own the zone completely so that it could know it's safe to remove records that aren't configured, but later work with filters and processors does allow softening that requirement. I've always had similar feelings about Terraform's state, in fact I started in on a prototype of an IaC system specifically to see if it was workable to avoid external state. The answer as far as that POC made it was yes. I was able to create VPCs, subnets, instances, to the point I stopped working on it. It was generally straightforward, but there was a hiccup for things that don't have a place to store metadata of some sort and the biggest issue was knowing when they were "owned" by the IaC. I think some of the other issues that things would eventually run into would be around finding out what exists (again to be able to delete things.) The system would essentially have to list/iterate every object of every possible type in order to decide whether or not to delete them. Similar to octoDNS this could be simplified by making the assumption that anything that exists is managed, but that's not workable unless you're starting greenfield and it would still require calling every possible API to list things. Anyway, I see why Terraform went the way it did, but I still wish it wasn't so. Thinking about it now makes me want to pick the POC back up... (maintainer of octoDNS)
• Too 4y ago
  Avoiding state is theoretically a nice goal. But a generic “none” backend as described in the OP would simply be impossible. The support for diffing desired vs actual state must be implemented in the resource provider, which in turn need to be supported by the cloud API. Ubiquitous labels, namespaces and global query performance seem to be the primary blockers today, judging by most other comments here. Interesting thought if someone would attempt to make such a provider. Also interesting to look at existing providers if they avoid state internally when possible or just use it because it is available. Looking at the challenges with kubectl apply --purge, already having all enablers laid out, it would require big effort.
• deleted 4y ago
  [deleted]
• renaudg 4y ago
  >Also, if Terraform configuration is refactored, for example, to wrap a bunch of frequently copy-pasted resources into a module, state must be manually reconciled before proceeding. Not true since Terraform 1.1 introduced moved blocks : https://www.terraform.io/language/modules/develop/refactoring https://www.terraform.io/language/modules/develop/refactorin...
• gkop 4y ago
  Thank you OP for answering a question I’ve been long curious about but never bothered to look into, and sharing here. I love/hate Terraform. It’s better than any other tool I’ve used for what it does, but the abundance of subtly leaky abstractions is tedious. And then when you mess up your state occasionally, yea that’s super annoying too.
  • smcleod 4y ago
    After spending the last 6 months with AWS CDK - I'd kill to go back to using Terraform which is far from perfect but light years ahead of CDK which is the most consuming time sink I've ever had to work on, it's truly dreadful. Give me Terraform any day!
    • jen20 4y ago
      “Broken time sink” is more inherent to CloudFormation than CDK per se - if you want to write code instead of config Pulumi is likely a better choice.
    • w3456yhbvr5yh 4y ago
      "time sink" is also how i would describe CF templates and wrappers like SAM. I'd be grateful if you could elaborate on the CDK issues. I'm also curious to know how many people in this thread consider themselves developers as opposed to devops/cloud engineers.
      • smcleod 4y ago
        There is so much toil involved in the software framework(s) around CDK, the most common CDK language is typescript - which while in many ways is better than JavaScript still suffers from the same garbage ecosystem. The serious problem with CDK is that it's just a wrapper for CloudFormation - so you have all the limitations of a CFn backend with the added complexity of a full general programming language and ecosystem. While Terraform is far from perfect you can land in a team, look at the terraform repo and immediately know what's going on, you've got very commonly used patterns / templates that can very quickly spin up your platform. With CDK almost anything you create is a pet, a special snowflake that almost immediately becomes technical debt. You end up spending a lot of time on maintenance and upkeep, updating libraries for (really bad!) node security patches, you have to test those updates across whatever packages and repos you're using them in, maintain software build/test/secure/deploy CI pipelines. I've seen CDK at reasonable scale - it's painful with more than a couple of repos, even worse when they're spread across teams and products. With Terraform at least you just update Terraform itself after checking for breaking changes. I could keep on ranting but I think no matter what I have seen and say - many developers will take a default position to arguing that infra/platform code is always better written in their language than any DSL, wrapper or templating system, I've heard this time and time again and only really agree with it when it's with Lambda only workloads, Often people cite "a real language is so much more powerful" etc... but in reality 99% of the time you just don't need highly complex programming logic when creating reliable and scalable platform using something like Terraform. (Insert "any fool can create something complex" quote here). The one and only place I think CDK is genuinely a half-decent tool for the job is with Lambda only deployments as long as you keep it pretty lean don't don't over cook it with complexity and keep in mind that it's just CFn in the background so it has the same limitations. I'm in a "DevOps" / platform engineering / automation role. </poorly written midnight rant> :)
• loxias 4y ago
  Completely agree. Thanks for writing this up. Specifically because I've always thought this, but also don't value my opinion on the topic. I was first exposed to devops a few years ago, and as a result was learning docker, k8s, terraform, salt, &c. at the same time. (I hated it, and now I'm happy writing C++ again) What I could never wrap my head around was why the heck the tools had to expose so much complexity. I want the description of my infrastructure to be a single, or set of text files, written preferably in JSON or YAML like everything else, and I want to run a command that behaves in the same way as GNU make.
  • NovemberWhiskey 4y ago
    That seems like madness to me: YAML? Really? Is this some variant of K8s Stockholm syndrome? It's always seemed to me that one of the virtues of Terraform was that the language was just declarative, and rich enough to express what you needed without becoming a Turing tarpit.
  • x86_64Ubuntu 4y ago
    >... want the description of my infrastructure to be a single, or set of text files, written preferably in JSON or YAML That's CloudFormation, there is a reason that people migrated away from it.
  • knckl 4y ago
    Have you tried Pulumi? You can use TypeScript, Python, C#, Go, Java, YAML etc to define your infrastructure. Almost all Terraform providers are ported to Pulumi or have Pulumi-native alternatives.
  • nathants 4y ago
    this is the ideal approach for most scenarios. fewer knobs, simpler tooling.
• pram 4y ago
  When was Terraform stateless? I've been using it since 2015 and it has had state files since then. Must have been a very brief period!
  • mlvljr 4y ago
    [dead]
• chucky_z 4y ago
  You can use terraform in a "stateless" manner! Define everything in like cdk (... I use ruby to generate the tf.json), generate the code, import everything you can without error, and apply the rest. Performance will be _bad_ but that will completely eliminate state problems.
  • infogulch 4y ago
    I think the reason why terraform did this is that if you have a medium-large deployment that you plan/run often you'll probably run face first into cloud management api limits, it's not just bad performance.
    • ericbarrett 4y ago
      You're absolutely right. I started using Terraform around 2016 and it was pretty common to get barked at by the AWS API if you were repeatedly running plans, even with state. I bet the cloud providers have had to make significant infra changes to support the growing number of customers using TF.
      • scarface74 4y ago
        There are still plenty of services in AWS that have painfully low service limits for API calls. The first one that bit me was creating parameters in SSM Parameter Store using CloudFormation. CF tries to create as many resources in parallel as possible based on the dependency graph it creates. The only way around it was making each Parameter resource dependent on the other (using DependsOn) to force the resources to be created sequentially. Before the CF vs TF holy wars began, this is an API limitation that you would hit regardless of your IAC.
  • cmclaughlin 4y ago
    cdktf doesn’t really help with state problems in my opinion. Under the hood terraform is still storing state and if you’re working on a team you’ll need to share state, e.g to s3. If you don’t have any state and try to make a change, terraform will try to recreate and fail. Importing is a pain. cdktf is great, but I would also rather do away with the state. I’ve gotten into too many problems that were only resolved by deleting everything and starting over.
• anothernewdude 4y ago
  State or Idempotency. Pick one. If you want stateless, then you can use Ansible and use their providers. Enjoy spawning new instances everytime you change your infrastructure, rather than having existing ones change.
  • aussiedude 4y ago
    You can also use Ansible with Jinja2 templating to generate your Cloudformation templates. Describe loops and other properties in Ansible and have it build out a complex Cloudformation template for it to deploy. Use an Ansible variable for your stack name so you update/delete an existing stack and your good to go. Could even break it down to environments so dev = smaller instances vs stage/prod etc.
    • anothernewdude 4y ago
      That just shifts where the state is being stored. Once you use cloudformation, it's being stored on S3, which you can do with terraform anyway.
    • takeda 4y ago
      Wouldn't using troposphere[1] be easier? [1] https://github.com/cloudtools/troposphere https://github.com/cloudtools/troposphere
• janosdebugs 4y ago
  I respectfully disagree. If TF was stateless, you'd have to manage situations by hand that involve changing the idempotency key, such as the name of a VM. You'd also have to manage situations by hand where a resource is removed from the config. The whole point of TF is that it has state and doesn't require workarounds for these scenarios. Yes, you have to maintain state, but the state problems usually come from buggy providers, not Terraform itself. For example, try and use the GitHub provider to create a repo with GitHub Pages enabled from the gh-pages branch. It won't work because the authors didn't respect a fundamental rule of writing a provider: one state-changing API call = one resource. If you don't respect that, you have to do state handling yourself and you're almost certain to have state-related bugs.
  • Jasp3r 4y ago
    Why would renaming be impossible to solve by a stateless terraform? In a cloud setup with 3 VMs, and you rename 1 VM from A to B: Measured cloud setup would have VMs A + X + Y New cloud setup would specify VMs B + X + Y You can easily identity X and Y as their names go unchanged, A and B would have similar config/metadata, instead of assuming A would be added and B would be removed you can ask the user if a rename happened. This is not a new problem to solve by the way, this is how database migration tools handle column renames.
    • cassianoleal 4y ago
      In this scenario, you'll have to manage your entire infrastructure from a single Terraform run. In most places I've been, Terraform is scattered, each run managing its own corner of the infrastructure. In this case, each Terraform run would delete everybody else's infrastructure. Also, how do you know when to re-create things like random strings or numbers? Or null resources? Presently it's a mix of keepers (requires state) and taint (also requires state).
    • ImPostingOnHN 4y ago
      as a user, I don't want to be asked to interactively guide what is supposed to be automated as its entire raison d'etre
• NovemberWhiskey 4y ago
  Uh, how do you delete resources with this model? If you don't have any state, and you have an empty module, did you just create it, or did you just remove all the resources from it? The former requires no action, the latter requires API calls to delete something that I no longer have a record of. More generally, do I have to completely enumerate the entire state of every service available to my AWS account to determine whether there's something that shouldn't be there vs. the contents of my Terraform modules?
  • tbrownaw 4y ago
    Maybe don't scope it to "everything I can see"? A model of "this current module maps to such-and-such container" could work, with the mapping vs the module working vaguely like git remotes vs your code tree.
  • c3534l 4y ago
    If its not specified to be there, it shouldn't be there.
    • NovemberWhiskey 4y ago
      That sounds logical, but it's not very practical due to the cost of enumeration; plus, it requires providers to gain deep knowledge into things that are implied dependencies ... like "hrmmm, should I try to delete this service-linked role or do I actually need it?"
    • TameAntelope 4y ago
      The way Terraform works makes that not feasible. As it is, you can introduce Terraform incrementally to your infra, and it won’t wipe away anything it’s not aware of. If you delete everything not explicitly specified, Terraform would have to describe literally everything, which could be a gigantic barrier to entry.
      • takeda 4y ago
        We know that if state is not used we can't handle deletions without affecting everything else. So here's a solution that could satisfy stateless proponents: we don't keep state for anything that we deploy and instead we should store state for everything else so we will know what not to remove /s But seriously, as things are right now, state is necessary, it could be possible to get rid of it, but that would require cloud providers to designate their infrastructure that way. And if they do that, in practice (because components building the cloud service likely are stateful) we would end up with a system with declarative language which in reality just abstracts away the state from us. Kind of what AWS CloudFormation essentially is.
• ldoughty 4y ago
  Basically agree with the article. I've used direct cloud formation, AWS SAM, Ansible, terraform, and AWS CDK to spin up infrastructure... My hard line opinion is that if something NEEDS state management to exist and update, it's a pet, treat it like a pet. Don't mix pets with the rest of your automated machinery except to the minimum extent required, when absolutely necessary. We had to rewrite an Ansible role because a patch level version upgrade on the recommended community package started destroying security groups... Another time we had to roll back our deployment Ansible image and update 30 repos because a minor level version bump in another Ansible recommended package suddenly required log groups to have an expiration value set in AWS or the module fell over with an null reference on the AWS lookup/comparison... So we couldn't use the latest version to fix it. I've almost never had this happen with any AWS tool... Sure, there's drift possibilities, but those are controllable by mainly not letting humans do things, and not having multiple cooks in the kitchen changing things in automation, which are good ideas for terraform and Ansible to... I also encourage modular designs, any of which (except cdn/db, and dns related stuff, in my use cases) can be torn down and re-built with only the brief outage nonexistence causes.. we only have done that once in 3 years, and we believe the issue was actually on AWS's internal side anyways. I've spun up over 260,000 vms over 4-5 years with one cloud formation template and the basic SDK call, and we've never bothered to convert it to another tool because it's never broken.. we occasionally tweak it, use gp3 instead of gp2, etc, but it's never needed us to unexpectedly side track a sprint for 1-3 days
  • dangus 4y ago
    > I've almost never had this happen with any AWS tool... Sure, there's drift possibilities, but those are controllable by mainly not letting humans do things, and not having multiple cooks in the kitchen changing things in automation, which are good ideas for terraform and Ansible to... This is also what you do with Terraform: that's what the Terraform Cloud product is about (or you can just build a CI pipeline in your tool of choice with a blog post's amount of work). It also sounds like what you're saying is that you can just avoid all these problems by having everything automated from day one, but that's not the reality in any employer I've ever worked for. Unless you're starting a company today and happen to have an experienced infrastructure engineer on staff from day one you're not getting that world. Terraform state also comes in handy as an audit trail. Through versions of your state file, you can see how your infrastructure changed in the past (You could use S3 versioning or you could have Terraform Cloud manage the state file automatically). Terraform Cloud turns that concept into a relatively powerful feature. It becomes a compliance record: who changed what and how it changed, along with diff files. As another commenter pointed out, it really seems like the alternative tool you're using (CloudFormation) has its own concept of state, but it's just hiding it from you. Pulumi also has a concept of state. I haven't really seen an infrastructure automation tool without that concept. Ultimately, these tools need to have some way to track what resource the code is referencing. Call that metadata or state, you have to track something. And if you come up with something that somehow avoids it, I have to ask why? It seems like trying really hard to make a car without doors that also prevents rain from coming inside: why not just leave the doors on the car?
• rwiggins 4y ago
  Having worked with both Terraform and Ansible code that created AWS resources - and operates very similarly to the model described here [0, see `filters` arg] - I generally disagree. For example, if you changed an ID in your stateless Terraform, you'd have to insert some kind of code to destroy (or rename, if possible) the old resource. Or modify the Terraform DSL to include that kind of information, I suppose, and keep a historical record in the code perhaps. Then there's the question of what happens if someone modifies the physical resource out from under you -- could end up creating brand new resources rather than tracking the existing ones. Also, it's nice to know that your Terraform instance is what created a thing -- if you ran a stateless `terraform destroy`, it's possible you could be deleting resources that someone else created that happened to match what your Terraform code defined. More of an edge case, I admit, but at scale these things have a way of happening... That said, resources that don't have "physical" IDs work similarly to the stateless model by necessity. For example, VPC route table rules: [1, see the "Import" section]. Refactoring Terraform code is super annoying because of state, though, I'll give you that. [0]: https://docs.ansible.com/ansible/latest/collections/amazon/aws/ec2_instance_module.html https://docs.ansible.com/ansible/latest/collections/amazon/a... [1]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route https://registry.terraform.io/providers/hashicorp/aws/latest...
  • rtp4me 4y ago
    I am starting on a journey to deploy resources in the big three cloud providers (AWS, Azure, Google) and could use your input. So far, we have been working with Ansible to provision our AWS resources. Because we find Ansible to be a pain to use, we are considering Terraform - especially when working with multiple cloud providers. Do you have any advice or best practices to properly deploy and manage instances using Terraform? Also, what led you to use both Ansible and Terraform?
    • Aeolun 4y ago
      I would suggest maybe bypassing terraform entirely and going to something like Pulumi or CDK.
• aayjaychan 4y ago
  If Terraform is stateless, how does it know what it needs to / can delete? You'll either have to: - Move the state management elsewhere, and invoke different commands depends on what and how resources are changed. This will make automation difficult, and doesn't solve the problem. - Make Terraform assume that everything it sees is under its management, deleting everything not defined in the current configuration. This will make Terraform hard to adopt in an environment with existing infrastructure.
  • rwiggins 4y ago
    Presumably you'd encode removed resources somehow in the DSL. Maybe a flag like `removed = true`.
    • NovemberWhiskey 4y ago
      Congratulations; your state management is now part of your code.
      • GeneralMayhem 4y ago
        And that would be a huge improvement! Code has history. Code can be managed with sed and grep. Code can be generated by tools I write myself. Adding a tombstone for deletion, or a formerly-known-as tag for renames, is only "state" in the way that reserved tag numbers in protocol buffers are "state". It is a little annoying to have to do, and it creates clutter that you eventually have to go back and clean up, but neither of those is a dealbreaker, and in the meantime it solves the second-biggest problem with Terraform, which is the inscrutability of what it actually thinks it's doing when it comes up with a plan you don't expect. (The biggest problem is how ridiculously inexpressive HCL is as a language.)
        • InvertedRhodium 4y ago
          The best practices for terraform is to use a versioned state store, which covers history. Under the hood, the terraform file is just JSON, so sed, grep, jq etc can be used to manipulate it (as well as any other tools you'd care to write).
• deleted 4y ago
  [deleted]
• leetrout 4y ago
  Given the ability to import resources this actually wouldn't be that big of a lift for some providers like AWS. Wouldn't take much to hack something together to test this out, either... parse the TF for resources, lookup what is used for their IDs, run TF import with discovered IDs from the service provider and then your local state is up to date, run your plan / apply and blow away the state when you finish. But this is super gross IMO :)
• tedk-42 4y ago
  Ahahaha there are commenters that don't realise that Cloudformation IS the state for your infrastructure that you've provisioned. Ansible is stateless because every operation is suppose to be idempotent. Unless your ansible is doing a HTTP PUT request to an API I suspect you're misusing the tool for something it's not meant to do. State is a good thing with infrastructure and terraform got it right.
  • master_crab 4y ago
    Cloudformation is not the state of your architecture. It tracks the state. A very big difference. You can still make changes to resources within stacks outside CF, and Cloudformation can still be very unaware of the drift.
    • tedk-42 4y ago
      Then you're using cloudformation wrong. Unless you're arguing semantics to which I'm not interested.
  • pojzon 4y ago
    Author doesnt seem to understand all cases where state is simply required. Terraform is not perfect, but looking at solutions we have available on the market -> its years ahead of competition.
    • sjtindell 4y ago
      Author is more asking the question, probably also thinks “it needs state” could be valid.
• jvitor03 4y ago
  "Stateless is better", until you remove a tf file or resource from your code and Terraform have no way to tell if that resource ever existed in the provider during the apply to delete it, because there's nowhere to compare and it is impractical to query all existent resources and all tags/ids from those resources in the cloud provider on every infrastructure change (imagine multiple CIs doing that all day long, there's not enough api quota and your pipeline will take forever just for terraform know what to change in a medium/large organization). That deletion and modification problems happens in Ansible and other provisioning tools that relies in idempotency, and that is one of the things that makes Terraform different from them. A stateless Terraform is useless, it's better use other provisioning tools.
• throwaway892238 4y ago
  Having a state file isn't a bad idea. State files are a logical map from your code to the AWS resources. You can actually import a resource that Terraform never created into a state file so that Terraform can manage it. But of course, you could just write the code to explicitly include that pre-existing resource, rather than have to run a command to tell a state file to explicitly include it. The problem with Terraform isn't that it keeps its own state file. The problem is that Terraform is just really dumb, regardless of the state file. It doesn't know how to auto-import existing resources. It doesn't know how to overwrite existing resources. It doesn't know how to detect existing resources and incorporate them into its plan. It doesn't know how to delete resources that were unexpected. It's so stupid that it basically just gives up any time any complication happens. Puppet and Ansible aren't that stupid. They both will make a best effort to deal with existing resources and work around problems. It's not the presence or absence of an external state database that make those tools "more advanced", it's simple logic.
  • NovemberWhiskey 4y ago
    You might think that's useful behavior; personally I absolutely prefer the Terraform model of having to explicitly import things into state that weren't created by Terraform.
    • throwaway892238 4y ago
      Because you get paid more per hour?
      • NovemberWhiskey 4y ago
        Why so snarky? No; because if the goal is repeatable infrastructure, I don't want the bits that are non-repeatable to be silently assumed - you just end up with a cloud-centric version of "it works on my machine".
        • nathants 4y ago
          one still must test on multiple machines before being confident it doesn’t just work on one’s machine. cloud is no different. we must test.
• nathants 4y ago
  amen! as another commenter here rightly put: state or idempotency, pick one. hello? how is this even a question? i have a hard time believing that idempotent infrastructure mutations are not the right move almost always. shameless plug[1], i’ve been exploring an aws specific approach to infrastructure that is stateless and idempotent for exactly these reason. slow, finicky, stateful deploys are about as awful as it gets. add a pinch of lowest common denominator among all providers, and that’s a tough pill to swallow. there’s got to be another way, aws should be fun! 1. https://github.com/nathants/libaws https://github.com/nathants/libaws
• eljimmy 4y ago
  It's almost funny seeing almost every comment referencing AWS. Just another layer of abstraction...
  • erik_seaberg 4y ago
    We might take another look, but back when we launched an internal platform, it was literally impossible to import manually created resources into CloudFormation. Terraform was the only game in town. I also keep hearing about long delays in CF supporting new features; the TF provider is usually well maintained. I wish either of them had a more graceful plugin story, though.
  • NovemberWhiskey 4y ago
    Also all the comments are in English!
  • nathants 4y ago
    aws is absolutely the worst featured and least stable cloud provider. except for all the others. sometimes, being least bad is a great accomplishment.
• tokamak 4y ago
  AWS should really find a way to replace CF with something sane like TF so we can have both cloud resources and cloud workload described with native and managed tech. TF is pain but necessary.
  • takeda 4y ago
    I think your perception might be due to using TF. To me, once CF added YAML support it's much more straight forward to use.
    • erikerikson 4y ago
      Use the Serverless framework for your cFT (no serverless needed) to add in richer variables (e.g. config find fetching) capability too.
  • kevan 4y ago
    Have you checked out CDK or Pulumi? CDK still has a few warts but overall I've been pretty happy with it. Haven't used Pulumi but from chatting with friends it sounds like it has a similar feel as CDK but supports multi-cloud like Terraform.
    • tokamak 4y ago
      Pulumi is TF alternative but it is not managed like CF. CDK uses CF underneath with all its issues.
    • pojzon 4y ago
      Cdk is compiling into CF template. And Pulumi is using Terraform providers under the hood. Writing infra in imperative style can cause ALOT of unseen issues when developers start adding IFology or Design Patterns..
      • pid-1 4y ago
        Most CDK code is absolutely declarative. Unlike TF, you have an actual programming language as a escape hatch.
• smitty1e 4y ago
  The article mentions using tags in lieu of state. What are tags, if not state?
  • Rapzid 4y ago
    Yep and the configs are state too. These tools (Chef, Puppet, Ansible, TF, etc) are state replication engines; they replicate the state described in the config into other systems.
    • smitty1e 4y ago
      So, we can hate the state as much as the most rabid libertarian, but what is IT really if not state management?
• notorandit 4y ago
  I am not sure how feasible the proposal is. Certainly cloudformation is too archaic without Jinja & co. Terraform doesn't need extras, but provider dependencies can be painful.
• pst 4y ago
  I maintain a Terraform provider for Kubernetes. And one of the main reasons for that is because the Terraform state ensures purging of deleted resources. Something that kubectl is not capable of. The lastAppliedConfig annotation does not help for purging, because once the manifest has been deleted on disk, there is no way of knowing what to delete from the server. The unusable apply --purge flag is the best example of this issue I think the state mainly exist to know what has been created in the past but since been deleted from manifests and therefore needs to be purged. The caching/performance argument is rather weak, because Terraform refreshes by default anyway before any operation.
  • phrotoma 4y ago
    Could you elaborate on the poor usability of the --purge flag?
    • pst 4y ago
      It's not trivial to get the labels correct to avoid collateral deletions. Also, while it makes sense, I and many teams I consulted with found it rather unintuitive that apply --purge with a label selector will also only update resources with the label. Not all resources that are in the list of resources. Last time I checked it was also still marked experimental and has been for years.
  • anticristi 4y ago
    > I think the state mainly exist to know what has been created in the past but since been deleted from manifests and therefore needs to be purged. The caching/performance argument is rather weak, because Terraform refreshes by default anyway before any operation. Beautiful summary. For resources with flexible tags, one could easily imagine tags like Kubernetes's: terraform.io/name terraform.io/instance However, for tag-less resources you have no choice but to store state to map real-world IDs with what is in the config. I wish Terraform "tried harder" to avoid state when it can be avoided. Perhaps it could introduce some soft state, where deleted resources are refreshed by looking at tags and not state.
    • 4y ago
• m1keil 4y ago
  > Ansible, Puppet, etc. don’t have intermediate stores of the hosts’ configuration, but then again they are used for different things. Ansible, Chef, Puppet and co existed long before Terraform. If the stateless way would work better, these tools take over the cloud infrastructure space. But they didn't. To me it seems like a good sign that the their approaches didn't fit to infra. Additionally, one of the biggest terraform selling points in the early days was the change plan feature. The ability to see the entire change that is about to happen as a result of a config change. I don't think it's easy to implement such thing in a stateless system. Is it possible to create a stateless config for a subset of the resources/providers with a better functionality? Absolutely! Octodns seems like a good example. Another great example would be any of the serverless frameworks out there that do a much better job than Terraform at managing the lifecycle of the functions. But can this approach be applied to every provider and resource?
  • Xylakant 4y ago
    One of the fundamental issues that I’ve encountered in all of the puppet managed environments is that people would remove resources from the configuration without first explicitly setting them to “absent”, effectively leaving them in an unmanaged state of the previously management configuration. If that state differs from the default, a new machine would end up with a different configuration, potentially breaking things.
  • Jasp3r 4y ago
    Ansible and Puppet are for infrastructure automation across multiple VMs. Terraform is a way to declare cloud (native) infrastructure as code. Ansible and Puppet are not used anymore because of the shift to cloud native, not because they are stateless.
    • m1keil 4y ago
      I wasn't suggesting that CMS fell from grace because they were being stateless. I was saying that when TF just started, these tools were in the right position to take lead over the IaaC space and didn't because the stateless model didn't fit. Take a look at Ansible's cloud modules, they tried. TF's success, in my opinion, can be boiled down to: a) Having a state - allowed them to provide declarative config with plan functionality b) Choosing Go - removed the burden of installing or having to invest in self contained setup like they did with Vagrant & ruby. I'm not a TF zealot. I hate my life every time I need to use it. And yet I know that there are no better alternatives out there. I'm sure there are some nice things in Pulumi or cdk-tf but I doubt switching to any worth the investment with existing TF project. I would be glad to try and play with a tf-like project that works without state - "talk is cheap, show me the code".
• nathants 4y ago
  something that can help reduce the need for state is aggressively partitioning systems into separate aws accounts. then you can KNOW that no other random infrastructure should exist in an account. terraform definitely does help coordinate the bunk beds of room mates. wouldn’t want them to accidentally discard each other’s pillows as they move in and out of the shared space. separate billing is a nice bonus.
• nailer 4y ago
  Terraform it’s slowly losing relevance. You’re making a good point but they haven’t been open to chance for decades are are content to become another Sun Microsystems. Better idea: advocate the ‘state only if needed’ approach to Pulumi.
• koiueo 4y ago
  This looked suspicious when I started reading. But when I reached comparison to Puppet and Ansible my feeling about author's lack of understanding of TF got reinforced.
• rbirkby 4y ago
  “level triggered” vs “edge triggered” is the kubernetes term for stateless vs stateful. It’s one of the reasons why any cluster configuration drift becomes eventually consistent with desired state.
• pixelcort 4y ago
  Can the git repo that the terraform config lives inside of serve as state? For example, one commit ago this part of config was there and now it’s gone, showing temporal intent that it should be deleted in real life as well?
  • kubanczyk 4y ago
    I've played with your idea, hoping to easily demolish it. It is not bullet-proof at all, but I declare it brilliant. It uses simple tools and it does work around the cloud's inability to efficiently query all possible types of resources in existence.
• pharmakom 4y ago
  The article misses a key bit of information TF needs when making a plan: 4. The previous Terraform configuration This is effectively stored by state. We need this because if a resource is removed from the new config then Terraform needs to be able to delete the existing resource from the world. If we don’t have the state then Terraform must either: 1. Not delete it from the world 2. Or risk deleting something not managed by Terraform If everything were managed by Terraform then perhaps we would not need state, but this is not realistic in my view.
  • stingraycharles 4y ago
    The whole idea of the article is that “previous state” is something that can be derived from tags / metadata in the provider’s resources, rather than a standalone file. In other words, this metadata then is the state. Could you give an example of a situation / transition that cannot be captured correctly by managing the state using these types of tags?
    • wernerb 4y ago
      There are plenty of providers/resources that have no such metadata
    • deleted 4y ago
      [deleted]
    • EwanToo 4y ago
      This is the fundamental misconception of the article as far as I understand it. Cloud providers do not provide enough metadata to enable that mapping across all resources.
      • Jasp3r 4y ago
        That cloud providers provide such muddy/inadequate APIs that it is impossible to view the state they are in is a very bold claim to make. Do you have an example to back this up?
• 8organicbits 4y ago
  I haven't found a good way to handle resources that were created outside TF. Like a EC2 instance running in staging to debug an issue, but then forgotten about. They don't exist in TF state or config so TF simply ignores them. I've tried terraformer[1], but I can't tell how well maintained that is (it failed to get my creds, I had to modify the code to fix it, and then it crashed with an obscure error). Anyone have a good approach? 1. https://github.com/GoogleCloudPlatform/terraformer https://github.com/GoogleCloudPlatform/terraformer
• cybice 4y ago
  "None" state store definitely would be good for one of our case - automatic preview deploys for pull requests. Statefull approach works good 99.9% of time but rare errors like inability to destroy resource because of cloud provider issues cause that you need manual interventions - i.e. resource cant be created as not managed by state etc. Even rare errors are not rare when you have 20+ devs :-) That needs some manual fixes i.e import state, killing resources etc but not all devs has access rights or knowledge to do this.
• deleted 4y ago
  [deleted]
• rbongers 4y ago
  Is "remained stateless" accurate? The document only indicates that they used to try storing state in tags, not that it was stateless. "Stateless", to me, would mean that it tries to compare current resources with your current configuration without any metadata. If you mean that they should have tried to keep using tags, not every resource and cloud provider supporting tags pretty much ends the possibility of that. If you mean they should have gone completely stateless, I would say that state is great for detecting configuration drift. If Terraform tries to create a new resource, is it being created for the first time, or was it manually removed? You can only tell with state. Sure, you can get this information in a number of other ways, but one of the major strengths of Terraform is as an easy drift detection tool. I get the desire for state not to exist, but in reality it is essential for what Terraform does.
  • raffraffraff 4y ago
    It's not a drift detection tool for resources that are manually created.
    • rbongers 4y ago
      Are you saying it should? I think whether that's desirable or not is a matter of preference. The nice thing about Terraform is that you don't have to go all in. I don't really want to have to tell it to ignore every resource that I want to manually manage, and I don't want it wasting the extra API calls on trying to find every resource that's not in my configuration every time I want to deploy. Now, if they provided a separate tool for detecting manually created resources, that would be awesome, but it wouldn't fit into the typical flow of showing drift regularly just before deploying in my opinion. That's more about detecting whether you're about to overwrite something that was manually done, or if you're about to make changes assuming your infrastructure is as you last left it. I don't need to take inventory of every single resource I have every time before deploying.
      • raffraffraff 4y ago
        Not suggesting it "should", I was just pointing out that it's not a complete drift detection tool.
• Copenjin 4y ago
  The disclaimer on top is completely unnecessary.
• tragomaskhalos 4y ago
  Terraform has to have state, for the reasons outlined by other commenters. But, it is undoubtedly a pain in the arse in practice - invariably someone else is doing a change on a branch but has already applied it to the development infra using the shared state bucket, which then pollutes it for everyone else as you don't have those changes so your applys will now want to undo them. As a first timer with this stuff, I think a key lesson learned is to balkanise your terraform quite heavily - we have a tfstate per environment but that is nowhere near granular enough, slicing it into smaller pieces would obviate many of those 'pollution' problems.
  • ciceryadam 4y ago
    We use a single a single tf state per Google project. Instead of slicing up the state into multiple substates, we use modules as much as possible, and use "-target=module.module_name" to test our branches.
• benreesman 4y ago
  Given that there are well over 100 comments in this thread it’s possible that someone already mentioned this, but for people who are short-staffed but highly adventurous NixOS has an interesting take on all this. There’s even a Terraform binding called Terranix that isn’t too bad. It’s a real journey and not for people under short-term time pressure, but of all the pain-in-the-ass things I’ve learned in computing over the years it’s paid me back well-above average.
  • sitkack 4y ago
    interesting https://terranix.org/ https://terranix.org/ https://terranix.org/documentation/terranix-vs-hcl/ https://terranix.org/documentation/terranix-vs-hcl/
• franzwong 4y ago
  If you use Terraform, you should only update your infrastructure through Terraform and persist the state in a shared place (e.g. S3 versioning). I see people having hard time when they use both AWS CLI, AWS console and Terraform.
  • michaelsalim 4y ago
    This is my big pain point with Terraform. Sometimes I forgot to update only through Terraform. Especially when it's trivial to do the change through UI but not necessarily the config. I wish Terraform could automatically detect the change and convert it into code.
• robertlagrant 4y ago
  I think the best thesis for this is decentralisation. The stateless approach only works if this one tf file (or set of files) manage(s) the whole world. As soon as you get to multiple teams managing (say) AWS infra, you can't infer that a resource present in infra but not in tf file means a resource should be deleted.