8 ms·
The article misses a key bit of information TF needs when making a plan: 4. The previous Terraform configuration This is effectively stored by state. We n
by pharmakom 4y ago
The article misses a key bit of information TF needs when making a plan:
4. The previous Terraform configuration
This is effectively stored by state.
We need this because if a resource is removed from the new config then Terraform needs to be able to delete the existing resource from the world. If we don’t have the state then Terraform must either:
1. Not delete it from the world
2. Or risk deleting something not managed by Terraform
If everything were managed by Terraform then perhaps we would not need state, but this is not realistic in my view.
- stingraycharles 4y agoThe whole idea of the article is that “previous state” is something that can be derived from tags / metadata in the provider’s resources, rather than a standalone file. In other words, this metadata then is the state. Could you give an example of a situation / transition that cannot be captured correctly by managing the state using these types of tags?
- wernerb 4y agoThere are plenty of providers/resources that have no such metadata
- deleted 4y ago[deleted]
- EwanToo 4y agoThis is the fundamental misconception of the article as far as I understand it. Cloud providers do not provide enough metadata to enable that mapping across all resources.
- Jasp3r 4y agoThat cloud providers provide such muddy/inadequate APIs that it is impossible to view the state they are in is a very bold claim to make. Do you have an example to back this up?
- ntoskrnl 4y agoNo DNS provider I've ever used has a way to store metadata alongside a DNS record (like an individual A record)
- paulgb 4y agoI was just about to give the same example. I’m more familiar with GCP’s DNS, but I don’t see a place for structured metadata in Route 53 either. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-route53-recordset.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
- georgyo 4y agoWhy do I need metadata along with my A record? It either exists or it doesn't. That is the state required for Terraform. The article explicitly mentions OctoDNS as a stateless configuration management system for DNS as a good solution.
- KyeRussell 4y agoTo know if it’s managed by TF or not, to know whether or not to delete it. Exactly what’s being described.
- NateEag 4y agoThat's what TXT records are for: https://en.m.wikipedia.org/wiki/TXT_record https://en.m.wikipedia.org/wiki/TXT_record
- tomgouder 4y agoI agree that state can be removed in Terraform. But I do not believe leveraging tags and/or metadata is the right approach - configuration for these resources could potentially be large (e.g. GKE resources) and most providers will have a size constraint on their metadata and tag values. Creating a metadata/tag key for each configuration key would also get messy, but solves the value size problem. Why wouldn’t it be possible to not store the previous state at all? Terraform’s job is to reconcile what exists with what is declared - we should be able to rely on the provider’s APIs to understand what exists, perform the diff, and reconcile the changes.
- chrisweekly 4y ago"should be able to rely on the provider’s APIs to understand what exists" is the crux of it. Provider APIs, in praxis, are insufficient / impractical.
- ntoskrnl 4y agoHow would TF work without state and without tags? Let's say your TF config declares a database Foo. Your AWS account has databases Foo and Bar. How does TF know whether it's responsible for database Bar and whether to delete it?
- Aeolun 4y agoResource is tagged with “terraform stack x created this”?
- cassianoleal 4y agoInteresting. Disregarding for a moment that not all cloud resources supporting tags on all cloud providers (and let's not get started with all the non-cloud stuff Terraform is capable to manage), how does tf know that `stack x` is the thing that's currently running?
- gtirloni 4y agoBy the time the tag has enough information to be useful, you are storing a TF state file in a tag. Anyway, I don't think people complaining about a s3 bucket for tf state actually has experience creating tools like tf.
- pharmakom 4y agoSo the position is Terraform should have state, but rather than keep it in one place such as an S3 object, it should be spread across the metadata of many resources? Some issues with that: - Fetching the whole state would be hugely impractical due to the number of API calls - The risk of losing state information by a resource being deleted outside of Terraform is greater - Again, not all resources have metadata that could be used to store state
- deleted 4y ago[deleted]
- usrn 4y ago>- The risk of losing state information by a resource being deleted outside of Terraform is greater This isn't losing state information though! That is the state. If state information were kept outside this it would now be wrong which means terraform *would do the wrong thing.
- kkdaemas 4y agoNo because the metadata on the deleted resource is now lost. With the information being stored outside the resource, we know that it was deleted and the metadata about it.
- usrn 4y agoThat's not state information though, that's metadata.
- pharmakom 4y agoIf some state information is stored using metadata then we need that metadata to know the total state. The contents of TF state is more than what can be read from the provider APIs often.
- mountainriver 4y ago
- baobob 4y agoWhere would the metadata for the local_file resource be stored? How would Terraform find files it once created and must now delete?
- Too 4y agoSame famous problem for any kind of make-based build system, removing targets usually leave old files in your build directory. IIRC Bazel can avoid it but don’t know how, either it must keep a list of old files or compare the entirety of build directory to the build-graph.
- aequitas 4y ago> The whole idea of the article is that “previous state” is something that can be derived from tags / metadata in the provider’s resources Except for the Cloud's and API's that don't provide them but we stil need configuration managed. This is the world we live in, and in that world Terraform is the solution to the problems that we seek. In an ideal world Terraform would not be needed, but we don't live in that world.
- grumple 4y ago> Could you give an example of a situation / transition that cannot be captured correctly by managing the state using these types of tags? Tags do not solve the deletion issue. It would require two step deployments, for example by adding delete = true to the config, applying, then removing the resource from the config entirely and applying again. But I don't think that's too bad tbh.
- nanis 4y agoHow does that work when terraforming Artifactory?
- deleted 4y ago[deleted]
- hamandcheese 4y agoSo state, but probably way slower and varies from cloud to cloud.
- enobrev 4y agoThat's still storing state, except instead of keeping it within a single file which you have full control over, you're instead sprinkling that state all over your infrastructure and then hoping it doesn't get mangled in between invocations. And also hoping that service's API has all the tools needed to find your scattered state within a reasonable amount of time in order to diff any changes you make in your declarations
- mountainriver 4y agoYeah but then you don’t have to worry about the state getting out of sync which is one of the biggest problems. Also versioning is a pain
- ImPostingOnHN 4y agoyeah, but then you DO have to worry about scanning every single resource you control, every time you plan and run
- hellcow 4y agoI built pretty much the system you’re describing for my company (a stateless terraform alternative) and this scan that you cite as a negative happens in… 100ms in parallel? Roughly the same amount of time it would take to download a state file? Dunno about you but ensuring state is always accurate and in-sync is well worth the trade-off to me.
- enobrev 4y agoI'm curious, and not antagonistically, about the size and variability of the infrastructure you're working with, on how many platforms it's running, and how many people are in charge of it. Is this tool private or available for us to try out?
- hellcow 4y agoIt’s small, under 50 servers running OpenBSD and Linux under a wide variety of configurations (including gpu), 6 databases, and Redis. Though the system is easily extensible to other clouds and resource types, it only needs to work on GCP right now. Been running it in prod for the past 3 or 4 years—no outages, no downtime, no surprises. Doubled infra over the holidays then scaled everything back with no issue. The stateless approach has worked really well for us.
- rorychatt 4y agoWe use Terraform extensively at our organisation. Some examples come to mind that make this impractical: - For services that do support tags, we are already reaching limits on the number of tags that can be associated with a single resource. For example, in Azure, some resources still only support 10 unique key/values - Drift detection against write, but no read secrets mean that you cannot do drift detection over certificates, and secrets. Depending on your organisation and how they manage things like PKI, this may be impractical to track validity of the endpoint. - Many services we manage don't have tags. For example, we use Terraform to manage Github Repositories, Actions, AzDo Pipelines, and Permissions - Some object types simply don't have primary keys that are easily searchable by the provider, and requires some sort of composite key to be compiled and tracked. State gives us a common schema and playing field to significantly simply the generation of dependency graphs and show drift. I imagine that even without a 'statefile', you would end up having to generate a similar graph in memory anyway.
- devit 4y agoOr, you know, put an UUID in the configuration and tag everything with it so that it knows what it created by just querying the tag (or similarly, using a namespace if supported, or putting the information in the resource name itself if possible).
- vlunkr 4y agoThese arguments are just going in circles but here's why: You can't store metadata on every object in every provider, and often, resource names are restrictive in length and allowed characters. And what if something needs deleted? Now every time you run terraform you need to list every type of object the provider offers and check tags, that's a huge waste of time and bandwidth.
- ElectricalUnion 4y ago> UUID in the configuration If you turn the configuration into state, you don't need state anymore I guess?
- runlevel1 4y agoIn Chef, you have to change the resource to `action :remove` if you want it deleted. Couldn't you do something like that?
- zoul 4y agoYou could, but then you are losing what I think is the main point of TF – describing what the infra should look like and letting the tool figure out what the necessary steps are for reaching that state.
- runlevel1 4y agoIt can still be declarative for the things that it manages. Perhaps Puppet would have been the better analogy (i.e. `ensure => absent`). While Terraform providers can be built to do a lot of implicit work, in practice they don't. In my experience, you still have to specify all the pieces. The only place I can think of where auto-deletion came in handy was for infrastructure CI. However, that CI pipeline doesn't catch many critical issues, and it's very expensive to operate. So I can't say it's been a total win.
- jon_adler 4y agoOne of terraforms key design principles is that it is declarative whereas this instruction is imperative.
- mountainriver 4y agoI think the trade offs of a stateless terraform would be favored by many
- funcDropShadow 4y agoThen use ansible, or some other stateless tool in the same space.
- barefeg 4y agoYou can still store state but delegate it to a different system. One that is more robust and homogeneous like a kubernetes API. This is what crossplane does
- mountainriver 4y agoTerraform can use k8s as a state store. The issue is two sources of truth and everything that causes
- Faaak 4y agoTF noob here: is it not possible to store a "managed by terraform" annotation on the resources ? Or are not all resources able to store labels (extreme example: terraform-spotify plugin)
- FooBarWidget 4y agoMany resources don't support abitrary metadata.
- pharmakom 4y agoIt is possible for resources that support tags, but not everything supports tags.