7 ms·
The whole idea of the article is that “previous state” is something that can be derived from tags / metadata in the provider’s resources, rather than a standalo
by stingraycharles 4y ago
The whole idea of the article is that “previous state” is something that can be derived from tags / metadata in the provider’s resources, rather than a standalone file. In other words, this metadata then is the state.
Could you give an example of a situation / transition that cannot be captured correctly by managing the state using these types of tags?
- wernerb 4y agoThere are plenty of providers/resources that have no such metadata
- deleted 4y ago[deleted]
- EwanToo 4y agoThis is the fundamental misconception of the article as far as I understand it. Cloud providers do not provide enough metadata to enable that mapping across all resources.
- Jasp3r 4y agoThat cloud providers provide such muddy/inadequate APIs that it is impossible to view the state they are in is a very bold claim to make. Do you have an example to back this up?
- ntoskrnl 4y agoNo DNS provider I've ever used has a way to store metadata alongside a DNS record (like an individual A record)
- paulgb 4y agoI was just about to give the same example. I’m more familiar with GCP’s DNS, but I don’t see a place for structured metadata in Route 53 either. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-route53-recordset.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
- georgyo 4y agoWhy do I need metadata along with my A record? It either exists or it doesn't. That is the state required for Terraform. The article explicitly mentions OctoDNS as a stateless configuration management system for DNS as a good solution.
- KyeRussell 4y agoTo know if it’s managed by TF or not, to know whether or not to delete it. Exactly what’s being described.
- NateEag 4y agoThat's what TXT records are for: https://en.m.wikipedia.org/wiki/TXT_record https://en.m.wikipedia.org/wiki/TXT_record
- gtirloni 4y agoExposed to the world?
- NateEag 4y agoI guess that's not ideal, though I'm not clear what attack surface area is increased by storing creation/ deletion metadata in public. I guess it lets an attacker know that you're using Terraform, which might help them target their attacks.
- ntoskrnl 4y agoTerraform can be used to manage TXT records too. Where does it store metadata for them? Or is it TXT all the way down?
- NateEag 4y agoYou can have multiple TXT records for a given domain name, so it would be possible to store an arbitrary amount of metadata for whatever systems you desire, and just loop through the TXT records to figure out which ones are for the current system's purpose.
- deleted 4y ago[deleted]
- rwmcfa1 4y agoIf you look through the octoDNS providers there's a number of cases where extra info is stored for "dynamic" records. The metadata is often things like the pool name or rule number. In other cases it's details about the health check config/version. The extra info is sometimes stored into a "notes" field, other times it's encoded into the ID or similar. It's true that nothing extra is needed for simple/standard records, but once you start doing GeoDNS, failover, health check, etc. it's required. In all cases thus far we've been able to find a way to store/indicate whatever we need. (maintainer of octoDNS)
- orf 4y agoYou can create an s3 object in a bucket with terraform. There isn’t a way to go from “tag -> key”.
- moondev 4y agohttps://github.com/nat-henderson/terraform-provider-dominos https://github.com/nat-henderson/terraform-provider-dominos terraform is more than just cloud providers https://registry.terraform.io/browse/providers https://registry.terraform.io/browse/providers
- noirbot 4y agoI'm amused that you think "cloud providers have bad APIs" is a bold claim instead of the default state of things. Things as basic as IAM Groups in AWS aren't taggable: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_group https://registry.terraform.io/providers/hashicorp/aws/latest... IAM Users are taggable, but to get the tags on a given user, you must request them one user at a time from a known list of users. The "List all users" call doesn't return their tags. Obviously this is less of an issue for the TF state use case, but does add to the API call overhead for any tag-based approach.
- tedivm 4y agoMy favorite is that you can look up some resources by tag directly, but other resources have to use the special resource tagging api. The AWS API also silently fails, on purpose, for some things- try looking up quotas for Organizations as an example and you'll end up with an empty array instead of the actual quotas. Cloud providers having bad APIs is definitely the default state.
- tomgouder 4y agoI agree that state can be removed in Terraform. But I do not believe leveraging tags and/or metadata is the right approach - configuration for these resources could potentially be large (e.g. GKE resources) and most providers will have a size constraint on their metadata and tag values. Creating a metadata/tag key for each configuration key would also get messy, but solves the value size problem. Why wouldn’t it be possible to not store the previous state at all? Terraform’s job is to reconcile what exists with what is declared - we should be able to rely on the provider’s APIs to understand what exists, perform the diff, and reconcile the changes.
- chrisweekly 4y ago"should be able to rely on the provider’s APIs to understand what exists" is the crux of it. Provider APIs, in praxis, are insufficient / impractical.
- ntoskrnl 4y agoHow would TF work without state and without tags? Let's say your TF config declares a database Foo. Your AWS account has databases Foo and Bar. How does TF know whether it's responsible for database Bar and whether to delete it?
- Aeolun 4y agoResource is tagged with “terraform stack x created this”?
- cassianoleal 4y agoInteresting. Disregarding for a moment that not all cloud resources supporting tags on all cloud providers (and let's not get started with all the non-cloud stuff Terraform is capable to manage), how does tf know that `stack x` is the thing that's currently running?
- gtirloni 4y agoBy the time the tag has enough information to be useful, you are storing a TF state file in a tag. Anyway, I don't think people complaining about a s3 bucket for tf state actually has experience creating tools like tf.
- pharmakom 4y agoSo the position is Terraform should have state, but rather than keep it in one place such as an S3 object, it should be spread across the metadata of many resources? Some issues with that: - Fetching the whole state would be hugely impractical due to the number of API calls - The risk of losing state information by a resource being deleted outside of Terraform is greater - Again, not all resources have metadata that could be used to store state
- deleted 4y ago[deleted]
- usrn 4y ago>- The risk of losing state information by a resource being deleted outside of Terraform is greater This isn't losing state information though! That is the state. If state information were kept outside this it would now be wrong which means terraform *would do the wrong thing.
- kkdaemas 4y agoNo because the metadata on the deleted resource is now lost. With the information being stored outside the resource, we know that it was deleted and the metadata about it.
- usrn 4y agoThat's not state information though, that's metadata.
- pharmakom 4y agoIf some state information is stored using metadata then we need that metadata to know the total state. The contents of TF state is more than what can be read from the provider APIs often.
- mountainriver 4y ago
- baobob 4y agoWhere would the metadata for the local_file resource be stored? How would Terraform find files it once created and must now delete?
- Too 4y agoSame famous problem for any kind of make-based build system, removing targets usually leave old files in your build directory. IIRC Bazel can avoid it but don’t know how, either it must keep a list of old files or compare the entirety of build directory to the build-graph.
- aequitas 4y ago> The whole idea of the article is that “previous state” is something that can be derived from tags / metadata in the provider’s resources Except for the Cloud's and API's that don't provide them but we stil need configuration managed. This is the world we live in, and in that world Terraform is the solution to the problems that we seek. In an ideal world Terraform would not be needed, but we don't live in that world.
- grumple 4y ago> Could you give an example of a situation / transition that cannot be captured correctly by managing the state using these types of tags? Tags do not solve the deletion issue. It would require two step deployments, for example by adding delete = true to the config, applying, then removing the resource from the config entirely and applying again. But I don't think that's too bad tbh.
- nanis 4y agoHow does that work when terraforming Artifactory?
- deleted 4y ago[deleted]
- hamandcheese 4y agoSo state, but probably way slower and varies from cloud to cloud.
- enobrev 4y agoThat's still storing state, except instead of keeping it within a single file which you have full control over, you're instead sprinkling that state all over your infrastructure and then hoping it doesn't get mangled in between invocations. And also hoping that service's API has all the tools needed to find your scattered state within a reasonable amount of time in order to diff any changes you make in your declarations
- mountainriver 4y agoYeah but then you don’t have to worry about the state getting out of sync which is one of the biggest problems. Also versioning is a pain
- ImPostingOnHN 4y agoyeah, but then you DO have to worry about scanning every single resource you control, every time you plan and run
- hellcow 4y agoI built pretty much the system you’re describing for my company (a stateless terraform alternative) and this scan that you cite as a negative happens in… 100ms in parallel? Roughly the same amount of time it would take to download a state file? Dunno about you but ensuring state is always accurate and in-sync is well worth the trade-off to me.
- enobrev 4y agoI'm curious, and not antagonistically, about the size and variability of the infrastructure you're working with, on how many platforms it's running, and how many people are in charge of it. Is this tool private or available for us to try out?
- hellcow 4y agoIt’s small, under 50 servers running OpenBSD and Linux under a wide variety of configurations (including gpu), 6 databases, and Redis. Though the system is easily extensible to other clouds and resource types, it only needs to work on GCP right now. Been running it in prod for the past 3 or 4 years—no outages, no downtime, no surprises. Doubled infra over the holidays then scaled everything back with no issue. The stateless approach has worked really well for us.
- rorychatt 4y agoWe use Terraform extensively at our organisation. Some examples come to mind that make this impractical: - For services that do support tags, we are already reaching limits on the number of tags that can be associated with a single resource. For example, in Azure, some resources still only support 10 unique key/values - Drift detection against write, but no read secrets mean that you cannot do drift detection over certificates, and secrets. Depending on your organisation and how they manage things like PKI, this may be impractical to track validity of the endpoint. - Many services we manage don't have tags. For example, we use Terraform to manage Github Repositories, Actions, AzDo Pipelines, and Permissions - Some object types simply don't have primary keys that are easily searchable by the provider, and requires some sort of composite key to be compiled and tracked. State gives us a common schema and playing field to significantly simply the generation of dependency graphs and show drift. I imagine that even without a 'statefile', you would end up having to generate a similar graph in memory anyway.