5 ms·

Uh, how do you delete resources with this model? If you don't have any state, and you have an empty module, did you just create it, or did you just remove all

• tbrownaw 4y ago
  Maybe don't scope it to "everything I can see"? A model of "this current module maps to such-and-such container" could work, with the mapping vs the module working vaguely like git remotes vs your code tree.
• c3534l 4y ago
  If its not specified to be there, it shouldn't be there.
  • NovemberWhiskey 4y ago
    That sounds logical, but it's not very practical due to the cost of enumeration; plus, it requires providers to gain deep knowledge into things that are implied dependencies ... like "hrmmm, should I try to delete this service-linked role or do I actually need it?"
  • TameAntelope 4y ago
    The way Terraform works makes that not feasible. As it is, you can introduce Terraform incrementally to your infra, and it won’t wipe away anything it’s not aware of. If you delete everything not explicitly specified, Terraform would have to describe literally everything, which could be a gigantic barrier to entry.
    • takeda 4y ago
      We know that if state is not used we can't handle deletions without affecting everything else. So here's a solution that could satisfy stateless proponents: we don't keep state for anything that we deploy and instead we should store state for everything else so we will know what not to remove /s But seriously, as things are right now, state is necessary, it could be possible to get rid of it, but that would require cloud providers to designate their infrastructure that way. And if they do that, in practice (because components building the cloud service likely are stateful) we would end up with a system with declarative language which in reality just abstracts away the state from us. Kind of what AWS CloudFormation essentially is.
  • pst 4y ago
    This doesn't work unless your infrastructure is entirely static. As soon as you have active control loops like e.g. an autoscaler it will create resources that you don't know about and would then delete.
• shepherdjerred 4y ago
  I'm most familiar with AWS, but I assume Azure and GCP have similar features. Use some tag unique to the project and tag all resources on creation. When deleting resources delete everything not in your stack that has the tag. You can find these resources using the provider's tagging APIs, e.g. https://aws.amazon.com/blogs/aws/new-aws-resource-tagging-api/ https://aws.amazon.com/blogs/aws/new-aws-resource-tagging-ap...
  • NovemberWhiskey 4y ago
    That probably does help; I still feel like you're going to end up sacrificing a lot of the drift-detection capabilities as well though - like unless you know a priori what the default values of every optional parameter are, how do you know whether the values in your resources are "correct"?
  • InvertedRhodium 4y ago
    There are quite a few AWS resources that don't support tags.
• tuckerman 4y ago
  This isn’t about terraform, but in other systems I’ve had to put in tombstones to explicitly turn down a service. This allowed for services to exist that weren’t managed by that partition of state while still allowing for the automation to handle turndowns.
• nathants 4y ago
  deletion is more complicated than creation. what if it’s stateful infrastructure like a db, and contains important state? is it already backed up? would it be expensive to restore? what else aren’t we thinking of. even in a stateful model, deletion of top level, especially stateful, infrastructure should ALWAYS happen later, and maybe not happen at all. creation is easy. deletion is it depends.
  • taeric 4y ago
    Sorta. Creation is only easy if you know what the expected state is. Did you just create a new table, or was it already created? This is especially hard when you /want/ table names that are not stable.
    • jakelazaroff 4y ago
      You mean with regard to renaming, right? If your database has table `foo` and your config has table `bar`, did you mean to create a new table `bar` or rename your existing table `foo` to `bar`?
      • taeric 4y ago
        That, but also just managing that I want a stack to create a new table that I will logically refer to as "foo" in the stack. Outside the stack, I actually /don't/ want anyone referencing the table. Such that I expect the physical name to have a guid or some such. If you do that, it /has/ to keep track of which table it created, versus which were already there. (Right?)
    • nathants 4y ago
      using names to uniquely identify infrastructure is very convenient and how most of aws already works. why not take advantage of that instead mapping name to uuid via state? in what scenario are non-stable infrastructure names useful?
      • taeric 4y ago
• scapecast 4y ago
  That is correct, plus you'd have to also collect their delete dependencies so you can delete resources in the order required.
• GeneralMayhem 4y ago
  I'm in the process of building a tool that's basically Terraform, but for database schemas - you define the tables and columns in something like a proto or JSON, and it will do exactly what this article describes: read the current state of the world, plan out a minimal series of updates (avoiding destructive updates, and keeping dependencies in mind for things like foreign keys), and then apply them. The solutions for deletion with incremental adoption is tombstones. If there's a thing in reality and you don't have a config matching it, ignore it. If you want to delete it, add a special "delete this" block to the config. Once you've fully adopted, then you can flip a flag to "I own the whole world" mode, where the default is to delete anything not found in the canonical configuration. The solution for renames, both before and after full adoption, is similar - if you want to rename X to Y, change the canonical definition to Y, but add a tag saying "when you look at the current state of reality, you might find a thing called X; that's the old name for this, so rename X instead of creating from scratch".
  • scapecast 4y ago
    Literally 5 minutes ago I made a comment on LinkedIn on a Terraform Redshift provider how there's a need for a "Terraform for Analytics Infrastructure", where you define e.g. tables and the column names. And then also include everything that happens before and after the warehouse. I think it would sell like hotcakes.
    • GeneralMayhem 4y ago
      Standardized code generation in general is a huge opportunity. My preferred solution would be for everything to speak protobufs natively, and then you wouldn't need to do any other generation - you'd do what they do internally at Google and have tables with 2 columns (one for the key, and one for a fat protobuf that holds all the actual data), file formats like RecordIO as the default pipeline building block, and Capacitor [1] for columnar storage. But in the absence of good query syntax and columnar file formats that can handle rich data types, code generation it is - take the proto file, flatten it out (this is the tricky part, if you have repeated field names in a nested object model), and then you can generate all kinds of stuff from that: * A table/column schema, which you can automatically synchronize into any DB backend you want via plugins * Read/write logic in various languages - not an ORM, but a struct that represents a single row and handles the boilerplate. * Maybe some kinds of richer query/join logic? If you go too far this becomes another ActiveRecord, but I think there's a middle ground. * Batch pipelines with standard semantics - sort of like a materialized view, but computed via your big-data pipeline of choice rather than in-engine. Imagine that you have table A with 30 fields, table B with 15 fields, and you want to generate a downstream table with all 45. I think it's feasible to have a composable, declarative syntax that lets you create the 45-column table plus the pipeline that populates it with about 3 lines of configuration. Hard to turn into a product, because so much of that pipeline will depend on org-specific tech stack choices, but at the limit the "data platform engineer" could be entirely automated out of 80% of their job (and therefore be able to focus on more interesting things). [1] https://cloud.google.com/blog/products/bigquery/inside-capacitor-bigquerys-next-generation-columnar-storage-format https://cloud.google.com/blog/products/bigquery/inside-capac...
• nailer 4y ago
  Make it so that TF deletes anything not in your config unless whitelisted next question.
  • jen20 4y ago
    Next question: what if your database is managed by a separate team in the same account? Just delete it eh?
    • nailer 4y ago
      Yes. The tool warned you it would do this during the plan phase.
      • jen20 4y ago
        This is absolutely the worst design for a tool imaginable. Any tooling fit for purpose must make peace with the fact that one team, and thus application of a tool, do not own an entire account.
        • nailer 4y ago
          Ultimately if you’re managing your infrastructure as code you must manage your infrastructure as code. Cattle not pets.
  • moondev 4y ago
    What about the opposite which is how kubectl works. "terraform delete" only deletes what is in your config. "terraform apply" only creates and/or updates what is in your config. If you remove a resource from your config after applying it, it's your responsibility to manually delete it.
    • pharmakom 4y ago
      Not having to manually delete is one of the reasons I use Terraform. Managing state is much less painful than that would be.