6 ms·

Here's my #1 tip, most important: Try to keep your stateful resources / services in different "stacks" than your stateful things. Absolutely 100% completely o

Here's my #1 tip, most important: Try to keep your stateful resources / services in different "stacks" than your stateful things. Absolutely 100% completely obvious, maybe too obvious? Because none of these guides ever mention it. If you have state in a stack it becomes 10x more expensive and difficult to "replace your way out of trouble" (aka destroy and recreate as last resort). You want as much as possible in stateless, disposable stacks. DONT put that customer exposed bucket or DB in the same state/stack as app server resources. I don't care about your folder structure, I care about what % of the infra I can reliably burn to the ground and replace using pipelines and no manual actions.

sverhagen 3y ago
Is a "stack" here a (root) folder on which you'd do a "terraform apply"? I've never know what to call those, surely they aren't "modules". And, so, you're saying: try to have a separate deployment (stack then?) that contains the state, so you can wipe away everything else if you want to, without having to manage the state?
- xyzzy123 3y ago
  It's not exactly about the folder, the IaC from a single folder / project can be instantiated in multiple places. Each time you do that, it has a unique state file, so I usually hear it referred to as a "state". In cfn you can similarly deploy the same thing lots of times and each instantiation is called a "stack", so stack/state tend to get used inter-changeably. And yes, that's a succinct rephrasing. When you first use iac it maybe seems logical to put your db and app server in the same "thing" (stack or state file) but now that thing is "pet like" and you have to take care of it forever. You can't safely have a "destroy" action in your pipeline as a last resort. If you put the stateful stuff in a separate stack you can freely modify the things in the stateless one with much less worry.
  - swozey 3y ago
    Can you elaborate on this? I've never heard of this IAC structure and I'm trying to figure out what the benefit/cons are. Maybe it's just Friday and I'm checked out already. If you run a terraform apply and only update microservices but you also have your dbs/stateful things in the same stack/app, you're only updating the microservices so how would this affect the db/stateful at all? On the opposite end - I feel like there would be scenarios where I needed to update the stateful AND stateless services with the same terraform apply. Maybe I'm adding a new cluster and adding a db region/replica/securitygroup and that new cluster needs to point at the new db region. In your scenario I would have updated microservices trying to reach out to a db in a region that doesn't exist yet because I have to terraform apply two different stacks. How would you deal with a depends_on? Maybe I'm misunderstanding this.
    - rcrowley 3y ago
      (Hi, I’m one of the authors of the article at the root of this thread.) Considering your hypothetical stateless microservice change in the same root module as stateful services, problems arise when _someone else_ has merged changes that concern the stateful services, leaving you little room to apply your changes individually. It’s also worth remembering that, even if a stateless service and a stateful service are managed in the same root module, applying changes is absolutely not atomic. Applying tightly coupled changes to two services “at the same time” is likely to result in brief service interruptions, even if everything returns to normal as soon as the whole changeset is applied.
      - swozey 3y ago
        Ok I think we're talking about two separate things here - you're referencing a root module and not a "stack", as in a stack is a full service/application that uses multiple modules to deploy. Your db module, eks module, etc. All independent modules, not combined into one singular module. Say it's sitting in the /terraform/app1/services/db(&)app folders type of scenario. I think you're talking about putting stateful and stateless objects inside of a single module. So you've got /terraform/modules/mybigapp/main.tf that has your microservice + database inside of it. If I'm right and that's what you mean that's really interesting I don't think I've ever seen or done that but now I'm curious. I'm pretty sure I've never created an "app1" module with all of its resources. Am I totally off here?
  - waffletower 3y ago
    That's right, stacks can be instantiated across repos even depending upon the organization (both meanings of 'organization' are valid here).
  - pharmakom 3y ago
    what about prevent_destroy lifecycle?
- GauntletWizard 3y ago
  I have adopted the term "Root Module" vs "Submodule" because those line up with terraform's own definitions, but I agree that they're terribly, terribly named.
- raffraffraff 3y ago
  I have the same issue. They're all modules, but the ones at the tip of the directory tree (right at the end of your env/region/stack) are called root modules. Which makes no sense because the term "root" always implies that they are at the beginning, not the tippy-toe end. So I call mine "stacks". But as another answer suggested, "states" is also fine. Even though the actual state isn't inside that directory, it's probably in an object store. At the end of the day I don't care what other people call them.
robertlagrant 3y ago
Makes sense, but how do you connect the two so e.g. credentials from one are surfaced in the other?
- dharmab 3y ago
  Use Data Sources to reference resources in a different state: https://developer.hashicorp.com/terraform/language/data-sources https://developer.hashicorp.com/terraform/language/data-sour...
  - robertlagrant 3y ago
    Ah - I didn't think about that. Wrapping a remote call with a data source. Thanks.
- deleted 3y ago
  [deleted]
- paulddraper 3y ago
  terraform_remote_state The root module can have outputs just like any other module. These outputs can be accessed from other stacks from the backend. And if you use CDKTF the references are handled transparently.
  - robertlagrant 3y ago
    Thanks - another good option! I seem to get slight shivers down my spine when I read terraform_remote_state, but I never actually used it, so I might be half-remembering an old opinionated blog post.
- no_circuit 3y ago
  IMO you shouldn't be storing credentials in shared state, as suggested by the other comments, since that means that the principals able to read the state to deploy their service can also read the credentials for other services bundled in that state file. This could be the case if one had broken down the root modules into scopes/services like the linked page suggests. It is reasonable to assume if you are using Terraform to manage your infra, than your infra likely has access to a secrets manager from your infra vendor, e.g., AWS. Instead I'd recommend using a Terraform data resource to pull a credential from the secret manager by name -- and the name doesn't even necessarily have to be communicated through Terraform state. Then the credentials could directly be fed into where it is needed, e.g., a resource like a Kubernetes Secret. One can even skip this whole thing if the service can use the secret manager api itself. Finally access to the credentials itself would be locked down with IAM/RBAC.
hansoolo 3y ago
Just stopped here because I had said XYZZY way too often in the last three hours xD
coryrc 3y ago
You mean keep stateless separate from stateful? Everyone else seems to be reading over the typo or I'm more confused than I thought.
- wahnfrieden 3y ago
  Yes
thwway23432 3y ago
The "stack" nomenclature used here is jarring since it is unrepresented in Terraform HCL literature. A CDK stack, (assuming that's what is used here), would be loosely equivalent to a Terraform HCL module.
harha_ 3y ago
I've never used terraform, but I have used CloudFormation and AWS CDK. It's been a while though, is there a clear indication on the major cloud provider docs which resources are stateful? Or is it always obvious?
- tyingq 3y ago
  Difficult question, as people mean different things when they say state. One example might be a relatively simple AWS Lambda. Most people would say that's easily stateless. But, what if that Lambda depends on a VPC with some specific networking config to allow it to connect to some partner company private network? And, it's difficult to recreate that VPC without service disruption for a variety of reasons that are out of your control. Well, now you have state because you need to track which existing VPC the Lambda needs if you tear the Lambda down and recreate it.
  - giaour 3y ago
    Usually people use "state" to mean that the resource accumulates data. E.g., a VPC is stateless, but the s3 bucket containing flow logs for the VPC is not. Tearing down and recreating a VPC can be disruptive, but that doesn't make it a stateful resource.
    - tyingq 3y ago
      In my example, the "state" part is someone storing the specific VPC reference to re-apply to a newly re-created lambda. Rather than recreating both the VPC and the Lambda. Many people would refer to that as a bad practice because it's carrying state. e.g., the kind of stuff that terraform puts in .tfstate files. There's definitely a purist camp that advocates completely stateless infra. Though there's also what you're describing, where some people consider state to only refer to things like databases.

sverhagen 3y ago

Is a "stack" here a (root) folder on which you'd do a "terraform apply"? I've never know what to call those, surely they aren't "modules". And, so, you're saying: try to have a separate deployment (stack then?) that contains the state, so you can wipe away everything else if you want to, without having to manage the state?

xyzzy123 3y ago
It's not exactly about the folder, the IaC from a single folder / project can be instantiated in multiple places. Each time you do that, it has a unique state file, so I usually hear it referred to as a "state". In cfn you can similarly deploy the same thing lots of times and each instantiation is called a "stack", so stack/state tend to get used inter-changeably. And yes, that's a succinct rephrasing. When you first use iac it maybe seems logical to put your db and app server in the same "thing" (stack or state file) but now that thing is "pet like" and you have to take care of it forever. You can't safely have a "destroy" action in your pipeline as a last resort. If you put the stateful stuff in a separate stack you can freely modify the things in the stateless one with much less worry.
- swozey 3y ago
  Can you elaborate on this? I've never heard of this IAC structure and I'm trying to figure out what the benefit/cons are. Maybe it's just Friday and I'm checked out already. If you run a terraform apply and only update microservices but you also have your dbs/stateful things in the same stack/app, you're only updating the microservices so how would this affect the db/stateful at all? On the opposite end - I feel like there would be scenarios where I needed to update the stateful AND stateless services with the same terraform apply. Maybe I'm adding a new cluster and adding a db region/replica/securitygroup and that new cluster needs to point at the new db region. In your scenario I would have updated microservices trying to reach out to a db in a region that doesn't exist yet because I have to terraform apply two different stacks. How would you deal with a depends_on? Maybe I'm misunderstanding this.
  - rcrowley 3y ago
    (Hi, I’m one of the authors of the article at the root of this thread.) Considering your hypothetical stateless microservice change in the same root module as stateful services, problems arise when _someone else_ has merged changes that concern the stateful services, leaving you little room to apply your changes individually. It’s also worth remembering that, even if a stateless service and a stateful service are managed in the same root module, applying changes is absolutely not atomic. Applying tightly coupled changes to two services “at the same time” is likely to result in brief service interruptions, even if everything returns to normal as soon as the whole changeset is applied.
    - swozey 3y ago
      Ok I think we're talking about two separate things here - you're referencing a root module and not a "stack", as in a stack is a full service/application that uses multiple modules to deploy. Your db module, eks module, etc. All independent modules, not combined into one singular module. Say it's sitting in the /terraform/app1/services/db(&)app folders type of scenario. I think you're talking about putting stateful and stateless objects inside of a single module. So you've got /terraform/modules/mybigapp/main.tf that has your microservice + database inside of it. If I'm right and that's what you mean that's really interesting I don't think I've ever seen or done that but now I'm curious. I'm pretty sure I've never created an "app1" module with all of its resources. Am I totally off here?
- waffletower 3y ago
  That's right, stacks can be instantiated across repos even depending upon the organization (both meanings of 'organization' are valid here).
- pharmakom 3y ago
  what about prevent_destroy lifecycle?
GauntletWizard 3y ago
I have adopted the term "Root Module" vs "Submodule" because those line up with terraform's own definitions, but I agree that they're terribly, terribly named.
raffraffraff 3y ago
I have the same issue. They're all modules, but the ones at the tip of the directory tree (right at the end of your env/region/stack) are called root modules. Which makes no sense because the term "root" always implies that they are at the beginning, not the tippy-toe end. So I call mine "stacks". But as another answer suggested, "states" is also fine. Even though the actual state isn't inside that directory, it's probably in an object store. At the end of the day I don't care what other people call them.

robertlagrant 3y ago

Makes sense, but how do you connect the two so e.g. credentials from one are surfaced in the other?

dharmab 3y ago
Use Data Sources to reference resources in a different state: https://developer.hashicorp.com/terraform/language/data-sources https://developer.hashicorp.com/terraform/language/data-sour...
- robertlagrant 3y ago
  Ah - I didn't think about that. Wrapping a remote call with a data source. Thanks.
deleted 3y ago
[deleted]
paulddraper 3y ago
terraform_remote_state The root module can have outputs just like any other module. These outputs can be accessed from other stacks from the backend. And if you use CDKTF the references are handled transparently.
- robertlagrant 3y ago
  Thanks - another good option! I seem to get slight shivers down my spine when I read terraform_remote_state, but I never actually used it, so I might be half-remembering an old opinionated blog post.
no_circuit 3y ago
IMO you shouldn't be storing credentials in shared state, as suggested by the other comments, since that means that the principals able to read the state to deploy their service can also read the credentials for other services bundled in that state file. This could be the case if one had broken down the root modules into scopes/services like the linked page suggests. It is reasonable to assume if you are using Terraform to manage your infra, than your infra likely has access to a secrets manager from your infra vendor, e.g., AWS. Instead I'd recommend using a Terraform data resource to pull a credential from the secret manager by name -- and the name doesn't even necessarily have to be communicated through Terraform state. Then the credentials could directly be fed into where it is needed, e.g., a resource like a Kubernetes Secret. One can even skip this whole thing if the service can use the secret manager api itself. Finally access to the credentials itself would be locked down with IAM/RBAC.

hansoolo 3y ago

Just stopped here because I had said XYZZY way too often in the last three hours xD

coryrc 3y ago

You mean keep stateless separate from stateful? Everyone else seems to be reading over the typo or I'm more confused than I thought.

wahnfrieden 3y ago
Yes

thwway23432 3y ago

The "stack" nomenclature used here is jarring since it is unrepresented in Terraform HCL literature. A CDK stack, (assuming that's what is used here), would be loosely equivalent to a Terraform HCL module.

harha_ 3y ago

I've never used terraform, but I have used CloudFormation and AWS CDK. It's been a while though, is there a clear indication on the major cloud provider docs which resources are stateful? Or is it always obvious?

tyingq 3y ago
Difficult question, as people mean different things when they say state. One example might be a relatively simple AWS Lambda. Most people would say that's easily stateless. But, what if that Lambda depends on a VPC with some specific networking config to allow it to connect to some partner company private network? And, it's difficult to recreate that VPC without service disruption for a variety of reasons that are out of your control. Well, now you have state because you need to track which existing VPC the Lambda needs if you tear the Lambda down and recreate it.
- giaour 3y ago
  Usually people use "state" to mean that the resource accumulates data. E.g., a VPC is stateless, but the s3 bucket containing flow logs for the VPC is not. Tearing down and recreating a VPC can be disruptive, but that doesn't make it a stateful resource.
  - tyingq 3y ago
    In my example, the "state" part is someone storing the specific VPC reference to re-apply to a newly re-created lambda. Rather than recreating both the VPC and the Lambda. Many people would refer to that as a bad practice because it's carrying state. e.g., the kind of stuff that terraform puts in .tfstate files. There's definitely a purist camp that advocates completely stateless infra. Though there's also what you're describing, where some people consider state to only refer to things like databases.