8 ms·

You can (and always should) decouple the plan and apply steps. For a non terminal approach this can be done by outputting the binary plan to a plan file (with -

You can (and always should) decouple the plan and apply steps. For a non terminal approach this can be done by outputting the binary plan to a plan file (with -out=path) and reviewing the plan terminal output which will show all actions Terraform will perform in human readable format. Then have the plan file be used as input for the apply step. The apply won't perform any other action that was not in the plan file (which matches the output that was reviewed) and if state of the environment it would apply to has changed in the meantime it will abort without causing undesired changes and you can restart the process again. There is also the prevent_destroy[0] meta argument for resources but afaik it has no effect when you remove the resource from your .tf files[1], so it would not have helped in this case. [0] https://www.terraform.io/docs/language/meta-arguments/lifecycle.html#prevent_destroy https://www.terraform.io/docs/language/meta-arguments/lifecy... [1] https://github.com/hashicorp/terraform/issues/17599 https://github.com/hashicorp/terraform/issues/17599

operatorius 5y ago
This! In addition to that things can be made even less error prone. Ive done this using yaml pipeline in azure devops. The plan task can be used to set an output variable which indicates if the generated plan contains any changes. That boolean value is used as a condition to trigger a manual verification task which basically prevents apply running if there are any changes without reviewing it first. As the op mentions, the generated plan is an artifact itself that is used in a following apply task
thedougd 5y ago
I'm not very familiar with Azure anymore, but AWS also offers deletion protection on RDS and other services. This forces a two-phase operation with Terraform. You must first apply configuration to remove the deletion protection, then in the second phase you may delete the RDS instance. Unfortunately, I've never been able to get much mileage from the Terraform prevent_destroy lifecycle option because it can't be set from a variable. Most of my configurations use a module and pass different variable values per each environment. I'd want the lifecycle flag in production, but maybe not dev.
- Arnavion 5y ago
  Yes, Azure has a similar thing for resources where you can set a "CanNotDelete" lock on one. You have to delete the lock before you can delete the resource.
- usr1106 5y ago
  > Unfortunately, I've never been able to get much mileage from the Terraform prevent_destroy lifecycle option because it can't be set from a variable There is a trick around this. Code 2 mostly identical resources, one with prevent_destroy true and the other with prevent_destroy false. Give both resources a count. You can make the count dependent on interpolation. And the count can be zero. So depending on the interpolation result you have resource(s) with prevent_destroy or without. For the "wrong" ones you just create 0 of them. https://stackoverflow.com/questions/53727357/terraform-how-to-get-a-boolean-from-interpolation https://stackoverflow.com/questions/53727357/terraform-how-t... https://stackoverflow.com/questions/53744441/how-to-refer-to-alternate-resources-in-terraform/ https://stackoverflow.com/questions/53744441/how-to-refer-to...
  - terom 5y ago
    That just means that enabling `prevent_destroy` later on will a) destroy the resource b) create a new resource with a different name and `prevent_destroy` enabled. Plus the combinatorial explosion of all possible conditions. Not a good trick.
    - usr1106 5y ago
      Why would I want to enable it later? When I deploy production they ave enable_destroy. It's the same resource all the time and it must not be destroyed. When I want to deploy a test system the resource can be destroyed after I am done. Of course this is not really elegant code. It violates DRY, I need to make sure that my test resources have identical configuration as my production resources, except this one attribute. Maybe your use case is different. I never change the prevent_destroy attribute during the lifetime of the resource.
  - darkwater 5y ago
technological 5y ago
This is exactly what we do also. Apart from prevent destroy, you can also have termination protection enabled on the actual instance. Not sure if azure offers it but aws definitely does
manish_gill 5y ago
What was actually needed here was a Resource Lock on the Resource Group in question.
superduperycomb 5y ago
“I” create dozens of ephemeral terraform deployments for every iaas every day using terraform so I don’t have this luxury. Do you have any suggestions for keeping tabs on automated systems beyond spend anomalies?

operatorius 5y ago

This! In addition to that things can be made even less error prone. Ive done this using yaml pipeline in azure devops. The plan task can be used to set an output variable which indicates if the generated plan contains any changes. That boolean value is used as a condition to trigger a manual verification task which basically prevents apply running if there are any changes without reviewing it first. As the op mentions, the generated plan is an artifact itself that is used in a following apply task

thedougd 5y ago

I'm not very familiar with Azure anymore, but AWS also offers deletion protection on RDS and other services. This forces a two-phase operation with Terraform. You must first apply configuration to remove the deletion protection, then in the second phase you may delete the RDS instance. Unfortunately, I've never been able to get much mileage from the Terraform prevent_destroy lifecycle option because it can't be set from a variable. Most of my configurations use a module and pass different variable values per each environment. I'd want the lifecycle flag in production, but maybe not dev.

Arnavion 5y ago
Yes, Azure has a similar thing for resources where you can set a "CanNotDelete" lock on one. You have to delete the lock before you can delete the resource.
usr1106 5y ago
> Unfortunately, I've never been able to get much mileage from the Terraform prevent_destroy lifecycle option because it can't be set from a variable There is a trick around this. Code 2 mostly identical resources, one with prevent_destroy true and the other with prevent_destroy false. Give both resources a count. You can make the count dependent on interpolation. And the count can be zero. So depending on the interpolation result you have resource(s) with prevent_destroy or without. For the "wrong" ones you just create 0 of them. https://stackoverflow.com/questions/53727357/terraform-how-to-get-a-boolean-from-interpolation https://stackoverflow.com/questions/53727357/terraform-how-t... https://stackoverflow.com/questions/53744441/how-to-refer-to-alternate-resources-in-terraform/ https://stackoverflow.com/questions/53744441/how-to-refer-to...
- terom 5y ago
  That just means that enabling `prevent_destroy` later on will a) destroy the resource b) create a new resource with a different name and `prevent_destroy` enabled. Plus the combinatorial explosion of all possible conditions. Not a good trick.
  - usr1106 5y ago
    Why would I want to enable it later? When I deploy production they ave enable_destroy. It's the same resource all the time and it must not be destroyed. When I want to deploy a test system the resource can be destroyed after I am done. Of course this is not really elegant code. It violates DRY, I need to make sure that my test resources have identical configuration as my production resources, except this one attribute. Maybe your use case is different. I never change the prevent_destroy attribute during the lifetime of the resource.
- darkwater 5y ago

technological 5y ago

This is exactly what we do also. Apart from prevent destroy, you can also have termination protection enabled on the actual instance. Not sure if azure offers it but aws definitely does

manish_gill 5y ago

What was actually needed here was a Resource Lock on the Resource Group in question.

superduperycomb 5y ago

“I” create dozens of ephemeral terraform deployments for every iaas every day using terraform so I don’t have this luxury. Do you have any suggestions for keeping tabs on automated systems beyond spend anomalies?