6 ms·

Hashicorp's stuff always struck me as pretty hacky with awkward design decisions. For Terraform (at least a few years ago) a badly reviewed PR could cause catas

Hashicorp's stuff always struck me as pretty hacky with awkward design decisions. For Terraform (at least a few years ago) a badly reviewed PR could cause catastrophic data loss because resources are deleted without requiring an explicit tombstone. Then they did the license change, which didn't reflect well on them. Now it's being sold to IBM, which is essentially a consulting company trying to pivot to mostly undifferentiated software offerings. So I guess Hashicorp is basically over. I suspect the various forks will be used for a while.

JohnMakin 2y ago
> For Terraform (at least a few years ago) a badly reviewed PR could cause catastrophic data loss because resources are deleted without requiring an explicit tombstone. There have been lifecycle rules in place for as long as I can remember to prevent stuff like this. I'm not sure this is a "problem" unique to terraform.
- ants_everywhere 2y ago
  What happens if you forget the lifecycle annotations or put them in the wrong place or you accidentally delete them? Last time I checked it was data loss, but that was a few years ago.
  - anonfordays 2y ago
    "What happens if I turn a table saw on and start breakdancing on it?" Of course you're going to hurt yourself. If you didn't put lifecycle blocks on your production resources, you weren't organizationally mature enough to be using Terraform in production. Take an associate Terraform course, this specific topic is covered in it.
    - dpkirchner 2y ago
      I'm not familiar with every lifecycle argument but I don't know of any that prevent resources being destroyed if they are removed from the tf file (what the parent was talking about). prevent_destroy, per docs, only applies as long as the resource is defined. I think the only way to avoid accidentally destroying a resource is to refer to it somewhere else, like in a depends_on array. At least that would block the plan.
      - anonfordays 2y ago
        >I don't know of any that prevent resources being destroyed if they are removed from the tf file (what the parent was talking about). Azure Locks (which you can also manage with Terraform), Open Policy Agent, Sentinel rules, etc. will prevent a destroy even if you remove the definition from your Terraform codebase. Again, if you're not operationally mature enough, the problem isn't the tool, it's you.
        deleted 2y ago
        [deleted]
        regularfry 2y ago
        "Operationally mature" is code here for "the gun starts out loaded and pointed at your foot". It's fine to point out that that's a suboptimal design for a tool.
        anonfordays 2y ago
        >Operationally mature" is code here for "the gun starts out loaded and pointed at your foot" No, it's code for "don't build a load bearing bridge if you don't understand structural engineering." > It's fine to point out that that's a suboptimal design for a tool. This isn't "suboptimal" though. If you delete a stored procedure in your RDBMS and it causes an outage, it's not because SQL/PostgreSQL is suboptimal. Similarly if you accidentally delete files from your file system, it's not because file systems are "suboptimal". It's because you weren't operationally mature enough to have proper testing and backups in place.
        regularfry 2y ago
        Defaults matter. I don't know how else to express that a system which fails safe in the face of fallible humans is strictly better, all else being equal, than the alternative.
        anonfordays 2y ago
  - JohnMakin 2y ago
    The same as in any other language when what you wrote was not what you intended? Sorry, I’m really confused what your complaint here is or how you’d prefer it to work. If you make a sensitive resource managed by any kind of IAC, of course the IAC can destroy it in a manner that would result in irretrievable data loss. The language has for forever put semantics in place to prevent that, and I’m not sure as a power user I’d want it any other way, I’m explicit with what I want it to do and dont want it making crazy assumptions that I didnt write. like, what happens if you forget to free a pointer in c? sorry for snark but there are an unbelievably numerous amount of things to complain about in tf, never heard this one.
    - yyyyz 2y ago
      [dead]
    - defrost 2y ago
      > what happens if you forget to free a pointer in c? Assuming you mean 'forget' to free malloc'd space referenced by at least one pointer, that's an easy one .. it's reclaimed by the OS when the process ends. Whether that's a bad thing or not really depends on context - there are entire suites of interlocked processing pipelines built about the notion of allocating required resources, throughputing data, and terminating on completion - no free()'s
      - JohnMakin 2y ago
        surely my salient point is recognized regardless of semantics, but thanks for the correction. To use another example in another post - what happens if you DROP TABLE in sql?
        shermantanktop 2y ago
        What happens is that you call Iron Mountain and find out that those tapes don't actually have anything useful on them.
        fc417fc802 2y ago
  - acedTrex 2y ago
    I mean its also data loss if you run DROP DATABASE when you shouldn't. thats not sqls fault
    - slivanes 2y ago
      I think in this context it's that your database server is lost if you accidentally forget to write KEEP DATABASE.
- goodoldneon 2y ago
  IIRC, the lifecycle hook only prevents destruction of the resource if it needs to be replaced (e.g. change an immutable field). If you outright delete the resource declaration in code then it’s destroyed. I may be misremembering though
  - jmholla 2y ago
    I'm pretty sure you are. I've had it protect me from `terraform destroy`.
    - jtimdwyer 2y ago
      I think the previous post is saying a resource removed from a configuration file rather than an invocation explicitly deleting the resource in a command line. Of course if it’s removed from the config file, presumably the lifecycle configuration was as well!
      - glenngillen 2y ago
        Yeah, that's a legit challenge that it would be great if there was a better built-in solution for (I'm fairly sure you can protect against it with policy as code via Sentinel or OPA, but now you're having to maintain a list of protected resources too). That said the failure mode is also a bit more than "a badly reviewed PR". It's: * reviewing and approving a PR that is removing a resource * approving a run that explicitly states how many resources are going to be destroyed, and lists them * (or having your runs auto approve) I've long theorised the actual problem here is that in 99% of cases everything is fine, and so people develop a form of review fatigue and muscle memory for approving things without actually reviewing them critically.
  - csomar 2y ago
    This is not a terraform problem. This is your problem. Theoretically, you should be able to recreate the resource back with only a downtime or some services affected. You should centralize/separate state and have stronger protections for it.
  - 2y ago
gue5t 2y ago
I concur. I looked pretty hard into adapting Serf as part of a custom service mesh and it had some bonkers designs such as a big "everything" interface used just to break a cyclic module dependency (perhaps between the CLI and the library? I don't recall exactly), as well as lots of stuff that only made sense if you wanted "something to run Consul on top of" rather than a carefully-designed tool of its own with limited but cohesive scope. It seemed like a lot of brittle "just-so" code, which to some extent is probably due to how Go discourages abstraction, but really rubbed me the wrong way.
SOLAR_FIELDS 2y ago
An easy way to get someone to admit that terraform is a hacky child’s language is to ask how to simply print out the values of variables and resources you are using in terraform easily. This basic programming language 101 functionality is not present in the language
- Xelynega 2y ago
  I agree that terraform is hacky, but is "terraform output {variable name}" not how you would do that?
  - SOLAR_FIELDS 2y ago
    You have to be able to actually specify the output. And that does not handle all use cases. And it has requirements on how it can be run. And it takes the full lifecycle of the plan. And it won’t work in many circumstances without an apply. So no. Terraform has the information internally in many cases. There’s just no easy way to print it out.
- OJFord 2y ago
  $terraform console >var.whatever "its value" >whatever_resource.foo.whatever_attr "its value" If you mean somehow printing things when the configuration is being applied... I think you just need to understand that it's neither a procedural language (it's declarative) nor general-purpose (it's infrastructure configuration).
  - nightpool 2y ago
    This works great for toy examples and fails the moment you have 1 (one) module
    - OJFord 2y ago
      It doesn't fail, but you're right you can't reach inside submodules with terraform console; I wish you could.
XorNot 2y ago
My hot take is just that Vault isn't a good solution, and the permissions model is wholly inadequate. Except for not "feeling" secure, the only thing everyone wants is a Windows AD file share with ACLs. Just no one realises this: all the Vault on disk encryption and unsealing stuff is irrelevant - it's solving a problem handled at an entirely different level.
infraking 2y ago
Your team should be reviewing Terraform plans before merging any infra changes. I find this to be a very strange criticism and is probably indicative of a poor workflow or CI/CD system if anything.
- ants_everywhere 2y ago
  It's a design flaw. It should be nearly impossible to delete data accidentally. And it should be possible to roll back state in IIAC. No serious organization with any scale is going to have the only thing standing between them and a production database deletion being two over tired engineers rubber stamping each other's code changes. Bad config pushes to prod do happen and they can cause outages like the 2024 Cloudstrike outage. You don't want a tool that takes a minor (but significant) error and turns it into a catastrophic one because of poorly thought out semantics. It's better to just start with a tool that requires at least two engineers to explicitly sign off on deletion.
  - infraking2 2y ago
    > It's a design flaw. It should be nearly impossible to delete data accidentally. And it should be possible to roll back state in IIAC. Still not envisioning what your expected solution would look like. You check the plan, you asses the actions it will take and you either abandon the plan or apply it. You can't roll back destructive changes in cloud environments that's not possible today for obvious reasons. What IaC does provide is a mechanism to quickly rebuild if you do accidentally wipe out resources. I've worked in environments were our entire fleet was wiped out by AWS (thousands of VMs) and we were able to rebuild in hours because of IaC. > No serious organization with any scale is going to have the only thing standing between them and a production database deletion being two over tired engineers rubber stamping each other's code changes. Most "serious organizations" either have policy as code (Sentinel) or are running Terraform with credentials/roles that have reduced capabilities. > Bad config pushes to prod do happen and they can cause outages like the 2024 Cloudstrike outage. You don't want a tool that takes a minor (but significant) error and turns it into a catastrophic one because of poorly thought out semantics. It's better to just start with a tool that requires at least two engineers to explicitly sign off on deletion. This is a criticism of all software/infrastructure deployments with no guardrails. There's nothing stopping you from having two engineers sign off on a TF plan. You can absolutely build that system on top of whatever pipeline you are running. * Funny you mention databases because that's one of the few AWS resources that can be guarded in TF directly - https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/db_instance#deletion_protection-1 https://registry.terraform.io/providers/hashicorp/aws/latest...

JohnMakin 2y ago

> For Terraform (at least a few years ago) a badly reviewed PR could cause catastrophic data loss because resources are deleted without requiring an explicit tombstone. There have been lifecycle rules in place for as long as I can remember to prevent stuff like this. I'm not sure this is a "problem" unique to terraform.

ants_everywhere 2y ago
What happens if you forget the lifecycle annotations or put them in the wrong place or you accidentally delete them? Last time I checked it was data loss, but that was a few years ago.
- anonfordays 2y ago
  "What happens if I turn a table saw on and start breakdancing on it?" Of course you're going to hurt yourself. If you didn't put lifecycle blocks on your production resources, you weren't organizationally mature enough to be using Terraform in production. Take an associate Terraform course, this specific topic is covered in it.
  - dpkirchner 2y ago
    I'm not familiar with every lifecycle argument but I don't know of any that prevent resources being destroyed if they are removed from the tf file (what the parent was talking about). prevent_destroy, per docs, only applies as long as the resource is defined. I think the only way to avoid accidentally destroying a resource is to refer to it somewhere else, like in a depends_on array. At least that would block the plan.
    - anonfordays 2y ago
      >I don't know of any that prevent resources being destroyed if they are removed from the tf file (what the parent was talking about). Azure Locks (which you can also manage with Terraform), Open Policy Agent, Sentinel rules, etc. will prevent a destroy even if you remove the definition from your Terraform codebase. Again, if you're not operationally mature enough, the problem isn't the tool, it's you.
      - deleted 2y ago
        [deleted]
      - regularfry 2y ago
        "Operationally mature" is code here for "the gun starts out loaded and pointed at your foot". It's fine to point out that that's a suboptimal design for a tool.
        anonfordays 2y ago
        >Operationally mature" is code here for "the gun starts out loaded and pointed at your foot" No, it's code for "don't build a load bearing bridge if you don't understand structural engineering." > It's fine to point out that that's a suboptimal design for a tool. This isn't "suboptimal" though. If you delete a stored procedure in your RDBMS and it causes an outage, it's not because SQL/PostgreSQL is suboptimal. Similarly if you accidentally delete files from your file system, it's not because file systems are "suboptimal". It's because you weren't operationally mature enough to have proper testing and backups in place.
        regularfry 2y ago
        Defaults matter. I don't know how else to express that a system which fails safe in the face of fallible humans is strictly better, all else being equal, than the alternative.
        anonfordays 2y ago
- JohnMakin 2y ago
  The same as in any other language when what you wrote was not what you intended? Sorry, I’m really confused what your complaint here is or how you’d prefer it to work. If you make a sensitive resource managed by any kind of IAC, of course the IAC can destroy it in a manner that would result in irretrievable data loss. The language has for forever put semantics in place to prevent that, and I’m not sure as a power user I’d want it any other way, I’m explicit with what I want it to do and dont want it making crazy assumptions that I didnt write. like, what happens if you forget to free a pointer in c? sorry for snark but there are an unbelievably numerous amount of things to complain about in tf, never heard this one.
  - yyyyz 2y ago
    [dead]
  - defrost 2y ago
    > what happens if you forget to free a pointer in c? Assuming you mean 'forget' to free malloc'd space referenced by at least one pointer, that's an easy one .. it's reclaimed by the OS when the process ends. Whether that's a bad thing or not really depends on context - there are entire suites of interlocked processing pipelines built about the notion of allocating required resources, throughputing data, and terminating on completion - no free()'s
    - JohnMakin 2y ago
      surely my salient point is recognized regardless of semantics, but thanks for the correction. To use another example in another post - what happens if you DROP TABLE in sql?
      - shermantanktop 2y ago
        What happens is that you call Iron Mountain and find out that those tapes don't actually have anything useful on them.
      - fc417fc802 2y ago
- acedTrex 2y ago
  I mean its also data loss if you run DROP DATABASE when you shouldn't. thats not sqls fault
  - slivanes 2y ago
    I think in this context it's that your database server is lost if you accidentally forget to write KEEP DATABASE.
goodoldneon 2y ago
IIRC, the lifecycle hook only prevents destruction of the resource if it needs to be replaced (e.g. change an immutable field). If you outright delete the resource declaration in code then it’s destroyed. I may be misremembering though
- jmholla 2y ago
  I'm pretty sure you are. I've had it protect me from `terraform destroy`.
  - jtimdwyer 2y ago
    I think the previous post is saying a resource removed from a configuration file rather than an invocation explicitly deleting the resource in a command line. Of course if it’s removed from the config file, presumably the lifecycle configuration was as well!
    - glenngillen 2y ago
      Yeah, that's a legit challenge that it would be great if there was a better built-in solution for (I'm fairly sure you can protect against it with policy as code via Sentinel or OPA, but now you're having to maintain a list of protected resources too). That said the failure mode is also a bit more than "a badly reviewed PR". It's: * reviewing and approving a PR that is removing a resource * approving a run that explicitly states how many resources are going to be destroyed, and lists them * (or having your runs auto approve) I've long theorised the actual problem here is that in 99% of cases everything is fine, and so people develop a form of review fatigue and muscle memory for approving things without actually reviewing them critically.
- csomar 2y ago
  This is not a terraform problem. This is your problem. Theoretically, you should be able to recreate the resource back with only a downtime or some services affected. You should centralize/separate state and have stronger protections for it.
- 2y ago

gue5t 2y ago

I concur. I looked pretty hard into adapting Serf as part of a custom service mesh and it had some bonkers designs such as a big "everything" interface used just to break a cyclic module dependency (perhaps between the CLI and the library? I don't recall exactly), as well as lots of stuff that only made sense if you wanted "something to run Consul on top of" rather than a carefully-designed tool of its own with limited but cohesive scope. It seemed like a lot of brittle "just-so" code, which to some extent is probably due to how Go discourages abstraction, but really rubbed me the wrong way.

SOLAR_FIELDS 2y ago

An easy way to get someone to admit that terraform is a hacky child’s language is to ask how to simply print out the values of variables and resources you are using in terraform easily. This basic programming language 101 functionality is not present in the language

Xelynega 2y ago
I agree that terraform is hacky, but is "terraform output {variable name}" not how you would do that?
- SOLAR_FIELDS 2y ago
  You have to be able to actually specify the output. And that does not handle all use cases. And it has requirements on how it can be run. And it takes the full lifecycle of the plan. And it won’t work in many circumstances without an apply. So no. Terraform has the information internally in many cases. There’s just no easy way to print it out.
OJFord 2y ago
$terraform console >var.whatever "its value" >whatever_resource.foo.whatever_attr "its value" If you mean somehow printing things when the configuration is being applied... I think you just need to understand that it's neither a procedural language (it's declarative) nor general-purpose (it's infrastructure configuration).
- nightpool 2y ago
  This works great for toy examples and fails the moment you have 1 (one) module
  - OJFord 2y ago
    It doesn't fail, but you're right you can't reach inside submodules with terraform console; I wish you could.

XorNot 2y ago

My hot take is just that Vault isn't a good solution, and the permissions model is wholly inadequate. Except for not "feeling" secure, the only thing everyone wants is a Windows AD file share with ACLs. Just no one realises this: all the Vault on disk encryption and unsealing stuff is irrelevant - it's solving a problem handled at an entirely different level.

infraking 2y ago

Your team should be reviewing Terraform plans before merging any infra changes. I find this to be a very strange criticism and is probably indicative of a poor workflow or CI/CD system if anything.

ants_everywhere 2y ago
It's a design flaw. It should be nearly impossible to delete data accidentally. And it should be possible to roll back state in IIAC. No serious organization with any scale is going to have the only thing standing between them and a production database deletion being two over tired engineers rubber stamping each other's code changes. Bad config pushes to prod do happen and they can cause outages like the 2024 Cloudstrike outage. You don't want a tool that takes a minor (but significant) error and turns it into a catastrophic one because of poorly thought out semantics. It's better to just start with a tool that requires at least two engineers to explicitly sign off on deletion.
- infraking2 2y ago
  > It's a design flaw. It should be nearly impossible to delete data accidentally. And it should be possible to roll back state in IIAC. Still not envisioning what your expected solution would look like. You check the plan, you asses the actions it will take and you either abandon the plan or apply it. You can't roll back destructive changes in cloud environments that's not possible today for obvious reasons. What IaC does provide is a mechanism to quickly rebuild if you do accidentally wipe out resources. I've worked in environments were our entire fleet was wiped out by AWS (thousands of VMs) and we were able to rebuild in hours because of IaC. > No serious organization with any scale is going to have the only thing standing between them and a production database deletion being two over tired engineers rubber stamping each other's code changes. Most "serious organizations" either have policy as code (Sentinel) or are running Terraform with credentials/roles that have reduced capabilities. > Bad config pushes to prod do happen and they can cause outages like the 2024 Cloudstrike outage. You don't want a tool that takes a minor (but significant) error and turns it into a catastrophic one because of poorly thought out semantics. It's better to just start with a tool that requires at least two engineers to explicitly sign off on deletion. This is a criticism of all software/infrastructure deployments with no guardrails. There's nothing stopping you from having two engineers sign off on a TF plan. You can absolutely build that system on top of whatever pipeline you are running. * Funny you mention databases because that's one of the few AWS resources that can be guarded in TF directly - https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/db_instance#deletion_protection-1 https://registry.terraform.io/providers/hashicorp/aws/latest...