4 ms·

Y'know, normally I'd be relatively forgiving as none of us are perfect, but jeesus this is a clusterf** of an article. The author, a "Director of Consulting" m

Y'know, normally I'd be relatively forgiving as none of us are perfect, but jeesus this is a clusterf** of an article. The author, a "Director of Consulting" might want to do some training (Both Hashicorp and Microsoft have free training). Why in the hell would you have the same statefile for different environments? Why would you have these environments in the same Azure subscription? Why would you run your terraform so infrequently that you'd forget about a botched statefile move? Why would you not read TERRAFORM PLAN (It's LITERALLY WHAT TERRAFORM DOES)? I also suspect that while the author's probably heard of a CI/CD pipeline, they're running their IaC from their local machine, given the tone of the article. Why, for a production database that contains information not contained elsewhere, would you not configure an Azure Recovery Services Vault? Like I get it, people make mistakes. I once did something similar with an overenthusiastic use of terraform destroy, but this guy just seems like an absolute cowboy who doesn't really know what he's doing.

benjaminwootton 5y ago
The problem with Terraform and IAC is that there’s a big gap between learning how to use it and then learning how to use it in a safe, scalable way. It’s the same with something like a programming language where there are thousands of best practices and foot guns, but infrastructure as code is more dangerous and much newer. There are also less books, venues and even training courses to learn these practices. There are also a vanishingly small set of engineers who have actually done this in production at scale so it can be hard to find experienced people.
- thedeusx 5y ago
  I've worked with at least several hundred of them over the past 5-10 years. Granted, I'm in Europe and the author is in the US, perhaps you are too. I'm not sure what talent/skills are like there. In several firms I've worked in now someone missing so many things wouldn't be above consultant level, and wouldn't be approving PRs let alone deleting a live prod off from their local laptop without some significant questions being asked. I get everyone must learn things, but when you position yourself as a technical expert (Director, in this case) you should have enough experience to be a bit more thorough with your work so if a mistake happens, there's a way out, or just not make what amounts to several design and implementation mistakes. Part of what I think makes this a little egregious is the author didn't f** up his own systems, he f**'d his clients. I'd understand a little more if it were "I'm the in-house guy upskilling" rather than "look at this mistake I made as a (presumably) highly paid outside consultant literally brought in to make sure stuff like this doesn't happen". However, I still must give absolute kudos for sharing mistakes publicly. We all do make mistakes, and most people try and hide it. When the author realised there was an issue, every step after that was handled like a pro.
- rjzzleep 5y ago
  Both Terragrunt[0] and Terraspace[1] address this by providing a way to manage different environments. I do however think that Terraform should have a first class support for different environment without the need of a wrapper to facilitate that. [0] https://terragrunt.gruntwork.io/ https://terragrunt.gruntwork.io/ [1] https://terraspace.cloud/ https://terraspace.cloud/
  - Huggernaut 5y ago
    Is what you're describing different from the idea of Terraform Workspaces? https://www.terraform.io/docs/language/state/workspaces.html https://www.terraform.io/docs/language/state/workspaces.html > Certain backends support multiple named workspaces, allowing multiple states to be associated with a single configuration. The configuration still has only one backend, but multiple distinct instances of that configuration to be deployed without configuring a new backend or changing authentication credentials.
darkwater 5y ago
Totally agree with you. And this makes me think about blogging about this kind of things. On one side I understand it can be "liberating" and also it normalizes accepting that we are humans, we do mistakes etc but when it's a so basic error, I will personally feel really ashamed by it and would probably learn the lesson and hide it under a rock, instead of blogging about it andd having it on the HN front page. There might be a few people/companies doing this level of TF bad practices out there that might find this helpful, but I also think that these kind of companies don't have employees reading random tech blogs to learn things and from others' mistakes.
deleted 5y ago
[deleted]
space_defence 5y ago
Normally I'd just read attacks on useful blog posts from anonymous accounts, but jeesus this is one cluster* of a comment. You might want to actually read the entire blog post before going off on a tirade. I am not OP btw. > Why, for a production database that contains information not contained elsewhere * This is a dev environment - the one that he uses to to dev work where its perfectly okay to corrupt, which is why dev environments exists. > Why would you run your terraform so infrequently that you'd forget about a botched statefile move? * Why on earth is it surprising to you that theres a dev environment sitting somewhere that could be unused for some time? > I also suspect that while the author's probably heard of a CI/CD pipeline, they're running their IaC from their local machine * It is perfectly okay to develop and test changes on a local dev environment before putting it on a CI/CD pipeline. OP mentions that there is an Azure Devops pipeline in place. > this guy just seems like an absolute cowboy who doesn't really know what he's doing. * In dev yes. He clearly does know what he is doing. This of course would never happen with AWS and cloudformation changesets. OP is sharing gotchas that arise when using Azure and Terraform, this is useful.

benjaminwootton 5y ago

The problem with Terraform and IAC is that there’s a big gap between learning how to use it and then learning how to use it in a safe, scalable way. It’s the same with something like a programming language where there are thousands of best practices and foot guns, but infrastructure as code is more dangerous and much newer. There are also less books, venues and even training courses to learn these practices. There are also a vanishingly small set of engineers who have actually done this in production at scale so it can be hard to find experienced people.

thedeusx 5y ago
I've worked with at least several hundred of them over the past 5-10 years. Granted, I'm in Europe and the author is in the US, perhaps you are too. I'm not sure what talent/skills are like there. In several firms I've worked in now someone missing so many things wouldn't be above consultant level, and wouldn't be approving PRs let alone deleting a live prod off from their local laptop without some significant questions being asked. I get everyone must learn things, but when you position yourself as a technical expert (Director, in this case) you should have enough experience to be a bit more thorough with your work so if a mistake happens, there's a way out, or just not make what amounts to several design and implementation mistakes. Part of what I think makes this a little egregious is the author didn't f** up his own systems, he f**'d his clients. I'd understand a little more if it were "I'm the in-house guy upskilling" rather than "look at this mistake I made as a (presumably) highly paid outside consultant literally brought in to make sure stuff like this doesn't happen". However, I still must give absolute kudos for sharing mistakes publicly. We all do make mistakes, and most people try and hide it. When the author realised there was an issue, every step after that was handled like a pro.
rjzzleep 5y ago
Both Terragrunt[0] and Terraspace[1] address this by providing a way to manage different environments. I do however think that Terraform should have a first class support for different environment without the need of a wrapper to facilitate that. [0] https://terragrunt.gruntwork.io/ https://terragrunt.gruntwork.io/ [1] https://terraspace.cloud/ https://terraspace.cloud/
- Huggernaut 5y ago
  Is what you're describing different from the idea of Terraform Workspaces? https://www.terraform.io/docs/language/state/workspaces.html https://www.terraform.io/docs/language/state/workspaces.html > Certain backends support multiple named workspaces, allowing multiple states to be associated with a single configuration. The configuration still has only one backend, but multiple distinct instances of that configuration to be deployed without configuring a new backend or changing authentication credentials.

darkwater 5y ago

Totally agree with you. And this makes me think about blogging about this kind of things. On one side I understand it can be "liberating" and also it normalizes accepting that we are humans, we do mistakes etc but when it's a so basic error, I will personally feel really ashamed by it and would probably learn the lesson and hide it under a rock, instead of blogging about it andd having it on the HN front page. There might be a few people/companies doing this level of TF bad practices out there that might find this helpful, but I also think that these kind of companies don't have employees reading random tech blogs to learn things and from others' mistakes.

deleted 5y ago

[deleted]

space_defence 5y ago

Normally I'd just read attacks on useful blog posts from anonymous accounts, but jeesus this is one cluster* of a comment. You might want to actually read the entire blog post before going off on a tirade. I am not OP btw. > Why, for a production database that contains information not contained elsewhere * This is a dev environment - the one that he uses to to dev work where its perfectly okay to corrupt, which is why dev environments exists. > Why would you run your terraform so infrequently that you'd forget about a botched statefile move? * Why on earth is it surprising to you that theres a dev environment sitting somewhere that could be unused for some time? > I also suspect that while the author's probably heard of a CI/CD pipeline, they're running their IaC from their local machine * It is perfectly okay to develop and test changes on a local dev environment before putting it on a CI/CD pipeline. OP mentions that there is an Azure Devops pipeline in place. > this guy just seems like an absolute cowboy who doesn't really know what he's doing. * In dev yes. He clearly does know what he is doing. This of course would never happen with AWS and cloudformation changesets. OP is sharing gotchas that arise when using Azure and Terraform, this is useful.