21 ms·
A Month of Terraform
- danpalmer 6y agoThe arguments around the language and syntax inconsistencies here really echo my experience. I can't help but think it just wasn't designed, but patched together by different people over time, with no over-arching vision. There's nothing in Terraform that couldn't just be YAML with some semantics (like Ansible which I find much more approachable), and while a custom language can absolutely do better than that, it really needs to justify itself, and I'm not sure Terraform does yet. There's some hints of why it should be a custom language, but I honestly think it needs an overhaul focusing on consistency and clarity. These things really make a difference, I'd say it's one of the main reasons Go took off in such a big way for example.
- thraxil 6y ago> There's nothing in Terraform that couldn't just be YAML with some semantics (like Ansible which I find much more approachable) One secret of Terraform is that you can define resources with JSON instead of HCL (https://www.terraform.io/docs/configuration/syntax-json.html https://www.terraform.io/docs/configuration/syntax-json.html). The catch is that all the interpolation and iteration stuff is implemented in HCL, so if you use JSON, you probably want something else to do that. But, eg, Jsonnet works beautifully for that kind of thing: https://github.com/google/jsonnet/blob/master/examples/terraform/aws-two-tier.jsonnet https://github.com/google/jsonnet/blob/master/examples/terra...
- _5yoy 6y agoIf your only comparison is "I used to work with Heroku", Terraform might seem not great. I would argue this is a blub example, in that this person's lens does not contain enough context or experience to be able to assess value. Terraform's value becomes clear when moving in scenarios like moving from CloudFormation to Terraform, or when trying to integrate a second cloud's resources into your infrastructure workflow. Without experience in anything other than Heroku, of course a tool designed to do many many things is going to seem complex and at times frustrating. > Iteration times are way longer than with even mobile apps. Like, “you’re liable to task-switch while waiting to see plan output” longer. I'm over here laughing in my "over an hour of CloudFormation only to have a run fail, and then a rollback fail, and have to contact AWS support" life.
- pm90 6y ago> I'm over here laughing in my "over an hour of CloudFormation only to have a run fail, and then a rollback fail, and have to contact AWS support" life. Heh, same. Any of the cloud-provided orchestration tools (Cloudformation, Openstack Heat etc.) are great only for the most basic tasks; using them to provision complex infrastructure is just begging for a world of hurt. That being said. I think terraform could do better. I use terraform a lot, and yet I agree with the authors complaints that the syntax can be super confusing, is not documented very well, and the providers have their own idiosyncrasies. The last one isn't strictly an issue with terraform, its with the provider implementation. But... if terraform aims to be the Swiss army knife of infrastructure provisioning, I think criticism that its hard to standardize even within the same provider is fair.
- ciceryadam 6y ago> Any of the cloud-provided orchestration tools (Cloudformation, Openstack Heat etc.) are great only for the most basic tasks; using them to provision complex infrastructure is just begging for a world of hurt. It's also fun when you have to standardize something that was either created by hand in the Web UI (and has a bunch of hidden default values set), or was by some other orchestration tool, and then you have to import/recreate it in terraform. It's usually a PITA, but you'll learn a lot!
- pm90 6y agoYes! Its great to capture a bespoke configuration in code. You may already know this but there is a tool called terraformer that has limited support for doing just this: https://github.com/GoogleCloudPlatform/terraformer https://github.com/GoogleCloudPlatform/terraformer
- wgjordan 6y ago> Heh, same. Any of the cloud-provided orchestration tools (Cloudformation, Openstack Heat etc.) are great only for the most basic tasks; using them to provision complex infrastructure is just begging for a world of hurt. I disagree- I can't speak to OpenStack Heat (and I have no idea what you're referring to by 'cloud-provided orchestration tools' beyond these two specifically), but my own experience using CloudFormation to provision complex infrastructure is that it is in fact great for all but the most basic tasks (where any orchestration tool would just add unnecessary overhead).
- pqb 6y ago> terraform-lsp is supposed to provide autocomplete, but it mostly doesn’t, in my experience. I find Terraform for Jetbrains IDEs to work hundred times better than the VSCode one. I really can recommend it, does magic to autocompletion, peek definition, snippets insertion and linting TF/HCL files. Author of this blogpost have not mentioned Terragrunt [0], while I think it is worth mentioning. It is nice tool, especially if you work on bigger projects having multiple modules and per-environment variables. Also, a tip from a person, who works in team using terraform - use brew/apt repositories to keep the binaries up-to-date or at least on the same version as your team mates. I remember at least few situations, when patch update of a terraform binary was crucial to make some issues disappear. [0]: https://terragrunt.gruntwork.io/ https://terragrunt.gruntwork.io/ Psst... I am also using Terraform to bootstrap Pop_OS! with my dotfiles and it is surprisingly nice enough to mimic declarative and atomic NixOS configuration :)
- ciceryadam 6y agoFor distributed work on environments you ideally you want to have something like a gitlab pipeline that does the tf plan for you automatically, with a manual approve button of sorts. That way you a consistent tf version defined in .gitlab-ci.yml, plus due to centralized logging it's easier to find when and what changed in the envs.
- pqb 6y agoI assume you are talking about a similar tool to the Atlantis [0] / Terraform SaaS offering [1]. [0]: https://www.runatlantis.io/ https://www.runatlantis.io/ [1}: https://www.hashicorp.com/blog/terraform-collaboration-for-everyone https://www.hashicorp.com/blog/terraform-collaboration-for-e...
- ciceryadam 6y agoYeah, exactly the same functionality, but just implemented with a different toolset: https://docs.gitlab.com/ee/user/infrastructure/ https://docs.gitlab.com/ee/user/infrastructure/
- arkadiyt 6y agoSome secrets (like IAM access keys) can be encrypted with a public PGP key. Some other secrets (like RDS instance master passwords) can't be encrypted, but I like to use a trick where you have a local provisioner that runs after the instance gets created, which updates the just created DB and sets the password to a random value and prints it to stdout so you can save it in your secrets management tool of choice. The value saved in the terraform state is no longer valid. Some other other secrets, like elasticache auth tokens - there's no solution, they'll always be available as plaintext in the statefile. Overall agreed that Terraform needs to do better here.
- time0ut 6y agoI wish they had better coverage for PGP keys. On RDS, my typical pattern is to use Secrets Manager to auto rotate the master password immediately. AWS has sample Lambdas for that various databases.
- gtsteve 6y agoI have a lambda that runs every day to cycle the RDS master password. I create the password using the random provider, and save it to a secrets manager secret - the first time the environment is created, the secret is in plaintext in the state, but it will not be valid the next time the lambda runs (less than 24 hours). You can do the same with ElastiCache auth tokens, and ensure your application reads the token from a value in secrets manager.
- ec109685 6y agoIs there a race condition with the way you rotate the passwords?
- gtsteve 6y agoNo, once you've logged in with MySQL, changing the password doesn't close the connection. For rotating application passwords we use the same technique but we update the usernames, i.e. app_1@'%' becomes app_2@'%', and then rotates back to app_1@'%' to prevent issues with unsynced config files.
- nerdbaggy 6y agoI have been using more and more of https://www.pulumi.com/ https://www.pulumi.com/ for my deployments. It’s nice that the deployment code is written in js/python/go/.net so I can make it as customizable as I want.
- diveanon 6y agoI agree, I'm using pulumi with ts and it's pretty great to be able to develop / test my deployment pipeline using the same tools as my core services.
- nunez 6y agoSeconded. Pulumi is great.
- thayne 6y agoI'm seriously considering switching to pulumi, mostly because it seems a lot more flexible. Not to mention that developers are already familiar with the language rather than having to learn all the quirks and inconsistencies of hcl on top of using the tool itself
- nerdbaggy 6y agoYup that is why we use it. And it’s great since anything can be a data source
- chucky_z 6y agotry out https://github.com/hashicorp/terraform-cdk https://github.com/hashicorp/terraform-cdk i've had success with it thus far. i'm mostly using mainstream providers at this point though.
- macca321 6y agoIn a polyglot shop, it can actually be quite handy that HCL is it's own relatively simple language.
- thayne 6y ago
- sleepy_keita 6y ago> It mostly works! > When it doesn’t, it generally fails in a useful way, and then I can fix it and try again. This is the "killer feature" for me when comparing Terraform and CloudFormation. I know they've been working on making CF better, but the way Terraform handles errors (and allows you to pick up from where you left off, instead of waiting around for stuff to roll back) is a lot better suited to tight experimentation and feedback loops.
- jeremya 6y agoI think this feature lends itself well to tight feedback loops, as you said. However, after I crossed the painful divide from CloudFormation novice to... well, whatever is after that, CloudFormation rollbacks became a friend instead of an enemy. The ability to return to the last known good checkpoint is a powerful feature that is extremely useful in production systems and the pipelines that deploy them. In those scenarios, you're not experimenting and learning, but enhancing and evolving existing infrastructure.
- zug_zug 6y agoFor me the biggest disappointment is it won't import an existing aws stack into config files. The idea of declarative infrastructure seems amazing, but this implementation hasn't overwhelmed me.
- rwoll 6y agoCheck out Terraformer: https://github.com/GoogleCloudPlatform/terraformer https://github.com/GoogleCloudPlatform/terraformer This has been instrumental in importing and generating configs from existing infrastructure.
- latchkey 6y agoTerraform died for me, not because it didn't work well to setup one set of resources, but because when I went back to update things much later and a backwards breaking new version of TF totally broke my state file and the 3rd party resources I was using, had not caught up to the new version of TF yet. Short of starting all over again, I was dead in the water. As much as I really liked how quickly I was originally able to get my infra up and running with TF, if I'm ever in the need to have similar functionality again, I'll find something else to use.
- dj_mc_merlin 6y agoIt is in v0, breaking changes are expected. If the new version is not compatible, wait until it is and then upgrade. It is also completely reasonable to not want to use it until it is stable.
- HatchedLake721 6y agov0 or not doesn't matter, Terraform doesn't follow semantic versioning.
- kubanczyk 6y agoOf course it does follow semantic versioning, so far. > Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable. https://semver.org/#spec-item-4 https://semver.org/#spec-item-4 > At HashiCorp we take the idea of a v1.0 very seriously. https://github.com/hashicorp/terraform/issues/15839#issuecomment-323106524 https://github.com/hashicorp/terraform/issues/15839#issuecom... > Terraform 0.14 release. That's our last chance to deprecate features or introduce breaking changes, before 1.0. Will we have 0.15? That's anyone's guess at this point. I honestly don't know. https://www.hashicorp.com/resources/the-path-to-terraform-1-0 https://www.hashicorp.com/resources/the-path-to-terraform-1-...
- HatchedLake721 6y agoCalling a software that's been out for 6+ years powering everyone's IaaC from startups to multi-billion companies around the world as "initial development" to fit into SemVer is a stretch for me.
- sali0 6y agoThere are definitely some oddities in HCL. Regarding the dynamic blocks, it gets weirder. If you have a dynamic block inside of a resource that is using for_each, you can still reference the 'each' value alongside the value that the 'dynamic' block is looking at. resource "example_resource" { for_each = example_map dynamic "foo" { for_each = bar content { example_setting = each.value.setting new_baz = foo.baz } } } The original for_each is referencing its object as 'each', while the for_each inside the dynamic is referencing the iteration as 'foo'. Why not just name the for_each inside the dynamic block as something else? With that said, I absolutely love Terraform, and it is IMO the most powerful tool I've had the pleasure to work with. I also am a fan of DSLs if they are thoughtfully done. I think with some improvements, like some of the ones the author suggests, HCL could be a great language to work with.
- m1keil 6y agoAgree with pretty much every point in the blog. I would add to the list the fact that for_each cannot be used with pre-computed values. So if you have a module/resource that outputs a list of resources, you can't just plug it into another resource and use for_each to iterate on. I also think that using multiple state files is the only way to keep the config from being too fragile. Which is another point in which Terraform provides very little help and why Terragrunt is a popular combo.
- nikolay 6y agoI am using Terraform daily, and it's excruciating at times. First, HCL is very limited - HashiCorp didn't even use it for Sentinel! I won't switch to TypeScript either - I wish they've adopted an embedded language like Starlark like in this POC [0]. There are many inconsistencies; refactoring is very, very painful. While `terraform import` and `terraform state mv` work, it's all manual, and the first doesn't work with their SaaS - Terraform Cloud as imports, unlike runs, are always local, but sensitive variables are only available in the cloud. As pointed out, secrets are in plain text - GCP state even had my personal access token, which it's not supposed to as it's not something that has to be shared with the team. The good thing is that the velocity is greatly improving - we had v0.13 this year, and v0.14 is almost out - many features of v0.15 are done, so that pace compared to v0.12 is new! My biggest issue is how much copypasta you need for just reusing modules! There's no clean way to do Dependency Inject, and you have to replicate code all over the place as you can't "export" resources or data sources - you can only export primitive values, and then you need to have a data source, which uses that primary key of the resource. Terraform could turn into a huge, missed opportunity. I hope they realized the potential and that they need to execute fast. Also, it's very maddening that their prioritization is based on the number of thumbs-ups! Last but not least - many key providers are run by people who don't follow standards and are working part-time on key projects instead of HashiCorp making sure there's great DX with major vendors. One such example is the GitHub provider. A huge fiasco recently had a backward-incompatible minor release, which also didn't update the docs. After three weeks, the maintainer was still considering unreleasing when people already have found workarounds and reverse-engineered the issues and the missing documentation. Same provider still does not provide organization-level secrets even though this features has been available for more than 6 months! [0]: https://ascode.run/ https://ascode.run/
- kubanczyk 6y ago> you can only export primitive values That's seriously outdated. The v0.12.0 changelog: "Resource and module object values: An entire resource or module can now be treated as an object value within expressions, including passing them through input variables and output values to other modules, using an attribute-less reference syntax, like `aws_instance.foo`"
- baxtr 6y agoToo bad. I thought this is about changing the landscape
- pst 6y agoTerraform certainly has its fair share of quirks, but that's no different from any other language. However, I think it's fair to say that the infrastructure as code ecosystem is still much younger and therefore less mature. And there are various things that are standard in application development toolchains that don't exist for IaC yet. Take my focus, use-case specific frameworks for example. If I'm building a web application, I don't write my own request routing or authentication. But for Terraform, the majority of teams have to start from scratch, even for common use-cases. Yes, there are re-usable modules, but they're comparable to libraries and integrating various modules is still a lot of effort. If my use-case is building a Jamstack website, doing so using Gatsby gets me started faster, gives me a modern developer experience and means I can re-use community tested and maintained components to reduce the bespoke code I have to write and maintain. I'm trying to do the exact same thing for Iac with Kubestack. Kubestack is an open-source Terraform framework for managed Kubernetes. If your use-case is provisioning and maintaining EKS, AKS or GKE using Terraform, Kubestack may be worth trying. In my obviously creator-biased opinion. It helps you with the typical framework like workflow to get started faster and scaffold a repository with one command, then bring up a local development environment with the next. Yes, the long feedback cycles of IaC can be annoying. That's why I'm trying to improve the developer experience by providing an auto updating local development environment. For all modules (EKS, AKS, GKE) I maintain as part of the Kubestack framework, I also maintain a local variant. These accept the same input variables, but instead of provisioning cloud resources, provisions "mock" clusters locally using Docker containers as the cluster nodes. The kbst CLI watches for changes in the repository, and then runs Terraform locally and dynamically replaces the module source of the real cluster module with the local variant. Here's a video showing that in action: https://youtu.be/_VtakP6AdCs https://youtu.be/_VtakP6AdCs Similarly, I provide a Docker image for each framework release to provide a tested combination of versions of Terraform, it's providers and the cloud CLI (aws, gcloud, az). The images are used to bootstrap, for CI/CD runs and for the occasionally required manual tasks (state mv, etc.) or disaster recovery. Just to name two examples from the discussion here where the developer experience of Terraform lacks behind the equivalent application development tooling. Many people underestimate IaC, probably because of all the 5 minute Terraform tutorials and demos out there. But what these miss is that the real work only starts when you have to get your automation ready for day-2 operations. This is where I'm trying to provide a better developer experience through re-usable use-case specific modules, inheritance based environment configuration to avoid drift, and integration into a convenient but reliable GitOps workflows for teams from local development all the way to production. Kubestack's code is open-source for two years in December. But I only got around writing documentation after leaving the DevOps consulting job that inspired the framework. Anyone interested can give it a try: https://www.kubestack.com/ https://www.kubestack.com/ The site has links to the Slack channel (#kubestack on the Kubernetes Slack) and also the source on Github.
- kubanczyk 6y agoFor a 2020 piece, two things seem outdated: > And you can’t even use the for_each trick with module imports. It just isn’t supported. Works since Terraform 0.13 (released in August 2020). > data "provider_thingy" "my_name_for_this_data" > Though, like, why the quotes around the provider thingy? Since 0.12 you can just write data provider_thingy my_data {} resource provider_thingy my_resource {} They seem to keep the quotes for compatibility with v0.11. Yes, they should have updated the intro/examples to the simpler format. > Required vs optional parameters are not very clearly called out and are not at all segregated. Yes, in AWS provider. Other providers format the docs better and clearly segregate the required from the optional.
- igetspam 6y agoIf you've been using TF for any amount of time though, 0.13 is a breaking change (because semver... oh wait...). The migration from 0.11 to 0.12 was a PITA and if you're using any third party modules or resources, they still don't have consistent support for 0.13. Usung 0.12 currently gives you the most consistent experience. This is not bashing TF, it's just reality. I wish I had time to update everything to 0.13. The third party provider support alone makes it valuable.
- kubanczyk 6y agoI read you - I'm currently still going through 0.13 upgrade. The pain however is considerably smaller than 0.12 upgrade and I've already observed two 0.12 bugs disappearing on 0.13. The bugfix-backporting seems sluggish and it doesn't encourage me at all to keep using 0.12 despite it is "the most consistent experience".
- elango 6y agoA very good summary, this resonates well with my views. Terraform survived because devops folks mostly come from non- programming background and have low expectations from a DSL. When developers look at TF, there are obvious things that glare