7 ms·

Perhaps off-topic but how have people upgraded TF codebases to new versions? Just last year we had a big effort to upgrade a huge code-base from 0.11 to 0.12. I

Perhaps off-topic but how have people upgraded TF codebases to new versions? Just last year we had a big effort to upgrade a huge code-base from 0.11 to 0.12. I feel like it should be a lot smoother than a full-team full-sprint effort.

_joel 6y ago
With lots of swearing, at least in my experience when we did the same upgrade as you. I too am hoping this will be smoother in future and the changes stabilise. I guess if you're doing very simple stuff then it's not an issue but the larger the estate and 'custom' stuff that gets added the more difficult any upgrade becomes (tf or not)
scottsuch 6y ago
Our team just did a big effort to get from 11 to 12. After that the effort was quite minimal. Some little gotchas re: providers in 13 but we're just today finished the 14 upgrade and will probably let 15 marinate until we upgrade to that.
- llbeansandrice 6y ago
  I think we're dealing with the providers now as we get warnings on 0.12.29 that they'll be deprecated in 13 so nice to know that we're over the biggest hill.
mohsen0 6y ago
It becomes much easier between other versions. Though upgrading 0.12 to 0.13 I remember, i had to pull the state and change the provider field manually to avoid recreation of some resources.
- dastx 6y ago
  > i had to pull the state and change the provider field manually to avoid recreation of some resources. Terraform CLI introduced an upgrade command (can't remember what it's called) that automatically does this for you.
  - erik_seaberg 6y ago
    In 0.14 that is no longer available.
    - dastx 6y ago
      Terraform 0.14 is pretty much fully compatible with 0.13, so no such command is necessary. All you have to do is make sure your state is for version 0.13. Outside that it's a bunch of usability changes that do not affect your tf code.
jokethrowaway 6y ago
Someone not in our team upgraded version by mistake from 0.12 to 0.13 (he was contributing something small and used the latest), the CTO got involved and made us update everything and it was a big undertaking. Personally I have a nix shell file pinned to the exact version of terraform (as in, commit hash on the nix-packages repo) we use in every repo and just switch to that shell before doing anything.
- _joel 6y ago
  We use this https://github.com/tfutils/tfenv https://github.com/tfutils/tfenv
- yjftsjthsd-h 6y ago
  I'm a fan of tfenv for this; it's really easy to use and makes it trivial to pin each stack to an exact version of TF.
  - toyg 6y ago
    Between rbenv, tfenv, pyenv, sdkman and so on and so forth, maybe it's time for some sort of common OS-level env-management interface...?
    - yjftsjthsd-h 6y ago
      That is a selling point of nix and docker, yes:) The catch is that doing it generically makes the whole thing more complicated (although I suspect nix and docker are more complex than is strictly required for that use case)
    - darkwater 6y ago
      Maybe asdf? https://github.com/asdf-vm/asdf https://github.com/asdf-vm/asdf
      - diroussel 6y ago
        I use asdf and it’s great. You keep your versions in .tool-versions abs when you switch for branches you automatically get the right versions of node, java, terraform etc on your path.
yjftsjthsd-h 6y ago
On the bright side, now that they've committed to a stable state file format it should get better
mitchellh 6y ago
I'm one of the HashiCorp founders. Terraform 0.11 to 0.12 is by far the most difficult of the versions to upgrade between. I am really sorry about that. The other upgrades should be relatively minor as long as you read and follow the upgrade guides and upgrade one minor version at a time (0.11 => .12 => .13 etc.). There are rough edges for very specific cases but most of our customers were able to upgrade from 0.12 to subsequent versions same day without issue. Breaking changes and difficult upgrades is not something we want to do with Terraform (0.12 being a big exception as that was a very core "reset" so to speak). The reason there have been these changes in these recent releases is that we've been ensuring Terraform is in a place for 1.0 that we don't have to have difficult upgrades. You can see this path happening in each release: - Terraform 0.15: state file format stability - Terraform 0.14: provider dependency lock file - Terraform 0.13: required provider block (for more deterministic providers) - Terraform 0.12: stable JSON formats for plans/config parsing/etc. (particularly useful for interop with things like Terraform Cloud and 3rd party tooling) This was all done to lead up to a stable 1.0 release. As noted in the blog post, 0.15 is effectively a 1.0 pre-release so if everything goes well, we've MADE IT. For 1.0, we'll be outlining a number of very detailed compatibility promises which will make upgrades much easier going forward. :)
- llbeansandrice 6y ago
  Thanks, good to know that the upgrade to 12 is the biggest jump.
  - john37386 6y ago
    I had the same question or concern. I also realized too late that 0.12 is a bigger one than first thought. I was not severely impacted in the end, but boy it was a long time that I didn't experience such a tough upgrade. Happy to know that the hardest is behind and looking forward to try 0.15. Thanks
- 0xbadcafebee 6y ago
  Our teams have something like 100,000 LOC in Terraform 0.12, and it's not all in one big monorepo. At that scale there is no such thing as a relatively minor version upgrade. We want to upgrade to get away from some persistent 0.12 bugs, but we literally don't have the time. We have to change all of the code, and then test every single project that uses that code in non-prod, and pray that the testing finds most of the problems that will appear in production. And it's all owned by different groups and used in different projects, so that makes things longer/more complex. We also have to deal with provider version changes, upgrading CI pipelines and environments to be able to switch between Terraform binaries, and conventions to switch between code branches. I am already looking around for some way to remove Terraform from our org because it is slowly strangling our productivity. It's way too slow, there's too many footguns, it doesn't reliably predict changes, it breaks on apply like half of the time, and it's an arduous manual process to fix and clean up its broken state when it does eventually break. Not to mention just writing and testing the stuff takes forever, and the very obvious missing features like auto-generation and auto-import. I keep a channel just to rant about it. After Jenkins, Ansible and Puppet, it's one of those tools I dread but can't get away from.
jasonvorhe 6y ago
terraform 0.1xupgrade went pretty smoothly most of the time, with some minor manual changes and fixes here and there. Most of them were easily done in batch over the repository using sed. This approach worked for me in various setups from a small SaaS company to a major travel company as well as personal projects without issues.
uglygoblin 6y ago
I started using Terraform on our project in early 2019 which was version 0.11.13. The upgrade to 0.12.x seemed non-trivial so I put it off... now 2 years later and we're at 0.15.x. Looks like I need to clear my schedule in a upcoming sprint to get this done so the pain doesn't get even worse :)
dguaraglia 6y ago
As everyone else said: 0.11 to 0.12 was painful. After that, you just need to deal with converting the state whenever a new version comes out, but that's basically automated so not a biggie.
tetha 6y ago
From our experience. 0.11 -> 0.12: We're not attempting. We're in the process of changing out config management anyway and we're unhappy with a bunch of decisions in that terraform stack, so it's a good point to get rid of it. 0.12 -> 0.13: Well, we had to add what feels like a million required-provider-blocks and then some more. And sometimes it was tricky to pinpoint the module pulling in a default provider and crashing - `terraform providers` and `terraform graph` help there. The graph is easy to grep through to find the resources and modules pulling in wrong providers. And in the beginning the error message we got when we had to run 'terraform state upgrade-providers' was .. obscure. In newer versions, that message is much better. 0.13 -> 0.14 just happened and now the lock files are slowly piling in on demand.
garymoon 6y ago
I upgraded from 11 to 12 like one year ago and from 12 to 13 some days ago (upgrade to 14 seems that will be straightforward) but in my case what I did: - Don't upgrade directly from 12 to 14, go to 13 first - If you have warnings after moving from 11, fix them first - Run the 0.13upgrade command in your code that will generate the required_providers - Run terraform-v13 init - Change to correct workspace if using some - Run terraform-v13 plan which will probably fail due to the new explicit required-providers rule, if that happens you need to modify the state with the correct providers https://www.terraform.io/upgrade-guides/0-13.html#why-do-i-see-provider-during-init- . In my case I have a lot of modules so I created an script that automated that process - Execute again terraform-v13 plan and verify that it will not make uncommon changes - Then run terraform-v13 apply.
- matwood 6y ago
  > Don't upgrade directly from 12 to 14, go to 13 first The line above, plus running apply in each version is key. I literally just did the update from 11->latest for 3 different repos a couple of weeks ago. And tbh, its was only the first update where I had to make any code changes. The rest mostly worked.
skywhopper 6y ago
After 0.12 it has been much easier.
capableweb 6y ago
As no one seems to have mentioned anything, version numbers below 1.0 are generally considered unstable in terms of API interface (like HCL in this case) so if you'd like to avoid similar things in the future (with either Terraform or other tools), you're best to wait until they at least release version 1.0.

_joel 6y ago

With lots of swearing, at least in my experience when we did the same upgrade as you. I too am hoping this will be smoother in future and the changes stabilise. I guess if you're doing very simple stuff then it's not an issue but the larger the estate and 'custom' stuff that gets added the more difficult any upgrade becomes (tf or not)

scottsuch 6y ago

Our team just did a big effort to get from 11 to 12. After that the effort was quite minimal. Some little gotchas re: providers in 13 but we're just today finished the 14 upgrade and will probably let 15 marinate until we upgrade to that.

llbeansandrice 6y ago
I think we're dealing with the providers now as we get warnings on 0.12.29 that they'll be deprecated in 13 so nice to know that we're over the biggest hill.

mohsen0 6y ago

It becomes much easier between other versions. Though upgrading 0.12 to 0.13 I remember, i had to pull the state and change the provider field manually to avoid recreation of some resources.

dastx 6y ago
> i had to pull the state and change the provider field manually to avoid recreation of some resources. Terraform CLI introduced an upgrade command (can't remember what it's called) that automatically does this for you.
- erik_seaberg 6y ago
  In 0.14 that is no longer available.
  - dastx 6y ago
    Terraform 0.14 is pretty much fully compatible with 0.13, so no such command is necessary. All you have to do is make sure your state is for version 0.13. Outside that it's a bunch of usability changes that do not affect your tf code.

jokethrowaway 6y ago

Someone not in our team upgraded version by mistake from 0.12 to 0.13 (he was contributing something small and used the latest), the CTO got involved and made us update everything and it was a big undertaking. Personally I have a nix shell file pinned to the exact version of terraform (as in, commit hash on the nix-packages repo) we use in every repo and just switch to that shell before doing anything.

_joel 6y ago
We use this https://github.com/tfutils/tfenv https://github.com/tfutils/tfenv
yjftsjthsd-h 6y ago
I'm a fan of tfenv for this; it's really easy to use and makes it trivial to pin each stack to an exact version of TF.
- toyg 6y ago
  Between rbenv, tfenv, pyenv, sdkman and so on and so forth, maybe it's time for some sort of common OS-level env-management interface...?
  - yjftsjthsd-h 6y ago
    That is a selling point of nix and docker, yes:) The catch is that doing it generically makes the whole thing more complicated (although I suspect nix and docker are more complex than is strictly required for that use case)
  - darkwater 6y ago
    Maybe asdf? https://github.com/asdf-vm/asdf https://github.com/asdf-vm/asdf
    - diroussel 6y ago
      I use asdf and it’s great. You keep your versions in .tool-versions abs when you switch for branches you automatically get the right versions of node, java, terraform etc on your path.

yjftsjthsd-h 6y ago

On the bright side, now that they've committed to a stable state file format it should get better

mitchellh 6y ago

I'm one of the HashiCorp founders. Terraform 0.11 to 0.12 is by far the most difficult of the versions to upgrade between. I am really sorry about that. The other upgrades should be relatively minor as long as you read and follow the upgrade guides and upgrade one minor version at a time (0.11 => .12 => .13 etc.). There are rough edges for very specific cases but most of our customers were able to upgrade from 0.12 to subsequent versions same day without issue. Breaking changes and difficult upgrades is not something we want to do with Terraform (0.12 being a big exception as that was a very core "reset" so to speak). The reason there have been these changes in these recent releases is that we've been ensuring Terraform is in a place for 1.0 that we don't have to have difficult upgrades. You can see this path happening in each release: - Terraform 0.15: state file format stability - Terraform 0.14: provider dependency lock file - Terraform 0.13: required provider block (for more deterministic providers) - Terraform 0.12: stable JSON formats for plans/config parsing/etc. (particularly useful for interop with things like Terraform Cloud and 3rd party tooling) This was all done to lead up to a stable 1.0 release. As noted in the blog post, 0.15 is effectively a 1.0 pre-release so if everything goes well, we've MADE IT. For 1.0, we'll be outlining a number of very detailed compatibility promises which will make upgrades much easier going forward. :)

llbeansandrice 6y ago
Thanks, good to know that the upgrade to 12 is the biggest jump.
- john37386 6y ago
  I had the same question or concern. I also realized too late that 0.12 is a bigger one than first thought. I was not severely impacted in the end, but boy it was a long time that I didn't experience such a tough upgrade. Happy to know that the hardest is behind and looking forward to try 0.15. Thanks
0xbadcafebee 6y ago
Our teams have something like 100,000 LOC in Terraform 0.12, and it's not all in one big monorepo. At that scale there is no such thing as a relatively minor version upgrade. We want to upgrade to get away from some persistent 0.12 bugs, but we literally don't have the time. We have to change all of the code, and then test every single project that uses that code in non-prod, and pray that the testing finds most of the problems that will appear in production. And it's all owned by different groups and used in different projects, so that makes things longer/more complex. We also have to deal with provider version changes, upgrading CI pipelines and environments to be able to switch between Terraform binaries, and conventions to switch between code branches. I am already looking around for some way to remove Terraform from our org because it is slowly strangling our productivity. It's way too slow, there's too many footguns, it doesn't reliably predict changes, it breaks on apply like half of the time, and it's an arduous manual process to fix and clean up its broken state when it does eventually break. Not to mention just writing and testing the stuff takes forever, and the very obvious missing features like auto-generation and auto-import. I keep a channel just to rant about it. After Jenkins, Ansible and Puppet, it's one of those tools I dread but can't get away from.

jasonvorhe 6y ago

terraform 0.1xupgrade went pretty smoothly most of the time, with some minor manual changes and fixes here and there. Most of them were easily done in batch over the repository using sed. This approach worked for me in various setups from a small SaaS company to a major travel company as well as personal projects without issues.

uglygoblin 6y ago

I started using Terraform on our project in early 2019 which was version 0.11.13. The upgrade to 0.12.x seemed non-trivial so I put it off... now 2 years later and we're at 0.15.x. Looks like I need to clear my schedule in a upcoming sprint to get this done so the pain doesn't get even worse :)

dguaraglia 6y ago

As everyone else said: 0.11 to 0.12 was painful. After that, you just need to deal with converting the state whenever a new version comes out, but that's basically automated so not a biggie.

tetha 6y ago

From our experience. 0.11 -> 0.12: We're not attempting. We're in the process of changing out config management anyway and we're unhappy with a bunch of decisions in that terraform stack, so it's a good point to get rid of it. 0.12 -> 0.13: Well, we had to add what feels like a million required-provider-blocks and then some more. And sometimes it was tricky to pinpoint the module pulling in a default provider and crashing - `terraform providers` and `terraform graph` help there. The graph is easy to grep through to find the resources and modules pulling in wrong providers. And in the beginning the error message we got when we had to run 'terraform state upgrade-providers' was .. obscure. In newer versions, that message is much better. 0.13 -> 0.14 just happened and now the lock files are slowly piling in on demand.

garymoon 6y ago

I upgraded from 11 to 12 like one year ago and from 12 to 13 some days ago (upgrade to 14 seems that will be straightforward) but in my case what I did: - Don't upgrade directly from 12 to 14, go to 13 first - If you have warnings after moving from 11, fix them first - Run the 0.13upgrade command in your code that will generate the required_providers - Run terraform-v13 init - Change to correct workspace if using some - Run terraform-v13 plan which will probably fail due to the new explicit required-providers rule, if that happens you need to modify the state with the correct providers https://www.terraform.io/upgrade-guides/0-13.html#why-do-i-see-provider-during-init- . In my case I have a lot of modules so I created an script that automated that process - Execute again terraform-v13 plan and verify that it will not make uncommon changes - Then run terraform-v13 apply.

matwood 6y ago
> Don't upgrade directly from 12 to 14, go to 13 first The line above, plus running apply in each version is key. I literally just did the update from 11->latest for 3 different repos a couple of weeks ago. And tbh, its was only the first update where I had to make any code changes. The rest mostly worked.

skywhopper 6y ago

After 0.12 it has been much easier.

capableweb 6y ago

As no one seems to have mentioned anything, version numbers below 1.0 are generally considered unstable in terms of API interface (like HCL in this case) so if you'd like to avoid similar things in the future (with either Terraform or other tools), you're best to wait until they at least release version 1.0.