4 ms·

What are the arguments for TF being the better option for a new infrastructure build today? Given that a) Cloudformation with Custom Resources and Macros is ex

What are the arguments for TF being the better option for a new infrastructure build today? Given that a) Cloudformation with Custom Resources and Macros is extensible, AWS supported, and free and b) its hard to leverage many of the AWS services while still maintaining provider-neutrality (e.g Serverless Framework 2.0 on AWS runs on top of cloudformation templates now)

choochootrain 8y ago
i last used cloudformation over a year ago and it was a pretty shitty experience overall. bread and butter resources like vpcs would fail to provision in time which would cause a cascading failure and result in the entire stack failing and rolling back. also cleaning up stacks that failed to roll back was a bit of a pain when there were intermingled dependencies - the dashboard was about as unhelpful as possible in determining what order to delete things so i could carry on doing what i was actually trying to do. i chalk that up to aws resources being backed by an eventually consistent db and the fact that aws has never built dashboards that are better than "acceptable". it's possible that i'm missing something here but i've found terraform to be orders of magnitude better on both fronts. it takes a much more defensive stance on resource provisioning with retry logic and picking up on a previous `terraform apply` which timed out instead of rolling back the entire thing. and for whatever reason, terraform is usually way faster than cloudformation for provisioning the same resources.
chucky_z 8y ago
I used TF very early on, for versions 0.3 and 0.4. It was unpleasant at best, but very functional. I've began using it again with 0.11, and with 0.12 on the horizon it's much easier to use than it ever has been in the past. It used to be very difficult to do one-off tasks with Terraform but I've began managing things piecemeal and the community around importing everything into existing TF things are great. The statefile borking itself from time to time is still an issue, but no more than any other operational tool of this magnitude. The biggest downside to Terraform currently (0.11) is that you need to get extremely creative in ugly ways to accomplish some tasks. This can been seen very specifically in the AWS Security Group official TF module, which I could not live without, but would not want to write/maintain.
briffle 8y ago
I would say its definitely better for infrastructure building for anyone who does not use AWS..
djohnston 8y ago
i've used both CFN and TF on the same infra so i think i have a reasonable (although dated by 1 yr) perspective on this. my main gripe with CFN is that when things get into a weird state, it is extremely difficult to do anything about it. You are basically stuck waiting at the hands of the black magic box of AWS to either revert or complete the change set. i found that TF was much more clear about what went wrong when a changeset got into a bad state. i've also had no problems using AWS services via TF, with the added bonus that it's easy to get heterogeneous cloud deployments working.
kkapelon 8y ago
Not everybody is 100% on AWS. For people that work on multiple clouds, terraform is an obvious choice.
lostapathy 8y ago
Because not everyone runs everything on AWS?
fishnchips 8y ago
CloudFormation and Terraform are both excellent tools, and they're sufficiently different that I've seen folks successfully using both. The pattern I've seen and actually hope to implement at some point in my work is building a CloudFormation stack using Terraform and then yielding it to an external, specialised tool that will transactionally manage its lifecycle without all the faff of calling individual SDK methods.
outworlder 8y ago
If you are on AWS and don't expect to ever leave, then CloudFormation is ok. Not as elegant, but does the work well enough. If you are not on AWS you are out of luck.
Vinnl 8y ago
I've used TerraForm to string together (subsets of) AWS, Heroku and CloudFlare. Having set up quite a few greenfield projects recently, being able to re-use the same tool is great. The amount of effort required is so low now, that every project I set up has review apps - which really is a fantastic concept. TerraForm has been key. If you only use AWS, I'm sure CloudFormation is great too.
kokey 8y ago
Terraform, certainly. It was a bit rough around the edges about a year ago but it's been getting really good. I've only been using the free cli version. Number one advantage is that it's much better at telling you what's changing. For example, where CF will tell you it will change a security group, Terraform will tell you what specific rule it's going to update with what content. Terraform will tell you if it's going to have to wipe and replace an EC2 instance, or not, where CF will tell you it's 'conditional'. Number two advantage is that you can go from any state to getting back to managing things with Terraform again. You can start with a hand managed setup, one created by CF, or one created with TF and then broken by manual modifications, you name it, there is a way to get it back to being managed with Terraform. Sometimes it's not easy, but it's always possible. With Cloudformation there are many things that can be done where the only fix is to delete the stack and create it again. Then for the Serverless specific example, I recently replaced something that was deployed with that with Terraform. The Serverless deployment had some big problems because of Cloudformation, one was updating stacks with where Lambda functions needed interfaces in many VPCs, the other was some Cloudformation string limit issue where the only workaround was to delete the stack and create it again, which is super slow because of those network interfaces and there would be downtime while it's doing all this. With Terraform this went from being a risky deployment with potential downtime of many minutes, to Terraform only changing the resources that needed changing with each update, taking only seconds.
- ReidZB 8y ago
  I believe CloudFormation change-sets have more detailed information about "conditional" replacement cases. I also prefer Terraform. However, both Terraform and CloudFormation are warty in their own ways—this is a space where there's a lot of room for improvement. Fortunately, the as-yet-unreleased Terraform 0.12 [1] looks as though it'll resolve most of my complaints... [1]: https://www.hashicorp.com/blog/terraform-0-1-2-preview https://www.hashicorp.com/blog/terraform-0-1-2-preview
  - kokey 8y ago
    I've been generally impressed with how quickly niggly quirks with Terraform seems to go away with new versions. My main concern with Terraform is that there's nothing anywhere near a decent competitor out there.
wgjordan 8y ago
I've used CloudFormation to maintain a large-scale web-application on AWS infrastructure since 2016. I've been continuously evaluating Terraform over the years (it is indeed maturing quickly), and my team has stuck with CloudFormation for now though we're considering a potential future migration. A year ago I would have recommended CloudFormation for anyone managing an AWS-exclusive (or mostly-AWS) deployment, but today I would say both are equally good choices (and Terraform wins handily if your use-case includes _anything_ non-AWS.) In addition to being able to manage non-AWS infrastructure, here are some more arguments for TF being the better option for a new infrastructure build today: - CloudFormation is indeed AWS-supported, but it's also closed-source, which means that you never know exactly what is happening under the hood in resource implementations. I've spent days of back-and-forth communication with AWS support (weeks in one case when working with the AutoScalingRollingUpdate UpdatePolicy) trying to clarify/confirm implementation details for various resource-implementation edge-cases that were too time-consuming and/or risky to exhaustively test myself. With Terraform, I could just read the provider implementation code directly and confidently know exactly what it's going to do. - CloudFormation has an inconsistent and opaque development roadmap. Though CloudFormation generally has more complete coverage of AWS-service resources than Terraform's AWS provider (at least as of a year ago, I haven't done a more recent comparison), you will still encounter situations where a new service, or new API properties (reflecting new/updated features) on existing services, do not yet have a CloudFormation resource to match. Unlike the AWS SDKs which have super-quick, mostly automated release cycles, CloudFormation resource updates are developed separately and can take months or years until AWS happens to update the resource. Sure, you can deploy a Custom Resource to fill in the gaps with the SDK, but since it's community-supported, Terraform development is both much more responsive to incremental updates to evolving services and since the development is done in an open-source project it's much easier to track and know when to expect upcoming feature support. - CloudFormation doesn't support importing existing infrastructure. This adds a big extra barrier to evolving a complex system, as it prevents you from being able to incrementally create/modify resources through the console and then 'lock down' existing resources into your state/configuration after the fact. Instead, you need to re-create new resources within the stack and then migrate any dependencies over to the new resource. This migration effort can be substantial when dealing with live production systems- I once spent several weeks carefully migrating several old production CloudFront distributions to CloudFormation-managed resources, and still incurred several minutes of unexpected downtime along the way. This effort is completely eliminated using Terraform's import feature.
baylisscg 8y ago
I primarily deal with a weird OpenStack system but also bare metal, AWS, Azure, and whatever someone in authority decides we need to use this week. Terraform works, kinda, which is more than can be said for most of the alternatives.
devonkim 8y ago
CloudFormation is worse than Terraform in terms of tooling and simple expressions. We had to use a framework like Sceptre and write a bunch of Python functions, and Lambda functions for custom resources (and seriously, testing custom resources is not a quick turnaround thing if your function / endpoint doesn’t send a reply back with the right JSON because you have to wait about an hour for it to time out now) to get what data providers and some convenience functions are already baked into Terraform. Furthermore, AWS change sets even when using frameworks / wrappers are pretty clunky to use compared to TF style refresh && plan. Where CloudFormation wins by design is in deployment strategies like B/G deploys, rolling updates, etc. Nothing stops one from using a combination of Terraform for everything besides applications and to use CloudFormation for app-specific deployment resources like ASGs and Target Groups.
marenkay 8y ago
Never be bound to a single cloud provider. Ever. Same as before the cloud, whatever is important shall have multiple vendors offering it. Terraform is a) reliable and b) allows for 1-n providers. It's not just custom AWS, it's custom "ALL THE THINGS"

choochootrain 8y ago

i last used cloudformation over a year ago and it was a pretty shitty experience overall. bread and butter resources like vpcs would fail to provision in time which would cause a cascading failure and result in the entire stack failing and rolling back. also cleaning up stacks that failed to roll back was a bit of a pain when there were intermingled dependencies - the dashboard was about as unhelpful as possible in determining what order to delete things so i could carry on doing what i was actually trying to do. i chalk that up to aws resources being backed by an eventually consistent db and the fact that aws has never built dashboards that are better than "acceptable". it's possible that i'm missing something here but i've found terraform to be orders of magnitude better on both fronts. it takes a much more defensive stance on resource provisioning with retry logic and picking up on a previous `terraform apply` which timed out instead of rolling back the entire thing. and for whatever reason, terraform is usually way faster than cloudformation for provisioning the same resources.

chucky_z 8y ago

I used TF very early on, for versions 0.3 and 0.4. It was unpleasant at best, but very functional. I've began using it again with 0.11, and with 0.12 on the horizon it's much easier to use than it ever has been in the past. It used to be very difficult to do one-off tasks with Terraform but I've began managing things piecemeal and the community around importing everything into existing TF things are great. The statefile borking itself from time to time is still an issue, but no more than any other operational tool of this magnitude. The biggest downside to Terraform currently (0.11) is that you need to get extremely creative in ugly ways to accomplish some tasks. This can been seen very specifically in the AWS Security Group official TF module, which I could not live without, but would not want to write/maintain.

briffle 8y ago

I would say its definitely better for infrastructure building for anyone who does not use AWS..

djohnston 8y ago

i've used both CFN and TF on the same infra so i think i have a reasonable (although dated by 1 yr) perspective on this. my main gripe with CFN is that when things get into a weird state, it is extremely difficult to do anything about it. You are basically stuck waiting at the hands of the black magic box of AWS to either revert or complete the change set. i found that TF was much more clear about what went wrong when a changeset got into a bad state. i've also had no problems using AWS services via TF, with the added bonus that it's easy to get heterogeneous cloud deployments working.

kkapelon 8y ago

Not everybody is 100% on AWS. For people that work on multiple clouds, terraform is an obvious choice.

lostapathy 8y ago

Because not everyone runs everything on AWS?

fishnchips 8y ago

CloudFormation and Terraform are both excellent tools, and they're sufficiently different that I've seen folks successfully using both. The pattern I've seen and actually hope to implement at some point in my work is building a CloudFormation stack using Terraform and then yielding it to an external, specialised tool that will transactionally manage its lifecycle without all the faff of calling individual SDK methods.

outworlder 8y ago

If you are on AWS and don't expect to ever leave, then CloudFormation is ok. Not as elegant, but does the work well enough. If you are not on AWS you are out of luck.

Vinnl 8y ago

I've used TerraForm to string together (subsets of) AWS, Heroku and CloudFlare. Having set up quite a few greenfield projects recently, being able to re-use the same tool is great. The amount of effort required is so low now, that every project I set up has review apps - which really is a fantastic concept. TerraForm has been key. If you only use AWS, I'm sure CloudFormation is great too.

kokey 8y ago

Terraform, certainly. It was a bit rough around the edges about a year ago but it's been getting really good. I've only been using the free cli version. Number one advantage is that it's much better at telling you what's changing. For example, where CF will tell you it will change a security group, Terraform will tell you what specific rule it's going to update with what content. Terraform will tell you if it's going to have to wipe and replace an EC2 instance, or not, where CF will tell you it's 'conditional'. Number two advantage is that you can go from any state to getting back to managing things with Terraform again. You can start with a hand managed setup, one created by CF, or one created with TF and then broken by manual modifications, you name it, there is a way to get it back to being managed with Terraform. Sometimes it's not easy, but it's always possible. With Cloudformation there are many things that can be done where the only fix is to delete the stack and create it again. Then for the Serverless specific example, I recently replaced something that was deployed with that with Terraform. The Serverless deployment had some big problems because of Cloudformation, one was updating stacks with where Lambda functions needed interfaces in many VPCs, the other was some Cloudformation string limit issue where the only workaround was to delete the stack and create it again, which is super slow because of those network interfaces and there would be downtime while it's doing all this. With Terraform this went from being a risky deployment with potential downtime of many minutes, to Terraform only changing the resources that needed changing with each update, taking only seconds.

ReidZB 8y ago
I believe CloudFormation change-sets have more detailed information about "conditional" replacement cases. I also prefer Terraform. However, both Terraform and CloudFormation are warty in their own ways—this is a space where there's a lot of room for improvement. Fortunately, the as-yet-unreleased Terraform 0.12 [1] looks as though it'll resolve most of my complaints... [1]: https://www.hashicorp.com/blog/terraform-0-1-2-preview https://www.hashicorp.com/blog/terraform-0-1-2-preview
- kokey 8y ago
  I've been generally impressed with how quickly niggly quirks with Terraform seems to go away with new versions. My main concern with Terraform is that there's nothing anywhere near a decent competitor out there.

wgjordan 8y ago

I've used CloudFormation to maintain a large-scale web-application on AWS infrastructure since 2016. I've been continuously evaluating Terraform over the years (it is indeed maturing quickly), and my team has stuck with CloudFormation for now though we're considering a potential future migration. A year ago I would have recommended CloudFormation for anyone managing an AWS-exclusive (or mostly-AWS) deployment, but today I would say both are equally good choices (and Terraform wins handily if your use-case includes _anything_ non-AWS.) In addition to being able to manage non-AWS infrastructure, here are some more arguments for TF being the better option for a new infrastructure build today: - CloudFormation is indeed AWS-supported, but it's also closed-source, which means that you never know exactly what is happening under the hood in resource implementations. I've spent days of back-and-forth communication with AWS support (weeks in one case when working with the AutoScalingRollingUpdate UpdatePolicy) trying to clarify/confirm implementation details for various resource-implementation edge-cases that were too time-consuming and/or risky to exhaustively test myself. With Terraform, I could just read the provider implementation code directly and confidently know exactly what it's going to do. - CloudFormation has an inconsistent and opaque development roadmap. Though CloudFormation generally has more complete coverage of AWS-service resources than Terraform's AWS provider (at least as of a year ago, I haven't done a more recent comparison), you will still encounter situations where a new service, or new API properties (reflecting new/updated features) on existing services, do not yet have a CloudFormation resource to match. Unlike the AWS SDKs which have super-quick, mostly automated release cycles, CloudFormation resource updates are developed separately and can take months or years until AWS happens to update the resource. Sure, you can deploy a Custom Resource to fill in the gaps with the SDK, but since it's community-supported, Terraform development is both much more responsive to incremental updates to evolving services and since the development is done in an open-source project it's much easier to track and know when to expect upcoming feature support. - CloudFormation doesn't support importing existing infrastructure. This adds a big extra barrier to evolving a complex system, as it prevents you from being able to incrementally create/modify resources through the console and then 'lock down' existing resources into your state/configuration after the fact. Instead, you need to re-create new resources within the stack and then migrate any dependencies over to the new resource. This migration effort can be substantial when dealing with live production systems- I once spent several weeks carefully migrating several old production CloudFront distributions to CloudFormation-managed resources, and still incurred several minutes of unexpected downtime along the way. This effort is completely eliminated using Terraform's import feature.

baylisscg 8y ago

I primarily deal with a weird OpenStack system but also bare metal, AWS, Azure, and whatever someone in authority decides we need to use this week. Terraform works, kinda, which is more than can be said for most of the alternatives.

devonkim 8y ago

CloudFormation is worse than Terraform in terms of tooling and simple expressions. We had to use a framework like Sceptre and write a bunch of Python functions, and Lambda functions for custom resources (and seriously, testing custom resources is not a quick turnaround thing if your function / endpoint doesn’t send a reply back with the right JSON because you have to wait about an hour for it to time out now) to get what data providers and some convenience functions are already baked into Terraform. Furthermore, AWS change sets even when using frameworks / wrappers are pretty clunky to use compared to TF style refresh && plan. Where CloudFormation wins by design is in deployment strategies like B/G deploys, rolling updates, etc. Nothing stops one from using a combination of Terraform for everything besides applications and to use CloudFormation for app-specific deployment resources like ASGs and Target Groups.

marenkay 8y ago

Never be bound to a single cloud provider. Ever. Same as before the cloud, whatever is important shall have multiple vendors offering it. Terraform is a) reliable and b) allows for 1-n providers. It's not just custom AWS, it's custom "ALL THE THINGS"