42 ms·

Terraform vs. AWS CloudFormation

• johnl1479 5y ago
  I can appreciate the author's criticisms of the shortcomings of Cloudformation, but this is really just a "Why you should use Terraform" post. e: Title was changed
• thecopy 5y ago
  > With Terraform, your local executable makes rest calls to each service’s REST API for you, meaning no intermediary sits between you and the service you’re controlling. Want an RDS instance? Terraform will make calls directly to the RDS API. How is this different than CloudFormation making the same calls?
  • lykr0n 5y ago
    You give CloudFormation a list of instructions. It accepts it and gives you an ID to watch for updates, then it goes off and executes them. Terraform executes a list of instructions. It executes them in front of you while you wait. Both are fine until you run into something like this: I'm pushing a Elastic Container Service Task Definition change via CDK. A CloudFormation change is submitted, and I wait for it to finish. In the background, it's trying to do the update but the update fails due to some misconfiguration with the new container. CloudFormation doesn't fail or return an error. It times out after an hour and reverts the change. I have to know to dig into the AWS console to find my failed tasks to view the error. If I did this update via Terraform, I would get the error back in my console quickly as Terraform is directly telling ECS to make the change. With CDK, the CloudFormation changeset is generated, it is submitted to CloudFormation, then the tool polls the AWS API for progress updates. Sometimes you get specific messages back, sometimes it fails and you need to go in and see what it failed on.
• kennu 5y ago
  That's right - use AWS CDK instead. You don't have to worry about the low-level CloudFormation syntax and details. I switched a few years ago and haven't looked back. CDK keeps getting better and better, also handling things like asset deployments (Docker images, S3 content, bundling), quick Lambda updates with --hotswap, quick stack debugging with --no-rollback, etc.
  • fdgsdfogijq 5y ago
    I'm always surprised that more people arent aware of CDK. Its an extremely powerful way to write software. Especially once you get good at it. CFN pales in comparison, CDK to me feels like the future of software development.
    • kevinsundar 5y ago
      My comments are my own and don't represent my employer. Almost every new service built inside Amazon uses CDK. I too am surprised that more people aren't aware of it. And you're right once you get good at it you can spin up infrastructure incredibly quickly with minimal guess work.
      • shepherdjerred 5y ago
        CDK was not used extensively at my org a couple of months ago. I agree it’s light years ahead of everything else though.
        • kevinsundar 5y ago
          Oh that's interesting, in my experience across the Books Org and InfoSec its being widely used.
    • jamesfinlayson 5y ago
      Maybe CloudFormation is too entrenched at the moment? I think CDK became publicly available in late 2019 or early 2020. I haven't used it because I'm currently working on a project using Serverless (thin layer on top of CloudFormation) with no easy way to slip any CDK in.
    • takeda 5y ago
      I was at a point willing to try it and what dissuade me was just that I also needed to have nodejs added as a dependency to my project.
      • 5y ago
• draklor40 5y ago
  CloudFormation, with its HORRIBLE YAML templating (whatever dsl/language) and arcane error messages is a horror story. I hate it so much that I'd rather quit my job than debug why CloudFormation decided for no reason to update my RDS instance for a PR that was just a README file update.
  • emmanueloga_ 5y ago
    How about Pulumi? [1] Seems compatible with CF and supports TypeScript as configuration language. Any fans? 1: https://www.pulumi.com/docs/guides/adopting/from_aws/ https://www.pulumi.com/docs/guides/adopting/from_aws/
    • draklor40 5y ago
      Interesting. Worth taking a look at
• singlewind 5y ago
  To be honest, I don't agree this. Manage an infrastructure need evidence and trace how this get created. I've been in the situation a few times. Have been threw projects terraform code doesn't match aws infrastructure. We don't know when an how the drift happen. At least, cloudformation can have some feature to detect the difference and help me trace back which commit actually has been deployed. CDK make the job easier for developers because it deliver some convenience and offer more pattern to write code. I like both.
• Hikikomori 5y ago
  Vanilla cloudformation is bad, but so is terraform (for my use case anyway). We wrap our cloudformation with python, you need something similar for terraform to make it less terrible (cdktf, terragrunt, terrascript).
• nzoschke 5y ago
  Counterpoint... Use CloudFormation! Managed services offer big benefits over software. With CF, new stacks, change sets, updates, rollbacks and drift detection are an API call away. Managed service providers offer big benefits over software. With CF and AWS support, help with problems are a support ticket away. Using a single cloud provider has a big benefit over a multi-cloud tooling. I only run workloads on AWS, so the CF syntax, specs and docs unlocks endless first party features. A portable Terraform + Kubernetes contraption is a lowest common denominator approach. Of course everything depends. I've configured literally 1000s of systems with CloudFormation with very few problems. I have seen Terraform turn into a tire-fire of migrations from state files to Terraform enterprise to Atlantis that took an entire DevOps team to care for.
  • dolni 5y ago
    > I've configured literally 1000s of systems with CloudFormation with very few problems. This is a great way of saying "I've never used CloudFormation" without stating it directly.
    • macintux 5y ago
      > Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith. Hacker News is not a site for snarky accusations of deception. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  • ldoughty 5y ago
    Agree. CF is not a magic bullet, but neither is ansible or terraform. We used ansible heavily with AWS for 2 years. Then we decided to gut it out and do CF directly. Why? If we want to switch clouds, it's not like the ansible or terraform modules are transferable ... So might as well go the native supported route. I agree with the article, messages can be cryptic, but at the end of the day, I have a CF stack that represents an entity. I can blow away the stack, and if there's any failure or issue, I can escalate my permissions and kill it again. Still a problem? Then it's AWS's fault and a ticket away (though I've only had to do this once in 5 years and > 150,000 CF stacks. I also would argue, if a stack deletion stalls development, you are probably using hard-coded stack names, which isn't wise. Throw in a "random" value like a commit or pipeline identifier. I've had far less issues with CF than terraform or ansible. I have yet to see CF break backward compatibility, while I had a nightmare day when I couldn't run any playbooks in ansible because the module had a new required parameter on a minor or patch version bump.l (which was when I called it quits on ansible, I then relooked at terraform, and decided to go native) I will caveat that our use case for AWS involves LOTS of creation and deletion, so I find it super helpful to manage my infrastructure in "stacks" that are created and deleted as a unit.. I dont need to worry about partial creations or deletions.. like ever... It basically never fails redoing known-working stuff... Only "first time" and usually because we follow least-privilege heavily
• booleanbetrayal 5y ago
  Yeah, importing existing resources into Cloudformation is a nightmare in "Am I going to break everything? Fingers Crossed". It is also very possible to get into very bad situations if your settings drift and you attempt to reconcile those changes.
• robohoe 5y ago
  That's right, don't use CloudFormation. Use CDK which will generate and obfuscate CF for you and you won't have to worry about it.
  • Arelius 5y ago
    I'm not sure I understand... Is obfuscating the CF a good thing?
    • yjftsjthsd-h 5y ago
      I'm pretty sure that was sarcasm. I disagree with said sarcasm, because CDK takes you one layer away from the actual thing that gets run but gives you a much nicer thing to work with so it can still be a good trade off; writing rust (or whatever) "obfuscates" the underlying CPU instructions but it still turns out to be a good idea.
• robohoe 5y ago
  I will admit that troubleshooting permissions-related deployment issues in StackSets are a super nightmare inducing events.
• zenux 5y ago
  Fun fact: in the leak of the Twitch (Amazon) repositories of this morning, I saw that the developers use Terraform !
• cube2222 5y ago
  You can use more than one tool. CloudFormation is great because of its transactionality, so it lends itself nicely to deploying multiple services which are versioned together. You either succeed fully, or all services will be rolled back. This way you can deploy your whole infra with Terraform, and then deploy to your i.e. ECS cluster using CloudFormation. Works great in practice.
  • zapt02 5y ago
    The rollback functionality of CF is a blessing. We use both CF and Terraform at my company and i vividly recall multiple times where my connection had cut out during "terraform apply" and left the Terraform infrastructure in a half-finished state.
    • nickjj 5y ago
      Rollback doesn't always work with CF. I've noticed so many times that it would mostly delete everything but not certain things once in a while. Then you're left having to play detective to manually figure out what you need to delete while having to delete dependencies by hand in a specific order. I've spent hours just waiting for CF to fail deleting EKS or RDS related resources then I end up getting billed for $30+ a month sometimes because I forgot to manually delete a NAT gateway.
    • acdha 5y ago
      > The rollback functionality of CF is a blessing When it works, which is a big caveat: we had far more cases where it failed in a way which required manual remediation and the gaps in validation meant that you'd be in a “apply / error / rollback” loop requiring 20+ minutes before you could try again. Terraform was always considerably faster but it was especially the orders of magnitude improvement in retry time which convinced most of us to switch. The CloudFormation team has been working on this so it's possible that experience has improved but the scar tissue will take time to fade.
      • knodi 5y ago
        This!! when you stack get stuck in error state is the worst!! Hello kinesis stream rollbacks.
• l0b0 5y ago
  Unless things have changed in the meantime, the killer feature of CloudFormation for me is that I don't have to keep track of the state locally. Having to set up tracking of the infra state in Terraform is a huge pain, since it should be stored independently of both the infra code (to allow deploying anything but HEAD) and the infra itself (duh). As long as Terraform doesn't query the existing infra to work out what needs doing I don't want to go back to it.
  • satyrnein 5y ago
    Terraform Cloud will store the state for you, for what it's worth.
    • l0b0 5y ago
      On a platform I don't control, separate from the cloud provider I'm deploying to and requiring both billing and PII, I suppose? No thanks.
• Pensacola 5y ago
  While the read was interesting and informative, something about the tone made me search for a disclaimer/disclosure of interest. Are you an "influencer?"
• orf 5y ago
  I spent a bit of time trying to deploy a lambda app with Cloudformation. I wanted to use a relational database, so I needed to handle migrations. Ok, so apparently I need to write a custom Cloudformation resource to execute a lambda function that will run the migrations prior to deploying the new version of the lambda. Kind of neat that you can do that. Except I messed up the output of the custom resource lambda and Cloudformation completely locked my deployment up for 3 hours. 3 hours. I couldn't do _anything_ - rollback, update, whatever. Cloudformation via a CDK is interesting, and I don't hate it, but oh boy if it gets into a weird state it can completely kill your iteration loop. And the docs say something along the lines of "if it's stuck for too long contact support". No thanks.
  • zapt02 5y ago
    CF does have a lot of quirks (especially stacks locking up for various reasons, or rollbacks taking hours). I find it easiest to run migrations when an application is first starting up (with an appropriate transaction lock so other instances won't cause the migration to run more than once), this way you don't have to do a lot of devops magic for it to work.
    • orf 5y ago
      That’s not a fantastic model, especially for short lived lambdas. In some small environments just running a migration check on boot is passable, especially if you don’t have better tooling to handle this, but I don’t want to double my execution time working out if I need to run a migration or not in a lambda. I guess I could do the migration check in the lambda startup rather than in the request path, but it’s still not ideal especially when handling rollbacks.
  • mellavora 5y ago
    I might have preferred to use SAM, which uses CF under the hood but is much more focused on getting lambdas to work. I'd pass in DB settings as parameters to the SAM file, rendering them as env. variables for the lambdas. with DB access managed by IAM roles (so you don't need sensitive info in the lambda env). And depending on size/scope of project/team, might have one CF for the DB infra, and another for the app lambdas. Using IaC services means I can have both dev and prod infra, so I can test DB migrations. And I can do this as a solo dev on a side project where I'm also responsible for front-end and the other back-end stuff plus keeping clients happy on evenings/weekends.
• easton 5y ago
  Something funny (well, kind of sad) about CloudFormation I noticed this summer was that if you deploy a CloudFormation stack which updates a ECS service and deploys tasks which then fail health checks, CloudFormation will do nothing about this and just let ECS keep killing and restarting tasks for.. well, at least several hours. You have to know to go into ECS and drain the tasks manually and then initiate a rollback from CF to get your service back into a good state. The bug reports about this I found were going back years. The upside is that I got really well acquainted with how ECS worked.
  • tkahnoski 5y ago
    100x this. Prior company committed to doing Infrastructure as Code and CloudFormation worked well except for this hiccup. We didn't even have that many services on ECS but we probably had 1 ticket a week asking support to help us with a 'stuck' stack. Our commitment to CloudFormation was doubled down on that we could do containers, Lambda, and 95% of any other AWS Services.... However, in hidsight using SAM and the ECS CLI probably would have resulted in a more predictable CI/CD process as we weren't fighting deploy semantics through CloudFormation abstraction.
    • viraptor 5y ago
      What we ended up doing is mostly decoupling CF and tasks. CF manages the infrastructure. Software deployments / tasks are done by another system which understands issues like this one.
  • fictionfuture 5y ago
    I had this same bug!! Cost us like $1000 before we fixed it;
    • misiti3780 5y ago
      Why did it cost so much money ?
      • deleted 5y ago
• cmaggiulli 5y ago
  Writing Terraform scripts for AWS is 70% of my job. I do have some issues with the AWS provider in Terraform. Firstly, there are bugs. I ran into a bug a few days ago where the ARN attribute on a Lamba alias was resolving to the ARN of the Lambda, not it’s alias. I only figured it out because I found a GitHub Issue. Additionally, Hashicorp is often playing catch-up with Amazon. A few days ago AWS released a new instruction set architecture for Lambdas that would save my org a lot of money. However after I saw the announcement in AWS I see tons of different GitHub issues created to add this functionality. So I start editing my files based off the documentation only for that issue to be closed and pointed to a new one with different syntax. So I start working off the new syntax only for that issue to close and be pointed to a different one
  • softveda 5y ago
    This will slowly change with AWS Cloud Control API. https://aws.amazon.com/blogs/aws/announcing-aws-cloud-control-api/ https://aws.amazon.com/blogs/aws/announcing-aws-cloud-contro...
• mylons 5y ago
  "But CDK transpiles into CloudFormation templates. For that reason alone I can’t recommend it." CDK is superior to terraform for a glaring reason: it's a first class citizen in AWS' eyes and terraform is not.
  • erik_seaberg 5y ago
    Their service owners don’t seem to feel that way. Terraform’s AWS provider often adds support for new services and features weeks or months before CloudFormation does.
  • nsonha 5y ago
    I've seen this argument everywhere. If this is the web, people would say "use the platform", because aparently any abstraction made on top of that is automatically worse, if this is .Net it would be "use Azure". If this is desktop applications... use C++? That's just not how reality works. First class citizens can suck, because guess what, they're designed by mortals
    • mylons 5y ago
      not the case with cdk. i hate IAC in general. terraform being one of my least favorites due to mostly being declarative, verbose, and the way it manages state. cdk is a large improvement and moves towards just being code.
• flurie 5y ago
  One of the most amazing things I saw at AWS reInvent was an advanced talk on IaC that provided the code of a lambda function inline in a CloudFormation template. I realize that this is just one talk, and there are plenty of ways to structure things well, but this practice is directly encouraged by the design of CloudFormation[1]. AWS has attempted redefining the lambda deployment story multiple times, there are multiple companies whose primary offering is providing a better way to deploy code to serverless offerings, but this still stands out to me as one of the most terrible ways to do things, and I blame the design of CloudFormation. [1] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-code.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
  • xyzzy123 5y ago
    I'm going off track here but Pulumi have a totally mind-bending feature where you can write the code of a lambda function not only inline, but such that it captures the value of variables from the surrounding infra code at the time the function is serialized. See: https://www.pulumi.com/docs/intro/concepts/function-serialization/ https://www.pulumi.com/docs/intro/concepts/function-serializ... Seeing the specific examples they use it for (AWS infra glue) makes me think that there is room for infrastructure related lambdas to be defined right in cfn or infra code, with very low ceremony, even if you wouldn't want to deploy "applications" like that.
    • galaxyLogic 5y ago
      I'm working with what I think is a similar problem-space, trying to create a Node.js program that produces code to be run on the browser. The code runs in Node.js at "build time" which essentially means constructing the static web-site, which means some JavaScript must be somehow "deployed" to the website to be then later run at "run time". So deploying to an AWS etc. Lambda seems conceptually similar to deploying from static-site-generator to a static web-site source-code, which is then what the browser will execute. Or is it?
• jasonhansel 5y ago
  Can we just agree that none of these tools (Terraform, CloudFormation, CDK) are particularly good? None of them really achieve the goal of purely declarative, immutable, stateless infrastructure-as-code and all of them have significant disadvantages.
  • ahzhou 5y ago
    Agreed. We use Pulumi and it's fine, but the state-managing feels brittle. I think the state-of-the-art is Kubernetes + jsonnet, though you still need to suffer through the learning curve and you'll still need TF and company to provision some of your cloud resources.
• kyleblarson 5y ago
  I've recently started using CDK (with Typescript) on a new project and find it to be great, much more so than Terraform.
  • Aeolun 5y ago
    I’ve started using Pulumi, which I understand is basically the same, except multi cloud/service.
• alde 5y ago
  The C in IaC stands for Code, and code is best written in proper programming languages. Both CF and Terraform are greatly limited by their use of yaml and hcl. Tools like AWS CDK and Pulumi, especially when used with a typed language like TS, are the future of IaC.
  • nsonha 5y ago
    you are arguing about another thing, Teraform also has a CDK. But the question isn't whether to use CDK, it's "native" vs third party (Terraform & Pulumi)
• zenlot 5y ago
  I came here to say that Pulumi is the answer, but people already mentioned it. It's TF under the hood, some of the docs links even refer to the links in TF documentation [not intentionally, just by copy and paste], but they made the experience so much better. Writing code in Typescript and seeing your infrastructure provisioned with all the goodies Pulumi added on top of TF is pure joy. No more HCL, or other "here's my DSL, learn it" scenarios.
  • Hawxy 5y ago
    I recently did a comparison of Pulumi and CDK. Pulumi's AWS abstractions are sadly nowhere near as good as CDK patterns, resulting in far more code needing to be written for the same result. I've also noticed that Pulumi seems to have a hard time accurately tracking state for AWS resources. Pulumi does work very well in the Azure space as Azure resources are already well abstracted.
  • nsonha 5y ago
    if it's Tf under the hood then could it be better than than TF's own CDK?
    • narism 5y ago
      The Pulumi providers for major clouds (AWS/GCP/Azure) have native versions instead of using the TF providers under the hood. Caveat: some are still in preview. https://www.pulumi.com/blog/pulumiup-native-providers/ https://www.pulumi.com/blog/pulumiup-native-providers/
  • bleepblooop 5y ago
    Pulumi is not "Terraform under the hood" in any sense than that some of the providers are forks of Terraform providers. The engine is entirely separate and, now that they have a foothold with it, they are building out providers that are built around the Pulumi execution model first-class, rather than an adapter layer. One notable core difference in the Pulumi model is that it doesn't have "plan" in the same sense that Terraform does, where the set of actions is fixed and just executed once you accept it, returning an error if that turns out not to be possible. "pulumi preview" just runs the program in a dry-run mode, so there's no guarantee that the subsequent apply will match unless you've been careful to write a totally deterministic program that always behaves the same with placeholder values as it does for concrete values. That's a typical tradeoff for using a DSL vs. a general-purpose language though, and one that seems to pay off for a lot of teams.
• shog_hn 5y ago
  I personally prefer aws-cdk over Terraform for the specific use case of writing complex Step Function State Machines. Using code to link and chain steps and logic together is much easier than HCL imo. For everything else I prefer Terraform though.
• zmmmmm 5y ago
  I find it mildly disturbing to see how willing people seem to be at a personal level to attach themselves skill-wise to single-vendor solutions. I get that CloudFormation may be a bit better than Terraform in this way or that way. But I have no personal interest in learning it. All that investment of time and effort and when I need to do a project on a different vendor I have nothing, no skills. What's the point? To be clear: I understand why companies and even specific teams or projects may choose to go all in on vendor-lock-in. It's often the most direct line to the business value they want and as people point out, it's rare to never that most cloud native applications will shift cloud vendors, and even if they do most of a Terraform setup will have to be reworked. So I get why companies and managers want that. I just don't understand why software engineers in those teams are so happy to go along.
  • Kreotiko 5y ago
    I don’t see how you couldn’t learn both. Terraform is dead simple to learn and cloud formation is the best way to define your infrastructure in code in AWS. Even more wrapped by CDK. I don’t think you would go that far career wise knowing only with Terraform for IaC. I also don’t understand those organisations that use every single AWS managed solution (RDS, Redis, Elasticsearch, EKS) but then they feel the need to manage their infrastructure state themselves with Terraform instead of choosing the managed solution for this.
    • kenerwin88 5y ago
      As a former AWS employee, Terraform is MUCH better at scale for AWS resources than Cloudformation or even AWS CDK. CDKs reliance on CloudFormation underneath means being limited to the exact same problems that are mentioned in the article, as well as managing resources at enormous scale incredibly painful.
  • deleted 5y ago
    [deleted]
  • derefr 5y ago
    Often, these single-vendor technologies are thin syntactic and nomenclaturial veneers over actually-quite-general abstractions. E.g. it's very easy, if you want or if you need, to rewrite a CloudFormation stack as a Terraform project, or a even a set of Kubernetes custom-resource-controller resource manifests. They're all analogous technologies; the and the "hard part" of an implementation in any of them isn't to do with the things that make them different, but rather the things they share in common — the formal modelling of a digraph of dependent declarative infrastructure resources.
• manca 5y ago
  There is a local company here in Seattle called Pulumi[1]. We've been using their tool extensively over the last 18+ months and it's pleasure to use. It's built on top of Terraform, but it simplifies IaC because you can now write your infra components in one of few high-level languages they support (JS, Go, Python, etc). CDK is similar, but biased towards AWS. With Pulumi you can provision your infra stuff in multiple clouds easily. [1] https://www.pulumi.com/ https://www.pulumi.com/
• nanis 5y ago
  terraform plan has the benefit of showing me whether what I did is at least internally logically consistent before making any attempts to change the state of the world. It doesn't mean things can't go wrong, but it is harder to make silly mistakes that are bound to happen when you try to inject logic into JSON or YAML which are just text. "HealthCheck": { "Target": { "Fn::Join": [ "", [ "HTTP:", { "Ref": "WebServerPort" }, "/" ] ] }, Sure, the equivalent terraform would not win a beauty context either but this is just an abomination. If someone is paying me well enough, I can live with it, especially if it worked perfectly, but it doesn't. Plus, standard CloudFormation crap Amazon will push on you tends to assume a bunch of things that may or may not be true in an account that has existed for a while before a particular feature became available.
  • chrisoverzero 5y ago
    I’m not so sure that’s a great example, since it could be written like this: HealthCheck: Target: !Sub HTTP:${WebServerPort}/ CloudFormation also has changesets which you can inspect to see what will change and why before applying them. `aws cloudformation deploy` is not all of CloudFormation.
    • nanis 5y ago
      I am not necessarily here to debate canonical or good CloudFormation ... I took one particularly unappealing snippet that does not reveal anything that I should not be disclosing from a bit of AWS provided CloudFormation. Regardless of how you write it, whether you write JSON or YAML, you are writing logic in pure text and the difference between that vs your code going through terraform's solver is night and day IMO. Plus, in source control, I find terraform diffs much easier to grok than JSON or YAML diffs. Especially since with the latter one misplaced invisible character can pass through without being noticed and cause disaster.
• nsonha 5y ago
  Terraform doesn't mean that you want the option to go with other cloud providers, it just mean that you want to treat your infrastructure the same way, regardless if it's is a cloud resource in AWS or something that AWS doesn even offer, such as email, analytics and so on. I don't know if it's still the case but AWS used to not have a rabbitmq offering.
• WatchDog 5y ago
  The state of things as I see it: Terraform - (bad) HCL sucks, not a programming language. - (good) One tool for multiple service providers is nice. - (good) Fully open source. - (good) directly calling service API's makes it more debuggable, and greater control over performance/time of deployments - (bad) directly calling service API's makes the agent running the tool a SPOF Cloudformation - (bad) YAML sucks, not a programming language. - (good) managed service, doesn't require an agent to run the deployment - (bad) often slow, no visibility into what it's doing under the hood - (bad) AWS specific(yes I know about public registry, but it still requires AWS) Pulumi - (good) can use an actual real programming language!! - (bad) directly calling service API's makes the agent running the tool a SPOF - (bad) need to be connected to, and have credentials for your environment to develop - (bad) running your code requires bi-directional communication with the pulumi GRPC api CDK - (good) can use an actual real programming language!! - (good) produces a declarative output - (bad) dependent on cloudformation - (unknown) CDK for terraform(I haven't tried this, but it sounds promising) I don't think we yet have the tool we need, but CDK is pretty close, having a real programming language, that produces a declarative intermediate format, that is then ran by another system, I think is the right approach.
  • jchanimal 5y ago
    The TypeScript CDK is the best use of TypeScript I've experienced. Just having all the documentation inline is hugely helpful, but the type checking also makes development faster by catching errors before you deploy (or even validate.)
  • jen20 5y ago
    You're not seeing them strictly correctly, then: All the tools described are declarative. If "directly calling service API's makes the agent running the tool a SPOF" is a "bad" for Terraform, it also must be for Pulumi. The idea that one "need[s] to be connected to, and have credentials for your environment to develop" to use Pulumi is equally true of Terraform, assuming by "credentials" you mean "credentials for the target provisioning platform". The idea that "running your code requires bi-directional communication with the pulumi GRPC api" is also untrue. It's important to be accurate when making these kinds of comparisons, lest you fuel memes instead of providing insight.
• ahzhou 5y ago
  I think the best solution is K8s and some tool to help you generate the yaml files. Terraform, CDK, and Pulumi all work similarly. At the end of the day, you declare a DAG that says how you want your infrastructure to work and use that DAG to come up with a list of changes you should make. Then you execute each of those changes via API calls and hope that reality on your cloud provider really does reflect the state DAG that you've built. CDK is better than the rest here, because it's got the best ability to measure the state. When you work with K8s, on the other hand, you tell your infrastructure what shape it should be in with yaml files and then let your infrastructure figure out the best way to get there. When combined with something like ArgoCD, you've got a really powerful deployment stack. The obvious downside is you've got to figure out how to get that all working.
  • orf 5y ago
    What does “measure the state” mean?
  • pid-1 5y ago
    I really didn't understand the state thing nor how CDK and Terraform aren't declarative. That said, I like how K8s + YAML allows a more "self service" like approach to IaC. In my org the goal of our devops team is to specify some common patterns for applications, then allow devs to use themselves. Both CDK and TF are really bad dealing with several apps spread over different repos that share resources (e.g.: ECS Clusters, VPCs, etc...).
    • tmerse 5y ago
      Both CDK and TF are really bad dealing with several apps spread over different repos that share resources (e.g.: ECS Clusters, VPCs, etc...). I had good experience by going a submodule route. One multirepo containing a CDK infra submodule and multiple repositories as submodules which each output resources (lambda code, etc.) which in turn can be imported more easily by cdk.
  • jpgvm 5y ago
    k8s is fundamentally just a much better API. YAML on the other hand is awful so I recommend Jsonnet instead to generate the k8s manifests. I currently use Tanka for this.
• vjust 5y ago
  I would prefer Terraform. I can transfer those skills to GCP, or Azure or something else. CloudFormation is probably advantageous if your infrastructure is 100% AWS..
• jamesmishra 5y ago
  I found both CloudFormation and Terraform's AWS provider to be too verbose for quickly spinning up a lot of infrastructure as a startup. So I ended up creating https://provose.com/ https://provose.com/ own high-level wrapper for Terraform's AWS provider. I can happily spin up VPCs with AWS ECS containers, RDS databases, EFS and FSx Lustre filesystems, and more. I really believe in the Terraform ecosystem, but I do have to admit that it is hard to write a Terraform module or provider that provides a significant level of abstraction over low-level cloud provider APIs. I've seen this issue drive a lot of other people to Pulumi or the AWS CDK, but I think Terraform is a better product specifically because Terraform HCL is not a (trivially) Turing-complete programming language.
  • nkotov 5y ago
    Awesome project, I'll check it out. We're building something similar but visually instead of code.
• ebfe1 5y ago
  Personally I like to use both and for specific jobs. For example, when we have tools that need to be deployed across 10+ AWS accounts managed by different Ops team, I hand them a CloudFormation template and they could run it, plug in the right parameters, pulling lambda code from the same S3 bucket we have etc... Totally agree with Writing Cloudformation is a pain but when you have it done once, it works consistently and we don't have to worry about terraform version, the tfstate etc... It just works. I use terraform for more complicated setup, an environment that we keep adding ontop, share & manage among our team and need rebuild/redeploy often/quickly or an environment we need to spin up for various tasks (eg incident handling VPC, interview challenges, CTF events...) ... Terraform and its powerful reusable terraform modules/module registry make spinning up these environments in minutes make it extremely attractive. Managing terraform version and tfstate is still a pain with terraform even with the help of remote S3 bucket & dynamodb lock but it is definitely better than when i first started and we had gpg encrypted tfstate. With that said, I work in security and only very occasionally I need a big deployment like scalable spark cluster or multi-zone elasticsearch cluster etc... so perhaps I don't have enough indepth knowledge about each tools.
• mikesabbagh 5y ago
  I stopped arguing about which is better CF or TF. Some people go crazy every time this topic is discussed. Me, I just hate all this IAC thing. I spend weeks sometimes to deliver a script that deploys an rds database with blah blah blah. Yeah it is nice to have all your infrastructure as code. It would be definitely useful if you are deploying the same structure again and again. or if you are running it frequently. But if you are deploying a structure once only, do you really need this? Do you have to spend 2 weeks on a script you will re-run once in 6 months? (and then you will find that many objects need to be updated to re-run without errors) It seems a lot of over-engineering for me (I know I will be hated so much for this). Sometimes when I run TF in production, I start praying that all goes well!
  • trhway 5y ago
    >I spend weeks sometimes to deliver a script that deploys an rds database with blah blah blah and this is $500K+/year in SV. What other area pays that well for such little delivery? Probably only AI :)
  • cliffy 5y ago
    So... what happens when the person who originally deployed everything eventually leaves or gets hit by a bus? Hey, it's six months later and you need to redeploy everything again. You need some automation or a really good how-to document, and hopefully both. It won't save you all the pain of learning how the deployment works but it will save you a lot of time and effort.
  • busterarm 5y ago
    This is why companies like Gruntwork have pre-written production-ready Terraform modules for just about any service/provider you would need. I have my own set of modules that I've written and use in my consulting work. It's repeatable work that I can have tailored to my clients in under an hour. Terraform puts dollars in my pocket.
    • raffraffraff 5y ago
      This. I haven't done that yet, but I'm thinking about going that route. My last job was with a start-up. I got to build everything from scratch, and I was 100% focused on infrastructure. I promised myself that everything new thing the team need would be deployed for them manually, then thoroughly researched and redeployed via IaaC asap. That meant that I was always delivering work that the team need. It also meant that I could take their request and build a really useful module from it. Example: nobody ever "just wanted" an SQS queue. They wanted a queue, various different sized workers in auto-scaling group with metrics to trigger scaling events. And logs. And probably an s3 bucket. And a deadletter queue. And all the security policies to allow those things to communicate with each other safely. Oh and custom metrics with alarms that notify you when your queries fill up. So I would build a single module that did all that stuff, using sensible and well-documented defaults. It was executed via for loop that iterated over a yaml structure. When the dev team wanted a new queue+workers, they edited the yaml (Hiera, in this case), added a few lines to the 'sqs-worker-groups:' block, and sent me a merge request. I would make sure they read the module readme, ask a few questions to make sure the foot-gun want cocked, and merge I know some people have said that the developers in their company write their own terraform. That's really neat, but do you get solutions like this, where a single person is focused on writing the DRYest possible code that can be re-used by everyone? When devs write infrastructure-as-code, there are also soooo many foot-guns that they all have to learn about over and over. Things that a seasoned infra engineer will have shot themselves with once and never again (s3 permissions, IOPS, egress costs, using us-east-1 etc). How many developers would bother studying the AWS infra-focused exam material?
• throwaway984393 5y ago
  This whole article is a case not to use Terraform. If you are creating an RDS cluster, CloudFormation will call the RDS REST endpoint for you. And so AWS takes care of things like retries, and you won't hit AWS's API call limit. With Terraform, the error messages you get are usually much more informative Often the errors are completely cryptic, or even misleading. Anyway, this isn't really a reason not to use CF or TF, it's a reason not to use AWS. With Terraform, you need to use valid credentials that grant access to multiple accounts. This is a security and reliability concern. Not only that most people end up making one credential which can access everything, but that people then end up running Terraform from one place, rather than running it per-account/per-region/per-AZ, which is much more resistant to failure and separates security domains. CloudFormation’s pseudo parameters and intrinsic functions are cruel jokes. By contrast, Terraform offers a rich set of data sources and transforming data with Terraform is a breeze. The more complicated you get with Terraform, the worse it works. This is actually a truism of all DSLs, but when you're using a DSL to talk to a program to talk to an API to talk to a program, it gets even worse. For example, some AWS resources will not work with Terraform properly unless you remember to apply the correct lifecycle policy (they'll go into a create/destroy loop, or never be able to be destroyed, or created, etc). With CF, AWS is taking care of that decision for you. AWS CloudFormation is S-L-O-W. Someone hasn't tried to apply a Terraform root module that manages hundreds of different resources, I see... Terraform can take dozens of minutes, to hours, depending on how carefully you've crafted your root modules, or your network latency, or even problems with AWS's TF provider. And if you're using STS creds that expire, it can leave you with broken infra / a broken remote state. Terraform, on the other hand, will occupy your shell until the directly-involved AWS service coughs up an error. No additional tooling is required. There's like 5 different hosted cloud services dedicated just to being able to run Terraform for large teams without it breaking or taking forever. On top of that, often Terraform will fail to apply, half way through applying, because Terraform doesn't ask APIs to validate all the operations before it starts performing them, and it often doesn't even check whether the operation it wants to do makes sense. It is very common for 'terraform apply' to break your Production infrastructure because it was literally impossible for you to know it would work until you tried it. If you learn AWS CloudFormation, then guess what: you can’t take your skills with you. This is almost a legitimate criticism, but it's not like using CF stops you from using Terraform too. You can use either, or both (just not on the same resources). Useful providers you can run on your own laptop include TLS key generators and random string generators. I highly recommend not using these. Makes it harder to run TF deterministically, and you want as much determinism as you can get in Immutable IaC. Importing Existing Resources This is a manual nightmare with TF. Terraformer, a completely different tool, makes this easier.
• eranation 5y ago
  I'm ex AWS so I used CloudFormation sort of because I had to (I guess, no one told me not to use terraform, but it felt wrong not to drink our own champaign). I left AWS to co-found a startup and when I had to pick between the two, I just used what I knew already (CloudFormation, but more specifically CDK). I have to say I am highly tempted to give Terraform a look, but if I do, it will be terraform + CDK probably: https://github.com/hashicorp/terraform-cdk https://github.com/hashicorp/terraform-cdk For me troubleshooting, speed, and that punch in the gut feeling when you see after 30 mins of crunching the dreadful - update failed rollback in progress - are great reasons to give Terraform a try. AWS should fix this by making the SDK/API be 1:1 with infra as code. The result of a describe call should === the stuff needed to create that thing declaratively. CDK is an amazing project, their high level constructs are making AWS SAM / Serverless framework / Amplify seem complex. With a line of code I get a best-practice opinionated VPC, an ECS cluster, a Fargate task with an ALB. (off topic - be careful - always ensure tasks return 200 OK on a GET / or it will thrash for hours, as others commented below, it's a known bug, sadly I knew about it only after wasting hours on troubleshooting it) So I agree perhaps CFN has native issues (speed, troubleshooting) but don't hate CDK just because it uses CFN in the back. So if you have to choose between CloudFormation or Terraform, I'd choose CDK.
  • wikibob 5y ago
    Terraform CDK is very very far behind AWS SDK. There’s essentially no Level 2 constructs.
    • eranation 5y ago
      Got it. Thanks for the heads-up! My hope is that CloudFormation will keep getting faster and easier to troubleshoot. And that the investment in CDK will pay off... but I'm going to start dabbing with terraform on the side... (although I rather my infrastructure as code to be in, well, code, I don't think yaml / json counts as code, but I won't let that distract me...)
      • robertlagrant 5y ago
        > although I rather my infrastructure as code to be in, well, code, I don't think yaml / json counts as code, but I won't let that distract me... Yes, quite.
• gattacamovie 5y ago
  maybe we skip the need and move to https://crossplane.io https://crossplane.io, to abstract all this part as well as gain 2nd day operations natively, and k8s readiness. (bonus: the closest path to cloud agnostic definition)
• pharmakom 5y ago
  Terraform is a better interface to AWS than CloudFormation. State is a crucial feature imo.
• jillesvangurp 5y ago
  The problem with both is that you quickly accumulate weeks/months of accumulated coding time that costs a pretty penny at market freelance rates. Spending a few hundred K on devops is routine for a lot of companies. My main issue with this is that a lot of that coding is just reinventing the same wheels over and over again. Jumping through many hoops along the way like a trained poodle. It's stupid. Why is this so hard & tedious. I've seen enough infrastructure as code projects over the last fifteen years to know that actually very little has changed in terms of the type of thing people build with these tools. There's always a vpc with a bunch of subnets for the different AZs. Inside those go vms that run whatever (typically dockerized these days) with a load balancer in front of it and some cruft on the side for managed services like databases, queues, etc. The LB needs a certificate. I've seen some minor variations of this setup but it basically boils down to that plus maybe some misc cruft in the form of lambdas, buckets, etc. So why is it so hard to get all of that orchestrated? Why do we have to boil the oceans to get some sane defaults for all of this. Why do we need to micromanage absolutely everything here? Monitoring, health checks, logging systems, backups, security & permissions. All of this needs to be micromanaged. All of it is disaster waiting to happen if you get it wrong. All of it is bespoke cruft that every project is reinventing from scratch. All I want is "hey AWS, Google, MS, ... go run this thing over there and tell me when its done. This is the domain it needs to run over. It's a bog standard web server expecting to serve traffic via http/websockets. Give me something with sane defaults. Not a disassembled jig saw puzzle with thousands of pieces. This stuff should not be this hard in 2021.
  • bradstewart 5y ago
    The AWS CDK is getting closer to this. Your standard VPC setup is extremely simple now. Like new Vpc() simple. The tooling definitely has its quirks, but is steadily improving, and you can drop into CloudFormation from CDK (with TypeScript type checking) when needed.
  • nprateem 5y ago
    Google Cloud Run basically does this
  • ukoki 5y ago
• shadycuz 5y ago
  If you are using cloudformation, I have a python testing library https://github.com/DontShaveTheYak/cloud-radar https://github.com/DontShaveTheYak/cloud-radar It allows unit testing of cloudformation templates locally without needing aws credentials or deploying anything. It also supports functional testing of cloudformation stacks once they are deployed. Full guide on my blog https://la-tech.co/post/hypermodern-cloudformation/getting-started/ https://la-tech.co/post/hypermodern-cloudformation/getting-s...
• desktopninja 5y ago
  Will be interesting to see how if other companies join these two in creating new tooling: https://aws.amazon.com/cloudcontrolapi/
• jaimehrubiks 5y ago
  What is the relationship between these thoughts and the new recent AWS API called "AWS Cloud Control API", which uses the Cloud Formation schema registry under the hood, and that will soon be available as a new aws terraform provider: https://www.hashicorp.com/blog/announcing-terraform-aws-cloud-control-provider-tech-preview https://www.hashicorp.com/blog/announcing-terraform-aws-clou...
• cdogl 5y ago
  We use terraform for our AWS infrastructure, but also for Datadog, PagerDuty, etc. That's where it starts to shine: consistent infrastructure as code across all your (provider-supported) vendors.