6 ms·

Have you ever set something up manually via AWS console, then 6 months later totally forgot the steps you took and end up wasting a lot of time reverse engineer

Have you ever set something up manually via AWS console, then 6 months later totally forgot the steps you took and end up wasting a lot of time reverse engineering what you did in order to make a comparatively small change? After that perhaps you vowed to take better notes, so the next time you do it that way, but then 6 months later you find that you missed some detail, or there was some changes in between that were not recorded. So then you decide you're going to do everything by API and save the commands, so you write a bunch of bash scripts that execute against the AWS CLI. Over time you add several more for different operations (eg. adding an instance, configuring new ssl cert, etc), but the list of scripts grows very long, and you find that each one makes a lot of implicit assumptions about the state of the infra when its run, so you end up with varying degrees of confidence in the scripts depending on how often you run them. Now you are primed for Terraform. At this point you realize the hard part about cloud configuration is state management. Furthermore, you realize there are some common patterns of how different components and APIs interact with regard to serial dependencies and operation idempotency, however the specifics vary by service and by use case. Terraform gives you a standard substrate for state management, and a framework for developing service "providers" that know how to interact with APIs and map them to state. All of this happens in code which is declarative, can be code reviewed, and state which can be centrally tracked and shared among a large team for a clear audit trail. There's definitely a learning curve, but once you learn it the overhead is pretty small compared to the benefits, even for small teams IMHO.

dkobran 6y ago
Great explanation!
aprdm 6y ago
The jump from what you mentioned in paragraph 2 to 3 is not necessarily Terraform. You can use other tools, like Ansible, that IMO have a much better framework than Terraform. I use Terraform for extremely simple stuff that is easy to destroy/recreate. Projects of bigger scale IMO are better served with Ansible and friends.
- themacguffinman 6y ago
  I'm no tooling expert but that seems like the opposite of what you should use Terraform for. Imperative frameworks like Ansible are basically fancy ways to organize and execute custom scripts, you still have to implement your own custom state management logic in Ansible commands/scripts. With larger & more complex infrastructure, you want to do this as little as possible because it's hard to consistently get it right, so Terraform basically steps in and implements state management for you. Even if you end up needing to write your own TF drivers, regular devs don't have to deal with it to make infra changes. Declarative templates are much simpler and easier to think about, all the complex state management stuff is hidden away.
  - aprdm 6y ago
    Ansible has a bunch of declarative modules. Writing imperative Ansible should basically never happen. My team maintains 50-80 galaxy_roles and it's super rare that we have to build any shared galaxy role on top of imperative logic.
    - themacguffinman 6y ago
      True, and interesting to see that it has worked out so well for your team. I suspect the difference I see in practice is that Ansible playbooks make it much easier to hack in imperative shell scripts in an Ansible task, so it's more likely to happen. I agree that a disciplined & experienced team can do the right thing with either tool and perhaps Ansible's API is nicer if you use it right. This makes me see Ansible in a slightly different light, perhaps I've been unfairly influenced by some Ansible playbooks I've seen in real life.
- weitzj 6y ago
  Ansible looks fine when you start, but gets painful. You need to write idempotent ansible from the start. No excuses. Then you would implement a retry logic in ansible since your cloud APIs will fail for weird reasons or you really must wait that a resource in AWS is truly ready before you run another task. ( for example vpn gateway in AWS must be ready and then you can attach a vpn connection, or a route53 zone record has to propagate before you do something with it. But in the meantime your other tasks can continue in parallel. Only the vpn tasks needs to wait) Then after a while with all your resources created with your idempotency implemented in ansible, your code will roughly first check ( with timeouts, retries) if you actually can skip tasks since there is nothing to do. The checks increase, your playbook takes longer and longer to execute. Maybe you try to parallelize it in some way. In the meantime a colleague of yours executes the same playbook in their machine or pipelines, and now you see weird side effects. Maybe you start implementing some locking behavior. At this point we did not do any code reviews yet and we did not answer the question: “how should the infrastructure look like?” This especially means: how do you actually delete resources with Ansible? So you Start to introduce a kind of “state” for your resource. So, if you squint hard enough, idempotent, resilient, parallelizable, thread-safe, fast ansible with a state is what terraform solves. If you look at tools like terratest you even get to do unit/integration testing for your infrastructure code. So as soon as you are more than 1 person handling infrastructure following best practices like code review and testing, without getting insane, use terraform. If you are alone and don’t want to follow the above best practices, you still have to be very good in writing idempotent, resilient ansible code. My hypothesis is that the intersection of people who write high quality (idempotent, resilient) ansible code but people who do not care about code review/testing, is quite small. Once you do not work in isolation any more, either in a team or as a consultant who needs to hand over their work (and Teach people how idempotent ansible works), I would favor terraform any time. You could even call ansible from terraform if you need some ansible integration, where ansible has a nicer api for you. But let terraform handle the retries, state management and ansible could just be the executing part. So ansible is more like a fancy bash script/function which you call in an orderly manner.
takeda 6y ago
I'm not parent poster, but I believe (when he mentioned looking at TF and AWS documentation) he most likely meant to use TF instead of CloudFormation. And frankly I think CF is far more robust.

dkobran 6y ago

Great explanation!

aprdm 6y ago

The jump from what you mentioned in paragraph 2 to 3 is not necessarily Terraform. You can use other tools, like Ansible, that IMO have a much better framework than Terraform. I use Terraform for extremely simple stuff that is easy to destroy/recreate. Projects of bigger scale IMO are better served with Ansible and friends.

themacguffinman 6y ago
I'm no tooling expert but that seems like the opposite of what you should use Terraform for. Imperative frameworks like Ansible are basically fancy ways to organize and execute custom scripts, you still have to implement your own custom state management logic in Ansible commands/scripts. With larger & more complex infrastructure, you want to do this as little as possible because it's hard to consistently get it right, so Terraform basically steps in and implements state management for you. Even if you end up needing to write your own TF drivers, regular devs don't have to deal with it to make infra changes. Declarative templates are much simpler and easier to think about, all the complex state management stuff is hidden away.
- aprdm 6y ago
  Ansible has a bunch of declarative modules. Writing imperative Ansible should basically never happen. My team maintains 50-80 galaxy_roles and it's super rare that we have to build any shared galaxy role on top of imperative logic.
  - themacguffinman 6y ago
    True, and interesting to see that it has worked out so well for your team. I suspect the difference I see in practice is that Ansible playbooks make it much easier to hack in imperative shell scripts in an Ansible task, so it's more likely to happen. I agree that a disciplined & experienced team can do the right thing with either tool and perhaps Ansible's API is nicer if you use it right. This makes me see Ansible in a slightly different light, perhaps I've been unfairly influenced by some Ansible playbooks I've seen in real life.
weitzj 6y ago
Ansible looks fine when you start, but gets painful. You need to write idempotent ansible from the start. No excuses. Then you would implement a retry logic in ansible since your cloud APIs will fail for weird reasons or you really must wait that a resource in AWS is truly ready before you run another task. ( for example vpn gateway in AWS must be ready and then you can attach a vpn connection, or a route53 zone record has to propagate before you do something with it. But in the meantime your other tasks can continue in parallel. Only the vpn tasks needs to wait) Then after a while with all your resources created with your idempotency implemented in ansible, your code will roughly first check ( with timeouts, retries) if you actually can skip tasks since there is nothing to do. The checks increase, your playbook takes longer and longer to execute. Maybe you try to parallelize it in some way. In the meantime a colleague of yours executes the same playbook in their machine or pipelines, and now you see weird side effects. Maybe you start implementing some locking behavior. At this point we did not do any code reviews yet and we did not answer the question: “how should the infrastructure look like?” This especially means: how do you actually delete resources with Ansible? So you Start to introduce a kind of “state” for your resource. So, if you squint hard enough, idempotent, resilient, parallelizable, thread-safe, fast ansible with a state is what terraform solves. If you look at tools like terratest you even get to do unit/integration testing for your infrastructure code. So as soon as you are more than 1 person handling infrastructure following best practices like code review and testing, without getting insane, use terraform. If you are alone and don’t want to follow the above best practices, you still have to be very good in writing idempotent, resilient ansible code. My hypothesis is that the intersection of people who write high quality (idempotent, resilient) ansible code but people who do not care about code review/testing, is quite small. Once you do not work in isolation any more, either in a team or as a consultant who needs to hand over their work (and Teach people how idempotent ansible works), I would favor terraform any time. You could even call ansible from terraform if you need some ansible integration, where ansible has a nicer api for you. But let terraform handle the retries, state management and ansible could just be the executing part. So ansible is more like a fancy bash script/function which you call in an orderly manner.

takeda 6y ago

I'm not parent poster, but I believe (when he mentioned looking at TF and AWS documentation) he most likely meant to use TF instead of CloudFormation. And frankly I think CF is far more robust.