7 ms·

The jump from what you mentioned in paragraph 2 to 3 is not necessarily Terraform. You can use other tools, like Ansible, that IMO have a much better framework

by aprdm 6y ago

The jump from what you mentioned in paragraph 2 to 3 is not necessarily Terraform. You can use other tools, like Ansible, that IMO have a much better framework than Terraform. I use Terraform for extremely simple stuff that is easy to destroy/recreate. Projects of bigger scale IMO are better served with Ansible and friends.

themacguffinman 6y ago
I'm no tooling expert but that seems like the opposite of what you should use Terraform for. Imperative frameworks like Ansible are basically fancy ways to organize and execute custom scripts, you still have to implement your own custom state management logic in Ansible commands/scripts. With larger & more complex infrastructure, you want to do this as little as possible because it's hard to consistently get it right, so Terraform basically steps in and implements state management for you. Even if you end up needing to write your own TF drivers, regular devs don't have to deal with it to make infra changes. Declarative templates are much simpler and easier to think about, all the complex state management stuff is hidden away.
- aprdm 6y ago
  Ansible has a bunch of declarative modules. Writing imperative Ansible should basically never happen. My team maintains 50-80 galaxy_roles and it's super rare that we have to build any shared galaxy role on top of imperative logic.
  - themacguffinman 6y ago
    True, and interesting to see that it has worked out so well for your team. I suspect the difference I see in practice is that Ansible playbooks make it much easier to hack in imperative shell scripts in an Ansible task, so it's more likely to happen. I agree that a disciplined & experienced team can do the right thing with either tool and perhaps Ansible's API is nicer if you use it right. This makes me see Ansible in a slightly different light, perhaps I've been unfairly influenced by some Ansible playbooks I've seen in real life.
weitzj 6y ago
Ansible looks fine when you start, but gets painful. You need to write idempotent ansible from the start. No excuses. Then you would implement a retry logic in ansible since your cloud APIs will fail for weird reasons or you really must wait that a resource in AWS is truly ready before you run another task. ( for example vpn gateway in AWS must be ready and then you can attach a vpn connection, or a route53 zone record has to propagate before you do something with it. But in the meantime your other tasks can continue in parallel. Only the vpn tasks needs to wait) Then after a while with all your resources created with your idempotency implemented in ansible, your code will roughly first check ( with timeouts, retries) if you actually can skip tasks since there is nothing to do. The checks increase, your playbook takes longer and longer to execute. Maybe you try to parallelize it in some way. In the meantime a colleague of yours executes the same playbook in their machine or pipelines, and now you see weird side effects. Maybe you start implementing some locking behavior. At this point we did not do any code reviews yet and we did not answer the question: “how should the infrastructure look like?” This especially means: how do you actually delete resources with Ansible? So you Start to introduce a kind of “state” for your resource. So, if you squint hard enough, idempotent, resilient, parallelizable, thread-safe, fast ansible with a state is what terraform solves. If you look at tools like terratest you even get to do unit/integration testing for your infrastructure code. So as soon as you are more than 1 person handling infrastructure following best practices like code review and testing, without getting insane, use terraform. If you are alone and don’t want to follow the above best practices, you still have to be very good in writing idempotent, resilient ansible code. My hypothesis is that the intersection of people who write high quality (idempotent, resilient) ansible code but people who do not care about code review/testing, is quite small. Once you do not work in isolation any more, either in a team or as a consultant who needs to hand over their work (and Teach people how idempotent ansible works), I would favor terraform any time. You could even call ansible from terraform if you need some ansible integration, where ansible has a nicer api for you. But let terraform handle the retries, state management and ansible could just be the executing part. So ansible is more like a fancy bash script/function which you call in an orderly manner.
- apple4ever 6y ago
  Thank you for this. I've been hemming and hawing over straight Ansible or Ansible + Terraform and you made it clear why the later makes sense.
darthrupert 6y ago
No, Ansible is not good for this. It's good for maintaining clusters of hosts, but its stateless nature makes it close to worthless for infrastructure.
- aprdm 6y ago
  Well I guess I beg to differ. Been creating and managing 10ks of hosts across multiple on prems datacenters and clouds for some time now.
  - darthrupert 6y ago
    Perhaps we have a different view of what infrastructure means? Managing hosts isn't it by my view.
    - aprdm 6y ago
      Possibly! Care to share your view?
  - apple4ever 6y ago
    How do you deal with this with Ansible instead of Terraform? I'd love to have a single tool, but Ansible's style seems like a poor fit especially compared to Terraform.

by aprdm 6y ago

themacguffinman 6y ago

I'm no tooling expert but that seems like the opposite of what you should use Terraform for. Imperative frameworks like Ansible are basically fancy ways to organize and execute custom scripts, you still have to implement your own custom state management logic in Ansible commands/scripts. With larger & more complex infrastructure, you want to do this as little as possible because it's hard to consistently get it right, so Terraform basically steps in and implements state management for you. Even if you end up needing to write your own TF drivers, regular devs don't have to deal with it to make infra changes. Declarative templates are much simpler and easier to think about, all the complex state management stuff is hidden away.

aprdm 6y ago
Ansible has a bunch of declarative modules. Writing imperative Ansible should basically never happen. My team maintains 50-80 galaxy_roles and it's super rare that we have to build any shared galaxy role on top of imperative logic.
- themacguffinman 6y ago
  True, and interesting to see that it has worked out so well for your team. I suspect the difference I see in practice is that Ansible playbooks make it much easier to hack in imperative shell scripts in an Ansible task, so it's more likely to happen. I agree that a disciplined & experienced team can do the right thing with either tool and perhaps Ansible's API is nicer if you use it right. This makes me see Ansible in a slightly different light, perhaps I've been unfairly influenced by some Ansible playbooks I've seen in real life.

weitzj 6y ago

Ansible looks fine when you start, but gets painful. You need to write idempotent ansible from the start. No excuses. Then you would implement a retry logic in ansible since your cloud APIs will fail for weird reasons or you really must wait that a resource in AWS is truly ready before you run another task. ( for example vpn gateway in AWS must be ready and then you can attach a vpn connection, or a route53 zone record has to propagate before you do something with it. But in the meantime your other tasks can continue in parallel. Only the vpn tasks needs to wait) Then after a while with all your resources created with your idempotency implemented in ansible, your code will roughly first check ( with timeouts, retries) if you actually can skip tasks since there is nothing to do. The checks increase, your playbook takes longer and longer to execute. Maybe you try to parallelize it in some way. In the meantime a colleague of yours executes the same playbook in their machine or pipelines, and now you see weird side effects. Maybe you start implementing some locking behavior. At this point we did not do any code reviews yet and we did not answer the question: “how should the infrastructure look like?” This especially means: how do you actually delete resources with Ansible? So you Start to introduce a kind of “state” for your resource. So, if you squint hard enough, idempotent, resilient, parallelizable, thread-safe, fast ansible with a state is what terraform solves. If you look at tools like terratest you even get to do unit/integration testing for your infrastructure code. So as soon as you are more than 1 person handling infrastructure following best practices like code review and testing, without getting insane, use terraform. If you are alone and don’t want to follow the above best practices, you still have to be very good in writing idempotent, resilient ansible code. My hypothesis is that the intersection of people who write high quality (idempotent, resilient) ansible code but people who do not care about code review/testing, is quite small. Once you do not work in isolation any more, either in a team or as a consultant who needs to hand over their work (and Teach people how idempotent ansible works), I would favor terraform any time. You could even call ansible from terraform if you need some ansible integration, where ansible has a nicer api for you. But let terraform handle the retries, state management and ansible could just be the executing part. So ansible is more like a fancy bash script/function which you call in an orderly manner.

apple4ever 6y ago
Thank you for this. I've been hemming and hawing over straight Ansible or Ansible + Terraform and you made it clear why the later makes sense.

darthrupert 6y ago

No, Ansible is not good for this. It's good for maintaining clusters of hosts, but its stateless nature makes it close to worthless for infrastructure.

aprdm 6y ago
Well I guess I beg to differ. Been creating and managing 10ks of hosts across multiple on prems datacenters and clouds for some time now.
- darthrupert 6y ago
  Perhaps we have a different view of what infrastructure means? Managing hosts isn't it by my view.
  - aprdm 6y ago
    Possibly! Care to share your view?
- apple4ever 6y ago
  How do you deal with this with Ansible instead of Terraform? I'd love to have a single tool, but Ansible's style seems like a poor fit especially compared to Terraform.