5 ms·

Very nice. I didn't know about terraform, and seems like a very powerful combo... Can anyone comment as to the differences in functionality between ansible AWS

by ivan_ah 10y ago

Very nice. I didn't know about terraform, and seems like a very powerful combo... Can anyone comment as to the differences in functionality between ansible AWS playbooks and terraform? The Segment Stack is a good showcase of best-practices for AWS ... only one thing I didn't understand, why a NAT instance is needed in each subnet? https://github.com/segmentio/stack/blob/master/vpc/main.tf#L54 https://github.com/segmentio/stack/blob/master/vpc/main.tf#L... It seems a little wasteful. Couldn't you just allow traffic between the subnets?

mikelinington 10y ago
The biggest thing I've noticed is that terraform doesn't, out of the box, have any way of bringing in your existing resources, and sharing the [tfstate file] that has the current state of your world is its own challenge. The ec2 module of ansible has `count_tag`, and that was exactly what I wanted: "I want this many boxes that are identified by these tags." Played with an external tool call [terraforming] that creates a terraform setup from your existing boxes, but that, too, was way more than I wanted. Terraform is a very neat tool, but learning the AWS ecosystem _and_ this third-party tool that has its own very strong set of best practices was too much for me. If you're experienced, and especially if you're already using CloudFormation heavily, you can probably get a lot of mileage out of terraform. Doing a quick search to make sure I'm not crazy, it looks like terraform 0.7 will start to support importing, so that's something: https://github.com/hashicorp/terraform/issues/581 https://github.com/hashicorp/terraform/issues/581 [tfstate file]: https://www.terraform.io/docs/state/remote https://www.terraform.io/docs/state/remote [terraforming]: https://github.com/dtan4/terraforming https://github.com/dtan4/terraforming
- jen20 10y ago
  I work on Terraform at HashiCorp - Terraform import is on the way, starting with 0.7. We have a multi-phase approach - first importing state with 0.7 and then moving on to generating configuration in subsequent releases. Many AWS resources are already supported for import, and we're intending to get full support over time. If there are specific resource types that you need that aren't currently supported, please open issues on the repository and we will endeavour to support them quickly!
  - fapjacks 10y ago
    Oh man, I just wanted to gush and say thanks for such awesome software!
bhahn 10y ago
> why a NAT instance is needed in each subnet? I don't think it's strictly needed, but it's best practice because instances in each AZ remain independent from failures in other AZs. Were the AZ with the single NAT to go down, then instances in the other AZs wouldn't be able to communicate outside the VPC (ie. to the rest of the internet) There's also a side benefit of much lower latency using a NAT in the same AZ vs going across AZ (unscientific benchmark is 0.1ms in same AZ vs 0.3ms across AZ)
- ivan_ah 10y ago
  > Were the AZ with the single NAT to go down, then instances in the other AZs wouldn't be able to communicate outside the VPC (ie. to the rest of the internet) Oh I see. Though assuming app servers are wired up behind an ELB, the service will only be partially degraded (no app server outbound connectivity, like you said). The one-NAT per AZ is a more robust design but at $30/mo each (for NAT gateway) seems expensive ;) Even at $10/mo (small do-it-yourself NAT instance) it's not free.
- vacri 10y ago
  Note that this is only important if your use-case requires your servers to have constant, direct 'phone out' access. If you're using ELBs or similar as your link to the outside world for your use-case, a NAT isn't really necessary for constant access. We use a single micro NAT in each VPC, which is only used for system updates (no, I don't have a package cache yet...) and for when we're manually in the server troubleshooting. If there's an outage, well, there's not much we can do in that case, and the NAT isn't needed for production use. And if we really need that NAT back up, just spin up another one and modify the VPC. As you say, it's not strictly needed. It really depends on your use case. If your use-case suffers for the NAT being down, then you need HA on it. If it can wait, then no. With the new managed NAT in AWS, you may as well go with that if you need HA - it's cheapest is roughly twice the price of a micro anyway, and it's one less bit of clutter in your instance list.
  - coredog64 10y ago
    If, for whatever reason, you can't run VPC endpoints then you also want the NAT to be able to reach S3 (and some other service endpoints)
jkern 10y ago
You only need a NAT associated to the top level VPC. The subnets that use it just need it to refer to it in their route table. It looks like that's what they've setup, but I'm not familiar enough with Terraform to say for sure
- jen20 10y ago
  I think the reason most people put a NAT per AZ is in case there is a whole AZ outage taking your NAT with it. However, these days I would argue that for most cases it is better to use NAT Gateway rather than NAT Instances.
  - coleca 10y ago
    And Terraform had support for the AWS NAT Gateway a long time before AWS' own CloudFormation did. I think there was a PR for TF a day or so after AWS added it to the API. Very impressed with Terraform.
    - jen20 10y ago
      It was about 5 hours after the API became available - I remember opening the PR ;-)
      - conorgil145 10y ago
        yup, I was on that early thread and really impressed how quickly support got into TF. I was working on a deadline and was able to use NAT Gateways for HA NAT instead of setting it all up myself. Super convenient :) https://github.com/hashicorp/terraform/issues/4374 https://github.com/hashicorp/terraform/issues/4374
joshpadnick 10y ago
- NAT Instances are placed in each Subnet for High Availability. This way, if one Availability Zone goes down and it happens to be where your NAT Instance is located, you don't lose connectivity. - Terraform vs. Ansible: /u/mitchellh (I think that's his HN name) started terraform and explained his "Terraform vs. Alternatives" in a Google Groups Post[1]. In addition to that post, Terraform also has nice support of multiple providers, so you can mix AWS, Azure, and others in one set of templates. In general, Terraform is doing a lot of cutting-edge thinking with orchestration tools and represents IMO a best-of-breed approach. That being said, there are a lot of bugs, especially around eventual consistency issues in AWS. They're getting better, and most of them are recoverable, though. [1] https://groups.google.com/d/msg/terraform-tool/6Fxnl_bejX4/0O-d17UwhHcJ https://groups.google.com/d/msg/terraform-tool/6Fxnl_bejX4/0...
- toomuchtodo 10y ago
  I believe you can use a Managed NAT Gateway now instead of NAT instances in each AZ for outbound VPC connectivity: https://aws.amazon.com/blogs/aws/new-managed-nat-network-address-translation-gateway-for-aws/ https://aws.amazon.com/blogs/aws/new-managed-nat-network-add... > In general, Terraform is doing a lot of cutting-edge thinking with orchestration tools and represents IMO a best-of-breed approach. Terraform isn't really doing anything that snazzy compared to Cloudformation [1] (an AWS tool) unless you're also orchestrating in concert with non-AWS services. [1] https://aws.amazon.com/cloudformation/details/ https://aws.amazon.com/cloudformation/details/
  - cddotdotslash 10y ago
    I agree managed NAT is probably better (although can be more costly), but you still need a managed NAT Gateway in each AZ.
    - conorgil145 10y ago
      Absolutely correct. We are using NAT Gateways in our architecture, and I love not having to manage it all myself, but it is definitely a tradeoff in costs. From the AWS documentation [1]: "Each NAT gateway is created in a specific Availability Zone and implemented with redundancy in that zone. You have a limit on the number of NAT gateways you can create in an Availability Zone. For more information, see Amazon VPC Limits. Note If you have resources in multiple Availability Zones and they share one NAT gateway, in the event that the NAT gateway’s Availability Zone is down, resources in the other Availability Zones lose Internet access. To create an Availability Zone-independent architecture, create a NAT gateway in each Availability Zone and configure your routing to ensure that resources use the NAT gateway in the same Availability Zone." [1]: https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-gateway.html https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-n...
dminor 10y ago
The idea with terraform is you describe your infrastructure as config (data), and terraform keeps track of the state within AWS (or whichever provider). So, as you change your config, terraform will come up with a plan to move from your present AWS state to the desired AWS state. Terraform allows you to easily express the dependencies in your infrastructure, and also knows about the various dependencies that naturally come with AWS services, which allows it to make better decisions in its planning compared to tools like ansible, chef, etc.

by ivan_ah 10y ago

mikelinington 10y ago

The biggest thing I've noticed is that terraform doesn't, out of the box, have any way of bringing in your existing resources, and sharing the [tfstate file] that has the current state of your world is its own challenge. The ec2 module of ansible has `count_tag`, and that was exactly what I wanted: "I want this many boxes that are identified by these tags." Played with an external tool call [terraforming] that creates a terraform setup from your existing boxes, but that, too, was way more than I wanted. Terraform is a very neat tool, but learning the AWS ecosystem _and_ this third-party tool that has its own very strong set of best practices was too much for me. If you're experienced, and especially if you're already using CloudFormation heavily, you can probably get a lot of mileage out of terraform. Doing a quick search to make sure I'm not crazy, it looks like terraform 0.7 will start to support importing, so that's something: https://github.com/hashicorp/terraform/issues/581 https://github.com/hashicorp/terraform/issues/581 [tfstate file]: https://www.terraform.io/docs/state/remote https://www.terraform.io/docs/state/remote [terraforming]: https://github.com/dtan4/terraforming https://github.com/dtan4/terraforming

jen20 10y ago
I work on Terraform at HashiCorp - Terraform import is on the way, starting with 0.7. We have a multi-phase approach - first importing state with 0.7 and then moving on to generating configuration in subsequent releases. Many AWS resources are already supported for import, and we're intending to get full support over time. If there are specific resource types that you need that aren't currently supported, please open issues on the repository and we will endeavour to support them quickly!
- fapjacks 10y ago
  Oh man, I just wanted to gush and say thanks for such awesome software!

bhahn 10y ago

> why a NAT instance is needed in each subnet? I don't think it's strictly needed, but it's best practice because instances in each AZ remain independent from failures in other AZs. Were the AZ with the single NAT to go down, then instances in the other AZs wouldn't be able to communicate outside the VPC (ie. to the rest of the internet) There's also a side benefit of much lower latency using a NAT in the same AZ vs going across AZ (unscientific benchmark is 0.1ms in same AZ vs 0.3ms across AZ)

ivan_ah 10y ago
> Were the AZ with the single NAT to go down, then instances in the other AZs wouldn't be able to communicate outside the VPC (ie. to the rest of the internet) Oh I see. Though assuming app servers are wired up behind an ELB, the service will only be partially degraded (no app server outbound connectivity, like you said). The one-NAT per AZ is a more robust design but at $30/mo each (for NAT gateway) seems expensive ;) Even at $10/mo (small do-it-yourself NAT instance) it's not free.
vacri 10y ago
Note that this is only important if your use-case requires your servers to have constant, direct 'phone out' access. If you're using ELBs or similar as your link to the outside world for your use-case, a NAT isn't really necessary for constant access. We use a single micro NAT in each VPC, which is only used for system updates (no, I don't have a package cache yet...) and for when we're manually in the server troubleshooting. If there's an outage, well, there's not much we can do in that case, and the NAT isn't needed for production use. And if we really need that NAT back up, just spin up another one and modify the VPC. As you say, it's not strictly needed. It really depends on your use case. If your use-case suffers for the NAT being down, then you need HA on it. If it can wait, then no. With the new managed NAT in AWS, you may as well go with that if you need HA - it's cheapest is roughly twice the price of a micro anyway, and it's one less bit of clutter in your instance list.
- coredog64 10y ago
  If, for whatever reason, you can't run VPC endpoints then you also want the NAT to be able to reach S3 (and some other service endpoints)

jkern 10y ago

You only need a NAT associated to the top level VPC. The subnets that use it just need it to refer to it in their route table. It looks like that's what they've setup, but I'm not familiar enough with Terraform to say for sure

jen20 10y ago
I think the reason most people put a NAT per AZ is in case there is a whole AZ outage taking your NAT with it. However, these days I would argue that for most cases it is better to use NAT Gateway rather than NAT Instances.
- coleca 10y ago
  And Terraform had support for the AWS NAT Gateway a long time before AWS' own CloudFormation did. I think there was a PR for TF a day or so after AWS added it to the API. Very impressed with Terraform.
  - jen20 10y ago
    It was about 5 hours after the API became available - I remember opening the PR ;-)
    - conorgil145 10y ago
      yup, I was on that early thread and really impressed how quickly support got into TF. I was working on a deadline and was able to use NAT Gateways for HA NAT instead of setting it all up myself. Super convenient :) https://github.com/hashicorp/terraform/issues/4374 https://github.com/hashicorp/terraform/issues/4374

joshpadnick 10y ago

- NAT Instances are placed in each Subnet for High Availability. This way, if one Availability Zone goes down and it happens to be where your NAT Instance is located, you don't lose connectivity. - Terraform vs. Ansible: /u/mitchellh (I think that's his HN name) started terraform and explained his "Terraform vs. Alternatives" in a Google Groups Post[1]. In addition to that post, Terraform also has nice support of multiple providers, so you can mix AWS, Azure, and others in one set of templates. In general, Terraform is doing a lot of cutting-edge thinking with orchestration tools and represents IMO a best-of-breed approach. That being said, there are a lot of bugs, especially around eventual consistency issues in AWS. They're getting better, and most of them are recoverable, though. [1] https://groups.google.com/d/msg/terraform-tool/6Fxnl_bejX4/0O-d17UwhHcJ https://groups.google.com/d/msg/terraform-tool/6Fxnl_bejX4/0...

toomuchtodo 10y ago
I believe you can use a Managed NAT Gateway now instead of NAT instances in each AZ for outbound VPC connectivity: https://aws.amazon.com/blogs/aws/new-managed-nat-network-address-translation-gateway-for-aws/ https://aws.amazon.com/blogs/aws/new-managed-nat-network-add... > In general, Terraform is doing a lot of cutting-edge thinking with orchestration tools and represents IMO a best-of-breed approach. Terraform isn't really doing anything that snazzy compared to Cloudformation [1] (an AWS tool) unless you're also orchestrating in concert with non-AWS services. [1] https://aws.amazon.com/cloudformation/details/ https://aws.amazon.com/cloudformation/details/
- cddotdotslash 10y ago
  I agree managed NAT is probably better (although can be more costly), but you still need a managed NAT Gateway in each AZ.
  - conorgil145 10y ago
    Absolutely correct. We are using NAT Gateways in our architecture, and I love not having to manage it all myself, but it is definitely a tradeoff in costs. From the AWS documentation [1]: "Each NAT gateway is created in a specific Availability Zone and implemented with redundancy in that zone. You have a limit on the number of NAT gateways you can create in an Availability Zone. For more information, see Amazon VPC Limits. Note If you have resources in multiple Availability Zones and they share one NAT gateway, in the event that the NAT gateway’s Availability Zone is down, resources in the other Availability Zones lose Internet access. To create an Availability Zone-independent architecture, create a NAT gateway in each Availability Zone and configure your routing to ensure that resources use the NAT gateway in the same Availability Zone." [1]: https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-gateway.html https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-n...

dminor 10y ago

The idea with terraform is you describe your infrastructure as config (data), and terraform keeps track of the state within AWS (or whichever provider). So, as you change your config, terraform will come up with a plan to move from your present AWS state to the desired AWS state. Terraform allows you to easily express the dependencies in your infrastructure, and also knows about the various dependencies that naturally come with AWS services, which allows it to make better decisions in its planning compared to tools like ansible, chef, etc.