7 ms·

AWS CloudFormation Best Practices

20 points by ryancoleman 6y ago

ben509 6y ago
Not a great article. Most of the content is far better explained in the AWS docs. To summarize their best practices: * don't put creds in templates * reference other parameters outside of the template, e.g. in Systems Manager Parameter Store * make your code readable * add comments * check your code Except for the second, all of those are fine practices for maintaining any code. The one best practice that's actually relevant to CF is to store parameters in SSM. Yet, they mention that drift is a problem with CloudFormation. So... why do you want to store parameters to your stack in a service that allows them to change independently of the stack? Maybe there's an argument for doing this, but they don't explain it. It's just a "best practice." They don't mention a major use case for CF, namely CI/CD. The article finishes up with a pitch for their template designer.
- throwaway894345 6y ago
  Man, I don't miss CloudFormation. If you want to do anything interesting at all (like being able to spin up turn-key ad-hoc stacks for developers that are mostly the same as your main environments), then you have to parameterize everything, and passing data around CloudFormation templates is just awful--what you want is functions and data structures but CloudFormation gives you "nested-stacks" and you have to encode most of your data structures as strings and "parse them out" with the YAML builtin functions that they provide (e.g., `{"Fn::Split": "foo,bar,baz"}` will evaluate to `["foo", "bar", "baz"]`, but good luck if you want a list of objects). Each stack also has a limit on the number of parameters you can pass, and since you can't really pass struct-like data (e.g., `"Foo": {"bar": "baz", "qux": "..."}`) you end up flattening that struct into N parameters (e.g., `"FooBar": "Baz", "FooQux": "..."`). You can try to plumb things through SSM or SecretsManager, but each stack also limits the number of parameters you can pass there as well. Basically, CloudFormation is like the shittiest programming language you've ever encountered. People will argue that it's supposed to be simple because it's just YAML, but that's bogus--we clearly need to be able to do complex things in this space or else CF wouldn't provide these hacky functions. We pretty clearly need some sort of expression language if not something more robust, since that's the direction all of these "it's just YAML!" tools are going (CloudFormation, Terraform, etc). CloudFormation might actually make a decent "assembly language" for an infrastructure-as-code backend (although extending CloudFormation for third party services was still far too difficult last I checked--basically you had to run your own lambdas) that some higher-level tools might generate, but it's a mess for anything that isn't a toy.
  - otterley 6y ago
    AWS solutions architect here! (Opinions are my own and not necessarily those of my employer.) Many of us now believe that CloudFormation is best described as the assembly language of AWS infrastructure: you can hand-code it, but it's challenging and verbose, and there's a lot of detail you have to get involved in. CDK is the high-level solution to this challenge. With CDK you describe your infrastructure in terms of high-level constructs such as classes. Your stacks are described as applications instead of YAML templates. You can use many different programming languages including TypeScript and Python, and code is reusable. You can even vendor custom construct libraries to share within your org, or openly via npm. I've become a convert since CDK came out - it's been ages since I've handwritten CloudFormation templates. https://aws.amazon.com/cdk/ https://aws.amazon.com/cdk/
    - twg_jack 6y ago
      I'll second CDK -- love it and it's so much easier than writing raw CFN.
    - throwaway894345 6y ago
      Yeah, I haven't given CDK a good shot yet. I did write something like it (for Python) before CDK existed, and it was very nice. I'd like to play with CDK, but I'm no longer working on AWS systems but rather GCP/Terraform. Best of luck to the CDK folks!
  - devonbleak 6y ago
    Yeah JSON/YAML is (mostly) great for data that needs to be parseable by machines and humans but trying to make a language out of it is horrible. Raw HCL2/Terraform is marginally better IMO with being able to pass around more complex structs and function calls that don't have to be expressed as JSON dicts, but still remaining declarative to be able to create a dependency graph internally. All of this is why CDK now exists to be able to express these things as TypeScript or Python and spit out the corresponding CFN or TF. It's also worth noting that AWS added support for additional providers a while back so the CustomResource+Lambda you're describing isn't the only way to interact with non-AWS APIs. I'm actually curious what you're using - you've shit all over a bunch of things without providing any real alternative.
- dragonwriter 6y ago
  > Not a great article Well, duh, it's a Stackery ad posing as an CF best practices article; if the superficial meat was any good, it would distract attention from the ad.
logicslave 6y ago
CDK is strictly superior to cfn
- nikolay 6y ago
  Not at all. I don't think most people really grasp what CloudFormation is.
  - logicslave 6y ago
    How so? CFN is terrible to write and maintain, cdk abstracts alot of the difficulty and makes reusable components possible. I work at Amazon...
    - stevekemp 6y ago
      Doing it by hand is certainly painful, but with libraries such as Troposphere it isn't so bad ..
      - simlevesque 6y ago
        Or such as CDK...
    - deleted 6y ago
      [deleted]
- Varriount 6y ago
  Reading CDK's introduction page, it seems that, similar to CloudFormation, it uses pre-made components, just at a higher level. Those components are then translated into a CloudFormation template[0]. Is that correct? Might it be more accurate to state that CDK is a higher level version of CloudFormation, that abstracts away unimportant details? [0] https://aws.amazon.com/cdk/ https://aws.amazon.com/cdk/
  - zoover2020 6y ago
    Yes, exactly. CDK is the abstraction which compiles down to CloudFormation. You can see CF here as the assembly that gets generated.
ManuelKiessling 6y ago
My personal AWS CloudFormation best practice is „use Terraform“. But I may be ignorant and would love to hear if and how CF beats TF.
- nikolay 6y ago
  Imagine CF being Terraform + Terraform Cloud (only free!) - but more reliable and having real changesets with more predictable behavior and the state being the true AWS state, not some projection of it within Terraform.
  - SteveNuts 6y ago
    The only problem is Cloudformation is strictly AWS resources, so unless you're absolutely 100% sure you'll never need anything outside of AWS ecosystem, I always recommend using TF.
    - nikolay 6y ago
      This is true, of course, but you can have a Terraform CloudFormation resource, too, and pass data from other resources back and forth. Since when we don't like using the best tool for the job?!
    - bassdropvroom 6y ago
      For companies the likelihood of not needing stuff outside of AWS is relatively low. Unless you're okay with managing other applications by hand, which is probably the wrong attitude.
    - Varriount 6y ago
      Not quite true, you can technically create your own CloudFormation resources[0]. It's a pain to do though. [0] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
      - 6y ago
twalla 6y ago
Here's a summary of my personal CFN best practices: 1. Don't
Arelius 6y ago
Honestly, I really like CFN, but raw templates are a bit of a disaster... Most of the rest of my infrastructure is powershell, so I acutally write the templates inline in Powershell something like this: New-AWSTemplate -Resources @{ $OAI = @{ Type = "AWS::CloudFront::CloudFrontOriginAccessIdentity" Properties = @{ CloudFrontOriginAccessIdentityConfig = @{ Comment = "Access to the bucket" } } } .... } If I were doing something without that context I'd likely use Javascript, and just stitch together template structures I've tried out CDK but honestly it seems very opaque and hard to understand, I want to give it another shot but it just doesn't seem to be a direct translation from CFN templates...
- ipsocannibal 6y ago
  Having written more CFN than I care to remember CDK is a breath of fresh air. It has a multi-level API that deals with the relationships between resources in a much cleaner way than plain CFN. However, at the lowest level you can still write basically CFN in code. But CDK has so much more. It includes multi-language support, unit testing, compile time checking for errors, etc.

20 points by ryancoleman 6y ago

Not a great article. Most of the content is far better explained in the AWS docs. To summarize their best practices: * don't put creds in templates * reference other parameters outside of the template, e.g. in Systems Manager Parameter Store * make your code readable * add comments * check your code Except for the second, all of those are fine practices for maintaining any code. The one best practice that's actually relevant to CF is to store parameters in SSM. Yet, they mention that drift is a problem with CloudFormation. So... why do you want to store parameters to your stack in a service that allows them to change independently of the stack? Maybe there's an argument for doing this, but they don't explain it. It's just a "best practice." They don't mention a major use case for CF, namely CI/CD. The article finishes up with a pitch for their template designer.

throwaway894345 6y ago
Man, I don't miss CloudFormation. If you want to do anything interesting at all (like being able to spin up turn-key ad-hoc stacks for developers that are mostly the same as your main environments), then you have to parameterize everything, and passing data around CloudFormation templates is just awful--what you want is functions and data structures but CloudFormation gives you "nested-stacks" and you have to encode most of your data structures as strings and "parse them out" with the YAML builtin functions that they provide (e.g., `{"Fn::Split": "foo,bar,baz"}` will evaluate to `["foo", "bar", "baz"]`, but good luck if you want a list of objects). Each stack also has a limit on the number of parameters you can pass, and since you can't really pass struct-like data (e.g., `"Foo": {"bar": "baz", "qux": "..."}`) you end up flattening that struct into N parameters (e.g., `"FooBar": "Baz", "FooQux": "..."`). You can try to plumb things through SSM or SecretsManager, but each stack also limits the number of parameters you can pass there as well. Basically, CloudFormation is like the shittiest programming language you've ever encountered. People will argue that it's supposed to be simple because it's just YAML, but that's bogus--we clearly need to be able to do complex things in this space or else CF wouldn't provide these hacky functions. We pretty clearly need some sort of expression language if not something more robust, since that's the direction all of these "it's just YAML!" tools are going (CloudFormation, Terraform, etc). CloudFormation might actually make a decent "assembly language" for an infrastructure-as-code backend (although extending CloudFormation for third party services was still far too difficult last I checked--basically you had to run your own lambdas) that some higher-level tools might generate, but it's a mess for anything that isn't a toy.
- otterley 6y ago
  AWS solutions architect here! (Opinions are my own and not necessarily those of my employer.) Many of us now believe that CloudFormation is best described as the assembly language of AWS infrastructure: you can hand-code it, but it's challenging and verbose, and there's a lot of detail you have to get involved in. CDK is the high-level solution to this challenge. With CDK you describe your infrastructure in terms of high-level constructs such as classes. Your stacks are described as applications instead of YAML templates. You can use many different programming languages including TypeScript and Python, and code is reusable. You can even vendor custom construct libraries to share within your org, or openly via npm. I've become a convert since CDK came out - it's been ages since I've handwritten CloudFormation templates. https://aws.amazon.com/cdk/ https://aws.amazon.com/cdk/
  - twg_jack 6y ago
    I'll second CDK -- love it and it's so much easier than writing raw CFN.
  - throwaway894345 6y ago
    Yeah, I haven't given CDK a good shot yet. I did write something like it (for Python) before CDK existed, and it was very nice. I'd like to play with CDK, but I'm no longer working on AWS systems but rather GCP/Terraform. Best of luck to the CDK folks!
- devonbleak 6y ago
  Yeah JSON/YAML is (mostly) great for data that needs to be parseable by machines and humans but trying to make a language out of it is horrible. Raw HCL2/Terraform is marginally better IMO with being able to pass around more complex structs and function calls that don't have to be expressed as JSON dicts, but still remaining declarative to be able to create a dependency graph internally. All of this is why CDK now exists to be able to express these things as TypeScript or Python and spit out the corresponding CFN or TF. It's also worth noting that AWS added support for additional providers a while back so the CustomResource+Lambda you're describing isn't the only way to interact with non-AWS APIs. I'm actually curious what you're using - you've shit all over a bunch of things without providing any real alternative.
dragonwriter 6y ago
> Not a great article Well, duh, it's a Stackery ad posing as an CF best practices article; if the superficial meat was any good, it would distract attention from the ad.

logicslave 6y ago

CDK is strictly superior to cfn

nikolay 6y ago
Not at all. I don't think most people really grasp what CloudFormation is.
- logicslave 6y ago
  How so? CFN is terrible to write and maintain, cdk abstracts alot of the difficulty and makes reusable components possible. I work at Amazon...
  - stevekemp 6y ago
    Doing it by hand is certainly painful, but with libraries such as Troposphere it isn't so bad ..
    - simlevesque 6y ago
      Or such as CDK...
  - deleted 6y ago
    [deleted]
Varriount 6y ago
Reading CDK's introduction page, it seems that, similar to CloudFormation, it uses pre-made components, just at a higher level. Those components are then translated into a CloudFormation template[0]. Is that correct? Might it be more accurate to state that CDK is a higher level version of CloudFormation, that abstracts away unimportant details? [0] https://aws.amazon.com/cdk/ https://aws.amazon.com/cdk/
- zoover2020 6y ago
  Yes, exactly. CDK is the abstraction which compiles down to CloudFormation. You can see CF here as the assembly that gets generated.

ManuelKiessling 6y ago

My personal AWS CloudFormation best practice is „use Terraform“. But I may be ignorant and would love to hear if and how CF beats TF.

nikolay 6y ago
Imagine CF being Terraform + Terraform Cloud (only free!) - but more reliable and having real changesets with more predictable behavior and the state being the true AWS state, not some projection of it within Terraform.
- SteveNuts 6y ago
  The only problem is Cloudformation is strictly AWS resources, so unless you're absolutely 100% sure you'll never need anything outside of AWS ecosystem, I always recommend using TF.
  - nikolay 6y ago
    This is true, of course, but you can have a Terraform CloudFormation resource, too, and pass data from other resources back and forth. Since when we don't like using the best tool for the job?!
  - bassdropvroom 6y ago
    For companies the likelihood of not needing stuff outside of AWS is relatively low. Unless you're okay with managing other applications by hand, which is probably the wrong attitude.
  - Varriount 6y ago
    Not quite true, you can technically create your own CloudFormation resources[0]. It's a pain to do though. [0] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
    - 6y ago

twalla 6y ago

Here's a summary of my personal CFN best practices: 1. Don't

Arelius 6y ago

Honestly, I really like CFN, but raw templates are a bit of a disaster... Most of the rest of my infrastructure is powershell, so I acutally write the templates inline in Powershell something like this: New-AWSTemplate -Resources @{ $OAI = @{ Type = "AWS::CloudFront::CloudFrontOriginAccessIdentity" Properties = @{ CloudFrontOriginAccessIdentityConfig = @{ Comment = "Access to the bucket" } } } .... } If I were doing something without that context I'd likely use Javascript, and just stitch together template structures I've tried out CDK but honestly it seems very opaque and hard to understand, I want to give it another shot but it just doesn't seem to be a direct translation from CFN templates...

ipsocannibal 6y ago
Having written more CFN than I care to remember CDK is a breath of fresh air. It has a multi-level API that deals with the relationships between resources in a much cleaner way than plain CFN. However, at the lowest level you can still write basically CFN in code. But CDK has so much more. It includes multi-language support, unit testing, compile time checking for errors, etc.