5 ms·

AWS Tagging Best Practices: Using Terraform and CloudFormation to Enforce Tags

63 points by toeknee123 6y ago

nprateem 6y ago
Use a tag policy and you can use whatever tool you want
ldoughty 6y ago
I really dislike that the solutions for AWS tag shortcomings is custom code. I'm a big AWS fan and heavy user for 5 years now, but it seems silly to me that you need to write a custom wrapper (e.g. force the creation through a managed script/template like Terraform/Ansible/CF)... Or write reactionary cloud trail policies to handle a situation where someone launches an EC2 instance without providing a specific tag. The fact IAM policies still can't deny requests missing a tag, or deny requests by tag-value condition seems silly to me... Or one step further: allowing some auto-populated tags like what principle was responsible for making the instance in the first place.
- chrisoverzero 6y ago
  > The fact IAM policies still can't deny requests missing a tag, or deny requests by tag-value condition seems silly to me... Can't it, though? "Condition": { "StringEquals": { "aws:RequestTag/key1": "value1", "aws:RequestTag/key2": "value2" }, "ForAllValues:StringEquals": { "aws:TagKeys": [ "key1", "key2" ] } }
  - wernerb 6y ago
    The problem is that all IAM conditionals just give a generic "Denied" statement which is hard to debug. You at left figuring out if you are missing a uppercase character in your tag key. AWS IAM is currently not meant to be used in a devops/enforcement of best practices way. They have AWS config for that, but that's always after the fact while immediate feedback is what you need to function at scale. A possible fix is to allow custom deny reasons but large changes like this to any already extremely large and entangled system like IAM is unlikely.
    - whoknew1122 6y ago
      > "A possible fix is to allow custom deny reasons but large changes like this to any already extremely large and entangled system like IAM is unlikely." This would allow an attack to enumerate permissions by seeing what's denied. Generic statements aren't a bug, they're a feature.
      - Spivak 6y ago
        Yes, but what they're talking about isn't a security denial but a configuration check. Can you imagine if the only output to a git commit pre-hook was "no" and you were left to figure out why?
        whoknew1122 6y ago
        > "Yes, but what they're talking about isn't a security denial but a configuration check." Configuration is a part of security though. Let's say AWS credentials are leaked. One of the most common uses for leaked credentials is to launch EC2 instance for crypto mining. A malicious actor tries to launch RunInstances on $20k worth of instances and gets denied because there isn't a tag on the instances. Would you want the error to say 'Access denied because you didn't don't have the following tag...'?
        Spivak 6y ago
        Yes, because lacking a tag isn't a security boundary. The account has permission to launch EC2 instances and knowing that instances need to have tags isn't a password designed to keep malicious actors out, it's a safeguard.
      - renewiltord 6y ago
        I feel like this is not a complex problem. You just ensure that someone has `iam:GetDetailedError`. Then you can turn it on for your role/user/whatever and if that doesn't work you can turn it on at the global level temporarily. After all, if you think about it, if you are unable to determine that you're denying the person for the right reason, you could easily later allow them for the wrong reason. e.g. A policy that's not meant to be applied to some IAM user is being applied to them but it's documented to be for a different purpose. When that purpose expires, you might remove that policy, and accidentally enable access. If you can get a trace of the denial then you know that it's for the right reasons.
  - mmbleh 6y ago
    Kinda, but it isn't consistent. They've been steadily improving/fixing it, but for some resources, tagging new resources via the console is a 2 step process - it creates the resource THEN adds the tags. They are fixing these things so it's all added at once. What this does is block the ability to create some resources via console since you can't add tags at creation
    - whoknew1122 6y ago
      This is also a problem when using CloudFormation. CloudFormation will often create resources without tags and then add the tags in a separate call.
      - ldoughty 6y ago
        That's the root of my complaint... This effectively means IAM cannot limit creation events by tag because it's two steps. Only solution is two steps
  - ldoughty 6y ago
    EC2, last I checked, does not allow tag conditionals at instance creation. You can have tag controls afterwards, but the instance creation call does not support tagging (the GUI masks this)
- twunde 6y ago
  FYI, IAM policies do support tag-based access control: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags.html#access_tags_control-tag-keys https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags.... This was introduced in the past 1-2 years.
  - ldoughty 6y ago
    That is access control when tags exist... You can't, for example, deny an EC2 instance creation of it is NOT tagged with Key X. The recommendation is to watch cloud trail and post-creation, terminate it. Creation events for EC2 do. It support tags, it's actually a 2 step process (the GUI console hides this, the SDK and IAM policy limits make this clear)
    - twunde 6y ago
      Ah, what you're looking for is SCP (service control policies). And AWS even has a handy example for denying creation when a tag is missing: https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps_examples.html#example-require-tag-on-create https://docs.aws.amazon.com/organizations/latest/userguide/o...
- donavanm 6y ago
  > The fact IAM policies still can't deny requests missing a tag, or deny requests by tag-value condition seems silly to me... Am I missing something here? I could have sworn that aws:RequestTag with a “Tag On Create” Actions would enforce the authorization check based on the in the Tags parameter? Similarly aws:TagKeys allows you to control the permitted values of those Tags. > allowing some auto-populated tags like what principle was responsible for making the instance in the first place. These do exist as “system tags”, but its on a per service basis. Migration Hib and ECS come to mind. I dont know of any cross cutting application (like IAM) where theyre applied universally. https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags.html#access_tags_control-requests https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags... https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags.html#access_tags_control-tag-keys https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags... https://aws.amazon.com/about-aws/whats-new/2020/04/aws-migration-hub-automatically-tags-migration-results-cost-tracking/ https://aws.amazon.com/about-aws/whats-new/2020/04/aws-migra...
kapilvt 6y ago
This is one of the bread butter use cases on the opensource cloud custodian project (auto tagging, tag enforcement workflows, retro-active tagging from cloudtrail, etc). https://cloudcustodian.io https://cloudcustodian.io (now a cncf sandbox project).
- toeknee123 6y ago
  100% agree. We're big fans of CloudCustodian and use it ourselves. However, it can be tough for some org to get approval with using open source tools.
tthayer 6y ago
We use cloudposse's label terraform module for everything. It works really well and it lets you use common values for everything. Paired with terragrunt it removes most of the pain of tagging for us.
mcintyre1994 6y ago
Something that's pretty nice in AWS CDK is that tagging is recursive - so if you add a tag to eg. a top-level ECS service object then the tags are applied to all of its children automatically. https://docs.aws.amazon.com/cdk/latest/guide/tagging.html https://docs.aws.amazon.com/cdk/latest/guide/tagging.html
- acdha 6y ago
  There's a long-running Terraform issue for the equivalent if you want to chime in: https://github.com/terraform-providers/terraform-provider-aws/issues/7926 https://github.com/terraform-providers/terraform-provider-aw...
yonixw 6y ago
Tags are not supported for some VPC components (on creation). In that sense, Azure Resource Group and Google Projects is way better for organising and project based permissions. Source: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-ec2-resources-table https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Ta...
- acdha 6y ago
  Can you provide a more detailed comparison? I've run into a variety of GCP services which can't be labeled so I'm curious that this would be described as “way better”.
  - yonixw 6y ago
    Let's say I want to create an IAM to allow creation for a specific project (Full deploy), I can't enforce it on VPC.
deleted 6y ago
[deleted]
deleted 6y ago
[deleted]

63 points by toeknee123 6y ago

Use a tag policy and you can use whatever tool you want

I really dislike that the solutions for AWS tag shortcomings is custom code. I'm a big AWS fan and heavy user for 5 years now, but it seems silly to me that you need to write a custom wrapper (e.g. force the creation through a managed script/template like Terraform/Ansible/CF)... Or write reactionary cloud trail policies to handle a situation where someone launches an EC2 instance without providing a specific tag. The fact IAM policies still can't deny requests missing a tag, or deny requests by tag-value condition seems silly to me... Or one step further: allowing some auto-populated tags like what principle was responsible for making the instance in the first place.

chrisoverzero 6y ago
> The fact IAM policies still can't deny requests missing a tag, or deny requests by tag-value condition seems silly to me... Can't it, though? "Condition": { "StringEquals": { "aws:RequestTag/key1": "value1", "aws:RequestTag/key2": "value2" }, "ForAllValues:StringEquals": { "aws:TagKeys": [ "key1", "key2" ] } }
- wernerb 6y ago
  The problem is that all IAM conditionals just give a generic "Denied" statement which is hard to debug. You at left figuring out if you are missing a uppercase character in your tag key. AWS IAM is currently not meant to be used in a devops/enforcement of best practices way. They have AWS config for that, but that's always after the fact while immediate feedback is what you need to function at scale. A possible fix is to allow custom deny reasons but large changes like this to any already extremely large and entangled system like IAM is unlikely.
  - whoknew1122 6y ago
    > "A possible fix is to allow custom deny reasons but large changes like this to any already extremely large and entangled system like IAM is unlikely." This would allow an attack to enumerate permissions by seeing what's denied. Generic statements aren't a bug, they're a feature.
    - Spivak 6y ago
      Yes, but what they're talking about isn't a security denial but a configuration check. Can you imagine if the only output to a git commit pre-hook was "no" and you were left to figure out why?
      - whoknew1122 6y ago
        > "Yes, but what they're talking about isn't a security denial but a configuration check." Configuration is a part of security though. Let's say AWS credentials are leaked. One of the most common uses for leaked credentials is to launch EC2 instance for crypto mining. A malicious actor tries to launch RunInstances on $20k worth of instances and gets denied because there isn't a tag on the instances. Would you want the error to say 'Access denied because you didn't don't have the following tag...'?
        Spivak 6y ago
        Yes, because lacking a tag isn't a security boundary. The account has permission to launch EC2 instances and knowing that instances need to have tags isn't a password designed to keep malicious actors out, it's a safeguard.
    - renewiltord 6y ago
      I feel like this is not a complex problem. You just ensure that someone has `iam:GetDetailedError`. Then you can turn it on for your role/user/whatever and if that doesn't work you can turn it on at the global level temporarily. After all, if you think about it, if you are unable to determine that you're denying the person for the right reason, you could easily later allow them for the wrong reason. e.g. A policy that's not meant to be applied to some IAM user is being applied to them but it's documented to be for a different purpose. When that purpose expires, you might remove that policy, and accidentally enable access. If you can get a trace of the denial then you know that it's for the right reasons.
- mmbleh 6y ago
  Kinda, but it isn't consistent. They've been steadily improving/fixing it, but for some resources, tagging new resources via the console is a 2 step process - it creates the resource THEN adds the tags. They are fixing these things so it's all added at once. What this does is block the ability to create some resources via console since you can't add tags at creation
  - whoknew1122 6y ago
    This is also a problem when using CloudFormation. CloudFormation will often create resources without tags and then add the tags in a separate call.
    - ldoughty 6y ago
      That's the root of my complaint... This effectively means IAM cannot limit creation events by tag because it's two steps. Only solution is two steps
- ldoughty 6y ago
  EC2, last I checked, does not allow tag conditionals at instance creation. You can have tag controls afterwards, but the instance creation call does not support tagging (the GUI masks this)
twunde 6y ago
FYI, IAM policies do support tag-based access control: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags.html#access_tags_control-tag-keys https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags.... This was introduced in the past 1-2 years.
- ldoughty 6y ago
  That is access control when tags exist... You can't, for example, deny an EC2 instance creation of it is NOT tagged with Key X. The recommendation is to watch cloud trail and post-creation, terminate it. Creation events for EC2 do. It support tags, it's actually a 2 step process (the GUI console hides this, the SDK and IAM policy limits make this clear)
  - twunde 6y ago
    Ah, what you're looking for is SCP (service control policies). And AWS even has a handy example for denying creation when a tag is missing: https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps_examples.html#example-require-tag-on-create https://docs.aws.amazon.com/organizations/latest/userguide/o...
donavanm 6y ago
> The fact IAM policies still can't deny requests missing a tag, or deny requests by tag-value condition seems silly to me... Am I missing something here? I could have sworn that aws:RequestTag with a “Tag On Create” Actions would enforce the authorization check based on the in the Tags parameter? Similarly aws:TagKeys allows you to control the permitted values of those Tags. > allowing some auto-populated tags like what principle was responsible for making the instance in the first place. These do exist as “system tags”, but its on a per service basis. Migration Hib and ECS come to mind. I dont know of any cross cutting application (like IAM) where theyre applied universally. https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags.html#access_tags_control-requests https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags... https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags.html#access_tags_control-tag-keys https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags... https://aws.amazon.com/about-aws/whats-new/2020/04/aws-migration-hub-automatically-tags-migration-results-cost-tracking/ https://aws.amazon.com/about-aws/whats-new/2020/04/aws-migra...

kapilvt 6y ago

This is one of the bread butter use cases on the opensource cloud custodian project (auto tagging, tag enforcement workflows, retro-active tagging from cloudtrail, etc). https://cloudcustodian.io https://cloudcustodian.io (now a cncf sandbox project).

toeknee123 6y ago
100% agree. We're big fans of CloudCustodian and use it ourselves. However, it can be tough for some org to get approval with using open source tools.

tthayer 6y ago

We use cloudposse's label terraform module for everything. It works really well and it lets you use common values for everything. Paired with terragrunt it removes most of the pain of tagging for us.

mcintyre1994 6y ago

Something that's pretty nice in AWS CDK is that tagging is recursive - so if you add a tag to eg. a top-level ECS service object then the tags are applied to all of its children automatically. https://docs.aws.amazon.com/cdk/latest/guide/tagging.html https://docs.aws.amazon.com/cdk/latest/guide/tagging.html

acdha 6y ago
There's a long-running Terraform issue for the equivalent if you want to chime in: https://github.com/terraform-providers/terraform-provider-aws/issues/7926 https://github.com/terraform-providers/terraform-provider-aw...

yonixw 6y ago

Tags are not supported for some VPC components (on creation). In that sense, Azure Resource Group and Google Projects is way better for organising and project based permissions. Source: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-ec2-resources-table https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Ta...

acdha 6y ago
Can you provide a more detailed comparison? I've run into a variety of GCP services which can't be labeled so I'm curious that this would be described as “way better”.
- yonixw 6y ago
  Let's say I want to create an IAM to allow creation for a specific project (Full deploy), I can't enforce it on VPC.

deleted 6y ago

[deleted]

deleted 6y ago

[deleted]