4 ms·

Two basic principles will take you far with regards to AWS security: 1. Avoid long lived credentials (access key/secret key pairs) and IAM users whenever possi

by sylens 4y ago

Two basic principles will take you far with regards to AWS security: 1. Avoid long lived credentials (access key/secret key pairs) and IAM users whenever possible, and instead have everything assume roles for access to resources in the same as well as other accounts. Temporary credentials from STS are the way to go. 2. Turn on IMDSv2 by default for EC2 instances. v1 is susceptible to SSRF attacks so even if you’re attaching an instance profile to an instance and doing everything else right, you’ve got a problem if somebody finds a weakness in the app running on the instance. Also heavily recommend checking out Scott Piper’s SCP Best Practices (https://summitroute.com/blog/2020/03/25/aws_scp_best_practices/ https://summitroute.com/blog/2020/03/25/aws_scp_best_practic...) that still hold up even if being a few years old at this point.

gray_-_wolf 4y ago
I agree with 1, but still it is slightly annoying that role chaining is limited to 1h duration.
technion 4y ago
Regarding 1, my own cli runs aws-vault set-up such that I need to enter a totp code to assume a privileged role. It sounds like really basic security - my saved credential can't go delete servers. Absolutely everyone, from aws certified people to apparently renowned developers have gotten stuck into me over this, with complaints from "I'd quit anywhere that did that" to "bad setup, proper iam never requires mfa". I'm sure there's going to be some form of blow up where long term saved credentials eventually come out as problematic, and it will blow people's minds.
- sylens 4y ago
  aws-vault is an amazing tool that really simplifies working with lots of aws accounts in a secure fashion
- jffry 4y ago
  Do you get such pushback over the need for MFA, or more generally for the use of roles, or something else? MFA seems like an essential defense-in-depth measure to ensure that a compromise of the locally held IAM key is not enough on its own to compromise your AWS account.
  - technion 4y ago
    It's just the workflow. Mfa is fine with passwords and console, but everyone has been educated to understand once they have a iam key saved on a disk its no longer correct to have.
    - jffry 4y ago
      Maybe that's because AWS' own aws-cli setup encourages you to store these credentials on disk in plaintext in a standard-named file in your home directory, and their best story for temporary roles is to invoke `aws sts get-session-token` and copy paste values from the JSON output to env vars. It's really disappointed that aws-cli doesn't easily support this type of workflow, when using MFA and setting up multiple AWS accounts with cross-account roles are two things recommended as security best practices by AWS themselves. Don't get me started on how you can only have a single U2F key attached to your root user.
      - technion 4y ago
        I agree with that being a major part of the problem. Regarding root, I always create an account with a console login that can remove the root user mfa or reset a password, it becomes the recovery account and I can put its own key on it, and ideally never gets used once tested.
      - layble 4y ago
        Fought with getting a nice workflow with the cli that supports MFA. I got it working and wrote it up here. https://www.nicks.io/simplifying-using-the-aws-cli-with-mfa/ https://www.nicks.io/simplifying-using-the-aws-cli-with-mfa/
- AtlasBarfed 4y ago
  MFA gets in the way of automation, and large scale adhoc actions. Say I want to load a huge amount of data, or do a ton of s3 uploads. My role assumption via MFA / TOTP lasts 4 hours. Now say you are doing lots of different "big data" flows, how do you assume privileges when you need them? I understand what MFA / TOTP brings to security, but security people pretend like my job is about logging into a box once-in-a-while for a couple minutes, and that is the ONLY USE CASE. Least privilege is another thing that sounds great to a security guy but is a nightmare for development. So whenever you do anything off the beaten path (diagnose new systems deployed, try new database backends, do some data moves or loads, backup/restore), uhoh, you spend a ton of time diagnosing shortfalls and decoding error messages. The security guy DOES NOT CARE. S3 is a perfect example. S3 should be simple. It is a nightmare of permissions on the user side, permissions on the bucket, and even worse across accounts. Tiresome. Magic numbers/dates for the "version" field policy, what is with THAT? ACLs/Policies stomping over each other. The funny thing is, ask a security person to first enable cloudwatch and dig through the data to find issues and vulnerabilities, or ask security people to actually produce well-tuned IAM roles for what you need. Guess what? They don't want to do it. They want to sit around making checklists and sending memos.
CodesInChaos 4y ago
Isn't IMDSv2 still insecure against attackers who can initiate arbitrary http requests? I really don't get why all big cloud providers are so keen to preserve that vulnerability.
- mnutt 4y ago
  The AWS article and docs on IDMSv2 really bury the lede and people may take away “protects against SSRF” when AWS should really say “protects against SSRF for GET/POST methods only”. It’s fine as an additional layer but they kind of oversell it.
  - amluto 4y ago
    I don’t get why IDMS is TCP at all. IMO these services should be designed so that the VM root user can access them using an out of band mechanism only and VM non-root users cannot access them at all without explicit action from root. It’s dorky, but 9pfs over virtio would work for this purpose, at least with a Linux guest.
    - sylens 4y ago
      At some point though you start making it too hard for legitimate applications and services to make use of the original purpose of the service. IMDSv2 is really aiming to do two things: 1. Require a session to access credentials. You can only initiate the session with a PUT request to prevent some reverse proxy and WAF misconfigurations from becoming the front door. If your reverse proxy still supports PUT requests, IMDSv2 is going to reject anything with an X-Forwarded-For header 2. Once the session is established, it can only be used from the instance where the session began as you get an instance-specific token - so you can't take credentials and then plug them into your own client to start exfiltrating data
      - amluto 4y ago
        Reading a value out of /sys/fs/idms/key seems like just the right amount of barrier to entry. Most languages and environments support this out of the box with a line or two of code. Some may treat it as a privileged operation, which is a good thing. Operations like fetch() can only read it if they’re wildly misconducted so they have local filesystem access, which is a rather higher barrier to entry than plain SSRF. And this supports regular file modes and ACLs, whereas getting this right with iptables/nftables is awkward at best and requires enabling a firewall.
AtlasBarfed 4y ago
Sure would help if I wasn't using some slapdash webscraping thing like gimme-aws-creds from Nike to do this. AWS isn't particularly helpful at providing role assumption, and really, neither is Okta. And it's hilarious that ssh is being treated as some massive vulnerability by the security folks, and they are pushing enterprisey crap like Teleport instead, which crashes / agents go dead / doesn't work in low memory.
- mdaniel 4y ago
  for my curiosity, which part of your comment (or maybe the one you replied to) involves screen scraping?
  - AtlasBarfed 4y ago
    I tried to reverse engineer the python code of gimme-aws-creds and it was parsing/scraping an HTML response from AWS. If that's what it has to do, well ok, but the fact it wasn't a json service was a bit WTF
    - mdaniel 4y ago
      Ah, it's SAML-specific (https://sourcegraph.com/github.com/Nike-Inc/gimme-aws-creds/-/blob/gimme_aws_creds/aws.py?L85 https://sourcegraph.com/github.com/Nike-Inc/gimme-aws-creds/...) although the majority of BS4 seems to be about driving Okta not AWS: https://sourcegraph.com/github.com/Nike-Inc/gimme-aws-creds/-/blob/gimme_aws_creds/okta.py?L149 https://sourcegraph.com/github.com/Nike-Inc/gimme-aws-creds/... Yeah, SAML is, as far as I know, always headed. My copy of aws-vault is bright enough to launch against our AWS SSO provider(1), which secretly is also SAML to Okta, but I guess if your company's setup doesn't use AWS SSO then Nike must have thought parsing HTML to be The Solution™ 1: https://sourcegraph.com/github.com/99designs/aws-vault/-/blob/cli/login.go?L103 https://sourcegraph.com/github.com/99designs/aws-vault/-/blo...

by sylens 4y ago

gray_-_wolf 4y ago

I agree with 1, but still it is slightly annoying that role chaining is limited to 1h duration.

technion 4y ago

Regarding 1, my own cli runs aws-vault set-up such that I need to enter a totp code to assume a privileged role. It sounds like really basic security - my saved credential can't go delete servers. Absolutely everyone, from aws certified people to apparently renowned developers have gotten stuck into me over this, with complaints from "I'd quit anywhere that did that" to "bad setup, proper iam never requires mfa". I'm sure there's going to be some form of blow up where long term saved credentials eventually come out as problematic, and it will blow people's minds.

sylens 4y ago
aws-vault is an amazing tool that really simplifies working with lots of aws accounts in a secure fashion
jffry 4y ago
Do you get such pushback over the need for MFA, or more generally for the use of roles, or something else? MFA seems like an essential defense-in-depth measure to ensure that a compromise of the locally held IAM key is not enough on its own to compromise your AWS account.
- technion 4y ago
  It's just the workflow. Mfa is fine with passwords and console, but everyone has been educated to understand once they have a iam key saved on a disk its no longer correct to have.
  - jffry 4y ago
    Maybe that's because AWS' own aws-cli setup encourages you to store these credentials on disk in plaintext in a standard-named file in your home directory, and their best story for temporary roles is to invoke `aws sts get-session-token` and copy paste values from the JSON output to env vars. It's really disappointed that aws-cli doesn't easily support this type of workflow, when using MFA and setting up multiple AWS accounts with cross-account roles are two things recommended as security best practices by AWS themselves. Don't get me started on how you can only have a single U2F key attached to your root user.
    - technion 4y ago
      I agree with that being a major part of the problem. Regarding root, I always create an account with a console login that can remove the root user mfa or reset a password, it becomes the recovery account and I can put its own key on it, and ideally never gets used once tested.
    - layble 4y ago
      Fought with getting a nice workflow with the cli that supports MFA. I got it working and wrote it up here. https://www.nicks.io/simplifying-using-the-aws-cli-with-mfa/ https://www.nicks.io/simplifying-using-the-aws-cli-with-mfa/
AtlasBarfed 4y ago
MFA gets in the way of automation, and large scale adhoc actions. Say I want to load a huge amount of data, or do a ton of s3 uploads. My role assumption via MFA / TOTP lasts 4 hours. Now say you are doing lots of different "big data" flows, how do you assume privileges when you need them? I understand what MFA / TOTP brings to security, but security people pretend like my job is about logging into a box once-in-a-while for a couple minutes, and that is the ONLY USE CASE. Least privilege is another thing that sounds great to a security guy but is a nightmare for development. So whenever you do anything off the beaten path (diagnose new systems deployed, try new database backends, do some data moves or loads, backup/restore), uhoh, you spend a ton of time diagnosing shortfalls and decoding error messages. The security guy DOES NOT CARE. S3 is a perfect example. S3 should be simple. It is a nightmare of permissions on the user side, permissions on the bucket, and even worse across accounts. Tiresome. Magic numbers/dates for the "version" field policy, what is with THAT? ACLs/Policies stomping over each other. The funny thing is, ask a security person to first enable cloudwatch and dig through the data to find issues and vulnerabilities, or ask security people to actually produce well-tuned IAM roles for what you need. Guess what? They don't want to do it. They want to sit around making checklists and sending memos.

CodesInChaos 4y ago

Isn't IMDSv2 still insecure against attackers who can initiate arbitrary http requests? I really don't get why all big cloud providers are so keen to preserve that vulnerability.

mnutt 4y ago
The AWS article and docs on IDMSv2 really bury the lede and people may take away “protects against SSRF” when AWS should really say “protects against SSRF for GET/POST methods only”. It’s fine as an additional layer but they kind of oversell it.
- amluto 4y ago
  I don’t get why IDMS is TCP at all. IMO these services should be designed so that the VM root user can access them using an out of band mechanism only and VM non-root users cannot access them at all without explicit action from root. It’s dorky, but 9pfs over virtio would work for this purpose, at least with a Linux guest.
  - sylens 4y ago
    At some point though you start making it too hard for legitimate applications and services to make use of the original purpose of the service. IMDSv2 is really aiming to do two things: 1. Require a session to access credentials. You can only initiate the session with a PUT request to prevent some reverse proxy and WAF misconfigurations from becoming the front door. If your reverse proxy still supports PUT requests, IMDSv2 is going to reject anything with an X-Forwarded-For header 2. Once the session is established, it can only be used from the instance where the session began as you get an instance-specific token - so you can't take credentials and then plug them into your own client to start exfiltrating data
    - amluto 4y ago
      Reading a value out of /sys/fs/idms/key seems like just the right amount of barrier to entry. Most languages and environments support this out of the box with a line or two of code. Some may treat it as a privileged operation, which is a good thing. Operations like fetch() can only read it if they’re wildly misconducted so they have local filesystem access, which is a rather higher barrier to entry than plain SSRF. And this supports regular file modes and ACLs, whereas getting this right with iptables/nftables is awkward at best and requires enabling a firewall.

AtlasBarfed 4y ago

Sure would help if I wasn't using some slapdash webscraping thing like gimme-aws-creds from Nike to do this. AWS isn't particularly helpful at providing role assumption, and really, neither is Okta. And it's hilarious that ssh is being treated as some massive vulnerability by the security folks, and they are pushing enterprisey crap like Teleport instead, which crashes / agents go dead / doesn't work in low memory.

mdaniel 4y ago
for my curiosity, which part of your comment (or maybe the one you replied to) involves screen scraping?
- AtlasBarfed 4y ago
  I tried to reverse engineer the python code of gimme-aws-creds and it was parsing/scraping an HTML response from AWS. If that's what it has to do, well ok, but the fact it wasn't a json service was a bit WTF
  - mdaniel 4y ago
    Ah, it's SAML-specific (https://sourcegraph.com/github.com/Nike-Inc/gimme-aws-creds/-/blob/gimme_aws_creds/aws.py?L85 https://sourcegraph.com/github.com/Nike-Inc/gimme-aws-creds/...) although the majority of BS4 seems to be about driving Okta not AWS: https://sourcegraph.com/github.com/Nike-Inc/gimme-aws-creds/-/blob/gimme_aws_creds/okta.py?L149 https://sourcegraph.com/github.com/Nike-Inc/gimme-aws-creds/... Yeah, SAML is, as far as I know, always headed. My copy of aws-vault is bright enough to launch against our AWS SSO provider(1), which secretly is also SAML to Okta, but I guess if your company's setup doesn't use AWS SSO then Nike must have thought parsing HTML to be The Solution™ 1: https://sourcegraph.com/github.com/99designs/aws-vault/-/blob/cli/login.go?L103 https://sourcegraph.com/github.com/99designs/aws-vault/-/blo...