6 ms·

The IMDS is useful, and these changes definitely make it safer, but it is also inherently risky. I wrote a blog post[0] about the risks it introduces. One of th

by cedws 3y ago

The IMDS is useful, and these changes definitely make it safer, but it is also inherently risky. I wrote a blog post[0] about the risks it introduces. One of them being that any process on your VM instances can hit the IMDS and get credentials, allowing it to perform any action that the assigned role can perform. A simple role that allows apps to write logs to a bucket could be used for exfiltration if the policy is not fine grained enough. [0] https://cedwards.xyz/instance-metadata-service-risks/ https://cedwards.xyz/instance-metadata-service-risks/

colmmacc 3y ago
I'm the technical lead for IMDS. Without an IMDS, the alternative is usually hard-coding long-lived credentials, which is much worse. We've looked a few times at different ways that IMDS could vend different credentials to different user-ids. We documented how to use local firewall rules, which you've linked to. This gives single uid restrictions, similar to filterd. We also have a BPF based tracer tool, https://github.com/aws/aws-imds-packet-analyzer https://github.com/aws/aws-imds-packet-analyzer which can monitor which user ids and processes are calling IMDS (and which version they are using). Our next thought was to expose IMDS as a filesystem. That way ordinary POSIX filesystem permissions could be used to control which user ids could read which credentials, and it would even work on Windows. But our research found that security issues (in applications and libraries that customers run) that grant local file reading privileges are even more common than SSRF issues (in part this because many SSRF issues allow "file://" urls, so they become a subset). We've looked at interfacing with the kernel keyring and the TPM 2.0 interface (and we now have Nitro TPM) ... but both are difficult to call to user space. The latter doesn't get user ID granularity, and the former is hard to coordinate with from the outside for revoking and rotating credentials.
- theossuary 3y ago
  Maybe consider having a simple config I can stick into cloud-init to restrict access to IMDS to a set of users? Seems like something generalizable enough to be very useful to the cloud ecosystem.
- cedws 3y ago
  Thanks for your comment, it's interesting to hear what you guys have tried out. Absolutely, long-lived credentials are worse. The only advantage to long-lived/file-sourced creds right now is granularity. Being able to lock them down to specific processes/UIDs and have a role per service. I guess what we want is the best of both worlds - short-lived per-service creds. It appears to be a difficult problem to solve. I was wondering if something could be done with kernel modules. Are you still working on this? Would really like to hear about future developments.
jiggawatts 3y ago
The problem is that the FAANGS run single-process servers at a scale of thousands of VMs, so in their mind 1 VM == 1 App role. They have no concept of sharing a single VM for dozens of apps.
- sofixa 3y ago
  > They have no concept of sharing a single VM for dozens of apps They do, it's called a container or a function, which have their own IAM/auth/role.
  - zokier 3y ago
    In the context of AWS, both containers (Fargate) and functions (Lambda) run on Firecracker, meaning separate VM for each container/function. No shared VMs there.
    - sofixa 3y ago
      Separate micro-VM which is just there as a security boundary and nothing else, so I'm not sure it's comparable to an EC2.
- oblio 3y ago
  Well, they're kind of right. Physical servers I get, you can't really do 1:1 without wasting a ton of resources. But VMs can be resized quickly. The problem is that many companies have very rigid IT departments so even though in theory it can be done somewhat easily, in some places it can be a months-long process to actually get it done.
  - jiggawatts 3y ago
    You can’t resize a VM below 1 vCPU. Actually in most clouds it’s 2 vCPUs minimum in practice. I have a government customer running 150 web apps in about 20 vCPUs. If you naively multiply 150 apps by the minimum scale for HA, non-prod, and the 2 vCPU minimum for cloud hosts you end up with 450-600 vCPUs! That’s expensive even for governments.
k8svet 3y ago
Just wanted to say, from the wording of your comment and the context, I didn't expect to learn as much as I did from your article. Very insightful with lots of good references. Top job. Thought I'd mention for others that think they know IMDS fairly well
- cedws 3y ago
  Thank you, I'm glad it was useful.
gray_-_wolf 3y ago
You can use iptables to allow access only from the root user. That is what we do. I feel like that help to a large degree, assuming you properly run the applications under separate users.
- cedws 3y ago
  I mention this in my blog post. You can do this, but then you're forced to run all apps that require the IMDS under that specific user. It's better than having no restrictions at all, but having a user shared between multiple apps is also problematic in itself.
  - gray_-_wolf 3y ago
    Not really. You can have a daemon, that fetches from IMDS just what the particular user needs and stores it in a way accessible to just that user.
    - cedws 3y ago
      Well yes, you can do that, but it won't be compatible with the cloud SDKs so you'd have to write your own code to fetch that stuff, as well as implement your own token renewal since IMDS credentials expire fairly quickly. This is easier said than done on a brownfield estate.
      - gray_-_wolf 3y ago
        > Well yes, you can do that, but it won't be compatible with the cloud SDKs I am confused why you just could not use the AWS_WEB_IDENTITY_TOKEN_FILE. You could just produce that file for every user, refresh it before expiration and set the environment variable for each user. I believe that should work with the default SDK just fine, no? I mean, sure, it is not exactly zero effort, but still seems fairly doable...
        cedws 3y ago
solatic 3y ago
> Furthermore, AWS and Google Cloud’s metadata services only support one role/service account per instance. If you have multiple services that require different permissions on an instance, they’ll have to share, which is poor security practice. A service may get access to something it doesn’t need and audit logs will be less useful. There’s no such limitation with static file-sourced credentials. Bit of a head-scratcher. If you run multiple services on a VM, why not issue each of them identities of their own, rather than relying on the IMDS identity? If you need separate identities on the same machine for your security posture, then the complexity of setting that up is largely handled for you by IRSA: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-f... Even the logging service should not be using the IMDS identity. The IMDS identity should be very much locked-down, something like AmazonSSMManagedInstanceCore if I recall correctly.
- scarface_74 3y ago
  Because not everyone uses EKS?
  - jsty 3y ago
    Then, as alluded to in the last section, you can link IAM to an OIDC identity provider to hand your workloads temporary role credentials https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_pr...
- kevincox 3y ago
  > why not issue each of them identities of their own Because the easy ways of doing this are not nearly as secure as the short-lived credentials provided by the metadata services in these clouds. With 5-min credential expiry you can know that if a developer logs into the VM (for debugging or troubleshooting or whatever) after 5min they can no longer invisibily impersonate that role. This makes audit logging and similar much more powerful. I've frequently seen long-lived credentials being used as an alternative and this way anyone who have ever logged into any VM with that key can invisibly impersonate that service forever. You can do this by writing your own credentials manager but this is much more work, harder to make as reliable as the one provided by the IAAP provider and you still need a root of trust somewhere. Relying on the metadata service moves the root of trust to something ephemeral and managed which is highly valuable.

by cedws 3y ago

colmmacc 3y ago

I'm the technical lead for IMDS. Without an IMDS, the alternative is usually hard-coding long-lived credentials, which is much worse. We've looked a few times at different ways that IMDS could vend different credentials to different user-ids. We documented how to use local firewall rules, which you've linked to. This gives single uid restrictions, similar to filterd. We also have a BPF based tracer tool, https://github.com/aws/aws-imds-packet-analyzer https://github.com/aws/aws-imds-packet-analyzer which can monitor which user ids and processes are calling IMDS (and which version they are using). Our next thought was to expose IMDS as a filesystem. That way ordinary POSIX filesystem permissions could be used to control which user ids could read which credentials, and it would even work on Windows. But our research found that security issues (in applications and libraries that customers run) that grant local file reading privileges are even more common than SSRF issues (in part this because many SSRF issues allow "file://" urls, so they become a subset). We've looked at interfacing with the kernel keyring and the TPM 2.0 interface (and we now have Nitro TPM) ... but both are difficult to call to user space. The latter doesn't get user ID granularity, and the former is hard to coordinate with from the outside for revoking and rotating credentials.

theossuary 3y ago
Maybe consider having a simple config I can stick into cloud-init to restrict access to IMDS to a set of users? Seems like something generalizable enough to be very useful to the cloud ecosystem.
cedws 3y ago
Thanks for your comment, it's interesting to hear what you guys have tried out. Absolutely, long-lived credentials are worse. The only advantage to long-lived/file-sourced creds right now is granularity. Being able to lock them down to specific processes/UIDs and have a role per service. I guess what we want is the best of both worlds - short-lived per-service creds. It appears to be a difficult problem to solve. I was wondering if something could be done with kernel modules. Are you still working on this? Would really like to hear about future developments.

jiggawatts 3y ago

The problem is that the FAANGS run single-process servers at a scale of thousands of VMs, so in their mind 1 VM == 1 App role. They have no concept of sharing a single VM for dozens of apps.

sofixa 3y ago
> They have no concept of sharing a single VM for dozens of apps They do, it's called a container or a function, which have their own IAM/auth/role.
- zokier 3y ago
  In the context of AWS, both containers (Fargate) and functions (Lambda) run on Firecracker, meaning separate VM for each container/function. No shared VMs there.
  - sofixa 3y ago
    Separate micro-VM which is just there as a security boundary and nothing else, so I'm not sure it's comparable to an EC2.
oblio 3y ago
Well, they're kind of right. Physical servers I get, you can't really do 1:1 without wasting a ton of resources. But VMs can be resized quickly. The problem is that many companies have very rigid IT departments so even though in theory it can be done somewhat easily, in some places it can be a months-long process to actually get it done.
- jiggawatts 3y ago
  You can’t resize a VM below 1 vCPU. Actually in most clouds it’s 2 vCPUs minimum in practice. I have a government customer running 150 web apps in about 20 vCPUs. If you naively multiply 150 apps by the minimum scale for HA, non-prod, and the 2 vCPU minimum for cloud hosts you end up with 450-600 vCPUs! That’s expensive even for governments.

k8svet 3y ago

Just wanted to say, from the wording of your comment and the context, I didn't expect to learn as much as I did from your article. Very insightful with lots of good references. Top job. Thought I'd mention for others that think they know IMDS fairly well

cedws 3y ago
Thank you, I'm glad it was useful.

gray_-_wolf 3y ago

You can use iptables to allow access only from the root user. That is what we do. I feel like that help to a large degree, assuming you properly run the applications under separate users.

cedws 3y ago
I mention this in my blog post. You can do this, but then you're forced to run all apps that require the IMDS under that specific user. It's better than having no restrictions at all, but having a user shared between multiple apps is also problematic in itself.
- gray_-_wolf 3y ago
  Not really. You can have a daemon, that fetches from IMDS just what the particular user needs and stores it in a way accessible to just that user.
  - cedws 3y ago
    Well yes, you can do that, but it won't be compatible with the cloud SDKs so you'd have to write your own code to fetch that stuff, as well as implement your own token renewal since IMDS credentials expire fairly quickly. This is easier said than done on a brownfield estate.
    - gray_-_wolf 3y ago
      > Well yes, you can do that, but it won't be compatible with the cloud SDKs I am confused why you just could not use the AWS_WEB_IDENTITY_TOKEN_FILE. You could just produce that file for every user, refresh it before expiration and set the environment variable for each user. I believe that should work with the default SDK just fine, no? I mean, sure, it is not exactly zero effort, but still seems fairly doable...
      - cedws 3y ago

solatic 3y ago

> Furthermore, AWS and Google Cloud’s metadata services only support one role/service account per instance. If you have multiple services that require different permissions on an instance, they’ll have to share, which is poor security practice. A service may get access to something it doesn’t need and audit logs will be less useful. There’s no such limitation with static file-sourced credentials. Bit of a head-scratcher. If you run multiple services on a VM, why not issue each of them identities of their own, rather than relying on the IMDS identity? If you need separate identities on the same machine for your security posture, then the complexity of setting that up is largely handled for you by IRSA: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-f... Even the logging service should not be using the IMDS identity. The IMDS identity should be very much locked-down, something like AmazonSSMManagedInstanceCore if I recall correctly.

scarface_74 3y ago
Because not everyone uses EKS?
- jsty 3y ago
  Then, as alluded to in the last section, you can link IAM to an OIDC identity provider to hand your workloads temporary role credentials https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_pr...
kevincox 3y ago
> why not issue each of them identities of their own Because the easy ways of doing this are not nearly as secure as the short-lived credentials provided by the metadata services in these clouds. With 5-min credential expiry you can know that if a developer logs into the VM (for debugging or troubleshooting or whatever) after 5min they can no longer invisibily impersonate that role. This makes audit logging and similar much more powerful. I've frequently seen long-lived credentials being used as an alternative and this way anyone who have ever logged into any VM with that key can invisibly impersonate that service forever. You can do this by writing your own credentials manager but this is much more work, harder to make as reliable as the one provided by the IAAP provider and you still need a root of trust somewhere. Relying on the metadata service moves the root of trust to something ephemeral and managed which is highly valuable.