7 ms·

It is a such tiny thing, but I really like that the metadata service is at fd00:ec2::254 On v4 it is 169.254.169.254 hence the final 254 (and other services li

by mcbain 5y ago

It is a such tiny thing, but I really like that the metadata service is at fd00:ec2::254 On v4 it is 169.254.169.254 hence the final 254 (and other services like DNS at 253), but using the ULA range of ec2 is a simple, effective touch.

CodesInChaos 5y ago
I really hate that the metadata service is reachable via IP at all, and not just via a file based unix socket or a similar endpoint. Or even similar to procfs, instead of http based at all.
- tedk-42 5y ago
  You'd rather mount a socket or file to a container that hit the endpoint directly over HTTP? I prefer the latter as it's far simpler.
  - johncolanduoni 5y ago
    The problem is the metadata endpoint gives any process on your VM that can make API calls to that address full AWS credentials for that instance’s role. That can turn out to be too simple to use.
    - nousermane 5y ago
      ...unless you use identity-aware firewall, such as iptables: iptables -A OUTPUT \ -d 169.254.0.0/16 \ -m owner -j REJECT \ '!' --uid-owner 0
      - pabs3 5y ago
        Hmm, so root processes within the container can still access it?
        yebyen 5y ago
        Yeah, if you're running containers with root access or processes with root running inside of them, there are a number of other things that will need to change about your security posture, least of which is probably how you provide or restrict access to IAM roles within the tenant boundary.
        nousermane 5y ago
        No, recent docker uses UID namespaces, where "root" inside a container translates to a high real UID as seen from the host OS (100000 or some such). iptables' "owner" extension would use that real UID to match. Which could be a problem if intention was to allow it, as your sibling comment points out.
        ak217 5y ago
        Yes - this is not really a Docker thing but a Linux kernel thing, although client-side support is of course needed from Docker and any other system that uses cgroups/namespaces. Also, one other thing to know is from the kernel's point of view "root" is not a thing. It has been unbundled into a set of capabilities (https://man7.org/linux/man-pages/man7/capabilities.7.html https://man7.org/linux/man-pages/man7/capabilities.7.html). When you launch a container, you specify which capabilities you want to drop (so even root in the container can't have them), and which UID mapping you want to use.
      - johncolanduoni 5y ago
        Sure, but what percentage of EC2 instances do you think have a firewall rule like that? Defaults are important. Not to mention the fact that the rule you listed breaks down if you’re running docker containers.
        ak217 5y ago
        ECS and EKS have configurable functionality to blackhole IMDS. You can also configure IMDSv2 to have a hop limit of 1, preventing any bridged networks (such as containers) from accessing IMDS.
        Spivak 5y ago
        There’s no good defaults here. From AWS’s perspective the whole VM is in the same security context. It’s all untrusted customer code from the kernel on up. That’s the granularity on which policy can be applied. Either the whole VM can do something or it can’t. How you define your own security boundaries inside is your business. You can help AWS by doing your own access keys but most people would rather just also make their security boundary the VM and not have to think about it. Treat a compromised processes and a compromised host the same.
        ak217 5y ago
        That's the "give up on container and process security" approach. It's a fine approach from any given user's standpoint. From an infra/tool provider standpoint, it ignores the intense demand that exists for establishing least privilege and defense-in-depth for containerized workloads. Fargate might be a good solution too.
        Spivak 5y ago
        This is backwards, you don't have to give up on container/process security just because AWS's own trust boundary is the VM. But to do that you can't assign VM-level privs. Something something eating cake. Put yourself in AWS's position. Your customer is running a VM which you have only hypervisor level control over. Could be Linux, could be AIX, could be an appliance. How could you possibly implement user/process level security from the outside? How could you know what processes inside the opaque black box are the privileged ones?
    - dekhn 5y ago
      That's the point of the endpoint, IMHO. It's a well known location where anything in the VM can locate the credentials.
    - 22c 5y ago
      IMDSv2 is supposed to solve that https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configur...
    - mvanbaak 5y ago
      There are solutions for this: http://www.daemonology.net/blog/2020-01-27-Announcing-imds-filterd.html http://www.daemonology.net/blog/2020-01-27-Announcing-imds-f...
  - nijave 5y ago
    Unauthenticated HTTP is harder to secure against SSRF When you use the AWS SDK, it abstracts it away anyway
- ak217 5y ago
  A file based unix socket or procfs node is an operating system implementation-specific concept. It requires a substantial amount of logic to be bundled with the guest OS. Maybe a serial device or something like that would be more appropriate.
- deleted 5y ago
  [deleted]
terom 5y ago
But 254 in hexadecimal is 0xfe :/ 0x0254 is 596, or .2.84 in IPv4 notation.
- Dagger2 5y ago
  You could write it as fd00:ec2::0.0.0.254...
  - dcow 5y ago
    The point is that the hex looks the same as the decimal. The actual address doesn't matter. They’re not doing for decimal 254. They’re going for “the last part of the address is familiar and typed the same on a keyboard”.
    - Dagger2 5y ago
      Sure. I was just pointing out that if you really wanted to use 254, you could use fd00:ec2::fe and write it as fd00:ec2::0.0.0.254 -- though mainly because I wanted people to know that doing so was possible, not because it was in any way a good idea in this case. (At least, I was trying to point that out, but re-reading my post it's obvious that I didn't do a very good job.)
      - dcow 5y ago
        No I get it. It’s a fun fact.
p1mrx 5y ago
These addresses technically violate https://datatracker.ietf.org/doc/html/rfc4193#section-3.2 https://datatracker.ietf.org/doc/html/rfc4193#section-3.2, unless "0x000ec20000" was an incredibly lucky RNG output.

by mcbain 5y ago

CodesInChaos 5y ago

I really hate that the metadata service is reachable via IP at all, and not just via a file based unix socket or a similar endpoint. Or even similar to procfs, instead of http based at all.

tedk-42 5y ago
You'd rather mount a socket or file to a container that hit the endpoint directly over HTTP? I prefer the latter as it's far simpler.
- johncolanduoni 5y ago
  The problem is the metadata endpoint gives any process on your VM that can make API calls to that address full AWS credentials for that instance’s role. That can turn out to be too simple to use.
  - nousermane 5y ago
    ...unless you use identity-aware firewall, such as iptables: iptables -A OUTPUT \ -d 169.254.0.0/16 \ -m owner -j REJECT \ '!' --uid-owner 0
    - pabs3 5y ago
      Hmm, so root processes within the container can still access it?
      - yebyen 5y ago
        Yeah, if you're running containers with root access or processes with root running inside of them, there are a number of other things that will need to change about your security posture, least of which is probably how you provide or restrict access to IAM roles within the tenant boundary.
      - nousermane 5y ago
        No, recent docker uses UID namespaces, where "root" inside a container translates to a high real UID as seen from the host OS (100000 or some such). iptables' "owner" extension would use that real UID to match. Which could be a problem if intention was to allow it, as your sibling comment points out.
        ak217 5y ago
        Yes - this is not really a Docker thing but a Linux kernel thing, although client-side support is of course needed from Docker and any other system that uses cgroups/namespaces. Also, one other thing to know is from the kernel's point of view "root" is not a thing. It has been unbundled into a set of capabilities (https://man7.org/linux/man-pages/man7/capabilities.7.html https://man7.org/linux/man-pages/man7/capabilities.7.html). When you launch a container, you specify which capabilities you want to drop (so even root in the container can't have them), and which UID mapping you want to use.
    - johncolanduoni 5y ago
      Sure, but what percentage of EC2 instances do you think have a firewall rule like that? Defaults are important. Not to mention the fact that the rule you listed breaks down if you’re running docker containers.
      - ak217 5y ago
        ECS and EKS have configurable functionality to blackhole IMDS. You can also configure IMDSv2 to have a hop limit of 1, preventing any bridged networks (such as containers) from accessing IMDS.
      - Spivak 5y ago
        There’s no good defaults here. From AWS’s perspective the whole VM is in the same security context. It’s all untrusted customer code from the kernel on up. That’s the granularity on which policy can be applied. Either the whole VM can do something or it can’t. How you define your own security boundaries inside is your business. You can help AWS by doing your own access keys but most people would rather just also make their security boundary the VM and not have to think about it. Treat a compromised processes and a compromised host the same.
        ak217 5y ago
        That's the "give up on container and process security" approach. It's a fine approach from any given user's standpoint. From an infra/tool provider standpoint, it ignores the intense demand that exists for establishing least privilege and defense-in-depth for containerized workloads. Fargate might be a good solution too.
        Spivak 5y ago
        This is backwards, you don't have to give up on container/process security just because AWS's own trust boundary is the VM. But to do that you can't assign VM-level privs. Something something eating cake. Put yourself in AWS's position. Your customer is running a VM which you have only hypervisor level control over. Could be Linux, could be AIX, could be an appliance. How could you possibly implement user/process level security from the outside? How could you know what processes inside the opaque black box are the privileged ones?
  - dekhn 5y ago
    That's the point of the endpoint, IMHO. It's a well known location where anything in the VM can locate the credentials.
  - 22c 5y ago
    IMDSv2 is supposed to solve that https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configur...
  - mvanbaak 5y ago
    There are solutions for this: http://www.daemonology.net/blog/2020-01-27-Announcing-imds-filterd.html http://www.daemonology.net/blog/2020-01-27-Announcing-imds-f...
- nijave 5y ago
  Unauthenticated HTTP is harder to secure against SSRF When you use the AWS SDK, it abstracts it away anyway
ak217 5y ago
A file based unix socket or procfs node is an operating system implementation-specific concept. It requires a substantial amount of logic to be bundled with the guest OS. Maybe a serial device or something like that would be more appropriate.
deleted 5y ago
[deleted]

terom 5y ago

But 254 in hexadecimal is 0xfe :/ 0x0254 is 596, or .2.84 in IPv4 notation.

Dagger2 5y ago
You could write it as fd00:ec2::0.0.0.254...
- dcow 5y ago
  The point is that the hex looks the same as the decimal. The actual address doesn't matter. They’re not doing for decimal 254. They’re going for “the last part of the address is familiar and typed the same on a keyboard”.
  - Dagger2 5y ago
    Sure. I was just pointing out that if you really wanted to use 254, you could use fd00:ec2::fe and write it as fd00:ec2::0.0.0.254 -- though mainly because I wanted people to know that doing so was possible, not because it was in any way a good idea in this case. (At least, I was trying to point that out, but re-reading my post it's obvious that I didn't do a very good job.)
    - dcow 5y ago
      No I get it. It’s a fun fact.

p1mrx 5y ago

These addresses technically violate https://datatracker.ietf.org/doc/html/rfc4193#section-3.2 https://datatracker.ietf.org/doc/html/rfc4193#section-3.2, unless "0x000ec20000" was an incredibly lucky RNG output.