4 ms·

I was able to follow this until the point of escaping the container. How did they get the user wouter created on the host and how did it have sudo access?

LethargicStud 6y ago
GCP supports remotely loading public ssh keys onto a box. They do this using the metadata endpoint - this is (in theory) a trusted API endpoint available to instances @ 169.254.169.254. IAM actually uses this - when you call other services, client libs reach out to the metadata endpoint and get IAM creds to send with each request. Anyway, they have a local process that polls the metadata endpoint and adds authorized keys on the host. So you can e.g. upload your public key in the web UI, their metadata endpoint will serve it up on your instance, the guest agent will poll the metadata endpoint and add your key to the authorized_keys file. These folks spoofed a response from the metadata endpoint. They used https://github.com/kpcyrd/rshijack https://github.com/kpcyrd/rshijack to inject their own hand-crafted public key, which the guest agent happily added to authorized_keys (and created the wouter user). They then ssh'd using their key: > ssh -i id_rsa -o StrictHostKeyChecking=no wouter@localhost > Once we accomplished that, we had full access to the host VM (Being able to execute commands as root through sudo). Looks like they had passwordless sudo as well.
- hnick 6y ago
  What's a mitigation for the spoofed packet? TLS or something of the sort?
  - paulfurtado 6y ago
    If the docker container was running with something other than --net=host, it could have been avoided easily with standard networking concepts (route tables with reverse path filtering or iptables rules), even if the attacker somehow managed to get CAP_NET_ADMIN in the container the host network namespace still would refuse the packets. Although, with --net=host you could actually add iptables rules that match based on the cgroup and limit the IPs/ports allowed. It'd also be possible to filter the container's syscalls with seccomp. I'm not entirely sure why the container had CAP_NET_ADMIN at all, which is required for tcpdump and the man in the middle. Also, using user namespaces would have limited the attacker's abilities even if they had root in the container. A lot of defense in depth techniques are possible here. There's also: simply not leaving the gcp login backdoor open. We run our gcp instances similar to ec2: on first boot we take the ssh keys from the metadata service and lay them down and do not run the GCP agent, we have standard config management + ldap for login after the first boot. This means that a hacker gaining access to your GCP credentials can't gain a shell on an existing instance trivially.
  - MertsA 6y ago
    Well for one not giving the container access to eth0 in the host. Ideally the container would be configured with its own network namespace, the portion of the article that mentions host network mode is talking about this. Instead of eth0 in the container just being able to see its own traffic due to how it was configured it could sniff and spoof traffic directly on the host's interface. But yeah, it seems strange to me that the metadata endpoint isn't secured via TLS. I guess they figured they had sufficiently prevented any kind of MitM attack (but obviously not in this case) so it was unnecessary?
Rotten194 6y ago
GCE uses this tool to allow a central metadata server to manage users: https://github.com/GoogleCloudPlatform/guest-agent#account-management https://github.com/GoogleCloudPlatform/guest-agent#account-m... The host VM had this running. Since they had access to the host's network (due to running in a `--network=host` container), they were able to spoof the response from the metadata server to say a new user should be added to `.authorized_keys`, with their supplied public key. The guest agent automatically adds the new users to the `sudoers` group, also giving them sudo access.
- SteveNuts 6y ago
  I'm curious why this host would have this running on it? Do GCP VMs co-locate with cloud SQL instances? I'd think it'd be separate infrastructure but maybe Google is just really good at binpacking (likely).
  - azurezyq 6y ago
    Cloud sql instances actually running on GCE VMs, as suggested by the post, which makes sense as well.

LethargicStud 6y ago

GCP supports remotely loading public ssh keys onto a box. They do this using the metadata endpoint - this is (in theory) a trusted API endpoint available to instances @ 169.254.169.254. IAM actually uses this - when you call other services, client libs reach out to the metadata endpoint and get IAM creds to send with each request. Anyway, they have a local process that polls the metadata endpoint and adds authorized keys on the host. So you can e.g. upload your public key in the web UI, their metadata endpoint will serve it up on your instance, the guest agent will poll the metadata endpoint and add your key to the authorized_keys file. These folks spoofed a response from the metadata endpoint. They used https://github.com/kpcyrd/rshijack https://github.com/kpcyrd/rshijack to inject their own hand-crafted public key, which the guest agent happily added to authorized_keys (and created the wouter user). They then ssh'd using their key: > ssh -i id_rsa -o StrictHostKeyChecking=no wouter@localhost > Once we accomplished that, we had full access to the host VM (Being able to execute commands as root through sudo). Looks like they had passwordless sudo as well.

hnick 6y ago
What's a mitigation for the spoofed packet? TLS or something of the sort?
- paulfurtado 6y ago
  If the docker container was running with something other than --net=host, it could have been avoided easily with standard networking concepts (route tables with reverse path filtering or iptables rules), even if the attacker somehow managed to get CAP_NET_ADMIN in the container the host network namespace still would refuse the packets. Although, with --net=host you could actually add iptables rules that match based on the cgroup and limit the IPs/ports allowed. It'd also be possible to filter the container's syscalls with seccomp. I'm not entirely sure why the container had CAP_NET_ADMIN at all, which is required for tcpdump and the man in the middle. Also, using user namespaces would have limited the attacker's abilities even if they had root in the container. A lot of defense in depth techniques are possible here. There's also: simply not leaving the gcp login backdoor open. We run our gcp instances similar to ec2: on first boot we take the ssh keys from the metadata service and lay them down and do not run the GCP agent, we have standard config management + ldap for login after the first boot. This means that a hacker gaining access to your GCP credentials can't gain a shell on an existing instance trivially.
- MertsA 6y ago
  Well for one not giving the container access to eth0 in the host. Ideally the container would be configured with its own network namespace, the portion of the article that mentions host network mode is talking about this. Instead of eth0 in the container just being able to see its own traffic due to how it was configured it could sniff and spoof traffic directly on the host's interface. But yeah, it seems strange to me that the metadata endpoint isn't secured via TLS. I guess they figured they had sufficiently prevented any kind of MitM attack (but obviously not in this case) so it was unnecessary?

Rotten194 6y ago

GCE uses this tool to allow a central metadata server to manage users: https://github.com/GoogleCloudPlatform/guest-agent#account-management https://github.com/GoogleCloudPlatform/guest-agent#account-m... The host VM had this running. Since they had access to the host's network (due to running in a `--network=host` container), they were able to spoof the response from the metadata server to say a new user should be added to `.authorized_keys`, with their supplied public key. The guest agent automatically adds the new users to the `sudoers` group, also giving them sudo access.

SteveNuts 6y ago
I'm curious why this host would have this running on it? Do GCP VMs co-locate with cloud SQL instances? I'd think it'd be separate infrastructure but maybe Google is just really good at binpacking (likely).
- azurezyq 6y ago
  Cloud sql instances actually running on GCE VMs, as suggested by the post, which makes sense as well.