16 ms·
Google Compute Engine VM takeover via DHCP flood
- rafaelturk 5y agoApparently Google Project Zero timeline only applies to others...
- Sebb767 5y agoHe reported the issue first on 2020-09-26 [0], nearly a year ago. [0] https://github.com/irsl/gcp-dhcp-takeover-code-exec#timeline https://github.com/irsl/gcp-dhcp-takeover-code-exec#timeline
- _trampeltier 5y agoLove the 2020-12-03: ... "holiday season coming up"
- deleted 5y ago[deleted]
- viraptor 5y agoI think you're mixing up the projects a bit. This vulnerability doesn't seem to be going via Project Zero. Other google teams are known to sometimes not react in time and when the reports are going through Project Zero, they are disclosed at 90 days, regardless if the project is internal or not. (I remember an unpatched Chrome vulnerability was published by Project Zero at the end of deadline) So the Zero timeline applies to everyone the same. It doesn't mean fixes actually happen on the same timeline.
- emayljames 5y agoparent comment is reffering to the double standards. One rule for google disclosing others vulns, another for their own vulns.
- atatatat 5y agowas this found by, or originally reported to, Project Zero?
- manquer 5y agoPrecisely the point OP is saying. The rules for project zero are different not that project zero is applying it differently.
- UncleMeat 5y ago90 days is fairly common in the industry, but not universal. GPZ is definitely not uniquely strict on disclosures.
- manquer 5y agoThsts not problem It is not that project zero is strict, The researcher here allowed 9 months for them to fix. Should he have disclosed after 90 days ? Clearly google didn't use the time to fix. It looks bad when you have a hard policy to disclose but not to fix
- UncleMeat 5y agoHe could have if he wanted to. He could have disclosed immediately if he wanted to. Google does not have a hard policy to disclose. GPZ does. Vulns in external products found through other groups within Google do not share all the same processes as GPZ.
- manquer 5y agoGPZ is not some independent entity google just funds, they are as much part of Google as any other team. If you want to be that precise, it is bad look for part of your organization to have hard policy that you expect external companies to follow, while parts of your organization itself cannot do the same. I am not saying Project Zero is wrong, clearly giving more time did not prod Google to actually fix timely, he certainly was being too polite and gave too much time, I don't know why, perhaps companies don't pay bounties if you disclose without their consent [2] ? All I am just saying Google as a company should hold itself to the same hard standard and fix issues in 90 days this is what Google Project Zero as a team expects other companies[1] to do so, they will even reject requests for extensions. As a company if they can't do it, they shouldn't expect others to do it either right? Or they should disclose reported vulnerabilities even if not fixed in 90 days. [1] Maybe they do it for internal teams as well, but that not relevant to us, all we should be concerned is how they behave externally with disclosing and solving issues. [2] Perhaps part of the reason GPZ is able to do this hard policy is because they don't depend on bug bounties as source of income as independent researchers do.
- breakingcups 5y agoTheir point is, Project Zero is a very public Google project to hold other companies (and yes, themselves too) accountable with disclosure policies Google (as a company) presumably stands behind. Thus it is quite ironic for an issue to not be fixed in a year. Yes, sometimes other Google teams miss the deadline set by Project Zero too. That's not the point.
- deleted 5y ago[deleted]
- jsnell 5y agoThe timeline for disclosure is set by the reporter, no? "We will disclose this bug on YYYY-MM-DD". The other side can ask for an extension, but has no intrinsic right to one. Unless I am missing something, this has nothing to do with PZ, so their default timeline is totally irrelevant.
- Cthulhu_ 5y agoI believe PZ can set the terms of reasonable disclosure (and the like) as a condition for paying out.
- manquer 5y agoDepends on if you want to get paid. Many organizations will not pay bug bounties if you disclose before they fix or without their consent. Project Zero probably doesn't care or doesn't even accept the bounties
- jsnell 5y agoThe post had no indication of a bounty being held hostage. From https://www.google.com/about/appsecurity/reward-program/ https://www.google.com/about/appsecurity/reward-program/ it seems like the requirement for a bounty is not for "don't disclose before fix is released", but "don't disclose without reasonable advance notice". So I just don't see the inconsistency here. Project Zero gives a disclosure deadline. The reporter here chose not to give one. When they said they wanted to disclose, there was no stalling for extra time. Just what is the expectation here?
- tptacek 5y agoThis superficial dismissal doesn't even make sense. Google didn't control the disclosure timeline here. The people who found these vulnerabilities could have published on T+90, or, for that matter, T+1. Meanwhile, the norm that does exist (it's a soft norm) is that you respect the disclosure preferences of the person who reports the vulnerability to the extent you reasonably can. I'm not sure I even understand the impulse behind writing a comment like this. Assume Google simply refuses to apply the P0 disclosure rule to themselves (this isn't the case, but just stipulate). Do you want them to stop funding Project Zero? Do you wish you knew less about high profile vulnerabilities?
- zokier 5y agoThe combination of dhcp, magic metadata servers, and cloud-init feels so awkward way of managing VM provisioning. I'm thinking would having a proper virtual device or maybe something on uefi layer clean up things?
- CodesInChaos 5y agoMetadata servers should not exist. An application having network access should not grant it any sensitive privileges.
- formerly_proven 5y agoSo I just read up on what this is and what they're for and can't help the feeling this is an "everything is a nail" design.
- CodesInChaos 5y agoThe traditional unix "everything is a file" approach would be much better than those metadata services.
- jcims 5y agoTotally agree. A block device with the contents of the metadata service in it would be nice! It becomes trivial to provide at least basic access control to the service.
- rkeene2 5y agoThis is done by some cloud software (e.g., OpenNebula). The downside is that modifying the metadata is now difficult since block devices being hot-plugged can cause issues if in use.
- edf13 5y agoIt's far worse... the Metadata can provision a new public ssh key to the VM (at will!). I wasn't aware of this single point of failure
- cle 5y agoThe more concerning security finding here is that Google sat on this for 9 months. Assuming the claims hold, this is a serious problem for any security-conscious GCP customers. What other vulnerabilities are they sitting on? Do they have processes in place to promptly handle new ones? Doesn’t look like it…
- saghm 5y agoThis is especially questionable given the much shorter deadline that Project Zero gives other companies to fix bugs before publishing their vulnerabilities (regardless of whether there's been a fix). It only seems fair that Google should hold itself to the same standard.
- VWWHFSfQ 5y agoGoogle doesn't hold itself to any standard. At least, not anymore.
- sirdarckcat 5y agohttp://g.co/appsecurity http://g.co/appsecurity has more details but TL;DR is that Google is supportive of people disclosing unfixed bugs after 90 days, which is what happened here.
- dataflow 5y agoProject Zero gives Google the same timeline. This had nothing to do with Project Zero from what I understand.
- breakingcups 5y agoProject Zero is a (very public) Google project though. If they stand behind their choices and policies, they should live by them.
- joshuamorton 5y agoIn what way is Google not standing by their policies (for example, have they criticized or tried to prevent this person from disclosing publicly)?
- gerdesj 5y agoThis is well worth reading. It describes how, through a series of well meaning steps, you shoot yourself in the face. It all starts with: "Note that the last 4 bytes (0a:80:00:02) of the MAC address (42:01:0a:80:00:02) are actually the same as the internal IP address of the box (10.128.0.2). This means, 1 of the 3 components is effectively public."
- markus_zhang 5y agoAs someone who knows nothing about networking, can you plz explain why they set up the config as this? Does that mean the third byte from top must be 0a? BTW I just checked my corporate intranet setup and MAC has nothing to do with the ip address.
- champtar 5y agoIn most (all?) public cloud you don't have real layer2, having the MAC computed from the IP allow to make a stateless ARP responder I guess
- markus_zhang 5y agoThanks! I don't understand the technical details but (combining multiple responds) I think I understand a bit now. I also remember that back when I was in the university we used sort of university intranet (every personal computer connected has to go through it to the Internet, but it's free anyway) and there was a lot of ARP attack then, and I learned to use arp -a and arp -v (maybe wrong).
- gerdesj 5y agoQuite. MAC addresses don't need to line up with IP addresses or vice versa. They are completely different things. However in IPv6 there is a standard for link local addresses that does have a correlation between IP and MAC but there you go. It's designed to avoid clashes and is not the finest design decision ever made! That last will probably creep into a CVE one day just like this chain of events. The config as designed probably looked like a good idea at the time. When you are worrying about millions of things in an address space like IPv4 and MAC, then having them tie up in some way may be useful for lookups in a very large database or two. However, giving information away about something that was never designed from the outset to do so is not a good idea. If you follow the chain of reasoning in the github post you can see that you can break the chain at multiple points by not "being clever". If you start by not making your IP addresses follow the MAC address you kill this problem off before it starts.
- Sebb767 5y agoIt's so strange to me that they have a process for adding a root key that involves no authentication at all. These are VMs with their images running their pre-installed software, it's not like this would have been a hard problem.
- champtar 5y agoIn GKE it allowed to go from a hostNetwork pod to root on the node: http://blog.champtar.fr/Metadata_MITM_root_EKS_GKE/ http://blog.champtar.fr/Metadata_MITM_root_EKS_GKE/
- willvarfar 5y agoHas it been verified that GCE is still vulnerable? There's clearly a communication gap between the researcher and Google. But perhaps the techies at Google saw it and fixed it and it just hasn't been communicated, or some other change in GCE has closed it or mitigated it?
- SahAssar 5y agoThe author asked for an update 2021-06-08, 8 months after reporting it. If they fixed it why wouldn't they say so?
- sparkling 5y agoVery creative approach, never thought of such an attack vector.
- tryauuum 5y agoYeah. I knew that some parts of DHCP are ipv4 and unicast... but to carry such a packet over the internet, what a bold move
- joelbondurant 5y agoCommunist data is public property.
- remus 5y agoWhile it certainly seems like a fairly serious vulnerability I think it's worth highlighting that this attack requires that either you already have access to a machine on the same subnet as the target machine or that the firewall in front of the target machine is very lax. That's a pretty high bar for getting the attack to work in the wild.
- tryauuum 5y agoI wonder if they have actually tested the last scenario: > Targeting a VM over the internet. This requires the firewall in front of the victim VM to be fully open. I mean, even if the firewall is non-existing, can you really carry a dhcp payload over the public internet? EDIT: still haven't read the whole thing, but looks like they did test it
- asah 5y ago-1: consider if you have complex vendor software on a GCE VM inside a larger GCP project... now a vulnerability in that vendor software means the whole GCP project is exposed. Vendor fixes are notoriously slow, so in practice you have to isolate vendor software to separate GCP projects. Real example: I have a client with precisely this situation, and elsewhere in the GCP project is PII consumer data requiring public disclosure if exposed.
- exitheone 5y ago
- asah 5y agoSame subnet = as another VM in your project? or random GCP VM that happens to share your subnet? Seems like pretty different risk levels... https://github.com/irsl/gcp-dhcp-takeover-code-exec#attack-scenarios https://github.com/irsl/gcp-dhcp-takeover-code-exec#attack-s...
- Rockslide 5y agoit literally says "same project" right there
- asah 5y agoThanks. Sorry, just wanted to be completely sure.
- kl4m 5y agoThe usual configuration is 1 or many subnets per VPC, and one or many VPC per project. a Shared VPC setup between projects is also possible but requires prior agreement from both projects.
- kobalsky 5y agoThe problem is that it means that any compromised app, container or chroot can now be escalated to a full fleet takeover unless you have the mentioned mitigations in place.
- deleted 5y ago[deleted]
- southerntofu 5y ago> any security-conscious GCP customers Does that exist? In my book, if you're security conscious, you can only do self-hosting whether on premises or in your own bay in a datacenter. Giving away your entire computing and networking to a third party such as Google is orthogonal to security.
- throwaway3699 5y agoMost people's threat models do not assume Amazon or Google are threats. Especially when you sign very large contracts with them, the law is enough to keep them in check.
- shiftpgdn 5y agoYou should absolutely consider your cloud provider a threat. What happens in a black swan even where a provider is completely compromised? Design around zero trust networks.
- dmos62 5y agoIt's unreasonable to always design around not trusting any third party.
- southerntofu 5y agoSure you must always put some levels of trust in 3rd parties. What level of trust is the important question. Ideally, you distribute that trust among several actors so a single compromise is not too much of a deal. That's why you use different hardware vendors for your routers and servers, another vendor for your network connectivity, and yet other vendors for your software. This way, MiTM is mitigated by TLS (or equivalent) and server compromise is mitigated by a good firewall and network inspection stack. Placing all your eggs in a single Google basket is giving a lot of power to a single "don't be evil" corporation, who may get hacked or compelled by law enforcement to spy on you and your clients.
- londons_explore 5y agoThis attack allows an adversary to move from root permissions on one VM in a GCP project to gaining root permissions on another VM in the same project. The attack is unlikely to succeed unless the attacker knows the exact time a particular VM was last rebooted, or can cause a reboot. Overall, this attack alone probably won't be the reason you will have to start filling in a GDPR notification to your users...
- darkwater 5y agoWell, it can be used in a chain of attack to jump from one compromised, external-facing machine to another only internal one that might hold more sensitive data. Or to run some inside-job (although in that case developers should not be able to spawn up a VM in production)
- addingnumbers 5y agoI'm super confused by the statement "The unix time component has a more broad domain, but this turns out to be not a practical problem (see later)." I don't know what exactly he expected us to "see later," except that he knows exactly when the target machine rebooted down to a 15 second window because he already had full control over it before starting the attack...
- londons_explore 5y agoI can imagine cases where an attacker could have good knowledge of the reboot time of a system. For example, they could ping it for months waiting for a maintenance reboot. Or they could flood the machine with requests, hoping to cause some monitoring system or human to reboot it. But overall, it seems unlikely...
- Arnavion 5y ago>I don't know what exactly he expected us to "see later," See all mentions of "Attack #2"
- addingnumbers 5y ago
- londons_explore 5y agoSo why isn't the metadata server authenticated? It would seem simple enough for googles metadata server to have a valid HTTPS certificate and be hosted on a non-internal domain. Or use an internal domain, but make pre-built images use a custom CA. Or Google could make a 'trusted network device', rather like a VPN, which routes traffic for 169.254.169.254 (the metadata server IP address) and add metadata.google.internal to the hosts file as 169.254.169.254.
- rantwasp 5y agohow do you get the certs to the machines? ever had to rotate certs for all the machine in a datacenter?
- londons_explore 5y agoTo the metadata servers? They presumably hold keys to access all kinds of backend systems anyway. The certs don't require any additional trust. There must already be infrastructure in place for deploying said keys.
- rantwasp 5y agoyes and no. when doing stuff like this you will always have a chicken and egg problem.
- nijave 5y agoYou could also do a hybrid where each machine gets a volume with an x509 cert and key only root has access to which can then be used to mTLS to a network service (which can then manage the certs) That'd be a hybrid of cloud init data volume and network service
- rantwasp 5y agoyou could, the problem with this approach is how do you manage these volumes and the infrastructure around it. How do you get the keys on that volume? Usually, this trust is established when the machine is built the first time and it gets an identity and a cert assigned to it. You have the same problems (of how you control the infra and ensure that you and only you can do this on the network).
- antoncohen 5y agoWhile there are a series of vulnerabilities here, none of them would be exploitable in this way if the metadata server was accessed via an IP instead of the hostname metadata.google.internal. The metadata server is documented to be at 169.254.169.25, always[1]. But Google software (agents and libraries on VMs) resolves it by looking up metadata.google.internal. If metadata.google.internal isn't in /etc/hosts, as can be the case in containers, this can result in actual DNS lookups over the network to get an address that should be known. AWS uses the same address for their metadata server, but accesses via the IP address and not some hostname[2]. I've seen Google managed DNS servers (in GKE clusters) fall over under the load of Google libraries querying for the metadata address[3]. I'm guessing Google wants to maintain some flexibility, which is why they are using a hostname, but there are tradeoffs. [1] https://cloud.google.com/compute/docs/internal-dns https://cloud.google.com/compute/docs/internal-dns [2] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance... [3] This is easily solvable with Kubernetes HostAliases that write /etc/hosts in the containers.
- tryauuum 5y agoI think even without metadata server replacement this attack would still be painful. The ability to reconfigure network on a victim sounds painful
- antoncohen 5y agoThat is true, I was thinking specifically about the metadata and SSH keys. But DHCP can also set DNS servers, NTP servers, and other things that can either cause disruptions or be used to facilitate a different attack. There might be a persistence issue, it seems like part of this attack was that the IP was persisted to /etc/hosts even after the real DHCP server took over again. But even just writing to /etc/hosts could open the door redirecting traffic to an attacker controlled server.
- aenis 5y agoIt does not even take a lot. I run a production service on cloud run, the typical load is around 500qps, and the dns queries to resolve metadata server do fail frequently enough for this to be noticeable.
- zomgwat 5y agoIf I understand correctly, the attack can be mitigated with the appropriate level of firewall rules. Both ingress and egress traffic should be blocked by default and selectively allowed based on need. In this case, DHCP traffic would only be allowed to 169.254.169.254. You still have somebody in your network though, so there's that.
- rantwasp 5y agothat’s not how DHCP works. in the context of a machine coming up/renewing a lease it’s basically a broadcast and anyone on the network can reply. the traffic needs to happen on the interface where you get the ip (guessing the main interface is also using dhcp)
- zomgwat 5y agoHow does traffic reach or leave the machine if a network level firewall is restricting access? It seems I have a fundamental misunderstanding of something.
- rantwasp 5y agoif you have a network level firewall blocking DHCP traffic, you will not be able to do DHCP. The way it works for physical hosts is that all machines in the same rack see the dhcp traffic and the TOR has a special ip helper configured where it's sending the DHCP traffic. So it's broadcast in the rack and after that point to point, but still there is zero to no security when it comes to DHCP traffic. For VMs, I guess the hypervisor acts as the TOR with the same limitations.
- res0nat0r 5y agoEssentially this: > The firewall/router of GCP blocks broadcast packets sent by VMs, so only the metadata server (169.254.169.254) receives them. However, some phases of the DHCP protocol don't rely on broadcasts, and the packets to be sent can be easily calculated and sent in advance. > To mount this attack, the attacker needs to craft multiple DHCP packets using a set of precalculated/suspected XIDs and flood the victim's dhclient directly (no broadcasts here). If the XID is correct, the victim machine applies the network configuration. This is a race condition, but since the flood is fast and exhaustive, the metadata server has no real chance to win. > Google heavily relies on the Metadata server, including the distribution of ssh public keys. The connection is secured at the network/routing layer and the server is not authenticated (no TLS, clear http only). The google_guest_agent process, that is responsible for processing the responses of the Metadata server, establishes the connection via the virtual hostname metadata.google.internal which is an alias in the /etc/hosts file. He appears to spoof the DHCP packets to get the victim to respond to his rogue metadata server, insert the IP address of his metadata server into the /etc/hosts entry for metadata.google.internal, then is able to have his ssh pubkey installed on the victim so he can ssh to the host.
- corty 5y agoWhy the hell isn't the metadata server authenticated, e.g. via TLS certificates?
- tytso 5y agoThere's a really simple (albeit hacky) workaround which can be deployed fairly quickly. In /etc/dhcp/dhclient-exit-hooks.d/google_set_hostname replace this line: if [ -n "$new_host_name" ] && [ -n "$new_ip_address" ]; then with this: if [ -n "$new_host_name" -a ! "$new_host_name" =~ metadata.google.internal ] && [ -n "$new_ip_address" ]; then (Yes, =~ is a bashism, but google_set_hostname is a bash script.) This prevents /etc/hosts from getting poisoned with a bogus entry for the metadata server. Of course, dhcpd should also be fixed to use a better random number generator, and the firewall should be default stop dhcp packets from any IP address other than Google's DHCP server. Belt and suspenders, after all. But fixing the dhclient exit hooks is a simple text edit.
- floatingatoll 5y ago!= would be a simple non-regex replacement, right? Or are there parts to the exploit hostname that aren’t a literal match?
- tytso 5y agoThe reason why I used the regex match is because the attacker might try to add one or spaces as a prefix and/or suffix, e.g " metadata.google.internal " which wouldn't match "metadata.google.internal" but the spaces in /etc/hosts name would be ignored and still be effective in poisoning the /etc/hosts lookup for metadata.google.internal.
- tasterbg 5y ago<a href="https://www.google.com/ https://www.google.com/" id="fuzzelement1">test</a>