5 ms·

CPU cache-based communication between two co-located virtual machines

223 points by j_s 10y ago

spullara 10y ago
One totally practical use case for this is to ensure that none of the virtual machines that you have allocated on your cloud provider are on the same physical hardware. This would be extremely useful for reliability purposes. I wonder how hard it would be to build something like that on top of this library?
- redrabbyte 10y ago
  not that hard, but keep in mind the current limitations on supported cpus
- zbjornson 10y ago
  Interesting idea. Don't think that would work for multi-socket boards though, which I think are most servers.
  - spullara 10y ago
    Ah, didn't see the restriction to single socket boards but that makes sense. Probably not going to work for any of the major cloud providers then.
    - inetknght 10y ago
      It could work intermittently depending on which vCPUs your VM or covert channel apps get assigned (intermittently as that could change unless your hypervisor uses CPU pinning...)
- westbywest 10y ago
  Given that it's not economical for cloud providers to let too many of their customers monkey with resource provisioning, I gather such efforts on the part of the customer can only be so effective.
- j_s 10y ago
  With virtual machines I control, I don't have to use covert channels? https://cseweb.ucsd.edu/~hovav/dist/cloudsec.pdf https://cseweb.ucsd.edu/~hovav/dist/cloudsec.pdf (2009) Edit: we describe an empirical mea-surement study focused on understanding VM placement in the EC2 system and achieving co-resident placement for an adversary. To do this, we make use of network probing both to identify public services hosted on EC2 and to provide evidence of co-residence (that two instances share the same physical server). In particular, we utilize nmap, hping, and wget to perform network probes to determine liveness of EC2 instances The hard disk contention channel was used in Section 6 for establishing co-residence of instances.
  - Florin_Andrei 10y ago
    That assumes the provider will disclose hardware info to you, which is rarely the case.
    - j_s 10y ago
      (I apologize in advance if I've missed recognition of your authority on this topic; as an outsider it appears you are wrong.) Are you speaking generally or specifically addressing a shortcoming of the linked paper which demonstrates a methodology to determine whether or not two AWS VMs are on the same physical host? After a brief review I still don't see any mention of "hardware info" you mention as assumed being available. Edit: If something has come up since 2009 with this specific paper, I'd appreciate the heads-up!
- angerbot 10y ago
  I don't know that it would be difficult to reproduce the test that is mentioned in the paper linked below, but they do mention the following anyways for EC2: > A single account was never seen to have two instances simultaneously running on the same physical machine, so running n instances in parallel under a single account results in placement on n separate machines
scott_s 10y ago
They also have an Network and Distributed System Security Symposium (NDSS) paper with a title that brought a smile to my face: https://cmaurice.fr/pdf/ndss17_maurice.pdf https://cmaurice.fr/pdf/ndss17_maurice.pdf
- redrabbyte 10y ago
  I suspect this might also make you smile then ;) https://www.youtube.com/watch?v=yPZmiRi_c-o&feature=youtu.be https://www.youtube.com/watch?v=yPZmiRi_c-o&feature=youtu.be , if you haven't seen it already
brendangregg 10y ago
I noticed the paper tested an Xeon E5-2670 on EC2, which is old (Sandy Bridge; eg, m2's) and pre-dates Intel Cache Allocation Technology (IntelCAT): https://software.intel.com/en-us/articles/introduction-to-cache-allocation-technology https://software.intel.com/en-us/articles/introduction-to-ca... Support for IntelCAT was added to Xen two years ago. Eg: [root@vmm-psr vmm]# xl psr-hwinfo --cat Cache Allocation Technology (CAT): Socket ID : 0 L3 Cache : 12288KB Maximum COS : 15 CBM length : 12 Default CBM : 0xfff And to Linux 4.10 a couple of months ago: https://lwn.net/Articles/694800/ https://lwn.net/Articles/694800/ While I suspect IntelCAT could disable this side channel, I'd personally hesitate before turning it on for everyone on the cloud. Does it out weigh the performance cost? ... Can customers opt-in to this?
- redrabbyte 10y ago
  If there are no more shared cache sets between VMs, then yes, it's dead ofc I would suspect if one VM would like to be separated from the others in terms of cache sets, the others really don't have the choice to opt out?
  - brendangregg 10y ago
    good point...
- j_s 10y ago
  Yes it was specifically called out in someone's tweet I'm trying to find but I believe this was mentioned in the paper. Edit: My mix-up on the paper! Malware Guard Extension: Using SGX to Conceal Cache Attacks (Extended Version) | https://arxiv.org/pdf/1702.08719.pdf https://arxiv.org/pdf/1702.08719.pdf (2017) Recently, Intel introduced an instruction set extension called CAT (cache allocation technology) [51]. With Intel CAT it is possible to restrict CPU cores to one of the slices of the last-level cache and even to pin cache lines. Liu et al. [83] proposed a system that uses CAT to protect general purpose software and cryptographic algorithms. Their approach can be directly applied to protect against a malicious enclave. However, this approach also does not allow to protect enclaves from an outside attacker. [...] Instead of using Intel CAT on the operating level it could also be used to protect enclaves on the hardware level. By changing the eenter instruction in a way that it implicitly activates CAT for this core, any cache sharing between SGX enclaves and the outside as well as co-located enclaves could be eliminated. Thus, SGX enclaves would be protected from outside attackers. Furthermore, it would protect co-located enclaves as well as the operating system and user programs against malicious enclaves. [83] https://www.researchgate.net/profile/Yuval_Yarom/publication/291830462_CATalyst_Defeating_Last-Level_Cache_Side_Channel_Attacks_in_Cloud_Computing/links/56a6b0d408aeded22e3544ff.pdf https://www.researchgate.net/profile/Yuval_Yarom/publication...
vgddvh 10y ago
Explanation anyone?
- wolf550e 10y ago
  Watch this: https://www.youtube.com/watch?v=IT6PFSRrvsU https://www.youtube.com/watch?v=IT6PFSRrvsU
jwilk 10y ago
https://media.ccc.de/v/33c3-8044-what_could_possibly_go_wrong_with_insert_x86_instruction_here https://media.ccc.de/v/33c3-8044-what_could_possibly_go_wron...
kirekhanki 10y ago
Watch Power Rangers Full Movie Online free http://car.247sports.com/Board/101487/Contents/UPDATED-SABAN-Watch-Power-Rangers-Full-Movie-Online-free-52061561 http://car.247sports.com/Board/101487/Contents/UPDATED-SABAN...
amenghra 10y ago
Does this have any practical applications? You might get a perf win, but the likelihood of being collocated is small.
- cowboysauce 10y ago
  I think at this point it's mostly of interest to security researchers and malware writers. It's not hard to imagine malware using this technique to transfer data out of an infected VM in a difficult to detect manner. If you can infect enough VMs it might be worth it. Also, this type of technique could be interesting to government employed malware writers.
  - sillysaurus3 10y ago
    Another interesting way to exfiltrate data is via DNS. Few networks block DNS, so you can set up a protocol on your own DNS server to transfer data. The bandwidth is pretty decent, too.
    - cmdrfred 10y ago
      I'm still surprised how well this classic attack works. I've seen $100k IDS and firewall packages not detect this. I use this personally. Works behind captive portals sometimes too: http://code.kryo.se/iodine/ http://code.kryo.se/iodine/
darkengine 10y ago
Reminds me a lot of this DEF CON talk: https://www.youtube.com/watch?v=7X772EBdvnM https://www.youtube.com/watch?v=7X772EBdvnM He managed to stream video across VMs by doing a similar cache timing trick, which brought a round of applause from the crowd. Very cool stuff.
- redrabbyte 10y ago
  I guess you could call it an evolution, because memory deduplication is not always enabled anymore (research like this might have played a role here ;) he talks about that @35:27 in the video

223 points by j_s 10y ago

One totally practical use case for this is to ensure that none of the virtual machines that you have allocated on your cloud provider are on the same physical hardware. This would be extremely useful for reliability purposes. I wonder how hard it would be to build something like that on top of this library?

redrabbyte 10y ago
not that hard, but keep in mind the current limitations on supported cpus
zbjornson 10y ago
Interesting idea. Don't think that would work for multi-socket boards though, which I think are most servers.
- spullara 10y ago
  Ah, didn't see the restriction to single socket boards but that makes sense. Probably not going to work for any of the major cloud providers then.
  - inetknght 10y ago
    It could work intermittently depending on which vCPUs your VM or covert channel apps get assigned (intermittently as that could change unless your hypervisor uses CPU pinning...)
westbywest 10y ago
Given that it's not economical for cloud providers to let too many of their customers monkey with resource provisioning, I gather such efforts on the part of the customer can only be so effective.
j_s 10y ago
With virtual machines I control, I don't have to use covert channels? https://cseweb.ucsd.edu/~hovav/dist/cloudsec.pdf https://cseweb.ucsd.edu/~hovav/dist/cloudsec.pdf (2009) Edit: we describe an empirical mea-surement study focused on understanding VM placement in the EC2 system and achieving co-resident placement for an adversary. To do this, we make use of network probing both to identify public services hosted on EC2 and to provide evidence of co-residence (that two instances share the same physical server). In particular, we utilize nmap, hping, and wget to perform network probes to determine liveness of EC2 instances The hard disk contention channel was used in Section 6 for establishing co-residence of instances.
- Florin_Andrei 10y ago
  That assumes the provider will disclose hardware info to you, which is rarely the case.
  - j_s 10y ago
    (I apologize in advance if I've missed recognition of your authority on this topic; as an outsider it appears you are wrong.) Are you speaking generally or specifically addressing a shortcoming of the linked paper which demonstrates a methodology to determine whether or not two AWS VMs are on the same physical host? After a brief review I still don't see any mention of "hardware info" you mention as assumed being available. Edit: If something has come up since 2009 with this specific paper, I'd appreciate the heads-up!
angerbot 10y ago
I don't know that it would be difficult to reproduce the test that is mentioned in the paper linked below, but they do mention the following anyways for EC2: > A single account was never seen to have two instances simultaneously running on the same physical machine, so running n instances in parallel under a single account results in placement on n separate machines

scott_s 10y ago

They also have an Network and Distributed System Security Symposium (NDSS) paper with a title that brought a smile to my face: https://cmaurice.fr/pdf/ndss17_maurice.pdf https://cmaurice.fr/pdf/ndss17_maurice.pdf

redrabbyte 10y ago
I suspect this might also make you smile then ;) https://www.youtube.com/watch?v=yPZmiRi_c-o&feature=youtu.be https://www.youtube.com/watch?v=yPZmiRi_c-o&feature=youtu.be , if you haven't seen it already

brendangregg 10y ago

I noticed the paper tested an Xeon E5-2670 on EC2, which is old (Sandy Bridge; eg, m2's) and pre-dates Intel Cache Allocation Technology (IntelCAT): https://software.intel.com/en-us/articles/introduction-to-cache-allocation-technology https://software.intel.com/en-us/articles/introduction-to-ca... Support for IntelCAT was added to Xen two years ago. Eg: [root@vmm-psr vmm]# xl psr-hwinfo --cat Cache Allocation Technology (CAT): Socket ID : 0 L3 Cache : 12288KB Maximum COS : 15 CBM length : 12 Default CBM : 0xfff And to Linux 4.10 a couple of months ago: https://lwn.net/Articles/694800/ https://lwn.net/Articles/694800/ While I suspect IntelCAT could disable this side channel, I'd personally hesitate before turning it on for everyone on the cloud. Does it out weigh the performance cost? ... Can customers opt-in to this?

redrabbyte 10y ago
If there are no more shared cache sets between VMs, then yes, it's dead ofc I would suspect if one VM would like to be separated from the others in terms of cache sets, the others really don't have the choice to opt out?
- brendangregg 10y ago
  good point...
j_s 10y ago
Yes it was specifically called out in someone's tweet I'm trying to find but I believe this was mentioned in the paper. Edit: My mix-up on the paper! Malware Guard Extension: Using SGX to Conceal Cache Attacks (Extended Version) | https://arxiv.org/pdf/1702.08719.pdf https://arxiv.org/pdf/1702.08719.pdf (2017) Recently, Intel introduced an instruction set extension called CAT (cache allocation technology) [51]. With Intel CAT it is possible to restrict CPU cores to one of the slices of the last-level cache and even to pin cache lines. Liu et al. [83] proposed a system that uses CAT to protect general purpose software and cryptographic algorithms. Their approach can be directly applied to protect against a malicious enclave. However, this approach also does not allow to protect enclaves from an outside attacker. [...] Instead of using Intel CAT on the operating level it could also be used to protect enclaves on the hardware level. By changing the eenter instruction in a way that it implicitly activates CAT for this core, any cache sharing between SGX enclaves and the outside as well as co-located enclaves could be eliminated. Thus, SGX enclaves would be protected from outside attackers. Furthermore, it would protect co-located enclaves as well as the operating system and user programs against malicious enclaves. [83] https://www.researchgate.net/profile/Yuval_Yarom/publication/291830462_CATalyst_Defeating_Last-Level_Cache_Side_Channel_Attacks_in_Cloud_Computing/links/56a6b0d408aeded22e3544ff.pdf https://www.researchgate.net/profile/Yuval_Yarom/publication...

vgddvh 10y ago

Explanation anyone?

wolf550e 10y ago
Watch this: https://www.youtube.com/watch?v=IT6PFSRrvsU https://www.youtube.com/watch?v=IT6PFSRrvsU

jwilk 10y ago

https://media.ccc.de/v/33c3-8044-what_could_possibly_go_wrong_with_insert_x86_instruction_here https://media.ccc.de/v/33c3-8044-what_could_possibly_go_wron...

kirekhanki 10y ago

Watch Power Rangers Full Movie Online free http://car.247sports.com/Board/101487/Contents/UPDATED-SABAN-Watch-Power-Rangers-Full-Movie-Online-free-52061561 http://car.247sports.com/Board/101487/Contents/UPDATED-SABAN...

amenghra 10y ago

Does this have any practical applications? You might get a perf win, but the likelihood of being collocated is small.

cowboysauce 10y ago
I think at this point it's mostly of interest to security researchers and malware writers. It's not hard to imagine malware using this technique to transfer data out of an infected VM in a difficult to detect manner. If you can infect enough VMs it might be worth it. Also, this type of technique could be interesting to government employed malware writers.
- sillysaurus3 10y ago
  Another interesting way to exfiltrate data is via DNS. Few networks block DNS, so you can set up a protocol on your own DNS server to transfer data. The bandwidth is pretty decent, too.
  - cmdrfred 10y ago
    I'm still surprised how well this classic attack works. I've seen $100k IDS and firewall packages not detect this. I use this personally. Works behind captive portals sometimes too: http://code.kryo.se/iodine/ http://code.kryo.se/iodine/

darkengine 10y ago

Reminds me a lot of this DEF CON talk: https://www.youtube.com/watch?v=7X772EBdvnM https://www.youtube.com/watch?v=7X772EBdvnM He managed to stream video across VMs by doing a similar cache timing trick, which brought a round of applause from the crowd. Very cool stuff.

redrabbyte 10y ago
I guess you could call it an evolution, because memory deduplication is not always enabled anymore (research like this might have played a role here ;) he talks about that @35:27 in the video