5 ms·

Rethinking the Linux cloud stack for confidential VMs

138 points by Bogdanp 1y ago

le-mark 1y ago
Timely considering the current (yet another) chip act. Presumably government mandated surveillance silicon would also require confidential compute capability. https://www.atlanticcouncil.org/blogs/geotech-cues/how-the-chip-security-act-could-usher-in-an-era-of-trusted-trade-with-us-partners/ https://www.atlanticcouncil.org/blogs/geotech-cues/how-the-c...
- Joel_Mckay 1y ago
  Funny, some people never consider burning goodwill with populations directly open a competitive advantage for competitors. =3
riedel 1y ago
Confidential computing is the straw for many people to overcome GDPR headaches in Europe. I know particularly medical researchers that hope that they get access to scalable infrastructure this way, because they can tick it as the only additional TOM on the processor side. As mentioned in the comments of OP though it is more a promise than a reality at the moment with very little actual benefit in term of reducing relevant attack vectors.
- procaryote 1y ago
  Yeah, much like the "sovereign cloud" stuff from amazon around where they pretend that setting up and independent advisory board with no real power is somehow a fix for the cloud act. It only fools people who want to be fooled, or genuiely have no idea.
evrennetwork 1y ago
[dead]
Bender 1y ago
Maybe this will check a box in some OpenStack cluster but it wont work for me personally. Anything sensitive I use physical servers. Once I am on a VM of a physical server that is not mine then my data is their data. It is just turtles all the way down and there will always be a way to obtain data. Whats more this is required for lawful intercept and authorities expect providers today to be able to live copy/clone a VM. There will always be a back door and when authorities can access the back door, so can the providers and malicious actors. Even more unpopular is that to me encryption is just mathematical obfuscation a.k.a. magic math and the devil is in the implementation details remember WEP and DVD encryption? Just like cell phones there will always be some simple "debugging" toggle function that can bypass it.
- Joel_Mckay 1y ago
  Unfortunately, if someone really wants into modern equipment it is rather trivial. As modern clouds often just used cost-optimized consumer grade CPUs/GPUs with sometimes minor conveniences like more ECC RAM, and backplane management options. In many ways, incident detection and automated-recovery is more important than casting your servers in concrete. Emulated VM can create read-only signed backing images, and thus may revert/monitor states. RancherVM is actually pretty useful when you dig into the architecture. Best policy is to waste as much time and money of the irrational, and interleave tantalizing payloads of costly project failures. Adversaries eventually realize the lame prize is just not worth the effort, or steal things that ultimately will cost them later. =3
- crote 1y ago
  Why do you trust your physical servers? Do you believe it is impossible for a backdoor to exist in the CPU's Management Engine? Do you inspect the contents of every single network packet entering and exiting? Do you have a way of blocking or inspecting all electromagnetic radiation? Confidential computing is trying to solve the very problem you are worried about. It is a way of providing compute as a service without the customer having to blindly trust the compute provider. It moves the line from "the host can do anything it wants" to "we're screwed if they are collaborating with Intel to bake a custom backdoor into their CPUs". To me that sounds like a very reasonable goal. Go much beyond that, and the only plausible attacker is going to be the kind of people who'll simply drag you to a black site and apply the big wrench until you start divulging encryption keys.
matthewfcarlson 1y ago
Years ago, I saw a demo for a confidential gaming VM with the idea that games could ship with a whole VM instead of an anti cheat engine. Most of the tech was around doing it performantly. I wonder why it was never productized.
- tomrod 1y ago
  I'd imagine cost is a big factor. You have to contend with a lot of bad drivers on gpus, right? (This isn't my arena, just speculating here).
- kg 1y ago
  My understanding is that some modern game DRM does use an approach like that. See https://connorjaydunn.github.io/blog/posts/denuvo-analysis/ https://connorjaydunn.github.io/blog/posts/denuvo-analysis/
  - Squeeze2664 1y ago
    Denuvo's is a virtual machine similar to Java's virtual machine, in that it executes bytecode specifically written for it, within an application's process. I believe the parent post was referring to something closer to a Hyper-V virtual machine, an entire virtual computer.
- steve1977 1y ago
  Isn’t that more or less what modern Xbox is doing? https://en.wikipedia.org/wiki/Xbox_system_software#System https://en.wikipedia.org/wiki/Xbox_system_software#System
  - dijit 1y ago
    Yes, the Xbox (since Xbox “One”; what a hilariously awful naming scheme) has software packages which are essentially Hyper-V VMs. This was nice as a developer because we were not forced to patch our games when the overlay or underlying operating system of the console changed. In fact, On The Division 1 we shipped with a patched/modified version of the SDK- this wasn’t possible on Playstation. Consequently, while the Xbox was marginally faster in a hardware sense, it was slower in reality. It even had the advantage of us using native rendering SDKs (Playstations OpenGL “with additions” was very much a bolted on second class citizen) and still we had higher quality and more consistency or our frame times on Playstation. No free lunch.
louwrentius 1y ago
I find the article a difficult read for someone not versed in “confidential computing”. It felt written for insiders and/or people smarter than me. However, I feel that “confidential computing” is some kind of story to justify something that’s not possible: keep data ‘secure’ while running code on hardware maintained by others. Any kind of encryption means that there is a secret somewhere and if you have control over the stack below the VM (hypervisor/hardware) you’ll be able to read that secret and defeat the encryption. Maybe I’m missing something, though I believe that if the data is critical enough, it’s required to have 100% control over the hardware. Now go buy an Oxide rack (no I didn’t invest in them)
- crote 1y ago
  The unique selling point here is that you don't need to trust the hypervisor or operator, as the separation and per-VM encryption is managed by the CPU itself. The CPU itself can attest that it is running your code and that your dedicated slice of memory is encrypted using a key inaccessible to the hypervisor. Provided you still trust AMD/Intel to not put backdoors into their hardware, this allows you to run your code while the physical machine is in possession of a less-trusted party. It's of course still not going to be enough for the truly paranoid, but I think it provides a neat solution for companies with security needs which can't be met via regular cloud hosting.
  - thrawa8387336 1y ago
    Hasn't that been exploited several times?
    - crote 1y ago
      I don't believe so? I have no doubt that there have been vulnerabilities, but the technology is quite new and barely used in practice, so I would be surprised if there have been significant exploits already - let alone ones applicable in the wild rather than a lab.
      - 1y ago
adam_gyroscope 1y ago
Apple has done a good job on the implementation and documentation for their confidential computing (https://security.apple.com/documentation/private-cloud-compute https://security.apple.com/documentation/private-cloud-compu...) but of course it’s Apple only. There’s a few folks working on a non-Apple version of this, eg https://confident.security/ https://confident.security/ and others (disclaimer that I helped work on a very early version of this. Read the Apple docs - they are very well written and accessible for the average HN reader.
- warkdarrior 1y ago
  Bit Google Cloud and AWS support confidential computing: https://cloud.google.com/security/products/confidential-computing https://cloud.google.com/security/products/confidential-comp... https://aws.amazon.com/confidential-computing/ https://aws.amazon.com/confidential-computing/
signalloss 1y ago
Someone willing to price this out?
Harvesterify 1y ago
I'd recommend anyone interested in Confidential Computing to read the work from Rodrigo Branco (@BSDaemon) to understand why it's mostly a failure and a PR stunt from cloud providers to give the illusion that the customer stays in control, while at the same time the hardware capabilities CC is built upon are unsecure (and can't be fixed by firmware or microcode update, most of the time). For example, a direct link to his keynote slides from ESA 3S conference last year (PDF): https://indico.esa.int/event/528/attachments/5988/10212/Keynote_Confidential_Computing_Hype_Future_and_Reality.pdf https://indico.esa.int/event/528/attachments/5988/10212/Keyn...
- mnahkies 1y ago
  The slides were an interesting read, I'd enjoy seeing the talk if it was recorded. They finish mentioning in "2023" though, we're in the back half of 2025 now - has anything changed significantly in the past couple of years? (I genuinely don't know)
  - wahern 1y ago
    Nope. Newer hardware, newer exploits.
  - JambalayaJimbo 1y ago
    I work in the space as a developer of an SGX based application. In the last few years, VM solutions have become much more popular, and our cloud provider has been pushing us to transition to AMD SEV-SNP. We haven't transitioned yet, so I cannot speak to them in great detail, but they certainly appear to greatly simplify app development.
procaryote 1y ago
Even if you were to trust secure boot and that there are no cpu bugs around the isolation, you're still running on someone else's hardware. The CPU and Secure boot has no reliable way to tell if the hardware was modded to allow bus snooping or a fake crash that still keeps the memory on a refresh loop. Don't put things in the cloud if your threat model doesn't allow you to trust the cloud provider, or whoever has the power to compell your cloud provider to do things.
- wat10000 1y ago
  Could this be solved with some sort of TPM-like secure attestation that can prove you’re running on the CPU you think you are, plus encrypted memory to defeat external memory reads?
  - throawayonthe 1y ago
    that's exactly what confidential vms are
  - procaryote 1y ago
    For it to work, the whole CPU would pretty much need to be a secure enclave. It puts very different requirements on the hardware than affordable high performance computing does. Even then, many secure enclaves have been compromised by people with enough time and motivation.
- trebligdivad 1y ago
  In AMD's SEV and Intel TDX, the memory is encrypted by the CPU before it hits the memory, so bus snooping outside of the CPU itself isn't useful. (And as that article says there's similar stuff for PCI). As I remember, there's some standards body that doesn't technically require the encryption as part of Confidential compute - but that's just silly standards.
  - michaelt 1y ago
    IMHO It still all hinges on users saying “I don’t trust this big American multinational, AWS, (who despite their policies and contracts could be subject to all manner of corrupt insiders and secret political pressures) - but I do trust this big American multinational, Intel, for some reason” To me, those trust boundaries are in the same place.

138 points by Bogdanp 1y ago

Timely considering the current (yet another) chip act. Presumably government mandated surveillance silicon would also require confidential compute capability. https://www.atlanticcouncil.org/blogs/geotech-cues/how-the-chip-security-act-could-usher-in-an-era-of-trusted-trade-with-us-partners/ https://www.atlanticcouncil.org/blogs/geotech-cues/how-the-c...

Joel_Mckay 1y ago
Funny, some people never consider burning goodwill with populations directly open a competitive advantage for competitors. =3

riedel 1y ago

Confidential computing is the straw for many people to overcome GDPR headaches in Europe. I know particularly medical researchers that hope that they get access to scalable infrastructure this way, because they can tick it as the only additional TOM on the processor side. As mentioned in the comments of OP though it is more a promise than a reality at the moment with very little actual benefit in term of reducing relevant attack vectors.

procaryote 1y ago
Yeah, much like the "sovereign cloud" stuff from amazon around where they pretend that setting up and independent advisory board with no real power is somehow a fix for the cloud act. It only fools people who want to be fooled, or genuiely have no idea.

evrennetwork 1y ago

[dead]

Bender 1y ago

Maybe this will check a box in some OpenStack cluster but it wont work for me personally. Anything sensitive I use physical servers. Once I am on a VM of a physical server that is not mine then my data is their data. It is just turtles all the way down and there will always be a way to obtain data. Whats more this is required for lawful intercept and authorities expect providers today to be able to live copy/clone a VM. There will always be a back door and when authorities can access the back door, so can the providers and malicious actors. Even more unpopular is that to me encryption is just mathematical obfuscation a.k.a. magic math and the devil is in the implementation details remember WEP and DVD encryption? Just like cell phones there will always be some simple "debugging" toggle function that can bypass it.

Joel_Mckay 1y ago
Unfortunately, if someone really wants into modern equipment it is rather trivial. As modern clouds often just used cost-optimized consumer grade CPUs/GPUs with sometimes minor conveniences like more ECC RAM, and backplane management options. In many ways, incident detection and automated-recovery is more important than casting your servers in concrete. Emulated VM can create read-only signed backing images, and thus may revert/monitor states. RancherVM is actually pretty useful when you dig into the architecture. Best policy is to waste as much time and money of the irrational, and interleave tantalizing payloads of costly project failures. Adversaries eventually realize the lame prize is just not worth the effort, or steal things that ultimately will cost them later. =3
crote 1y ago
Why do you trust your physical servers? Do you believe it is impossible for a backdoor to exist in the CPU's Management Engine? Do you inspect the contents of every single network packet entering and exiting? Do you have a way of blocking or inspecting all electromagnetic radiation? Confidential computing is trying to solve the very problem you are worried about. It is a way of providing compute as a service without the customer having to blindly trust the compute provider. It moves the line from "the host can do anything it wants" to "we're screwed if they are collaborating with Intel to bake a custom backdoor into their CPUs". To me that sounds like a very reasonable goal. Go much beyond that, and the only plausible attacker is going to be the kind of people who'll simply drag you to a black site and apply the big wrench until you start divulging encryption keys.

matthewfcarlson 1y ago

Years ago, I saw a demo for a confidential gaming VM with the idea that games could ship with a whole VM instead of an anti cheat engine. Most of the tech was around doing it performantly. I wonder why it was never productized.

tomrod 1y ago
I'd imagine cost is a big factor. You have to contend with a lot of bad drivers on gpus, right? (This isn't my arena, just speculating here).
kg 1y ago
My understanding is that some modern game DRM does use an approach like that. See https://connorjaydunn.github.io/blog/posts/denuvo-analysis/ https://connorjaydunn.github.io/blog/posts/denuvo-analysis/
- Squeeze2664 1y ago
  Denuvo's is a virtual machine similar to Java's virtual machine, in that it executes bytecode specifically written for it, within an application's process. I believe the parent post was referring to something closer to a Hyper-V virtual machine, an entire virtual computer.
steve1977 1y ago
Isn’t that more or less what modern Xbox is doing? https://en.wikipedia.org/wiki/Xbox_system_software#System https://en.wikipedia.org/wiki/Xbox_system_software#System
- dijit 1y ago
  Yes, the Xbox (since Xbox “One”; what a hilariously awful naming scheme) has software packages which are essentially Hyper-V VMs. This was nice as a developer because we were not forced to patch our games when the overlay or underlying operating system of the console changed. In fact, On The Division 1 we shipped with a patched/modified version of the SDK- this wasn’t possible on Playstation. Consequently, while the Xbox was marginally faster in a hardware sense, it was slower in reality. It even had the advantage of us using native rendering SDKs (Playstations OpenGL “with additions” was very much a bolted on second class citizen) and still we had higher quality and more consistency or our frame times on Playstation. No free lunch.

louwrentius 1y ago

I find the article a difficult read for someone not versed in “confidential computing”. It felt written for insiders and/or people smarter than me. However, I feel that “confidential computing” is some kind of story to justify something that’s not possible: keep data ‘secure’ while running code on hardware maintained by others. Any kind of encryption means that there is a secret somewhere and if you have control over the stack below the VM (hypervisor/hardware) you’ll be able to read that secret and defeat the encryption. Maybe I’m missing something, though I believe that if the data is critical enough, it’s required to have 100% control over the hardware. Now go buy an Oxide rack (no I didn’t invest in them)

crote 1y ago
The unique selling point here is that you don't need to trust the hypervisor or operator, as the separation and per-VM encryption is managed by the CPU itself. The CPU itself can attest that it is running your code and that your dedicated slice of memory is encrypted using a key inaccessible to the hypervisor. Provided you still trust AMD/Intel to not put backdoors into their hardware, this allows you to run your code while the physical machine is in possession of a less-trusted party. It's of course still not going to be enough for the truly paranoid, but I think it provides a neat solution for companies with security needs which can't be met via regular cloud hosting.
- thrawa8387336 1y ago
  Hasn't that been exploited several times?
  - crote 1y ago
    I don't believe so? I have no doubt that there have been vulnerabilities, but the technology is quite new and barely used in practice, so I would be surprised if there have been significant exploits already - let alone ones applicable in the wild rather than a lab.
    - 1y ago

adam_gyroscope 1y ago

Apple has done a good job on the implementation and documentation for their confidential computing (https://security.apple.com/documentation/private-cloud-compute https://security.apple.com/documentation/private-cloud-compu...) but of course it’s Apple only. There’s a few folks working on a non-Apple version of this, eg https://confident.security/ https://confident.security/ and others (disclaimer that I helped work on a very early version of this. Read the Apple docs - they are very well written and accessible for the average HN reader.

warkdarrior 1y ago
Bit Google Cloud and AWS support confidential computing: https://cloud.google.com/security/products/confidential-computing https://cloud.google.com/security/products/confidential-comp... https://aws.amazon.com/confidential-computing/ https://aws.amazon.com/confidential-computing/

signalloss 1y ago

Someone willing to price this out?

Harvesterify 1y ago

I'd recommend anyone interested in Confidential Computing to read the work from Rodrigo Branco (@BSDaemon) to understand why it's mostly a failure and a PR stunt from cloud providers to give the illusion that the customer stays in control, while at the same time the hardware capabilities CC is built upon are unsecure (and can't be fixed by firmware or microcode update, most of the time). For example, a direct link to his keynote slides from ESA 3S conference last year (PDF): https://indico.esa.int/event/528/attachments/5988/10212/Keynote_Confidential_Computing_Hype_Future_and_Reality.pdf https://indico.esa.int/event/528/attachments/5988/10212/Keyn...

mnahkies 1y ago
The slides were an interesting read, I'd enjoy seeing the talk if it was recorded. They finish mentioning in "2023" though, we're in the back half of 2025 now - has anything changed significantly in the past couple of years? (I genuinely don't know)
- wahern 1y ago
  Nope. Newer hardware, newer exploits.
- JambalayaJimbo 1y ago
  I work in the space as a developer of an SGX based application. In the last few years, VM solutions have become much more popular, and our cloud provider has been pushing us to transition to AMD SEV-SNP. We haven't transitioned yet, so I cannot speak to them in great detail, but they certainly appear to greatly simplify app development.

procaryote 1y ago

Even if you were to trust secure boot and that there are no cpu bugs around the isolation, you're still running on someone else's hardware. The CPU and Secure boot has no reliable way to tell if the hardware was modded to allow bus snooping or a fake crash that still keeps the memory on a refresh loop. Don't put things in the cloud if your threat model doesn't allow you to trust the cloud provider, or whoever has the power to compell your cloud provider to do things.

wat10000 1y ago
Could this be solved with some sort of TPM-like secure attestation that can prove you’re running on the CPU you think you are, plus encrypted memory to defeat external memory reads?
- throawayonthe 1y ago
  that's exactly what confidential vms are
- procaryote 1y ago
  For it to work, the whole CPU would pretty much need to be a secure enclave. It puts very different requirements on the hardware than affordable high performance computing does. Even then, many secure enclaves have been compromised by people with enough time and motivation.
trebligdivad 1y ago
In AMD's SEV and Intel TDX, the memory is encrypted by the CPU before it hits the memory, so bus snooping outside of the CPU itself isn't useful. (And as that article says there's similar stuff for PCI). As I remember, there's some standards body that doesn't technically require the encryption as part of Confidential compute - but that's just silly standards.
- michaelt 1y ago
  IMHO It still all hinges on users saying “I don’t trust this big American multinational, AWS, (who despite their policies and contracts could be subject to all manner of corrupt insiders and secret political pressures) - but I do trust this big American multinational, Intel, for some reason” To me, those trust boundaries are in the same place.