5 ms·

Linux address space isolation revived after lowering performance hit

195 points by teleforce 1y ago

api 1y ago
That's still really massive. It would only make sense in very high security environments. Honestly running system services in VMs would be cheaper and just as good, or an OS like Qubes. VM hit is much smaller, less than 1% in some cases on newer hardware.
- riedel 1y ago
  From reading the article that is the exactly also the feeling of the people involved. The question is if they are on track towards e.g. the 1% eventually.
- eptcyka 1y ago
  VMs suffer from memory use overhead. Would be cool if the guest kernel would cooperate with the host on that.
  - traverseda 1y ago
    It will! For Linux hosts and Linux guests, if you use virtio and memory ballooning.
    - shortrounddev2 1y ago
      This was an issue for me a few years ago running docker on macOS. macOS required you to allocate memory to docker ahead of time, whereas Windows/Hyper-V was able to use memory ballooning in WSL2
  - api 1y ago
    It's possible to address this to some extent with ballooning memory drivers, etc.
  - jeroenhd 1y ago
    There's KSM that should help: https://pve.proxmox.com/wiki/Kernel_Samepage_Merging_(KSM) https://pve.proxmox.com/wiki/Kernel_Samepage_Merging_(KSM) Probably works best running VMs with the same kernel and software version.
    - infogulch 1y ago
      But that just seems to reintroduce the same problem again: > However, while KSM can reduce memory usage, it also comes with some security risks, as it can expose VMs to side-channel attacks. ...
- gpapilion 1y ago
  It makes sense in any environment you have two workloads sharing compute from two parties, public clouds. The protection here is to ensure the vms are isolated. Without doing this there is the potential you can leak data via speculative execution across guests.
- russdill 1y ago
  Look at it this way, any time a new side channel attack comes out the situation changes. Having this as a mitigation that can be turned on is helpful
- bjackman 1y ago
  The next steps should make this much faster. Google's internal version generally gives us a sub-1% hit on everything we measure. If the community is up for merging this (which is a genuine question - the complexity hit is significant) I expect it to become the default everywhere and for most people it should be a performance win Vs the current default. But, yes. Not there right now, which is annoying. I'm hoping the community is willing to start merging this anyway with the trust we can get it to be really fast later. But they might say "no, we need a full prototype that's super fast right now", which would be fair.
Traubenfuchs 1y ago
Sometimes something in me starts thinking about if this regularly occurring slowing of chips through exploit mitigation is deliberate. All of big tech wins: CPUs get slower and we need more vcpu's and more memory to serve our javascript slop to end customers: The hardware companies sell more hardware, the cloud providers sell more cloud.
- bzzzt 1y ago
  Why would big tech do this when customers bring it upon themselves by building Javascript slop?
  - worthless-trash 1y ago
    Big tech isnt running their stack on js.
    - bzzzt 1y ago
      Maybe, but their cloud customers certainly are.
      - worthless-trash 1y ago
        As long as the customer pays, why wouldn't they promote an option which makes them more profit?
      - tatersolid 1y ago
        All the large cloud-hosted infra I’ve encountered in my career were written in JIT or AOT compiled languages (C#, Java, Golang, etc.) This is basically necessary at any sort of scale.
      - surajrmal 1y ago
        Cloud usage is dominated by larger companies with much older codebases that predates modern js backend development.
- 1y ago
kookamamie 1y ago
Windows suffers from similar effects when Virtualization-Based Security is active.
- Avamander 1y ago
  At the same time VBS is one of the biggest steps forward in terms of Windows kernel security. It's actually considered a proper security boundary.
  - munchlax 1y ago
    Funny that they called it VBS. That's not something I'd easily associate with a step forward in security.
- transpute 1y ago
  Hypervisor overhead should be low, https://www.howtogeek.com/does-windows-11-vbs-slow-pc-games/ https://www.howtogeek.com/does-windows-11-vbs-slow-pc-games/ What kind of workloads have noticeably lower performance with VBS?
  - kookamamie 1y ago
    We're working on HPC / graphics / computer-vision software and noticed a particularly nasty issue with VBS enabled just last week. Although, have to be mentioned it was on Win10 Pro.
    - kachapopopow 1y ago
      This most likely comes from IOMMU - disable it.
      - jychang 1y ago
        That’d break a lot of GPU setups
Eridrus 1y ago
My understanding was that many of the fixes for speculative execution issues themselves led to performance degradation, does anyone know the latest on that and how this compares? Are these performance hit numbers inclusive of turning off the other mitigations?
- snvzz 1y ago
  There's about one way[0] to fix timing side channels. The RISC-V ISA has an effort to standardize a timing fence[1][2], to take care of this once and for all. 0. https://tomchothia.gitlab.io/Papers/EuroSys19.pdf https://tomchothia.gitlab.io/Papers/EuroSys19.pdf 1. https://lf-riscv.atlassian.net/wiki/spaces/TFXX/pages/53837970/Ratification+Plan https://lf-riscv.atlassian.net/wiki/spaces/TFXX/pages/538379... 2. https://sel4.org/Summit/2024/slides/hardware-support.pdf https://sel4.org/Summit/2024/slides/hardware-support.pdf
  - eigenform 1y ago
    I'm all for giving programmers a way to flush state, and maybe this is just a matter of taste, but I wouldn't characterize this as "taking care of the problem once and for all" unless there's a [magic?] way to recover from the performance trade-off that you'd see in "normal" operating systems (ie. not seL4). It doesn't change the fact that when you implement a RISC-V core, you're going to have to partition/tag/track resources for threads that you want to be separated. Or, if you're keeping around shared state, you're going to be doing things like "flush all caches and predictors on every context switch" (can't tell if thats more or less painful). Anyway, that all still seems expensive and hard regardless of whether or not the ISA exposes it to you :(
    - snvzz 1y ago
      The research (multiple papers; note they have published and presented more than I linked) they've done prove that hardware help is necessary. i.e. not about reducing cost, but about being able to kill timing side channels at all.
- 0cf8612b2e1e 1y ago
gddbvxmm 1y ago
This week, Google Cloud paid out their highest bug bounty yet ($150k) for a vulnerability that could have been prevented with ASI [0]. Good to see that Google is pushing forward with ASI despite the performance impact, because it would benefit the security of all hosting companies that use Linux/KVM, not just the cloud providers of big tech. [0] https://cyberscoop.com/cloud-security-l1tf-reloaded-public-cloud-vulnerability-exploit/ https://cyberscoop.com/cloud-security-l1tf-reloaded-public-c...
WhyNotHugo 1y ago
When enabling this new protection, could we potentially disable other mitigation techniques which become redundant and therefore re-gain some performance?
- bjackman 1y ago
  Yes! The numbers in the posting don't account for this. Before doing this though, you need to be sure that ASI actually protects all the memory you care about. The version that currently exists protects all user memory but if the kernel copies something into its own memory it's now unprotected. So that needs to be addressed first (or some users might tolerate this risk).

195 points by teleforce 1y ago

api 1y ago

That's still really massive. It would only make sense in very high security environments. Honestly running system services in VMs would be cheaper and just as good, or an OS like Qubes. VM hit is much smaller, less than 1% in some cases on newer hardware.

riedel 1y ago
From reading the article that is the exactly also the feeling of the people involved. The question is if they are on track towards e.g. the 1% eventually.
eptcyka 1y ago
VMs suffer from memory use overhead. Would be cool if the guest kernel would cooperate with the host on that.
- traverseda 1y ago
  It will! For Linux hosts and Linux guests, if you use virtio and memory ballooning.
  - shortrounddev2 1y ago
    This was an issue for me a few years ago running docker on macOS. macOS required you to allocate memory to docker ahead of time, whereas Windows/Hyper-V was able to use memory ballooning in WSL2
- api 1y ago
  It's possible to address this to some extent with ballooning memory drivers, etc.
- jeroenhd 1y ago
  There's KSM that should help: https://pve.proxmox.com/wiki/Kernel_Samepage_Merging_(KSM) https://pve.proxmox.com/wiki/Kernel_Samepage_Merging_(KSM) Probably works best running VMs with the same kernel and software version.
  - infogulch 1y ago
    But that just seems to reintroduce the same problem again: > However, while KSM can reduce memory usage, it also comes with some security risks, as it can expose VMs to side-channel attacks. ...
gpapilion 1y ago
It makes sense in any environment you have two workloads sharing compute from two parties, public clouds. The protection here is to ensure the vms are isolated. Without doing this there is the potential you can leak data via speculative execution across guests.
russdill 1y ago
Look at it this way, any time a new side channel attack comes out the situation changes. Having this as a mitigation that can be turned on is helpful
bjackman 1y ago
The next steps should make this much faster. Google's internal version generally gives us a sub-1% hit on everything we measure. If the community is up for merging this (which is a genuine question - the complexity hit is significant) I expect it to become the default everywhere and for most people it should be a performance win Vs the current default. But, yes. Not there right now, which is annoying. I'm hoping the community is willing to start merging this anyway with the trust we can get it to be really fast later. But they might say "no, we need a full prototype that's super fast right now", which would be fair.

Traubenfuchs 1y ago

Sometimes something in me starts thinking about if this regularly occurring slowing of chips through exploit mitigation is deliberate. All of big tech wins: CPUs get slower and we need more vcpu's and more memory to serve our javascript slop to end customers: The hardware companies sell more hardware, the cloud providers sell more cloud.

bzzzt 1y ago
Why would big tech do this when customers bring it upon themselves by building Javascript slop?
- worthless-trash 1y ago
  Big tech isnt running their stack on js.
  - bzzzt 1y ago
    Maybe, but their cloud customers certainly are.
    - worthless-trash 1y ago
      As long as the customer pays, why wouldn't they promote an option which makes them more profit?
    - tatersolid 1y ago
      All the large cloud-hosted infra I’ve encountered in my career were written in JIT or AOT compiled languages (C#, Java, Golang, etc.) This is basically necessary at any sort of scale.
    - surajrmal 1y ago
      Cloud usage is dominated by larger companies with much older codebases that predates modern js backend development.
1y ago

kookamamie 1y ago

Windows suffers from similar effects when Virtualization-Based Security is active.

Avamander 1y ago
At the same time VBS is one of the biggest steps forward in terms of Windows kernel security. It's actually considered a proper security boundary.
- munchlax 1y ago
  Funny that they called it VBS. That's not something I'd easily associate with a step forward in security.
transpute 1y ago
Hypervisor overhead should be low, https://www.howtogeek.com/does-windows-11-vbs-slow-pc-games/ https://www.howtogeek.com/does-windows-11-vbs-slow-pc-games/ What kind of workloads have noticeably lower performance with VBS?
- kookamamie 1y ago
  We're working on HPC / graphics / computer-vision software and noticed a particularly nasty issue with VBS enabled just last week. Although, have to be mentioned it was on Win10 Pro.
  - kachapopopow 1y ago
    This most likely comes from IOMMU - disable it.
    - jychang 1y ago
      That’d break a lot of GPU setups

Eridrus 1y ago

My understanding was that many of the fixes for speculative execution issues themselves led to performance degradation, does anyone know the latest on that and how this compares? Are these performance hit numbers inclusive of turning off the other mitigations?

snvzz 1y ago
There's about one way[0] to fix timing side channels. The RISC-V ISA has an effort to standardize a timing fence[1][2], to take care of this once and for all. 0. https://tomchothia.gitlab.io/Papers/EuroSys19.pdf https://tomchothia.gitlab.io/Papers/EuroSys19.pdf 1. https://lf-riscv.atlassian.net/wiki/spaces/TFXX/pages/53837970/Ratification+Plan https://lf-riscv.atlassian.net/wiki/spaces/TFXX/pages/538379... 2. https://sel4.org/Summit/2024/slides/hardware-support.pdf https://sel4.org/Summit/2024/slides/hardware-support.pdf
- eigenform 1y ago
  I'm all for giving programmers a way to flush state, and maybe this is just a matter of taste, but I wouldn't characterize this as "taking care of the problem once and for all" unless there's a [magic?] way to recover from the performance trade-off that you'd see in "normal" operating systems (ie. not seL4). It doesn't change the fact that when you implement a RISC-V core, you're going to have to partition/tag/track resources for threads that you want to be separated. Or, if you're keeping around shared state, you're going to be doing things like "flush all caches and predictors on every context switch" (can't tell if thats more or less painful). Anyway, that all still seems expensive and hard regardless of whether or not the ISA exposes it to you :(
  - snvzz 1y ago
    The research (multiple papers; note they have published and presented more than I linked) they've done prove that hardware help is necessary. i.e. not about reducing cost, but about being able to kill timing side channels at all.
0cf8612b2e1e 1y ago

gddbvxmm 1y ago

This week, Google Cloud paid out their highest bug bounty yet ($150k) for a vulnerability that could have been prevented with ASI [0]. Good to see that Google is pushing forward with ASI despite the performance impact, because it would benefit the security of all hosting companies that use Linux/KVM, not just the cloud providers of big tech. [0] https://cyberscoop.com/cloud-security-l1tf-reloaded-public-cloud-vulnerability-exploit/ https://cyberscoop.com/cloud-security-l1tf-reloaded-public-c...

WhyNotHugo 1y ago

When enabling this new protection, could we potentially disable other mitigation techniques which become redundant and therefore re-gain some performance?

bjackman 1y ago
Yes! The numbers in the posting don't account for this. Before doing this though, you need to be sure that ASI actually protects all the memory you care about. The version that currently exists protects all user memory but if the kernel copies something into its own memory it's now unprotected. So that needs to be addressed first (or some users might tolerate this risk).