13 ms·

Why Xen Wasn't Hit by RETBleed on Intel CPUs

130 points by plam503711 4y ago

nano9 4y ago
Dear author, if you're the one sharing the article: you misspelled "retpoline" in your very first usage of the word.
- plam503711 4y ago
  A spectre silently fixed the spelling, thanks for the feedback ;)
  - ClassyJacket 4y ago
    I see you fixed rewrite as well, just as I was getting ready to point it out :P
    - plam503711 4y ago
      Yes, sorry for that, I was really more focused on getting the story details than the spelling (plus I'm not a native speaker as you probably guessed).
      - mmastrac 4y ago
        It wasn't obvious at all when reading - your written English is excellent. Only after revisiting and re-reading more carefully I noticed you used the construction "So ...." slightly more than a native speaker would have. Thanks for the great article.
        plam503711 4y ago
        Thank you, both for being kind and also providing a constructive feedback. "So" is very common trap for French speakers :D
- karamanolev 4y ago
  Must've been a rowhammer attack flipping some bits...
  - doubled112 4y ago
    Haven't there been solar storms this week? Maybe it wasn't malicious, maybe it was cosmic rays.
- deleted 4y ago
  [deleted]
kramerger 4y ago
So basically when people were fixing Spectre 2 Xen developers enabled an Intel thing that restricts speculation in indirect branches while Linux opted to not use that. What is the performance penalty for this thing? Was performance the reason for Linux's refusal?
- plam503711 4y ago
  You can read the original conversation from 2018 here: https://lkml.org/lkml/2018/1/21/192 https://lkml.org/lkml/2018/1/21/192
  - Sakos 4y ago
    Previous discussion (2018): https://news.ycombinator.com/item?id=16202205 https://news.ycombinator.com/item?id=16202205 I get the sense that IBRS wasn't used because Linus 1) didn't want to take on the responsibility of fixing something that needs to be fixed in hardware by Intel and 2) didn't want to help Intel avoid responsibility for shipping insecure hardware. The same way he's been antagonistic towards nVidia for being bad participants.
    - generalizations 4y ago
      Sounds like Linux/Linus figured they had enough clout to call bs on the patch, while Xen just focused on doing the best with what they had. I honestly don't blame either.
    - nousermane 4y ago
      > Linus (...) didn't want to help Intel avoid responsibility for shipping insecure hardware. Whatever is the saying about good intentions, you can argue he ended up doing Intel a favour here, by shipping patch that took away only a couple % of performance. Surely, it would've hurt them more, reputation-wise, if default workaround took larger chunk of CPU speed.
      - Sakos 4y ago
dontlaugh 4y ago
This absolutely vindicates Xen's approach vs Linux's. Performance gained by speculative execution sadly must be given up for safety as a default, to only be re-gained in specific cases where safety is certain. It also vindicates the naming of "Spectre", it most definitely keeps showing up. The article reads a bit too much like "I told you so", but ultimately the author is correct to say it.
- plam503711 4y ago
  The real golden-"I told you so" (that triggered the idea to write this very blog post) comes from a tweet of David Woodhouse last July: https://twitter.com/dwmw2/status/1549042968320811008 https://twitter.com/dwmw2/status/1549042968320811008
  - eru 4y ago
    Direct link: https://lkml.org/lkml/2018/1/22/598 https://lkml.org/lkml/2018/1/22/598
  - dwmw2 4y ago
    Hah, I wondered why there was an uptick of attention to that old tweet. We should be slightly careful — while I can't deny that there's a small element of "ITYS" about that tweet, should it really count if I then left it up to Intel to follow up? I whined that we hadn't shown that it was safe. The credit really does need to go to the RETbleed folks who put in the work to truly demonstrate that it wasn't.
- deleted 4y ago
  [deleted]
- ajross 4y ago
  > This absolutely vindicates Xen's approach vs Linux's Meh. It likely just points out that Xen is a much more limited software environment with much less dependence on indirect branching[1]. There are environments where IBRS has high cost and ones where it doesn't. Linux is in the former category. Xen also has the advantage of being a hypervisor, meaning that if all they do is expose IBRS to the guest, they can (somewhat cleverly) claim that any resulting vulnerabilities are the fault of the guest software not implementing them. Linux exposes a Unix userspace, and no one told userspace apps they need to use speculation barriers. Really this article is mostly just marketing. It's a win for Xen, sure, and they should crow about it. But we should recognize crowing vs. genuine security analysis, too. [1] Vs. say, Linux, which has an extremely robust polymorphic device/bus/probe model where all the methods are function pointers.
saagarjha 4y ago
Question: what’s the performance difference for enabling this in Linux versus in Xen? Naively I might expect that a system probably spends more time in Linux code, so the overall impact might be higher…
- bonzini 4y ago
  See https://news.ycombinator.com/item?id=32607709 https://news.ycombinator.com/item?id=32607709; in addition to what you say, enabling IBRS basically comes for free in Xen because its "userspace" is actually guest kernel code that can itself control processor speculation.
  - monocasa 4y ago
    Whatever comment you linked to seems to be deleted. Can you summarize?
    - glandium 4y ago
      The link is good, but a semi-colon got stuck to it. If you remove the semi-colon, you'll get to the comment.
Thaxll 4y ago
I forgot about Xen, is it still popular I thought everything moved to KVM in the last 10years?
- sofixa 4y ago
  Not really. VMware vSphere and KVM based virtualisations are by far the most popular. Xen's last holdout was AWS' custom version of it, but that has been replaced since a few generations ago with the KVM-based Nitro. So it's basically a niche thing, mostly used by those who already had it/know it (VMware vSphere is going down that road as well, it's basically legacy today).
- eixiepia 4y ago
  Yes it's still popular in some places, and lots of new development is going on. Xen is superior to KVM in my opinion.
  - AshamedCaptain 4y ago
    I'd actually _like_ to use Xen, but as far as I can see is just dead in all but name. To name one example, nested virtualization support is not only hopelessly broken, it's MORE broken in recent releases than it was a decade ago. You can see right here how the feature kept getting broken by every other release until nothing worked anymore: https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xe.... And Xen is literally the only virtualizer out there that does not support nested virtualization, which is a rather critical feature since many (dev) stacks assume one has hardware virtualization, and Windows is going to require it sooner than later.
    - naasking 4y ago
      If nested virtualization isn't used by their main customers I'm not sure it's that critical to them
    - plam503711 4y ago
      Xen Project is far being dead (there's a lot of activity in the mailing list, and now, thanks to new contributors like Vates/XCP-ng, there's also more initiatives to have a decent project tracking, see https://gitlab.com/groups/xen-project/-/epics?state=opened&page=1&sort=start_date_desc https://gitlab.com/groups/xen-project/-/epics?state=opened&p... for example). Regarding nested virt, you are mostly right: it's only "working-ish" for basic things, but indeed, it's broken when you start to use anything heavy in your nested VM. The main reason nobody fixed it is because it's not really used: as any other open source project, you find what you need if you contribute. Obviously, as soon someone will need this and willing to contribute, it will change :)
adultSwim 4y ago
David Woodhouse was right all along
api 4y ago
> Security is hard Unrelated but I am getting so sick of this cliche. Thinking of more insightful ways to explain why something is challenging is hard.
- effie 4y ago
  Also: hardware is hard, cryptography is hard. Doing anything new and important is hard.
Lind5 4y ago
More info here from the discoverers at ETH Zurich https://comsec.ethz.ch/research/microarch/retbleed/ https://comsec.ethz.ch/research/microarch/retbleed/ and here is the actual technical paper https://comsec.ethz.ch/wp-content/files/retbleed_sec22.pdf https://comsec.ethz.ch/wp-content/files/retbleed_sec22.pdf
bityard 4y ago
Has there ever been a practical speculative execution attack found in the wild?
- nilsbunger 4y ago
  Check out the video at https://comsec.ethz.ch/research/microarch/retbleed/ https://comsec.ethz.ch/research/microarch/retbleed/ , they capture the root password hash with their little utility.
effie 4y ago
> Mostly because people didn't believe that it was possible to exploit the retpoline limitations versus the performance penalty to mitigate them. I'm not a native speaker, but is this an acceptable written English? The part starting with "versus" seems out of place.
- ccooffee 4y ago
  It looks like the article may have updated this sentence. It now reads as: > Mostly because people didn't believe that it was possible to exploit the retpoline limitations, it was somehow logical -but risky- to refuse to implement a mitigation that would slow you down. In the version you quoted, I'd call it "clumsy" due to a lack of logical comparison from the left- and right-operands of "versus". I could understand what the author was getting at, however.
  - effie 4y ago
    Thanks.

130 points by plam503711 4y ago

nano9 4y ago

Dear author, if you're the one sharing the article: you misspelled "retpoline" in your very first usage of the word.

plam503711 4y ago
A spectre silently fixed the spelling, thanks for the feedback ;)
- ClassyJacket 4y ago
  I see you fixed rewrite as well, just as I was getting ready to point it out :P
  - plam503711 4y ago
    Yes, sorry for that, I was really more focused on getting the story details than the spelling (plus I'm not a native speaker as you probably guessed).
    - mmastrac 4y ago
      It wasn't obvious at all when reading - your written English is excellent. Only after revisiting and re-reading more carefully I noticed you used the construction "So ...." slightly more than a native speaker would have. Thanks for the great article.
      - plam503711 4y ago
        Thank you, both for being kind and also providing a constructive feedback. "So" is very common trap for French speakers :D
karamanolev 4y ago
Must've been a rowhammer attack flipping some bits...
- doubled112 4y ago
  Haven't there been solar storms this week? Maybe it wasn't malicious, maybe it was cosmic rays.
deleted 4y ago
[deleted]

kramerger 4y ago

So basically when people were fixing Spectre 2 Xen developers enabled an Intel thing that restricts speculation in indirect branches while Linux opted to not use that. What is the performance penalty for this thing? Was performance the reason for Linux's refusal?

plam503711 4y ago
You can read the original conversation from 2018 here: https://lkml.org/lkml/2018/1/21/192 https://lkml.org/lkml/2018/1/21/192
- Sakos 4y ago
  Previous discussion (2018): https://news.ycombinator.com/item?id=16202205 https://news.ycombinator.com/item?id=16202205 I get the sense that IBRS wasn't used because Linus 1) didn't want to take on the responsibility of fixing something that needs to be fixed in hardware by Intel and 2) didn't want to help Intel avoid responsibility for shipping insecure hardware. The same way he's been antagonistic towards nVidia for being bad participants.
  - generalizations 4y ago
    Sounds like Linux/Linus figured they had enough clout to call bs on the patch, while Xen just focused on doing the best with what they had. I honestly don't blame either.
  - nousermane 4y ago
    > Linus (...) didn't want to help Intel avoid responsibility for shipping insecure hardware. Whatever is the saying about good intentions, you can argue he ended up doing Intel a favour here, by shipping patch that took away only a couple % of performance. Surely, it would've hurt them more, reputation-wise, if default workaround took larger chunk of CPU speed.
    - Sakos 4y ago

dontlaugh 4y ago

This absolutely vindicates Xen's approach vs Linux's. Performance gained by speculative execution sadly must be given up for safety as a default, to only be re-gained in specific cases where safety is certain. It also vindicates the naming of "Spectre", it most definitely keeps showing up. The article reads a bit too much like "I told you so", but ultimately the author is correct to say it.

plam503711 4y ago
The real golden-"I told you so" (that triggered the idea to write this very blog post) comes from a tweet of David Woodhouse last July: https://twitter.com/dwmw2/status/1549042968320811008 https://twitter.com/dwmw2/status/1549042968320811008
- eru 4y ago
  Direct link: https://lkml.org/lkml/2018/1/22/598 https://lkml.org/lkml/2018/1/22/598
- dwmw2 4y ago
  Hah, I wondered why there was an uptick of attention to that old tweet. We should be slightly careful — while I can't deny that there's a small element of "ITYS" about that tweet, should it really count if I then left it up to Intel to follow up? I whined that we hadn't shown that it was safe. The credit really does need to go to the RETbleed folks who put in the work to truly demonstrate that it wasn't.
deleted 4y ago
[deleted]
ajross 4y ago
> This absolutely vindicates Xen's approach vs Linux's Meh. It likely just points out that Xen is a much more limited software environment with much less dependence on indirect branching[1]. There are environments where IBRS has high cost and ones where it doesn't. Linux is in the former category. Xen also has the advantage of being a hypervisor, meaning that if all they do is expose IBRS to the guest, they can (somewhat cleverly) claim that any resulting vulnerabilities are the fault of the guest software not implementing them. Linux exposes a Unix userspace, and no one told userspace apps they need to use speculation barriers. Really this article is mostly just marketing. It's a win for Xen, sure, and they should crow about it. But we should recognize crowing vs. genuine security analysis, too. [1] Vs. say, Linux, which has an extremely robust polymorphic device/bus/probe model where all the methods are function pointers.

saagarjha 4y ago

Question: what’s the performance difference for enabling this in Linux versus in Xen? Naively I might expect that a system probably spends more time in Linux code, so the overall impact might be higher…

bonzini 4y ago
See https://news.ycombinator.com/item?id=32607709 https://news.ycombinator.com/item?id=32607709; in addition to what you say, enabling IBRS basically comes for free in Xen because its "userspace" is actually guest kernel code that can itself control processor speculation.
- monocasa 4y ago
  Whatever comment you linked to seems to be deleted. Can you summarize?
  - glandium 4y ago
    The link is good, but a semi-colon got stuck to it. If you remove the semi-colon, you'll get to the comment.

Thaxll 4y ago

I forgot about Xen, is it still popular I thought everything moved to KVM in the last 10years?

sofixa 4y ago
Not really. VMware vSphere and KVM based virtualisations are by far the most popular. Xen's last holdout was AWS' custom version of it, but that has been replaced since a few generations ago with the KVM-based Nitro. So it's basically a niche thing, mostly used by those who already had it/know it (VMware vSphere is going down that road as well, it's basically legacy today).
eixiepia 4y ago
Yes it's still popular in some places, and lots of new development is going on. Xen is superior to KVM in my opinion.
- AshamedCaptain 4y ago
  I'd actually _like_ to use Xen, but as far as I can see is just dead in all but name. To name one example, nested virtualization support is not only hopelessly broken, it's MORE broken in recent releases than it was a decade ago. You can see right here how the feature kept getting broken by every other release until nothing worked anymore: https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xe.... And Xen is literally the only virtualizer out there that does not support nested virtualization, which is a rather critical feature since many (dev) stacks assume one has hardware virtualization, and Windows is going to require it sooner than later.
  - naasking 4y ago
    If nested virtualization isn't used by their main customers I'm not sure it's that critical to them
  - plam503711 4y ago
    Xen Project is far being dead (there's a lot of activity in the mailing list, and now, thanks to new contributors like Vates/XCP-ng, there's also more initiatives to have a decent project tracking, see https://gitlab.com/groups/xen-project/-/epics?state=opened&page=1&sort=start_date_desc https://gitlab.com/groups/xen-project/-/epics?state=opened&p... for example). Regarding nested virt, you are mostly right: it's only "working-ish" for basic things, but indeed, it's broken when you start to use anything heavy in your nested VM. The main reason nobody fixed it is because it's not really used: as any other open source project, you find what you need if you contribute. Obviously, as soon someone will need this and willing to contribute, it will change :)

adultSwim 4y ago

David Woodhouse was right all along

api 4y ago

> Security is hard Unrelated but I am getting so sick of this cliche. Thinking of more insightful ways to explain why something is challenging is hard.

effie 4y ago
Also: hardware is hard, cryptography is hard. Doing anything new and important is hard.

Lind5 4y ago

More info here from the discoverers at ETH Zurich https://comsec.ethz.ch/research/microarch/retbleed/ https://comsec.ethz.ch/research/microarch/retbleed/ and here is the actual technical paper https://comsec.ethz.ch/wp-content/files/retbleed_sec22.pdf https://comsec.ethz.ch/wp-content/files/retbleed_sec22.pdf

bityard 4y ago

Has there ever been a practical speculative execution attack found in the wild?

nilsbunger 4y ago
Check out the video at https://comsec.ethz.ch/research/microarch/retbleed/ https://comsec.ethz.ch/research/microarch/retbleed/ , they capture the root password hash with their little utility.

effie 4y ago

> Mostly because people didn't believe that it was possible to exploit the retpoline limitations versus the performance penalty to mitigate them. I'm not a native speaker, but is this an acceptable written English? The part starting with "versus" seems out of place.

ccooffee 4y ago
It looks like the article may have updated this sentence. It now reads as: > Mostly because people didn't believe that it was possible to exploit the retpoline limitations, it was somehow logical -but risky- to refuse to implement a mitigation that would slow you down. In the version you quoted, I'd call it "clumsy" due to a lack of logical comparison from the left- and right-operands of "versus". I could understand what the author was getting at, however.
- effie 4y ago
  Thanks.