6 ms·

Xen on Raspberry Pi 4

113 points by posix_me_less 6y ago

gorgoiler 6y ago
I reach for LXC if I need to contain something inside it’s own OS, on the condition that the contained OS is running the same kernel as the host. As I understand it, Xen would allow me to contain OSs that aren’t even Linux. What is available for Raspberry pi’s hardware, that I should try? Is there a compelling advantage to using Xen for a container Linux, versus LXC?
- deleted 6y ago
  [deleted]
- dylz 6y ago
  Xen is not a container, compare it to qemu/KVM more. I would not host untrusted customers in docker containers, LXC, but I would in Xen HVM any day.
  - NullPrefix 6y ago
    >I would [..] host untrusted customers in [...] Xen HVM any day. Intel doesn't recommend running untrusted code even in a separate VM, according to their official explanation of Spectre/Meltdown. At least we can do it on RPi now.
    - pengaru 6y ago
      > At least we can do it on RPi now. Couldn't we already do it on an RPi with linux/kvm+qemu?
    - SturgeonsLaw 6y ago
      Not to mention if you give the untrusted VM a network interface then it can be the equivalent of plugging an untrusted computer into your network
      - emteycz 6y ago
        Huh? You can't make a VLAN or use a firewall?
    - s1dev 6y ago
      The A72 has speculative execution so I don’t think that’s strictly speaking true
- danielheath 6y ago
  > What is available for Raspberry pi’s hardware, that I should try? Windows runs on the pi4 (and now has x86 emulation, IIRC). There's more than a few things that only run on windows.
  - WrtCdEvrydy 6y ago
    I wonder if HyperV is supported...
  - dmos62 6y ago
    > Windows runs on the pi4 (and now has x86 emulation, IIRC). That's new to me. Is the rpi4 and window's x86 emulation performant enough to use popular windows-only apps like the Adobe suite?
    - vetrom 6y ago
      Microsoft Surface units in non-US markets have been available in ARM variants for some time now. There was some different market name for the unit but I can't recall what they're called just now.
- t0astbread 6y ago
  Security is one advantage. If you're running a server with multiple containers and you're concerned about attackers taking over one of those then you can isolate them more from each other and the rest of the system using VMs. (Addendum: I'm not that experienced when it comes to container tech and I heard you can isolate your containers pretty well these days. Virtualization has also changed over time and isn't as "isolated" as it was once; for example virtualized guests are often executed directly on the host's CPU. I'd definitely look into this more before deciding on one approach.)
tosh 6y ago
> According to the RPi Foundation, over 35 million have been sold, with 44% of these sold into industry
- gwd 6y ago
  This was the thing that most surprised me: We tend to think of RPi as aimed at hobbyists, but it turns out that it's actually at a really good price / feature point for a lot of industrial applications.
als0 6y ago
Has anyone tried comparing Xen's performance on the Pi against KVM? Any reason to choose the former over the latter?
mercora 6y ago
is it possible to pass hardware peripherals to a virtual machine with this?
- dgfitz 6y ago
  I've done that with xen before. Of course I haven't tried on a pi yet.
noodlesUK 6y ago
Can someone point me to some literature that explains the benefits of using something like Xen over KVM/qemu? I’m mostly experienced with Linux systems, so KVM is generally what I reach for.
- mkesper 6y ago
  And that's the right thing to do! Xen features no practical advantages anymore, that may have been different historically.
  - CameronNemo 6y ago
    In the past (4+ years ago) I read that Xen was more performant than KVM due to tighter integration between the host and guest kernels. Is that no longer the case? Is paravirtualization no longer necessary, or does KVM do paravirtualization now?
    - mkesper 6y ago
      KVM supports paravirtualization of different hosts for years now. Some project history: https://lwn.net/Articles/705160/ https://lwn.net/Articles/705160/
- gwd 6y ago
  It really depends on your context. On the x86 side, Xen has two unique advantages over KVM in terms of security. The first is the mature XenProject Security Response process [1]. All known Xen-related security issues, even DoSes, are announced and documented, so it's easy to find out when you need to patch your software. If you're a cloud provider, or make software with Xen as a component, you can be notified under embargo before the public announcement, so you can have your cloud patched / have a software patch tested and ready to download on the day the announcement goes public. KVM doesn't have an equivalent process. Many high-profile KVM issues are issued under embargo on a mailing list, but 1) many are not 2) only distros are allowed to be on the list. The second thing on the x86 side are some additional defense-in-depth security measures, including driver domains and device model stub domains. Driver domains allow you to run device drivers in a completely separate VM; so (for instance) a privilege escalation in iptables would allow an attacker only to control the bridge and the network device, as opposed to being able to take over the whole system. Similarly, device model stubdomains run QEMU (or the emulator of your choice) in a separate VM; which means if there's a privilege escalation bug in QEMU, you've just broken into Yet Another VM; whereas in KVM you're now inside a Linux host process. KVM processes inside Linux can be restricted with things like SVirt, but it's fundamentally more difficult to isolate a process than a VM. These are some of the reasons why QubesOS [2] and OpenXT [3] both rely on Xen. On the embedded side, the distinctive that Xen has over KVM is that it's a microkernel-style hypervisor. This leads to a couple of advantages. First, Xen itself boots in less than a second, and using the "dom0less" feature, can direct-boot any number of other domains from the same initrd [4]. This means that if none of your VMs are Linux, you don't need to run Linux at all -- you can boot up all of your VMs and have them up and running in hundreds of milliseconds; or, if you need a single VM up and running quickly, you can start that one along with dom0, and start your other ones from dom0. Secondly, Xen is small enough to be safety certified. This is possible for a microkernel-style hypervisor like Xen, particularly with the "dom0less" direct-boot feature, in a way that would be impossible for KVM, since you'd have to not only certify enough of the Linux kernel, but all of the userspace which is running to start your other VMs. This is why Xen has been making significant inroads into the embedded space. It's been put on rockets [5], and was chosen by ARM to be part of their Automotive Reference Platform [6]. If you just want to run the occasional VM on your x86 desktop, then KVM is likely to be a better bet: There won't be a significant performance difference, and it's less effort to set up. But if you're making a product in which you want to embed virtualization, Xen has a lot of advantages. This to me is actually why the Xen for RPi is so interesting: Because actually 44% of RPi sales are actually for industrial use cases, and this port expands the market both for Xen and RPi. Obviously there are lots of other strengths and weaknesses, but that should give you an idea. [1] https://xenproject.org/developers/security-policy/ https://xenproject.org/developers/security-policy/ [2] https://www.qubes-os.org/ https://www.qubes-os.org/ [3] https://openxt.org/ https://openxt.org/ [4] https://xenproject.org/2019/12/16/true-static-partitioning-with-xen-dom0-less/ https://xenproject.org/2019/12/16/true-static-partitioning-w... [5] https://www.embedded-computing.com/guest-blogs/the-final-frontier-why-engineering-firm-dornerworks-is-taking-xen-hypervisor-to-space https://www.embedded-computing.com/guest-blogs/the-final-fro... [6] https://www.youtube.com/watch?v=boh4nqPAk50 https://www.youtube.com/watch?v=boh4nqPAk50
walterbell 6y ago
Last week's discussion: https://news.ycombinator.com/item?id=24627745 https://news.ycombinator.com/item?id=24627745

113 points by posix_me_less 6y ago

I reach for LXC if I need to contain something inside it’s own OS, on the condition that the contained OS is running the same kernel as the host. As I understand it, Xen would allow me to contain OSs that aren’t even Linux. What is available for Raspberry pi’s hardware, that I should try? Is there a compelling advantage to using Xen for a container Linux, versus LXC?

deleted 6y ago
[deleted]
dylz 6y ago
Xen is not a container, compare it to qemu/KVM more. I would not host untrusted customers in docker containers, LXC, but I would in Xen HVM any day.
- NullPrefix 6y ago
  >I would [..] host untrusted customers in [...] Xen HVM any day. Intel doesn't recommend running untrusted code even in a separate VM, according to their official explanation of Spectre/Meltdown. At least we can do it on RPi now.
  - pengaru 6y ago
    > At least we can do it on RPi now. Couldn't we already do it on an RPi with linux/kvm+qemu?
  - SturgeonsLaw 6y ago
    Not to mention if you give the untrusted VM a network interface then it can be the equivalent of plugging an untrusted computer into your network
    - emteycz 6y ago
      Huh? You can't make a VLAN or use a firewall?
  - s1dev 6y ago
    The A72 has speculative execution so I don’t think that’s strictly speaking true
danielheath 6y ago
> What is available for Raspberry pi’s hardware, that I should try? Windows runs on the pi4 (and now has x86 emulation, IIRC). There's more than a few things that only run on windows.
- WrtCdEvrydy 6y ago
  I wonder if HyperV is supported...
- dmos62 6y ago
  > Windows runs on the pi4 (and now has x86 emulation, IIRC). That's new to me. Is the rpi4 and window's x86 emulation performant enough to use popular windows-only apps like the Adobe suite?
  - vetrom 6y ago
    Microsoft Surface units in non-US markets have been available in ARM variants for some time now. There was some different market name for the unit but I can't recall what they're called just now.
t0astbread 6y ago
Security is one advantage. If you're running a server with multiple containers and you're concerned about attackers taking over one of those then you can isolate them more from each other and the rest of the system using VMs. (Addendum: I'm not that experienced when it comes to container tech and I heard you can isolate your containers pretty well these days. Virtualization has also changed over time and isn't as "isolated" as it was once; for example virtualized guests are often executed directly on the host's CPU. I'd definitely look into this more before deciding on one approach.)

tosh 6y ago

> According to the RPi Foundation, over 35 million have been sold, with 44% of these sold into industry

gwd 6y ago
This was the thing that most surprised me: We tend to think of RPi as aimed at hobbyists, but it turns out that it's actually at a really good price / feature point for a lot of industrial applications.

als0 6y ago

Has anyone tried comparing Xen's performance on the Pi against KVM? Any reason to choose the former over the latter?

mercora 6y ago

is it possible to pass hardware peripherals to a virtual machine with this?

dgfitz 6y ago
I've done that with xen before. Of course I haven't tried on a pi yet.

noodlesUK 6y ago

Can someone point me to some literature that explains the benefits of using something like Xen over KVM/qemu? I’m mostly experienced with Linux systems, so KVM is generally what I reach for.

mkesper 6y ago
And that's the right thing to do! Xen features no practical advantages anymore, that may have been different historically.
- CameronNemo 6y ago
  In the past (4+ years ago) I read that Xen was more performant than KVM due to tighter integration between the host and guest kernels. Is that no longer the case? Is paravirtualization no longer necessary, or does KVM do paravirtualization now?
  - mkesper 6y ago
    KVM supports paravirtualization of different hosts for years now. Some project history: https://lwn.net/Articles/705160/ https://lwn.net/Articles/705160/
gwd 6y ago
It really depends on your context. On the x86 side, Xen has two unique advantages over KVM in terms of security. The first is the mature XenProject Security Response process [1]. All known Xen-related security issues, even DoSes, are announced and documented, so it's easy to find out when you need to patch your software. If you're a cloud provider, or make software with Xen as a component, you can be notified under embargo before the public announcement, so you can have your cloud patched / have a software patch tested and ready to download on the day the announcement goes public. KVM doesn't have an equivalent process. Many high-profile KVM issues are issued under embargo on a mailing list, but 1) many are not 2) only distros are allowed to be on the list. The second thing on the x86 side are some additional defense-in-depth security measures, including driver domains and device model stub domains. Driver domains allow you to run device drivers in a completely separate VM; so (for instance) a privilege escalation in iptables would allow an attacker only to control the bridge and the network device, as opposed to being able to take over the whole system. Similarly, device model stubdomains run QEMU (or the emulator of your choice) in a separate VM; which means if there's a privilege escalation bug in QEMU, you've just broken into Yet Another VM; whereas in KVM you're now inside a Linux host process. KVM processes inside Linux can be restricted with things like SVirt, but it's fundamentally more difficult to isolate a process than a VM. These are some of the reasons why QubesOS [2] and OpenXT [3] both rely on Xen. On the embedded side, the distinctive that Xen has over KVM is that it's a microkernel-style hypervisor. This leads to a couple of advantages. First, Xen itself boots in less than a second, and using the "dom0less" feature, can direct-boot any number of other domains from the same initrd [4]. This means that if none of your VMs are Linux, you don't need to run Linux at all -- you can boot up all of your VMs and have them up and running in hundreds of milliseconds; or, if you need a single VM up and running quickly, you can start that one along with dom0, and start your other ones from dom0. Secondly, Xen is small enough to be safety certified. This is possible for a microkernel-style hypervisor like Xen, particularly with the "dom0less" direct-boot feature, in a way that would be impossible for KVM, since you'd have to not only certify enough of the Linux kernel, but all of the userspace which is running to start your other VMs. This is why Xen has been making significant inroads into the embedded space. It's been put on rockets [5], and was chosen by ARM to be part of their Automotive Reference Platform [6]. If you just want to run the occasional VM on your x86 desktop, then KVM is likely to be a better bet: There won't be a significant performance difference, and it's less effort to set up. But if you're making a product in which you want to embed virtualization, Xen has a lot of advantages. This to me is actually why the Xen for RPi is so interesting: Because actually 44% of RPi sales are actually for industrial use cases, and this port expands the market both for Xen and RPi. Obviously there are lots of other strengths and weaknesses, but that should give you an idea. [1] https://xenproject.org/developers/security-policy/ https://xenproject.org/developers/security-policy/ [2] https://www.qubes-os.org/ https://www.qubes-os.org/ [3] https://openxt.org/ https://openxt.org/ [4] https://xenproject.org/2019/12/16/true-static-partitioning-with-xen-dom0-less/ https://xenproject.org/2019/12/16/true-static-partitioning-w... [5] https://www.embedded-computing.com/guest-blogs/the-final-frontier-why-engineering-firm-dornerworks-is-taking-xen-hypervisor-to-space https://www.embedded-computing.com/guest-blogs/the-final-fro... [6] https://www.youtube.com/watch?v=boh4nqPAk50 https://www.youtube.com/watch?v=boh4nqPAk50

walterbell 6y ago

Last week's discussion: https://news.ycombinator.com/item?id=24627745 https://news.ycombinator.com/item?id=24627745