5 ms·

Xen Project officially ports its hypervisor to Raspberry Pi 4

147 points by sohkamyung 6y ago

vianpl 6y ago
https://lore.kernel.org/lkml/alpine.DEB.2.21.2007101521290.4124@sstabellini-ThinkPad-T480s/ https://lore.kernel.org/lkml/alpine.DEB.2.21.2007101521290.4...
- ncmncm 6y ago
  Is this a parallel effort? Or somebody who did it first? Or a snapshot of the work reported? And, if one of the former, how did they solve the problems reported in the article?
  - gwd 6y ago
    > Is this a parallel effort? Or somebody who did it first? It's not a parallel effort; Stefano Stabellini is actually quoted in The Register article. There's a tiny bit more technical detail in Stefano's own article here: https://www.linux.com/featured/xen-on-raspberry-pi-4-adventures/ https://www.linux.com/featured/xen-on-raspberry-pi-4-adventu... But for more details you'll have to either look through the git commits, or wait until the LWN article comes out.
LargoLasskhyfv 6y ago
So this is "Asymmetric Nested Virtualization", then?
nikisweeting 6y ago
Did ESXi on Raspi ever materialize? I remember seeing rumors about it being in the works last year.
- rwmj 6y ago
  There was another VMware demo recently. https://www.theregister.com/2020/09/24/vmworld_preview_esxi_on_arm/ https://www.theregister.com/2020/09/24/vmworld_preview_esxi_...
- sithadmin 6y ago
  Pay attention to VMworld this week :)
  - LeSaucy 6y ago
    :O
  - ProtoAES256 6y ago
    Thanks for the heads up, will certainly do!
rwmj 6y ago
RPi4 is interesting because it's the first Raspberry Pi with a serious amount of RAM (8GB), moderate speed CPUs (4 x Cortex A72), that can boot and run from external USB 3 SSDs, with UAS and UEFI+ACPI. Essentially you can now use them for aarch64 server SBSA/SBBR development. It's about a third of the speed of my Intel i7 laptop from 18 months ago, which considering the incredibly low price I consider to be fantastic. I recently got Fedora Server installed on one using Robert Grimm's instructions (linked from my blog along with other useful links here: https://rwmj.wordpress.com/2020/09/24/raspberry-pi-4-running-fedora-32/ https://rwmj.wordpress.com/2020/09/24/raspberry-pi-4-running...)
- kristopolous 6y ago
  Are you running a 64 bit kernel actually? Which distribution? I found I could either get a 64 bit kernel (lots of things) Or decent hardware support (raspbian) but not both.
  - iMerNibor 6y ago
    There is now beta (?) builds of a 64bit raspberry pi os (formerly known as raspbian) available: https://downloads.raspberrypi.org/raspios_arm64/images/ https://downloads.raspberrypi.org/raspios_arm64/images/ Not sure how well it works, but it is a thing now
    - StillBored 6y ago
      I'm not sure I understand what the purpose of the raspberry pi OS is anymore. In the past it was a pretty custom kernel, with a user environment stripped down to work with a minimal RAM configuration. Everyone put up with nonstandard SSH daemons/window managers/missing packages/etc because due to the constrained environment and the fact that arm wasn't a first class citizen with the likes of ubuntu/redhat/etc. These days, every single major distro has rpi instructions, and 8G of ram is more than some laptops come with. So, its a disservice that they don't just tell people, here are a list of disto's which have been qualified to run on the hardware, pick one, and oh BTW, you can run those OS's on your laptop too.
  - 6y ago
bullen 6y ago
Per watt, Raspberry 4 is probably peak CPU of our civilization. But 64-bit is only viable if you buy the twice as expensive 8GB model and have use for the extra bandwidth. I'm building hybrid clusters of 4 (2GB) & 2 (the older version) on 32-bit so I can run my PaaS at record flops/W/$: https://github.com/tinspin/rupy https://github.com/tinspin/rupy
- coder543 6y ago
  > But 64-bit is only viable if you buy the twice as expensive 8GB model and have use for the extra bandwidth. Benchmarks don't entirely agree: https://www.cnx-software.com/2016/03/01/64-bit-arm-aarch64-instructions-boost-performance-by-15-to-30-compared-to-32-bit-arm-aarch32-instructions/ https://www.cnx-software.com/2016/03/01/64-bit-arm-aarch64-i... If the difference between AArch64 and AArch32 were simply larger pointers, I would probably agree, but AArch64 includes a number of instructions that can do more work, so 64-bit programs can actually run faster than their 32-bit equivalents, even if you're not using much RAM.
  - als0 6y ago
    AArch64 also has a larger number of registers, which can make a big difference for workloads.
- deleted 6y ago
  [deleted]
- my123 6y ago
  Per watt, it's not even close. That's because RPis are a lot behind on manufacturing processes, with newer designs being more efficient too. (RPi4 is on 28nm)
- kev009 6y ago
  Apple has had the performance per watt crown for quite some time. These chips are not very good for anything outside of prototyping, otherwise hyperscalers would run a sea of small nodes.
- genpfault 6y ago
rhatr 6y ago
One of the guys behind this hack here. The other is Stefano (and most credit goes to him actually). Would love for everyone to give it a try and help us improve it. I'm working on a few ready-made VMs that you can easily deploy on Raspberry Pi 4 with Project EVE (including Windows 10 VM and Plan9 VM). If you're interested -- drop me a note (I may even be able to do a few bounties/etc. for things we need to improve ASAP). Otherwise -- feel free to ask questions, etc. in this thread. It looks small -- but the whole thing took us a month or so (not full time of course) so there's quite a bit of subtlety there.
rhatr 6y ago
Btw, I just cut EVE release 5.12.3-rpi-xen-arm64 (there's also 5.12.3-rpi-kvm-arm64 of course -- but this is Xen thread ;-))) that fixes a few issues that were reported ever since we got slashdotted this morning. This is your best bet in trying Xen/EVE/RPi4 combo. E.g. docker run lfedge/eve:5.12.3-rpi-xen-arm64 live > live.raw is literally all you need to get going.

147 points by sohkamyung 6y ago

https://lore.kernel.org/lkml/alpine.DEB.2.21.2007101521290.4124@sstabellini-ThinkPad-T480s/ https://lore.kernel.org/lkml/alpine.DEB.2.21.2007101521290.4...

ncmncm 6y ago
Is this a parallel effort? Or somebody who did it first? Or a snapshot of the work reported? And, if one of the former, how did they solve the problems reported in the article?
- gwd 6y ago
  > Is this a parallel effort? Or somebody who did it first? It's not a parallel effort; Stefano Stabellini is actually quoted in The Register article. There's a tiny bit more technical detail in Stefano's own article here: https://www.linux.com/featured/xen-on-raspberry-pi-4-adventures/ https://www.linux.com/featured/xen-on-raspberry-pi-4-adventu... But for more details you'll have to either look through the git commits, or wait until the LWN article comes out.

LargoLasskhyfv 6y ago

So this is "Asymmetric Nested Virtualization", then?

nikisweeting 6y ago

Did ESXi on Raspi ever materialize? I remember seeing rumors about it being in the works last year.

rwmj 6y ago
There was another VMware demo recently. https://www.theregister.com/2020/09/24/vmworld_preview_esxi_on_arm/ https://www.theregister.com/2020/09/24/vmworld_preview_esxi_...
sithadmin 6y ago
Pay attention to VMworld this week :)
- LeSaucy 6y ago
  :O
- ProtoAES256 6y ago
  Thanks for the heads up, will certainly do!

rwmj 6y ago

RPi4 is interesting because it's the first Raspberry Pi with a serious amount of RAM (8GB), moderate speed CPUs (4 x Cortex A72), that can boot and run from external USB 3 SSDs, with UAS and UEFI+ACPI. Essentially you can now use them for aarch64 server SBSA/SBBR development. It's about a third of the speed of my Intel i7 laptop from 18 months ago, which considering the incredibly low price I consider to be fantastic. I recently got Fedora Server installed on one using Robert Grimm's instructions (linked from my blog along with other useful links here: https://rwmj.wordpress.com/2020/09/24/raspberry-pi-4-running-fedora-32/ https://rwmj.wordpress.com/2020/09/24/raspberry-pi-4-running...)

kristopolous 6y ago
Are you running a 64 bit kernel actually? Which distribution? I found I could either get a 64 bit kernel (lots of things) Or decent hardware support (raspbian) but not both.
- iMerNibor 6y ago
  There is now beta (?) builds of a 64bit raspberry pi os (formerly known as raspbian) available: https://downloads.raspberrypi.org/raspios_arm64/images/ https://downloads.raspberrypi.org/raspios_arm64/images/ Not sure how well it works, but it is a thing now
  - StillBored 6y ago
    I'm not sure I understand what the purpose of the raspberry pi OS is anymore. In the past it was a pretty custom kernel, with a user environment stripped down to work with a minimal RAM configuration. Everyone put up with nonstandard SSH daemons/window managers/missing packages/etc because due to the constrained environment and the fact that arm wasn't a first class citizen with the likes of ubuntu/redhat/etc. These days, every single major distro has rpi instructions, and 8G of ram is more than some laptops come with. So, its a disservice that they don't just tell people, here are a list of disto's which have been qualified to run on the hardware, pick one, and oh BTW, you can run those OS's on your laptop too.
- 6y ago

bullen 6y ago

Per watt, Raspberry 4 is probably peak CPU of our civilization. But 64-bit is only viable if you buy the twice as expensive 8GB model and have use for the extra bandwidth. I'm building hybrid clusters of 4 (2GB) & 2 (the older version) on 32-bit so I can run my PaaS at record flops/W/$: https://github.com/tinspin/rupy https://github.com/tinspin/rupy

coder543 6y ago
> But 64-bit is only viable if you buy the twice as expensive 8GB model and have use for the extra bandwidth. Benchmarks don't entirely agree: https://www.cnx-software.com/2016/03/01/64-bit-arm-aarch64-instructions-boost-performance-by-15-to-30-compared-to-32-bit-arm-aarch32-instructions/ https://www.cnx-software.com/2016/03/01/64-bit-arm-aarch64-i... If the difference between AArch64 and AArch32 were simply larger pointers, I would probably agree, but AArch64 includes a number of instructions that can do more work, so 64-bit programs can actually run faster than their 32-bit equivalents, even if you're not using much RAM.
- als0 6y ago
  AArch64 also has a larger number of registers, which can make a big difference for workloads.
deleted 6y ago
[deleted]
my123 6y ago
Per watt, it's not even close. That's because RPis are a lot behind on manufacturing processes, with newer designs being more efficient too. (RPi4 is on 28nm)
kev009 6y ago
Apple has had the performance per watt crown for quite some time. These chips are not very good for anything outside of prototyping, otherwise hyperscalers would run a sea of small nodes.
genpfault 6y ago

rhatr 6y ago

One of the guys behind this hack here. The other is Stefano (and most credit goes to him actually). Would love for everyone to give it a try and help us improve it. I'm working on a few ready-made VMs that you can easily deploy on Raspberry Pi 4 with Project EVE (including Windows 10 VM and Plan9 VM). If you're interested -- drop me a note (I may even be able to do a few bounties/etc. for things we need to improve ASAP). Otherwise -- feel free to ask questions, etc. in this thread. It looks small -- but the whole thing took us a month or so (not full time of course) so there's quite a bit of subtlety there.

rhatr 6y ago

Btw, I just cut EVE release 5.12.3-rpi-xen-arm64 (there's also 5.12.3-rpi-kvm-arm64 of course -- but this is Xen thread ;-))) that fixes a few issues that were reported ever since we got slashdotted this morning. This is your best bet in trying Xen/EVE/RPi4 combo. E.g. docker run lfedge/eve:5.12.3-rpi-xen-arm64 live > live.raw is literally all you need to get going.