10 ms·

Virtual machines with KVM on Pixel 6 and Android 13 DP1

72 points by farmerbb 5y ago

aftbit 5y ago
How do I follow along?
- neogodless 5y ago
  Hmm Thread Reader gives me this (two short paragraphs and videos that are hidden, probably due to my Firefox containers.) https://threadreaderapp.com/thread/1492712401262710784.html https://threadreaderapp.com/thread/1492712401262710784.html Looks like the meat of this is just 4 photos, 2 videos, 3 tweets. Unless you want to dig through the rest of the conversation, in which case you could click through to the Twitter thread linked originally.
floatboth 5y ago
This is amazing news! I've been wanting this since the Nexus 5X where I built a kernel with KVM and was disappointed to see the "kvm failed to become hypervisor" message in dmesg.
arketyp 5y ago
I bought the Pixel 6 looking for some approximation of openness, so this makes me happy to see. Does anyone know what significance the specific hardware (Google Tensor) has in this?
- kdrag0n 5y ago
  In the Android 13 bootloader and firmware for Tensor, Google added the ability to expose the EL2 hypervisor privilege level to the kernel for their protected KVM feature (pKVM). Protected VMs seem to be an optional feature, however, so full KVM access is available as far as I can tell. On Qualcomm SoCs, EL2 is occupied by their HypX hypervisor, so KVM is impossible without exploiting HypX. I don't think OEMs are allowed to do much about it, and it seems to be a somewhat integral part of their platform, so removing it would likely be a big undertaking even if OEMs were allowed to do so. On Exynos SoCs, the kernel boots in EL1, but Samsung has a security feature called RKP that involves loading code in EL2. This means that it's theoretically possible to hack it up for enabling KVM, but KVM doesn't just work out-of-the-box. I'm not sure about other SoC vendors, but the most popular one by far (Qualcomm) effectively makes KVM impossible.
  - kbuck 5y ago
    For more background on the technical details, I found this presentation describing how KVM works with Arm EL2: https://events.static.linuxfound.org/sites/events/files/slides/To%20EL2%20and%20Beyond.pdf https://events.static.linuxfound.org/sites/events/files/slid... (pdf)
  - arketyp 5y ago
    Thank you for the elaborate answer, and for your admirable work.
  - my123 5y ago
    A note: > On Qualcomm SoCs, EL2 is occupied by their HypX hypervisor, so KVM is impossible without exploiting HypX. I don't think OEMs are allowed to do much about it, and it seems to be a somewhat integral part of their platform, so removing it would likely be a big undertaking even if OEMs were allowed to do so. For Chrome OS firmware stack, full EL2 is given to the kernel, the regular way. For the Windows on Arm64 stack (on currently shipped SoCs), a mechanism, Secure Launch, is provided to escalate from EL1 to EL2. bootmgfw issues a SMC call, the function is the same as the one used to initialise Intel ACM or AMD SKINIT. QHEE intercepts it, does some sanity and integrity checks then remaps memory to load the TCB launcher and jumps to the entry point. This means that if you run Linux on those, you don't have EL2.
    - londons_explore 5y ago
      It's a real shame the number of privilege levels is fixed. This seems anti-turing-completeness. Instead, any VM or hypervisor should be able to emulate an environment that is indistinguishable from the environment it itself is running in. That in turn means that hypervisors must be infinitely nestable, and there must be no way for software to know that is running in an emulated environment (unless the hypervisor above wants to reveal this info). I get that 'infinitely nestable' is hard to implement in hardware, and it's much easier to design things with a fixed nesting depth/number of privilege levels, but I really don't think it would have been much of a stretch to design the instruction set with traps in the right place to allow software to implement infinite nestability without too much of a performance hit.
sto_hristo 5y ago
It's ridiculous that you can run full blown OSes in your device inside a VM, but you can't do it on metal, because you're device is locked so much.
- DCKing 5y ago
  On many Android devices, especially Pixels, devices being 'locked' is not the problem at all. It is that historically Android was built to fundamentally different from mainstream Linux, and it is hard to make mainstream Linux work with your phone's hardware [1]. And the phone hardware providers are notoriously terrible about supporting anything. Since virtualization abstracts those historic incompatibilities and driver issues away, there's a lot more stability you can expect from there. That is in addition to some other legitimate changes Google is making to make Pixel phones somewhat closer to mainstream Linux. [1]: PostmarketOS shows that it's very possible to do so on plenty of devices, but just not without a lot of tradeoffs: https://postmarketos.org https://postmarketos.org
- b112 5y ago
  The downside with most phones too, is even if rooted, the radio firmware is not in control, along with a bootloader, meaning no real trust.
  - DCKing 5y ago
    There's an urban legend that the baseband processor is an all governing, closed source backdoored bad actor in your phone. This may have been true for 2005 era dumbphones, but in reality your smartphone's radio is attached like a USB device is. In phones made in at least the last 10 years, it doesn't have a privileged position at all. I don't know why that legend keeps on being perpetuated. There's a lot wrong with hardware openness on mobile devices, but closed bootloaders and radio firmwares are also true on most Linux laptops that can thrive nonetheless.
    - b112 5y ago
      The radio firmware can be remotely updated, without the OS being involved. EG, by carrier. It has direct access, without OS involvement, to many peripherals. At that stage, any further discussion (being able to read system ram or not, segregation or not) is moot, for, it's the ultimate in lack of trust.
csdvrx 5y ago
How good is Windows 11 on the Pixel 6? Is there any slowdown from using a KVM? Can you export USB devices say for the bluetooth adapter or a keyboard? It may sound weird, but I love Windows, and the opportunity to run Win11 on better hardware than a Lumia 950 is very tempting to me!!
- opencl 5y ago
  There have also been a few projects to boot the ARM version of Windows on various Android phones, no VM involved. None of them really have good enough driver support to use as a daily driver though. For example this one[1] for the Oneplus 6/6T. [1]https://forum.xda-developers.com/t/windows-10-arm-on-oneplus-6-t.4287683/ https://forum.xda-developers.com/t/windows-10-arm-on-oneplus...
udbhavs 5y ago
Does W11 in a VM mean you could effectively dock your phone into a monitor and use it as a proper PC with shared filesystem access to all your Android stuff?
- kbuck 5y ago
  It's in a VM, so it will only have (easy) access to its own virtual disk, not anything from Android. Check out Samsung's "DeX" feature[1], though -- it's pretty much exactly what you're describing. [1]: https://www.samsung.com/us/explore/dex/ https://www.samsung.com/us/explore/dex/
  - sipos 5y ago
    Not sure about this implementation, but most virtualization implementations, including most KVM/libvirtd/QEMU ones, support shared directories.
- settrans 5y ago
  Sadly not, as all Pixels have DisplayPort / HDMI out turned off in software. Google expects you to use Chromecast or buy an expensive DisplayLink adapter.
  - londons_explore 5y ago
    Have they made Chromecast performance decent yet? Every time I've used it, there is so much lag that it's unusable for a 2nd screen. Even for non-interactive screen mirroring like letting someone else watch you browse the web or play a game it's jerky and laggy.
- diffeomorphism 5y ago
  We already had that with maru OS, then again with Samsung Dex, with Ubuntu Touch, ... . Seems like nobody actually wants that.
  - nobodyshere 5y ago
Tijdreiziger 5y ago
> There's actually a permission for apps to run a native service in a Microdroid VM (system-managed crosvm + Binder vsock), though I haven't gotten it working on DP1. Direct KVM access requires root though. https://twitter.com/kdrag0n/status/1492790737192177668 https://twitter.com/kdrag0n/status/1492790737192177668

72 points by farmerbb 5y ago

aftbit 5y ago

How do I follow along?

neogodless 5y ago
Hmm Thread Reader gives me this (two short paragraphs and videos that are hidden, probably due to my Firefox containers.) https://threadreaderapp.com/thread/1492712401262710784.html https://threadreaderapp.com/thread/1492712401262710784.html Looks like the meat of this is just 4 photos, 2 videos, 3 tweets. Unless you want to dig through the rest of the conversation, in which case you could click through to the Twitter thread linked originally.

floatboth 5y ago

This is amazing news! I've been wanting this since the Nexus 5X where I built a kernel with KVM and was disappointed to see the "kvm failed to become hypervisor" message in dmesg.

arketyp 5y ago

I bought the Pixel 6 looking for some approximation of openness, so this makes me happy to see. Does anyone know what significance the specific hardware (Google Tensor) has in this?

kdrag0n 5y ago
In the Android 13 bootloader and firmware for Tensor, Google added the ability to expose the EL2 hypervisor privilege level to the kernel for their protected KVM feature (pKVM). Protected VMs seem to be an optional feature, however, so full KVM access is available as far as I can tell. On Qualcomm SoCs, EL2 is occupied by their HypX hypervisor, so KVM is impossible without exploiting HypX. I don't think OEMs are allowed to do much about it, and it seems to be a somewhat integral part of their platform, so removing it would likely be a big undertaking even if OEMs were allowed to do so. On Exynos SoCs, the kernel boots in EL1, but Samsung has a security feature called RKP that involves loading code in EL2. This means that it's theoretically possible to hack it up for enabling KVM, but KVM doesn't just work out-of-the-box. I'm not sure about other SoC vendors, but the most popular one by far (Qualcomm) effectively makes KVM impossible.
- kbuck 5y ago
  For more background on the technical details, I found this presentation describing how KVM works with Arm EL2: https://events.static.linuxfound.org/sites/events/files/slides/To%20EL2%20and%20Beyond.pdf https://events.static.linuxfound.org/sites/events/files/slid... (pdf)
- arketyp 5y ago
  Thank you for the elaborate answer, and for your admirable work.
- my123 5y ago
  A note: > On Qualcomm SoCs, EL2 is occupied by their HypX hypervisor, so KVM is impossible without exploiting HypX. I don't think OEMs are allowed to do much about it, and it seems to be a somewhat integral part of their platform, so removing it would likely be a big undertaking even if OEMs were allowed to do so. For Chrome OS firmware stack, full EL2 is given to the kernel, the regular way. For the Windows on Arm64 stack (on currently shipped SoCs), a mechanism, Secure Launch, is provided to escalate from EL1 to EL2. bootmgfw issues a SMC call, the function is the same as the one used to initialise Intel ACM or AMD SKINIT. QHEE intercepts it, does some sanity and integrity checks then remaps memory to load the TCB launcher and jumps to the entry point. This means that if you run Linux on those, you don't have EL2.
  - londons_explore 5y ago
    It's a real shame the number of privilege levels is fixed. This seems anti-turing-completeness. Instead, any VM or hypervisor should be able to emulate an environment that is indistinguishable from the environment it itself is running in. That in turn means that hypervisors must be infinitely nestable, and there must be no way for software to know that is running in an emulated environment (unless the hypervisor above wants to reveal this info). I get that 'infinitely nestable' is hard to implement in hardware, and it's much easier to design things with a fixed nesting depth/number of privilege levels, but I really don't think it would have been much of a stretch to design the instruction set with traps in the right place to allow software to implement infinite nestability without too much of a performance hit.

sto_hristo 5y ago

It's ridiculous that you can run full blown OSes in your device inside a VM, but you can't do it on metal, because you're device is locked so much.

DCKing 5y ago
On many Android devices, especially Pixels, devices being 'locked' is not the problem at all. It is that historically Android was built to fundamentally different from mainstream Linux, and it is hard to make mainstream Linux work with your phone's hardware [1]. And the phone hardware providers are notoriously terrible about supporting anything. Since virtualization abstracts those historic incompatibilities and driver issues away, there's a lot more stability you can expect from there. That is in addition to some other legitimate changes Google is making to make Pixel phones somewhat closer to mainstream Linux. [1]: PostmarketOS shows that it's very possible to do so on plenty of devices, but just not without a lot of tradeoffs: https://postmarketos.org https://postmarketos.org
b112 5y ago
The downside with most phones too, is even if rooted, the radio firmware is not in control, along with a bootloader, meaning no real trust.
- DCKing 5y ago
  There's an urban legend that the baseband processor is an all governing, closed source backdoored bad actor in your phone. This may have been true for 2005 era dumbphones, but in reality your smartphone's radio is attached like a USB device is. In phones made in at least the last 10 years, it doesn't have a privileged position at all. I don't know why that legend keeps on being perpetuated. There's a lot wrong with hardware openness on mobile devices, but closed bootloaders and radio firmwares are also true on most Linux laptops that can thrive nonetheless.
  - b112 5y ago
    The radio firmware can be remotely updated, without the OS being involved. EG, by carrier. It has direct access, without OS involvement, to many peripherals. At that stage, any further discussion (being able to read system ram or not, segregation or not) is moot, for, it's the ultimate in lack of trust.

csdvrx 5y ago

How good is Windows 11 on the Pixel 6? Is there any slowdown from using a KVM? Can you export USB devices say for the bluetooth adapter or a keyboard? It may sound weird, but I love Windows, and the opportunity to run Win11 on better hardware than a Lumia 950 is very tempting to me!!

opencl 5y ago
There have also been a few projects to boot the ARM version of Windows on various Android phones, no VM involved. None of them really have good enough driver support to use as a daily driver though. For example this one[1] for the Oneplus 6/6T. [1]https://forum.xda-developers.com/t/windows-10-arm-on-oneplus-6-t.4287683/ https://forum.xda-developers.com/t/windows-10-arm-on-oneplus...

udbhavs 5y ago

Does W11 in a VM mean you could effectively dock your phone into a monitor and use it as a proper PC with shared filesystem access to all your Android stuff?

kbuck 5y ago
It's in a VM, so it will only have (easy) access to its own virtual disk, not anything from Android. Check out Samsung's "DeX" feature[1], though -- it's pretty much exactly what you're describing. [1]: https://www.samsung.com/us/explore/dex/ https://www.samsung.com/us/explore/dex/
- sipos 5y ago
  Not sure about this implementation, but most virtualization implementations, including most KVM/libvirtd/QEMU ones, support shared directories.
settrans 5y ago
Sadly not, as all Pixels have DisplayPort / HDMI out turned off in software. Google expects you to use Chromecast or buy an expensive DisplayLink adapter.
- londons_explore 5y ago
  Have they made Chromecast performance decent yet? Every time I've used it, there is so much lag that it's unusable for a 2nd screen. Even for non-interactive screen mirroring like letting someone else watch you browse the web or play a game it's jerky and laggy.
diffeomorphism 5y ago
We already had that with maru OS, then again with Samsung Dex, with Ubuntu Touch, ... . Seems like nobody actually wants that.
- nobodyshere 5y ago

Tijdreiziger 5y ago

> There's actually a permission for apps to run a native service in a Microdroid VM (system-managed crosvm + Binder vsock), though I haven't gotten it working on DP1. Direct KVM access requires root though. https://twitter.com/kdrag0n/status/1492790737192177668 https://twitter.com/kdrag0n/status/1492790737192177668