17 ms·

MacOS in QEMU – ARM edition [pdf]

222 points by watusername 3y ago

ACAVJW4H 3y ago
KVM Forum 2023 Day 2 10:15 AM https://youtu.be/hyrw4j2D6I0?t=4684 https://youtu.be/hyrw4j2D6I0?t=4684
- p0w3n3d 3y ago
  thank you, without the video, slides do not make sense
- solarkraft 3y ago
  Thank you. Slides don't really tell the whole story.
wyldfire 3y ago
Alex's presentation was among some of the cooler ones at this year's KVM Forum. HVF sounds like it has some unique features that make it nicer than KVM for some use cases.
darkclouds 3y ago
The command line strings for Qemu can be enormous and complicated because they have changed over time, whats shown on page49 is very simplified, but should get some people going.
- deaddodo 3y ago
  Well, not just that, but Qemu isn't a platform virtualization utility like Hyper-V or Virtualbox. Qemu is an emulator of general machine platforms that happens to offer virtualization APIs for performance boosts in specific configurations (x86 on x86, most specifically). This is what allows it to simultaneously pretend to be a reference PREP platform, G3 machine, G4 machine, ARM9TDMI mobile device, ARM-Cortex EFI platform, SPARC server or even some hodge-podge machine of mixed emulated components. People treat it as equivalent to VBox or VMware; but it's not, it's a different tool that happens to offer a similar overlap of use cases (like using the butt of a screwdriver as a hammer).
  - darkclouds 3y ago
    > Qemu isn't a platform virtualization utility like Hyper-V or Virtualbox I know, thats why I could run arm based OS on those twos on an x86 or x64 cpu. Theres a lot of work thats gone into Qemu to emulate the different processors, question is have they found undocumented features of cpu's when trying to emulate them?
    - deaddodo 3y ago
      Technically, a good chunk of the ARM boards that Qemu supports are completely undocumented. So yes? The emudev scene is pretty close-knit, there are shared contributors amongst MESS, MAME, Qemu, etc that spread research advances. In addition, a lot of the breakthrough work in undocumented/edge features is done by decappers, hardware researchers and computer engineers. This research then distills down into the more advanced emulator platforms before working its way into commodity documentation and hobby emulators.
mickgardner 3y ago
Worth noting that you can’t use your appleid or icloud on a vm of macos
- coldtea 3y ago
  As in technically can't, or mere license-wise "you're not supposed to"?
  - saagarjha 3y ago
    Virtualization at least does not have a functional sign-in
  - 130e13a 3y ago
    i could be completely wrong here, but i do seem to remember something about there being some physical chip in the machine that needed to be present for e.g. iMessage to work. this was a while ago but iirc they also somehow incorporated local Apple ID account management into that system.
  - amelius 3y ago
    I wouldn't be surprised if there are fuses inside the processor that tie it to your Apple ID.
- alex7o 3y ago
  I don't use them on my real Mac why would I need them on my fake (VM) one
  - ziftface 3y ago
    I think that means no iOS development so if you need that you need a real Mac
    - qalmakka 3y ago
      Well, QEMU has no graphical acceleration for macOS (no virtio drivers for XNU and Quartz) so it's basically unusable - software rendering has been historically borked on macOS, and never well tested.
john_alan 3y ago
I’m using QEMU via UTM as my primary virtualisation solution for Linux and macOS guests on macOS.
- sneak 3y ago
  What guest OS do you use for GPU accelerated Linux guests? I have not been able to get GPU accelerated full res Linux guests working on M1/M2 via UTM.
irusensei 3y ago
I'm using UTM (nice frontend to Qemu) and the performance is okay-ish on a base mbp m1. The desktop is fluid enough for basic usage. I use it mainly for remote access client apps that I don't want running on my OS, like Citrix.
- nortonham 3y ago
  what operating systems are you running on utm? I've tried it, and for some it's great, for others, usually one of the bsd's, there's always some small problem. And the gui feels kind of incomplete. Weirdly, virtualbox on linux was usually my go to for trying out other operating systems. It also felt like the most straightforward to use.
- sandyarmstrong 3y ago
  UTM is fantastic for macOS-on-macOS emulation on Apple Silicon devices. Historically, macOS VMs have been so slow on every other virtualization platform. UTM uses Apple's native virtualization framework and it's a godsend for testing older or newer OS's.
  - larusso 3y ago
    I used parallels in the past. Mainly to spin up macOS VMs for setup testing with ansible (I maintain a macOS build farm for our studio) And I found the performance decent enough. It all tanks the moment more than one instance is running though.
mikkelam 3y ago
how well does this work? Is this the new hackintosh?
- kapildev 3y ago
  It is MacOS only. So, no, this is not the new Hackintosh.
sandreas 3y ago
There is also a nice little guide on how to get macOS Ventura working on Proxmox (using QEMU), although x64 this is really working like expected: https://www.nicksherlock.com/2022/10/installing-macos-13-ventura-on-proxmox/ https://www.nicksherlock.com/2022/10/installing-macos-13-ven...
sovietmudkipz 3y ago
I recently (yesterday) ran into emulation issues when I tried to run my dedicated game server build for Linux (that I produced from Unity) inside an amazon Linux docker container on my M2 Mac. I use docker for apple silicon that supposedly allows you to run amd64 images on arm (which M1/M2 macs are). Yes even specifying the correct —-platform and upgrading the emulation software. It was only when I opened up all logging in a dev build did I discover there was some syscall from the mono side of things that wasn’t emulated properly enough. I thought it ironic recalling how many people had sold containers as lighter weight VMs that remove “works on my machine” hassles. Anyways, I just spun up an AWS EC2 instance to test this nothing part of the process. I know podman uses qemu and maybe it could run this Unity based workload and maybe I could uninstall docker but eesh the boring image tech is starting to become a bit of a hassle. I’ll probably just keep using EC2 instances or write up a vagrant file to create a local VM. I’m sure eventually my workload will be fully supported eventually so I can optionally just wait. I love and hate knowing so much.
- ksala_ 3y ago
  Isn't Docker for Mac just using VMs in the background?
  - fathyb 3y ago
    It is, and Docker for Mac uses QEMU for `--platform`, just like Podman AFAIK. The issue here is that it's QEMU user-space emulation, which does not support some syscalls, like `ioctl`. The fix here would be to spin an amd64 VM and run the binary or install Docker there, instead of relying on the more convenient - but less compatible - user-space emulation of Docker. Not very related to this very interesting presentation on virtualizing ARM macOS guests on QEMU hosts..
    - zbentley 3y ago
      > Docker for Mac uses QEMU for `--platform` Only on older setups and on the default config. Updated MacOS (13+) and updated docker provides an option to use Rosetta and the new virtualization system. That, I believe, replaces QEMU for x86 emulation with the Apple emulator. I've had substantial success with that method; not only does it seem a bit faster, but some containerized apps that very frequently failed to start (e.g. Apache Pulsar) work consistently in that mode. I suspect this difference in experience speaks to bugs in QEMU that trigger in the presence of certain application behavior (in Pulsar's case, something weird happening with the JVM on startup). QEMU's still an incredible feat of engineering and something I frequently use on Linux, though. It just seems that the Apple emulation on Macs is better (which makes sense, given that it's built by the people who made the semi-proprietary silicon it's emulating from).
fredgrott 3y ago
It would be interesting for this audience to compare QEMU against VirtualBox.. My own bias is that I will be purchasing a Mac Studio around XMas as a flutter app build machine. I know that VirtualBox has guest OS additions to pass through graphics acceleration from the Host OS to the Guest OS. Does QEMU have this for MS Windows and Linux *6 guest OSes?
- xxpor 3y ago
  The quick summary is VirtualBox vs kvm/qemu is a classic commercial product vs open source comparison. Without going too deep into the actual architecture, VirtualBox is much easier to get started with (although check out something like Proxmox), but qemu is much more powerful. AFAIK, there are no features in VirtualBox that can't also be done with kvm/qemu, but there's a lot of things that can be done with kvm/qemu that can't be done in VirtualBox. I say kvm/qemu because the actual hypervisor is kvm, while qemu is used for control and device emulation. qemu can also emulate full machines, which is obviously much slower but can be extremely useful.
- roblabla 3y ago
  As a personal VM machine, vbox is fine. But for anything more involved, it's got a lot of rough edges. We used to use virtualbox at work to run our CI workload (spawning a fresh VM for every job). Long story short: It does not scale well at all. Its scripting API is terrible, we ran into endless network problems (for instance, its builtin dhcp server would sometimes fail to flush expired IPs, leading to running out of IPs to give), and it would sometimes just freeze up entirely, requiring us to restart the virtualbox services. We switched over to kubevirt[0] (essentially qemu/kvm inside a kube cluster) to handle our CI, and it's been a much, much smoother sailing since then. And it even allowed us to run macos VMs on linux hosts, something that we never managed to get working in vbox. > Does QEMU have this for MS Windows and Linux *6 guest OSes? QEmu has virgl[1] for paravirtualized graphics, but its windows drivers are unsigned afaict, so you can't really use it there. Works fine on linux, so long as your mesa and kernel aren't super ancient.
deleted 3y ago
[deleted]
password4321 3y ago
Semi-related (MacOS x86-64): Quickemu: Quickly create and run optimised Win-10,11/macOS/Linux on Linux https://news.ycombinator.com/item?id=28797129 https://news.ycombinator.com/item?id=28797129 (2021)

222 points by watusername 3y ago

ACAVJW4H 3y ago

KVM Forum 2023 Day 2 10:15 AM https://youtu.be/hyrw4j2D6I0?t=4684 https://youtu.be/hyrw4j2D6I0?t=4684

p0w3n3d 3y ago
thank you, without the video, slides do not make sense
solarkraft 3y ago
Thank you. Slides don't really tell the whole story.

wyldfire 3y ago

Alex's presentation was among some of the cooler ones at this year's KVM Forum. HVF sounds like it has some unique features that make it nicer than KVM for some use cases.

darkclouds 3y ago

The command line strings for Qemu can be enormous and complicated because they have changed over time, whats shown on page49 is very simplified, but should get some people going.

deaddodo 3y ago
Well, not just that, but Qemu isn't a platform virtualization utility like Hyper-V or Virtualbox. Qemu is an emulator of general machine platforms that happens to offer virtualization APIs for performance boosts in specific configurations (x86 on x86, most specifically). This is what allows it to simultaneously pretend to be a reference PREP platform, G3 machine, G4 machine, ARM9TDMI mobile device, ARM-Cortex EFI platform, SPARC server or even some hodge-podge machine of mixed emulated components. People treat it as equivalent to VBox or VMware; but it's not, it's a different tool that happens to offer a similar overlap of use cases (like using the butt of a screwdriver as a hammer).
- darkclouds 3y ago
  > Qemu isn't a platform virtualization utility like Hyper-V or Virtualbox I know, thats why I could run arm based OS on those twos on an x86 or x64 cpu. Theres a lot of work thats gone into Qemu to emulate the different processors, question is have they found undocumented features of cpu's when trying to emulate them?
  - deaddodo 3y ago
    Technically, a good chunk of the ARM boards that Qemu supports are completely undocumented. So yes? The emudev scene is pretty close-knit, there are shared contributors amongst MESS, MAME, Qemu, etc that spread research advances. In addition, a lot of the breakthrough work in undocumented/edge features is done by decappers, hardware researchers and computer engineers. This research then distills down into the more advanced emulator platforms before working its way into commodity documentation and hobby emulators.

mickgardner 3y ago

Worth noting that you can’t use your appleid or icloud on a vm of macos

coldtea 3y ago
As in technically can't, or mere license-wise "you're not supposed to"?
- saagarjha 3y ago
  Virtualization at least does not have a functional sign-in
- 130e13a 3y ago
  i could be completely wrong here, but i do seem to remember something about there being some physical chip in the machine that needed to be present for e.g. iMessage to work. this was a while ago but iirc they also somehow incorporated local Apple ID account management into that system.
- amelius 3y ago
  I wouldn't be surprised if there are fuses inside the processor that tie it to your Apple ID.
alex7o 3y ago
I don't use them on my real Mac why would I need them on my fake (VM) one
- ziftface 3y ago
  I think that means no iOS development so if you need that you need a real Mac
  - qalmakka 3y ago
    Well, QEMU has no graphical acceleration for macOS (no virtio drivers for XNU and Quartz) so it's basically unusable - software rendering has been historically borked on macOS, and never well tested.

john_alan 3y ago

I’m using QEMU via UTM as my primary virtualisation solution for Linux and macOS guests on macOS.

sneak 3y ago
What guest OS do you use for GPU accelerated Linux guests? I have not been able to get GPU accelerated full res Linux guests working on M1/M2 via UTM.

irusensei 3y ago

I'm using UTM (nice frontend to Qemu) and the performance is okay-ish on a base mbp m1. The desktop is fluid enough for basic usage. I use it mainly for remote access client apps that I don't want running on my OS, like Citrix.

nortonham 3y ago
what operating systems are you running on utm? I've tried it, and for some it's great, for others, usually one of the bsd's, there's always some small problem. And the gui feels kind of incomplete. Weirdly, virtualbox on linux was usually my go to for trying out other operating systems. It also felt like the most straightforward to use.
sandyarmstrong 3y ago
UTM is fantastic for macOS-on-macOS emulation on Apple Silicon devices. Historically, macOS VMs have been so slow on every other virtualization platform. UTM uses Apple's native virtualization framework and it's a godsend for testing older or newer OS's.
- larusso 3y ago
  I used parallels in the past. Mainly to spin up macOS VMs for setup testing with ansible (I maintain a macOS build farm for our studio) And I found the performance decent enough. It all tanks the moment more than one instance is running though.

mikkelam 3y ago

how well does this work? Is this the new hackintosh?

kapildev 3y ago
It is MacOS only. So, no, this is not the new Hackintosh.

sandreas 3y ago

There is also a nice little guide on how to get macOS Ventura working on Proxmox (using QEMU), although x64 this is really working like expected: https://www.nicksherlock.com/2022/10/installing-macos-13-ventura-on-proxmox/ https://www.nicksherlock.com/2022/10/installing-macos-13-ven...

sovietmudkipz 3y ago

I recently (yesterday) ran into emulation issues when I tried to run my dedicated game server build for Linux (that I produced from Unity) inside an amazon Linux docker container on my M2 Mac. I use docker for apple silicon that supposedly allows you to run amd64 images on arm (which M1/M2 macs are). Yes even specifying the correct —-platform and upgrading the emulation software. It was only when I opened up all logging in a dev build did I discover there was some syscall from the mono side of things that wasn’t emulated properly enough. I thought it ironic recalling how many people had sold containers as lighter weight VMs that remove “works on my machine” hassles. Anyways, I just spun up an AWS EC2 instance to test this nothing part of the process. I know podman uses qemu and maybe it could run this Unity based workload and maybe I could uninstall docker but eesh the boring image tech is starting to become a bit of a hassle. I’ll probably just keep using EC2 instances or write up a vagrant file to create a local VM. I’m sure eventually my workload will be fully supported eventually so I can optionally just wait. I love and hate knowing so much.

ksala_ 3y ago
Isn't Docker for Mac just using VMs in the background?
- fathyb 3y ago
  It is, and Docker for Mac uses QEMU for `--platform`, just like Podman AFAIK. The issue here is that it's QEMU user-space emulation, which does not support some syscalls, like `ioctl`. The fix here would be to spin an amd64 VM and run the binary or install Docker there, instead of relying on the more convenient - but less compatible - user-space emulation of Docker. Not very related to this very interesting presentation on virtualizing ARM macOS guests on QEMU hosts..
  - zbentley 3y ago
    > Docker for Mac uses QEMU for `--platform` Only on older setups and on the default config. Updated MacOS (13+) and updated docker provides an option to use Rosetta and the new virtualization system. That, I believe, replaces QEMU for x86 emulation with the Apple emulator. I've had substantial success with that method; not only does it seem a bit faster, but some containerized apps that very frequently failed to start (e.g. Apache Pulsar) work consistently in that mode. I suspect this difference in experience speaks to bugs in QEMU that trigger in the presence of certain application behavior (in Pulsar's case, something weird happening with the JVM on startup). QEMU's still an incredible feat of engineering and something I frequently use on Linux, though. It just seems that the Apple emulation on Macs is better (which makes sense, given that it's built by the people who made the semi-proprietary silicon it's emulating from).

fredgrott 3y ago

It would be interesting for this audience to compare QEMU against VirtualBox.. My own bias is that I will be purchasing a Mac Studio around XMas as a flutter app build machine. I know that VirtualBox has guest OS additions to pass through graphics acceleration from the Host OS to the Guest OS. Does QEMU have this for MS Windows and Linux *6 guest OSes?

xxpor 3y ago
The quick summary is VirtualBox vs kvm/qemu is a classic commercial product vs open source comparison. Without going too deep into the actual architecture, VirtualBox is much easier to get started with (although check out something like Proxmox), but qemu is much more powerful. AFAIK, there are no features in VirtualBox that can't also be done with kvm/qemu, but there's a lot of things that can be done with kvm/qemu that can't be done in VirtualBox. I say kvm/qemu because the actual hypervisor is kvm, while qemu is used for control and device emulation. qemu can also emulate full machines, which is obviously much slower but can be extremely useful.
roblabla 3y ago
As a personal VM machine, vbox is fine. But for anything more involved, it's got a lot of rough edges. We used to use virtualbox at work to run our CI workload (spawning a fresh VM for every job). Long story short: It does not scale well at all. Its scripting API is terrible, we ran into endless network problems (for instance, its builtin dhcp server would sometimes fail to flush expired IPs, leading to running out of IPs to give), and it would sometimes just freeze up entirely, requiring us to restart the virtualbox services. We switched over to kubevirt[0] (essentially qemu/kvm inside a kube cluster) to handle our CI, and it's been a much, much smoother sailing since then. And it even allowed us to run macos VMs on linux hosts, something that we never managed to get working in vbox. > Does QEMU have this for MS Windows and Linux *6 guest OSes? QEmu has virgl[1] for paravirtualized graphics, but its windows drivers are unsigned afaict, so you can't really use it there. Works fine on linux, so long as your mesa and kernel aren't super ancient.

deleted 3y ago

[deleted]

password4321 3y ago

Semi-related (MacOS x86-64): Quickemu: Quickly create and run optimised Win-10,11/macOS/Linux on Linux https://news.ycombinator.com/item?id=28797129 https://news.ycombinator.com/item?id=28797129 (2021)