11 ms·

Apple Virtualization Framework

320 points by serhack_ 3y ago

alphanullmeric 3y ago
How similar to WSL2 is this?
- vbezhenar 3y ago
  WSL2 is a complete solution. Virtualization framework is just a library for developers. One can build something like WSL2 on top of this library.
serhack_ 3y ago
I'm posting this because I've just found it. I'm trying to boot a custom toy kernel that I'm developing (with virtio support). Wish me good luck! OSDEV rocks!
- serhack_ 3y ago
  Just a question: would anyone be interested in a good blog post explaining my journey with OSDEV and Virtualization.framework ?
  - mike_hearn 3y ago
    Sure, the collective internet programming community can always use good blog posts, especially on relatively obscure topics!
  - mactunes 3y ago
    Absolutely, that would be great!
  - aa-jv 3y ago
    Don't ask, do. This is a fascinating subject. I'd be quite happy to see how a fat MacOS machine is being used as a build-server for multiple generations of MacOS versions, as well as how it could be used (via Linux VM's) to also build for Android, Linux, Windows, etc. Essentially, this subject is all about the build server destination; there is an award for how many different targets you can get from the same code-base, and being able to load up a fat Mac that approaches the nexus, is certainly of interest to us developers. VM's are key tools in modern build environments. Having a fat Mac running multiple VM's to solve the target/build infrastructure problem is fascinating on all fronts..
    - sroussey 3y ago
      Would be great to run GitHub actions locally that use whichever VM is required.
  - rtpg 3y ago
    Most definitely
  - andyjohnson0 3y ago
    Yes!
  - serhack_ 3y ago
    yay! Understood
- sweetjuly 3y ago
  Fair warning, you might be better suited by QEMU since debug support in Virtualization.framework is not great. QEMU has support for debug stubs out of the box. QEMU does support Hypervisor.framework so your performance should still be good through it, assuming you're using a matching architecture as your host (i.e. don't build an x64 kernel for your ARM Mac)
_joel 3y ago
Has there been a significant development? Been using it with https://mac.getutm.app/ https://mac.getutm.app/ for a while
- aa-jv 3y ago
  At first I thought this was being done to pump us up for new Mac Pro hardware that might be on the horizon, but then I wasn't sure, but now I'm also still not sure it isn't .. ;) But, regardless, the subject of using a Mac to host a plethora of different platforms (in my case, for build-server duties) is indeed a fascinating subject. If I can get my target/builds set up on the next Mac Pro, whatever that is going to be, I'll be quite happy to do so .. especially if it isn't just doing MacOS/iOS builds, but whatever else my framework of choice (JUCE/Tracktion) supports (Android, Linux .. Windows .. etc.)
  - detourdog 3y ago
    I think this is being done because the Mach microkernel was designed to run multiple OS's at the same time. I think Apple finally has enough developers to start pushing into new areas such as a framework to advantage Mach.
    - saagarjha 3y ago
      Virtualization is, of course, based on a very standard KVM-like API. There's not much Mach in its design at all.
      - detourdog 3y ago
        Mach is down below. This is what Mach is for the BSD that was originally part of the DARPA deliverable was just to prove that Mach could run on OS.
        saagarjha 3y ago
        There's not much down below it. It calls into the Hypervisor APIs, which directly touch platform virtualization features.
deleted 3y ago
[deleted]
daneel_w 3y ago
I suppose this is something else than the Hypervisor framework (https://developer.apple.com/documentation/hypervisor https://developer.apple.com/documentation/hypervisor) - its successor or a convenient complement? Does anyone know if there's a convenient command line utility for wrangling VMs built on this framework?
- sigmaris 3y ago
  This is higher-level than the Hypervisor framework; this Virtualization framework providers an entire VM with virtio peripherals including a display. https://github.com/lima-vm/lima https://github.com/lima-vm/lima can use Virtualization framework for creating VMs, there is also https://github.com/gyf304/vmcli https://github.com/gyf304/vmcli as a very simple CLI utility for running VMs, though it's not very actively maintained.
antrod 3y ago
Please God let them use this to virtualize macOS inside iPadOS. Not as a daily driver but for the occasional use of a full desktop OS (when plugged into a keyboard/mouse). Then I could stop traveling with two devices once and for all.
- matthew-wegner 3y ago
  I think more likely is that xrOS will use this to virtualize both iPadOS and macOS. Initial rumors are that it can run iPad apps out of the box...
- bwilliams 3y ago
  If we can’t have native macOS on the iPad this would be the next best thing. Imagine having Linux or macOS on the iPad that is treated like any other app (sans memory limits and etc.). You could have a full blown desktop at the ready when iPadOS isn’t up to the task. The iPad could finally utilize the m1 processor to its fullest.
  - moondev 3y ago
    FWIW this is how it works on ChromeOS today. From the Linux terminal app (crostini lxc container) qemu launches macOS, Windows and Linux vms. They even support nested virtualization on my Framework Chromebook - so vms can run inside those vms. I agree if Apple implemented virtualization on iPad/iPhone this same way it would be a huge for unlocking their full usefulness and capabilities while still maintaining host isolation/sandbox.
    - pjmlp 3y ago
      The irony of a monolithic kernel to use containers and VMs inside VMs all over the place.
_s 3y ago
https://developer.apple.com/videos/play/wwdc2022/10002/ https://developer.apple.com/videos/play/wwdc2022/10002/ ^^ For the curious ones :) More or less it's an abstraction on top of Hypervisor to make VMs much simpler in Swift. From what I remember, Docker tried to use this but suffered greatly on disk performance.
infosecb 3y ago
VirtualBuddy is a great open-source project that takes advantage of this framework: https://github.com/insidegui/VirtualBuddy https://github.com/insidegui/VirtualBuddy
- sea-gold 3y ago
  Also: https://github.com/abiosoft/colima https://github.com/abiosoft/colima https://github.com/rancher-sandbox/rancher-desktop https://github.com/rancher-sandbox/rancher-desktop
- jbergstroem 3y ago
  Orbstack is a relatively new player that offers both VMs and containerization on top of the virtualization framework: https://orbstack.dev https://orbstack.dev
  - psygn89 3y ago
    Side note: It's free for now but will be a paid service later.
Kalanos 3y ago
Where can we obtain a "macOS restore image" ? I imagine they are not as easy to come by as an Ubuntu ISO.
- lesquivemeau 3y ago
  It used to come on DVD's, now you can create one from within mac os
  - ChrisMarshallNY 3y ago
    Yeaahhh ... the secure enclave seems to have pooched the idea of a USB stick booter. Restore Mode works fine. I used to use CCC (Carbon Copy Cloner) to make a mirror, but it doesn't like to boot from the mirror (it does, but I can't use any of the secure stuff). Also CCC destabilizes my system. If I have it running all the time, I crash -a lot-. Since I yanked it, I haven't had a hard system crash in months. These days, I just make sure the Time Machine backups keep going, and I'll use CCC on a one-shot basis to make occasional disk clones (makes for a much faster restore).
    - microtonal 3y ago
      Yeaahhh ... the secure enclave seems to have pooched the idea of a USB stick booter. Restore Mode works fine. Maybe I misunderstood you, but you can make a bootable USB stick from the installer app: https://support.apple.com/en-us/HT201372 https://support.apple.com/en-us/HT201372
      - ChrisMarshallNY 3y ago
        Nah. That's not the issue. The issue is cloning my system, so I can basically have a portable workstation on a USB stick. It's been a while, since that was possible. The M-series just basically put a fork in it. That, and CCC kept crashing the machine.
        cprecioso 3y ago
xrd 3y ago
I tried a bunch of different options and landed on multipass. I'm curious whether someone has found a good option for running x86 images on m1 chips. I'm generally happy with the multipass VMs I'm creating but they are arm architecture so then if i use docker images inside them I'm limited to arm images. Maybe I just haven't explored how to run x86 correctly within multipass? I'm finding that usually arm images are a step behind the x86 images if they are official, or they are built by someone outside the organization, so I'm never sure I'm using a safe image. Do any of the options here make it performant to run x86 on apple silicon? Some interesting comments on Rosetta that I don't fully understand.
- joshcsimmons 3y ago
  I've had the best experience using Parallels. Not sure what exactly goes on under the hood but x86 performance on my M1 Mac is snappier than what I've experienced in VirtualBox (have never tried Multipass). Regardless, performance on my beefy M1 Max MacBook Pro is pitiful compared to running natively on an x86 system.
  - xrd 3y ago
    You pay for parallels, right? I always try to find something that's fully open so I'm able to control my destiny fully. That's a bit ironic trying to run on apple hardware, but if the entire stack above the bottom layer is open then I can easily switch hardware of course.
    - greggsy 3y ago
      That’s noble, but open source doesn’t always deliver the same benefits of COTS products, such as stability and support. That trumps ideology for many people and businesses.
      - xrd 3y ago
        Definitely true. But I also feel like open source stuff is more honest about the capabilities. That's important to me and often trumps the marketing effort by a commercial effort.
villgax 3y ago
Is there a headless mode to skip the BS when doing macOS VM or is something they think should only run on Apple Silicon alone?
abdellah123 3y ago
Can this run FreeBSD?
RobotToaster 3y ago
Haven't apple been spending a lot of money on lawyers trying to stop people virtualising OSX?
- r618 3y ago
  on non Apple hardware
  - MBCook 3y ago
    Right. I believe the idea is to virtualize macOS inside macOS. I think some developers use that to test on older OS releases.
jsd1982 3y ago
Been using vftool to host docker instead of Docker.app. It's been alright so far, got rosetta set up inside to run x86 images. All it takes is a little startup script and some up front work to prepare a disk image with Ubuntu server. Every once in a while, the guest Ubuntu kernel will oops about smp related things. Every time it boots its clock is reset almost a month in the past and I have to force sntp to correct it. After sleeping the host, the guest clock will not advance so it'll be behind several hours. I'm not sure where to assign these issues: Ubuntu kernel? virtualization framework? vftool? bad configuration?
- moondev 3y ago
  Does vftool use https://developer.apple.com/documentation/virtualization https://developer.apple.com/documentation/virtualization or https://developer.apple.com/documentation/hypervisor https://developer.apple.com/documentation/hypervisor ? https://github.com/evansm7/vftool https://github.com/evansm7/vftool appears to indicate the former, but I thought the latter was required for rosetta so interested to try this. EDIT - looks like I had this backwards, vftool indeed supports https://developer.apple.com/documentation/virtualization https://developer.apple.com/documentation/virtualization which is the newer framework that supports Rosetta. The older framework is https://developer.apple.com/documentation/hypervisor https://developer.apple.com/documentation/hypervisor
  - jsd1982 3y ago
    Right, it uses Virtualization Framework. I forked vftool [0] and added direct Rosetta support which was trivial to do according to the Apple documentation; just had to convert Swift example code to Obj-C. Have had a PR open for several months with no attention from the author. I almost forgot about it until seeing this HN headline. [0] https://github.com/JamesDunne/vftool-rosetta https://github.com/JamesDunne/vftool-rosetta The Rosetta support is surprisingly trivial to use. The virtualization framework exposes a mountable volume containing a single `rosetta` executable. In the Linux guest, you just register binfmt_misc support that recognizes an x86 ELF image and point it at the `rosetta` binary. It works for docker too so you can run arm64 and amd64 images.
timsutton 3y ago
Shameless plug for a couple blog posts I have written as explorations of the Virtualization Framework: - Introduction when Monterey was released: https://macops.ca/macos-monterey-apple-silicon-vms/ https://macops.ca/macos-monterey-apple-silicon-vms/ - New features added in Ventura: https://macops.ca/virtualization-updates-2022-06/ https://macops.ca/virtualization-updates-2022-06/
- john_alan 3y ago
  Cool! What’s the difference between Virtualisation.framework and Hypervisor.framework?
  - sweetjuly 3y ago
    Hypervisor.framework provides only the bare necessities: the ability to manage vCPUs, stage 2 page tables, and handle hypervisor exits. It does not do any sort of virtualization except for the CPU itself (which, on ARMv8, is very little due to hardware providing good primitives already). Virtualization.framework is full operating system virtualization suitable for running modern Linux and macOS guests.
  - vbezhenar 3y ago
    AFAIK Virtualization.framework is built on top of Hypervisor.framework. Former is high-level one allowing you to create Linux VM with few hundreds lines of code. Latter allows you to run your whatever you want with few tens of thousands lines of code, but absolute freedom when it comes to drivers and every low level detail.
    - john_alan 3y ago
      Very cool thanks!
betimsl 3y ago
This looks good.
SergeAx 3y ago
This is not news, right? And there was not any significant updates recently? So it is upvoted just because, right? This is not a rhetorical question, I am trying to understand if I missed something important.
- vbezhenar 3y ago
  I quickly glanced over docs and didn't find anything new since 2022. So it seems that nothing new indeed. Hopefully in a few weeks something new will appear.
olliej 3y ago
Is there something new here? I scanned through it briefly and couldn't see anything
eslaught 3y ago
It's kind of wild that Apple provides code samples you can use to directly create your own VMs. Has ended up being the most straightforward way to run virtualized macOS on a macOS host that I've found, and comes with the nice benefit that your can read the code yourself and fiddle with the knobs. https://developer.apple.com/documentation/virtualization/running_macos_in_a_virtual_machine_on_apple_silicon_macs https://developer.apple.com/documentation/virtualization/run...
- w10-1 3y ago
  I can't upvote this enough: just take the sample code and build on it. I tired of evaluating and learning various wrappers and helpful virtualization applications. Being able to configure directly in a normal language is great, and it eliminates software supply chain concerns. Plus, I'm finding programs are running faster in my virtualized ubuntu than on macOS "bare metal" (still on Intel, where big memory is cheap)
  - ayewo 3y ago
    > Plus, I'm finding programs are running faster in my virtualized ubuntu than on macOS "bare metal" (still on Intel, where big memory is cheap) Can you explain more? Sounds like you are saying your Ubuntu (guest) VM reacts faster to your actions than the underlying (host) macOS?
    - kdrag0n 3y ago
      It's possible because of differences in performance/efficiency core scheduling and QoS: https://eclecticlight.co/2022/07/18/virtualisation-on-apple-silicon-macs-4-core-allocation-in-vms/ https://eclecticlight.co/2022/07/18/virtualisation-on-apple-...
Theophrastos 3y ago
Apple Virtualization Framework is a nice tool (option offered in particular by the UTM application in addition to Qemu option). And I hope it will improve with the release of the successor to Ventura and offer, for example, nested virtualization. The only concern currently, besides for virtualized macOS the lack of support for iCloud accounts, is that it contains a number of bugs which make it difficult to use as is for non-English/US configurations. Thus, there is a bug with ISO keyboards, which are recognized by Apple virtualization framework as ANSI keyboards, which shifts or even makes a certain number of keys disappear (different depending on the language: French, German, Danish, etc.).
kdrag0n 3y ago
I use Virtualization.framework for OrbStack (https://orbstack.dev https://orbstack.dev), which is a Docker Desktop and WSL alternative. It's great overall, and fairly convenient, but it has a fair share of bugs and limitations. The Virtualization service crashes with various combinations of Linux kernel version, macOS version, and architecture (ARM/Intel), so making it stable in each setup takes quite a bit of work. Device support is limited (no USB, etc.), and workarounds have other limitations. Rosetta is really buggy. The popular belief is that Virtualization.framework is inherently faster/lighter than other VMMs like QEMU, but I haven't found evidence to back it up. I spent a day prototyping a custom VMM (with an open-source base) and was able to get dynamic memory allocation working [1], as well as faster file sharing in some cases. There's a lot of other things I could do better (faster, simpler, more reliable) with a custom, tightly-integrated replacement. But unfortunately, that's not an option. Apple doesn't allow third-party VMMs to set the necessary CPU flags for Rosetta. There are too many users relying on Rosetta for fast x86 emulation for me to ditch it, despite all its bugs. So yes, I still use Virtualization.framework, but only because I have no choice. Happy to answer other questions about the framework! [1] https://twitter.com/kdrag0n/status/1645645284721184768 https://twitter.com/kdrag0n/status/1645645284721184768
- transpute 3y ago
  > Apple doesn't allow third-party VMMs to set the necessary CPU flags for Rosetta. Would that change with Asahi Linux host OS and macOS guest?
  - kdrag0n 3y ago
    I can't speak for the feasibility of running a macOS guest on Asahi, but yes, Linux could be in control of the guest's CPU flags. Same goes for the m1n1 hypervisor. I can't find the exact post but I remember marcan saying that he doesn't want to poke the bear of using Apple's custom ISA extensions on Linux. So even though it's possible, I'm not sure whether it'll actually be done.
- stefandesu 3y ago
  > OrbStack (https://orbstack.dev https://orbstack.dev), which is a Docker Desktop and WSL alternative Wait why have I never heard of this? I just tried it out and it seems to work amazingly well.
vbezhenar 3y ago
Virtualization Framework is awesome and I'm slowly building my own VirtualBox-like app, learning Cocoa programming. Hopefully will release it some day. One big thing that's missing is better storage support. Right now it's just disk image and that's about it. No way to implement something like qcow2 with snapshots and whatnot. It's missing a lot of opportunity for VMs. Another big thing that's missing is USB pass-through. Other than that: it does absolutely everything I need. It even provides some kind of hook for network which should allow to build userspace router with any network architecture (didn't try it myself, though, but I think it should work). I was able to implement thing that I missed for years on M1 mac. Basically I run Fedora VM, I installed Rosetta helper inside, and docker. And then automagically my x86 images started to work with reasonable speed. I don't know if Docker Desktop supports it already, I don't want to touch it with a longest stick in the world. But for myself I solved this issue. It was rock stable in my experience and just worked. qemu on the other hand managed to crash my entire macOS (hopefully that bug was fixed). Another thing that I liked about it: protocols are "modern" and open-source. Almost every driver is virtio. No ancient cruft. No proprietary things. Eventually other operating systems could be run on this solution.
serhack_ 3y ago
Apple has updated the code to run macOS inside macOS. Now it can support recovering from previous saved states https://developer.apple.com/documentation/virtualization/running_macos_in_a_virtual_machine_on_apple_silicon https://developer.apple.com/documentation/virtualization/run...

320 points by serhack_ 3y ago

alphanullmeric 3y ago

How similar to WSL2 is this?

vbezhenar 3y ago
WSL2 is a complete solution. Virtualization framework is just a library for developers. One can build something like WSL2 on top of this library.

serhack_ 3y ago

I'm posting this because I've just found it. I'm trying to boot a custom toy kernel that I'm developing (with virtio support). Wish me good luck! OSDEV rocks!

serhack_ 3y ago
Just a question: would anyone be interested in a good blog post explaining my journey with OSDEV and Virtualization.framework ?
- mike_hearn 3y ago
  Sure, the collective internet programming community can always use good blog posts, especially on relatively obscure topics!
- mactunes 3y ago
  Absolutely, that would be great!
- aa-jv 3y ago
  Don't ask, do. This is a fascinating subject. I'd be quite happy to see how a fat MacOS machine is being used as a build-server for multiple generations of MacOS versions, as well as how it could be used (via Linux VM's) to also build for Android, Linux, Windows, etc. Essentially, this subject is all about the build server destination; there is an award for how many different targets you can get from the same code-base, and being able to load up a fat Mac that approaches the nexus, is certainly of interest to us developers. VM's are key tools in modern build environments. Having a fat Mac running multiple VM's to solve the target/build infrastructure problem is fascinating on all fronts..
  - sroussey 3y ago
    Would be great to run GitHub actions locally that use whichever VM is required.
- rtpg 3y ago
  Most definitely
- andyjohnson0 3y ago
  Yes!
- serhack_ 3y ago
  yay! Understood
sweetjuly 3y ago
Fair warning, you might be better suited by QEMU since debug support in Virtualization.framework is not great. QEMU has support for debug stubs out of the box. QEMU does support Hypervisor.framework so your performance should still be good through it, assuming you're using a matching architecture as your host (i.e. don't build an x64 kernel for your ARM Mac)

_joel 3y ago

Has there been a significant development? Been using it with https://mac.getutm.app/ https://mac.getutm.app/ for a while

aa-jv 3y ago
At first I thought this was being done to pump us up for new Mac Pro hardware that might be on the horizon, but then I wasn't sure, but now I'm also still not sure it isn't .. ;) But, regardless, the subject of using a Mac to host a plethora of different platforms (in my case, for build-server duties) is indeed a fascinating subject. If I can get my target/builds set up on the next Mac Pro, whatever that is going to be, I'll be quite happy to do so .. especially if it isn't just doing MacOS/iOS builds, but whatever else my framework of choice (JUCE/Tracktion) supports (Android, Linux .. Windows .. etc.)
- detourdog 3y ago
  I think this is being done because the Mach microkernel was designed to run multiple OS's at the same time. I think Apple finally has enough developers to start pushing into new areas such as a framework to advantage Mach.
  - saagarjha 3y ago
    Virtualization is, of course, based on a very standard KVM-like API. There's not much Mach in its design at all.
    - detourdog 3y ago
      Mach is down below. This is what Mach is for the BSD that was originally part of the DARPA deliverable was just to prove that Mach could run on OS.
      - saagarjha 3y ago
        There's not much down below it. It calls into the Hypervisor APIs, which directly touch platform virtualization features.

deleted 3y ago

[deleted]

daneel_w 3y ago

I suppose this is something else than the Hypervisor framework (https://developer.apple.com/documentation/hypervisor https://developer.apple.com/documentation/hypervisor) - its successor or a convenient complement? Does anyone know if there's a convenient command line utility for wrangling VMs built on this framework?

sigmaris 3y ago
This is higher-level than the Hypervisor framework; this Virtualization framework providers an entire VM with virtio peripherals including a display. https://github.com/lima-vm/lima https://github.com/lima-vm/lima can use Virtualization framework for creating VMs, there is also https://github.com/gyf304/vmcli https://github.com/gyf304/vmcli as a very simple CLI utility for running VMs, though it's not very actively maintained.

antrod 3y ago

Please God let them use this to virtualize macOS inside iPadOS. Not as a daily driver but for the occasional use of a full desktop OS (when plugged into a keyboard/mouse). Then I could stop traveling with two devices once and for all.

matthew-wegner 3y ago
I think more likely is that xrOS will use this to virtualize both iPadOS and macOS. Initial rumors are that it can run iPad apps out of the box...
bwilliams 3y ago
If we can’t have native macOS on the iPad this would be the next best thing. Imagine having Linux or macOS on the iPad that is treated like any other app (sans memory limits and etc.). You could have a full blown desktop at the ready when iPadOS isn’t up to the task. The iPad could finally utilize the m1 processor to its fullest.
- moondev 3y ago
  FWIW this is how it works on ChromeOS today. From the Linux terminal app (crostini lxc container) qemu launches macOS, Windows and Linux vms. They even support nested virtualization on my Framework Chromebook - so vms can run inside those vms. I agree if Apple implemented virtualization on iPad/iPhone this same way it would be a huge for unlocking their full usefulness and capabilities while still maintaining host isolation/sandbox.
  - pjmlp 3y ago
    The irony of a monolithic kernel to use containers and VMs inside VMs all over the place.

_s 3y ago

https://developer.apple.com/videos/play/wwdc2022/10002/ https://developer.apple.com/videos/play/wwdc2022/10002/ ^^ For the curious ones :) More or less it's an abstraction on top of Hypervisor to make VMs much simpler in Swift. From what I remember, Docker tried to use this but suffered greatly on disk performance.

infosecb 3y ago

VirtualBuddy is a great open-source project that takes advantage of this framework: https://github.com/insidegui/VirtualBuddy https://github.com/insidegui/VirtualBuddy

sea-gold 3y ago
Also: https://github.com/abiosoft/colima https://github.com/abiosoft/colima https://github.com/rancher-sandbox/rancher-desktop https://github.com/rancher-sandbox/rancher-desktop
jbergstroem 3y ago
Orbstack is a relatively new player that offers both VMs and containerization on top of the virtualization framework: https://orbstack.dev https://orbstack.dev
- psygn89 3y ago
  Side note: It's free for now but will be a paid service later.

Kalanos 3y ago

Where can we obtain a "macOS restore image" ? I imagine they are not as easy to come by as an Ubuntu ISO.

lesquivemeau 3y ago
It used to come on DVD's, now you can create one from within mac os
- ChrisMarshallNY 3y ago
  Yeaahhh ... the secure enclave seems to have pooched the idea of a USB stick booter. Restore Mode works fine. I used to use CCC (Carbon Copy Cloner) to make a mirror, but it doesn't like to boot from the mirror (it does, but I can't use any of the secure stuff). Also CCC destabilizes my system. If I have it running all the time, I crash -a lot-. Since I yanked it, I haven't had a hard system crash in months. These days, I just make sure the Time Machine backups keep going, and I'll use CCC on a one-shot basis to make occasional disk clones (makes for a much faster restore).
  - microtonal 3y ago
    Yeaahhh ... the secure enclave seems to have pooched the idea of a USB stick booter. Restore Mode works fine. Maybe I misunderstood you, but you can make a bootable USB stick from the installer app: https://support.apple.com/en-us/HT201372 https://support.apple.com/en-us/HT201372
    - ChrisMarshallNY 3y ago
      Nah. That's not the issue. The issue is cloning my system, so I can basically have a portable workstation on a USB stick. It's been a while, since that was possible. The M-series just basically put a fork in it. That, and CCC kept crashing the machine.
      - cprecioso 3y ago

xrd 3y ago

I tried a bunch of different options and landed on multipass. I'm curious whether someone has found a good option for running x86 images on m1 chips. I'm generally happy with the multipass VMs I'm creating but they are arm architecture so then if i use docker images inside them I'm limited to arm images. Maybe I just haven't explored how to run x86 correctly within multipass? I'm finding that usually arm images are a step behind the x86 images if they are official, or they are built by someone outside the organization, so I'm never sure I'm using a safe image. Do any of the options here make it performant to run x86 on apple silicon? Some interesting comments on Rosetta that I don't fully understand.

joshcsimmons 3y ago
I've had the best experience using Parallels. Not sure what exactly goes on under the hood but x86 performance on my M1 Mac is snappier than what I've experienced in VirtualBox (have never tried Multipass). Regardless, performance on my beefy M1 Max MacBook Pro is pitiful compared to running natively on an x86 system.
- xrd 3y ago
  You pay for parallels, right? I always try to find something that's fully open so I'm able to control my destiny fully. That's a bit ironic trying to run on apple hardware, but if the entire stack above the bottom layer is open then I can easily switch hardware of course.
  - greggsy 3y ago
    That’s noble, but open source doesn’t always deliver the same benefits of COTS products, such as stability and support. That trumps ideology for many people and businesses.
    - xrd 3y ago
      Definitely true. But I also feel like open source stuff is more honest about the capabilities. That's important to me and often trumps the marketing effort by a commercial effort.

villgax 3y ago

Is there a headless mode to skip the BS when doing macOS VM or is something they think should only run on Apple Silicon alone?

abdellah123 3y ago

Can this run FreeBSD?

RobotToaster 3y ago

Haven't apple been spending a lot of money on lawyers trying to stop people virtualising OSX?

r618 3y ago
on non Apple hardware
- MBCook 3y ago
  Right. I believe the idea is to virtualize macOS inside macOS. I think some developers use that to test on older OS releases.

jsd1982 3y ago

Been using vftool to host docker instead of Docker.app. It's been alright so far, got rosetta set up inside to run x86 images. All it takes is a little startup script and some up front work to prepare a disk image with Ubuntu server. Every once in a while, the guest Ubuntu kernel will oops about smp related things. Every time it boots its clock is reset almost a month in the past and I have to force sntp to correct it. After sleeping the host, the guest clock will not advance so it'll be behind several hours. I'm not sure where to assign these issues: Ubuntu kernel? virtualization framework? vftool? bad configuration?

moondev 3y ago
Does vftool use https://developer.apple.com/documentation/virtualization https://developer.apple.com/documentation/virtualization or https://developer.apple.com/documentation/hypervisor https://developer.apple.com/documentation/hypervisor ? https://github.com/evansm7/vftool https://github.com/evansm7/vftool appears to indicate the former, but I thought the latter was required for rosetta so interested to try this. EDIT - looks like I had this backwards, vftool indeed supports https://developer.apple.com/documentation/virtualization https://developer.apple.com/documentation/virtualization which is the newer framework that supports Rosetta. The older framework is https://developer.apple.com/documentation/hypervisor https://developer.apple.com/documentation/hypervisor
- jsd1982 3y ago
  Right, it uses Virtualization Framework. I forked vftool [0] and added direct Rosetta support which was trivial to do according to the Apple documentation; just had to convert Swift example code to Obj-C. Have had a PR open for several months with no attention from the author. I almost forgot about it until seeing this HN headline. [0] https://github.com/JamesDunne/vftool-rosetta https://github.com/JamesDunne/vftool-rosetta The Rosetta support is surprisingly trivial to use. The virtualization framework exposes a mountable volume containing a single `rosetta` executable. In the Linux guest, you just register binfmt_misc support that recognizes an x86 ELF image and point it at the `rosetta` binary. It works for docker too so you can run arm64 and amd64 images.

timsutton 3y ago

Shameless plug for a couple blog posts I have written as explorations of the Virtualization Framework: - Introduction when Monterey was released: https://macops.ca/macos-monterey-apple-silicon-vms/ https://macops.ca/macos-monterey-apple-silicon-vms/ - New features added in Ventura: https://macops.ca/virtualization-updates-2022-06/ https://macops.ca/virtualization-updates-2022-06/

john_alan 3y ago
Cool! What’s the difference between Virtualisation.framework and Hypervisor.framework?
- sweetjuly 3y ago
  Hypervisor.framework provides only the bare necessities: the ability to manage vCPUs, stage 2 page tables, and handle hypervisor exits. It does not do any sort of virtualization except for the CPU itself (which, on ARMv8, is very little due to hardware providing good primitives already). Virtualization.framework is full operating system virtualization suitable for running modern Linux and macOS guests.
- vbezhenar 3y ago
  AFAIK Virtualization.framework is built on top of Hypervisor.framework. Former is high-level one allowing you to create Linux VM with few hundreds lines of code. Latter allows you to run your whatever you want with few tens of thousands lines of code, but absolute freedom when it comes to drivers and every low level detail.
  - john_alan 3y ago
    Very cool thanks!

betimsl 3y ago

This looks good.

SergeAx 3y ago

This is not news, right? And there was not any significant updates recently? So it is upvoted just because, right? This is not a rhetorical question, I am trying to understand if I missed something important.

vbezhenar 3y ago
I quickly glanced over docs and didn't find anything new since 2022. So it seems that nothing new indeed. Hopefully in a few weeks something new will appear.

olliej 3y ago

Is there something new here? I scanned through it briefly and couldn't see anything

eslaught 3y ago

It's kind of wild that Apple provides code samples you can use to directly create your own VMs. Has ended up being the most straightforward way to run virtualized macOS on a macOS host that I've found, and comes with the nice benefit that your can read the code yourself and fiddle with the knobs. https://developer.apple.com/documentation/virtualization/running_macos_in_a_virtual_machine_on_apple_silicon_macs https://developer.apple.com/documentation/virtualization/run...

w10-1 3y ago
I can't upvote this enough: just take the sample code and build on it. I tired of evaluating and learning various wrappers and helpful virtualization applications. Being able to configure directly in a normal language is great, and it eliminates software supply chain concerns. Plus, I'm finding programs are running faster in my virtualized ubuntu than on macOS "bare metal" (still on Intel, where big memory is cheap)
- ayewo 3y ago
  > Plus, I'm finding programs are running faster in my virtualized ubuntu than on macOS "bare metal" (still on Intel, where big memory is cheap) Can you explain more? Sounds like you are saying your Ubuntu (guest) VM reacts faster to your actions than the underlying (host) macOS?
  - kdrag0n 3y ago
    It's possible because of differences in performance/efficiency core scheduling and QoS: https://eclecticlight.co/2022/07/18/virtualisation-on-apple-silicon-macs-4-core-allocation-in-vms/ https://eclecticlight.co/2022/07/18/virtualisation-on-apple-...

Theophrastos 3y ago

Apple Virtualization Framework is a nice tool (option offered in particular by the UTM application in addition to Qemu option). And I hope it will improve with the release of the successor to Ventura and offer, for example, nested virtualization. The only concern currently, besides for virtualized macOS the lack of support for iCloud accounts, is that it contains a number of bugs which make it difficult to use as is for non-English/US configurations. Thus, there is a bug with ISO keyboards, which are recognized by Apple virtualization framework as ANSI keyboards, which shifts or even makes a certain number of keys disappear (different depending on the language: French, German, Danish, etc.).

kdrag0n 3y ago

I use Virtualization.framework for OrbStack (https://orbstack.dev https://orbstack.dev), which is a Docker Desktop and WSL alternative. It's great overall, and fairly convenient, but it has a fair share of bugs and limitations. The Virtualization service crashes with various combinations of Linux kernel version, macOS version, and architecture (ARM/Intel), so making it stable in each setup takes quite a bit of work. Device support is limited (no USB, etc.), and workarounds have other limitations. Rosetta is really buggy. The popular belief is that Virtualization.framework is inherently faster/lighter than other VMMs like QEMU, but I haven't found evidence to back it up. I spent a day prototyping a custom VMM (with an open-source base) and was able to get dynamic memory allocation working [1], as well as faster file sharing in some cases. There's a lot of other things I could do better (faster, simpler, more reliable) with a custom, tightly-integrated replacement. But unfortunately, that's not an option. Apple doesn't allow third-party VMMs to set the necessary CPU flags for Rosetta. There are too many users relying on Rosetta for fast x86 emulation for me to ditch it, despite all its bugs. So yes, I still use Virtualization.framework, but only because I have no choice. Happy to answer other questions about the framework! [1] https://twitter.com/kdrag0n/status/1645645284721184768 https://twitter.com/kdrag0n/status/1645645284721184768

transpute 3y ago
> Apple doesn't allow third-party VMMs to set the necessary CPU flags for Rosetta. Would that change with Asahi Linux host OS and macOS guest?
- kdrag0n 3y ago
  I can't speak for the feasibility of running a macOS guest on Asahi, but yes, Linux could be in control of the guest's CPU flags. Same goes for the m1n1 hypervisor. I can't find the exact post but I remember marcan saying that he doesn't want to poke the bear of using Apple's custom ISA extensions on Linux. So even though it's possible, I'm not sure whether it'll actually be done.
stefandesu 3y ago
> OrbStack (https://orbstack.dev https://orbstack.dev), which is a Docker Desktop and WSL alternative Wait why have I never heard of this? I just tried it out and it seems to work amazingly well.

vbezhenar 3y ago

Virtualization Framework is awesome and I'm slowly building my own VirtualBox-like app, learning Cocoa programming. Hopefully will release it some day. One big thing that's missing is better storage support. Right now it's just disk image and that's about it. No way to implement something like qcow2 with snapshots and whatnot. It's missing a lot of opportunity for VMs. Another big thing that's missing is USB pass-through. Other than that: it does absolutely everything I need. It even provides some kind of hook for network which should allow to build userspace router with any network architecture (didn't try it myself, though, but I think it should work). I was able to implement thing that I missed for years on M1 mac. Basically I run Fedora VM, I installed Rosetta helper inside, and docker. And then automagically my x86 images started to work with reasonable speed. I don't know if Docker Desktop supports it already, I don't want to touch it with a longest stick in the world. But for myself I solved this issue. It was rock stable in my experience and just worked. qemu on the other hand managed to crash my entire macOS (hopefully that bug was fixed). Another thing that I liked about it: protocols are "modern" and open-source. Almost every driver is virtio. No ancient cruft. No proprietary things. Eventually other operating systems could be run on this solution.

serhack_ 3y ago

Apple has updated the code to run macOS inside macOS. Now it can support recovering from previous saved states https://developer.apple.com/documentation/virtualization/running_macos_in_a_virtual_machine_on_apple_silicon https://developer.apple.com/documentation/virtualization/run...