5 ms·

Show HN: Vmctl/Vmcli – Easily Run Linux VMs on M1 Macs

198 points by gyf304 6y ago

gyf304 6y ago
I recently got myself a M1 Mac and I want to run Linux on it. By reading the docker M1 tech preview release (https://www.docker.com/blog/download-and-try-the-tech-preview-of-docker-desktop-for-m1/ https://www.docker.com/blog/download-and-try-the-tech-previe...), I was made aware of the existence of Virtualization.framework on recent macOS releases. Virtualization.framework provides 99% of the building blocks for running a Linux VM on a Mac, but there’s no off-the-shelf CLI for running a VM. Inspired by https://github.com/evansm7/vftool https://github.com/evansm7/vftool, I wrote another CLI (vmcli) with some more features (better termios handling / escape sequence detection) in Swift, and build a script (vmctl) to daemon-ize vmcli and help manage VMs. I also made a script for provisioning a Ubuntu VM, so you can run a Ubuntu VM from scratch in minutes.
- mjard 6y ago
  That’s really cool, nice work!
- aaomidi 6y ago
  Would be super nice if you get this on homebrew
- pabs3 6y ago
  Check out Asahi Linux: https://asahilinux.org/ https://asahilinux.org/ https://github.com/AsahiLinux/ https://github.com/AsahiLinux/
  - cbhl 6y ago
    Asahi is for running Linux on M1 bare metal, no? Also super early stage, IIRC.
    - pabs3 6y ago
      Right.
- mbreese 6y ago
  >I also made a script for provisioning a Ubuntu VM FYI - for an Intel Mac at least, the Ubuntu script needed a little manual tweaking, but the script was more than enough to get a VM working. Thanks!
  - pdimitar 6y ago
    Can you share what needed tweaking?
    - mbreese 6y ago
      I needed to add a persistent network configuration (which I didn't really figure out -- I was okay with starting an interface manually. I also had to re-generate the SSH host keys for some reason. In order to do these, I loaded the image in initramfs (basically started the VM w/o specifying root=/dev/vda as the command line argument). Then I mounted /dev/vda and chroot'd to it. Then I could change the root password to something that I knew, and setup the keys / config. So, instead of tweaking the setup script, I ended up making the changes manually, but it should be scriptable. I also was able to use the Ubuntu kernel/initrd to load a Debian 10 image as well. The default Debian 10 cloud image doesn't include the necessary kernel modules (virtio_console might be the only one necessary to add). I've spent a few days testing out this and the linked vftool (https://github.com/evansm7/vftool https://github.com/evansm7/vftool) to try to get a Debian VM. It's not an easy thing, but it did eventually work. I ended up corrupting the disk image though, so that wasn't fun. In order to do this on my Mac w/o needing a Linux machine, I installed ext4fuse so that I could mount raw disk images and mount partitions. I followed instructions from this GH issue, which was a great help. https://github.com/evansm7/vftool/issues/2 https://github.com/evansm7/vftool/issues/2
ulzeraj 6y ago
This is cool. Definitely going to test it. Years ago I’ve looked into something like this but found only find xhyve which looked a bit crude. Ended using Parallels Desktop Lite which to my understanding is a GUI to VF. Used to be free for Linux VMs then they’ve changed to yearly subscriptions. And they charge premium for a year of their glorified virt-manager. To make things worse their 1 year subscription is blocking me from changing countries on app store. I hate that company to guts now.
- ajconway 6y ago
  > Parallels Desktop Lite which to my understanding is a GUI to VF No, Parallels has its own hypervisor as a kernel module (mainly used pre-Big Sur) and can use Hypervisor Framework (mainly used with Big Sur). It also emulates a lot of hardware. Virtualization Framework only supports booting Linux VMs with minimal hardware emulation at the moment. It is only available on Big Sur.
  - ulzeraj 6y ago
    You are right. I was thinking Hypervisor Framework and its usage by Parallels Desktop Lite from the App Store. Lite version doesn’t require additional drivers and bridge adapters. It runs on top of Hypervisor Framework.
boris 6y ago
Here is a previous discussion of a similar tool that also lists some of the limitations (like inability to use bridged networking without Apple's permission) of using Virtualization.framework: https://news.ycombinator.com/item?id=25382529 https://news.ycombinator.com/item?id=25382529 I am personally more interested in tools that are based on Hypervisor.framework (which as I understand offers lower-level support for virtualization). Anyone knows what is the current state of Qemu support for M1 (last time I heard there were some out-of-tree patches)?
- gorgoiler 6y ago
  I won’t repeat any more outrage on the matter, but the requirement for permission seems so unlike the way things used to be with everyone’s favourite translucent NextStepish FreeBSDlike OS. Oh how I long for the days when the most controversial thing about Mac OS X was that it included some, gasp, GNU tools.
  - ogre_codes 6y ago
    > the requirement for permission seems so unlike the way things used to be It's not 1997 anymore. Security needs have changed over time. With modern systems, running unknown code is vastly riskier than it was 20 years ago.
    - gorgoiler 6y ago
      I see what you mean but I don’t know it’s that simple. There have always been unspoken dividing lines between software based on reputation. Systems software. Utilities. Trusted apps. Speculative apps from less well know authors. Web Browsers that run the most dangerous code of all: webpages. Sliding down the loss-of-permissions slope seems to mean that everything is untrusted these days and nothing counts as part of the club anymore, unless it’s shipped by the vendor. What made Mac OS X so wonderful was that the vendor seemed humble. They shipped their glitter but it was on top of an open system. Third parties were kept at arms length if they were apps but underneath — if you knew what you were doing enough to know there was an underneath — the choices were yours to make. Those choices were taken away. I use Linux now.
6a74 6y ago
Wow. This is fantastic. I had no idea Virtualization.framework was a thing. I've installed and tested vmctl/vmcli out on my M1 MacBook Air. Your instructions worked like a charm. Edit: I don't see the option to list running VMs though. I was hoping to check that powering off the VM worked. The inability to ping the VM tells me it is off, but if I didn't have the IP, I don't know how I would check this.
- jbverschoor 6y ago
  It has been a thing for 3 releases. Unfortunately, only veertu used it, and xhyve. Veertu is not what it was anymore. Sorry.. that was HVK. VK is newer
amedvednikov 6y ago
Amazing, thank you! Looks much cleaner than virtualbox. I assume the performance is much better as well. Any way to build it without installing xcode, just with xcode command line tools?
e12e 6y ago
So looking at: https://developer.apple.com/documentation/virtualization https://developer.apple.com/documentation/virtualization I see that the host supports virtio for providing devices - but this is a hypervisor, not an emulator? You can't run amd64 VMs on m1 and vice-versa? Is there a "standard" 64bit arm target that means pre-built VMs are available? It should also be possible to run windows arm VMs?
- bonzini 6y ago
  There is no support for Windows VMs in Virtualization.framework and therefore in vmcli. Hopefully QEMU 6.0 will include ARM support for the low-level Hypervisor.framework and you should be able to run both Linux and Windows with it.
  - e12e 6y ago
    Isn't that the other way around - there aren't windows arm builds that run under virtualization.framework? I mean, there are windows builds for 64bit arm, just as there are Linux builds?
    - bonzini 6y ago
      Virtualization.framework has its own boot loader and it can only boot Linux. With Hypervisor.framework QEMU can boot to a bog standard UEFI firmware image, which then loads either Windows or GRUB.
GEBBL 6y ago
Thank you so much. I bit the bullet and bought an M1 Mac but I’ve been really missing a Linux VM for hackthebox etc and was having to route through a Dropbox droplet. This should solve my issue
proyb2 6y ago
A quick benchmark, Vmctl has fare slight lower/better latencies than Parallels.
prime31 6y ago
For those looking for a UI and full desktop environment UTM has full support for the M1 now: https://github.com/utmapp/UTM https://github.com/utmapp/UTM
stephenr 6y ago
I wonder how practical it would be to implement a vagrant provider on top of this. Has anyone tried running this on an intel Mac?

198 points by gyf304 6y ago

I recently got myself a M1 Mac and I want to run Linux on it. By reading the docker M1 tech preview release (https://www.docker.com/blog/download-and-try-the-tech-preview-of-docker-desktop-for-m1/ https://www.docker.com/blog/download-and-try-the-tech-previe...), I was made aware of the existence of Virtualization.framework on recent macOS releases. Virtualization.framework provides 99% of the building blocks for running a Linux VM on a Mac, but there’s no off-the-shelf CLI for running a VM. Inspired by https://github.com/evansm7/vftool https://github.com/evansm7/vftool, I wrote another CLI (vmcli) with some more features (better termios handling / escape sequence detection) in Swift, and build a script (vmctl) to daemon-ize vmcli and help manage VMs. I also made a script for provisioning a Ubuntu VM, so you can run a Ubuntu VM from scratch in minutes.

mjard 6y ago
That’s really cool, nice work!
aaomidi 6y ago
Would be super nice if you get this on homebrew
pabs3 6y ago
Check out Asahi Linux: https://asahilinux.org/ https://asahilinux.org/ https://github.com/AsahiLinux/ https://github.com/AsahiLinux/
- cbhl 6y ago
  Asahi is for running Linux on M1 bare metal, no? Also super early stage, IIRC.
  - pabs3 6y ago
    Right.
mbreese 6y ago
>I also made a script for provisioning a Ubuntu VM FYI - for an Intel Mac at least, the Ubuntu script needed a little manual tweaking, but the script was more than enough to get a VM working. Thanks!
- pdimitar 6y ago
  Can you share what needed tweaking?
  - mbreese 6y ago
    I needed to add a persistent network configuration (which I didn't really figure out -- I was okay with starting an interface manually. I also had to re-generate the SSH host keys for some reason. In order to do these, I loaded the image in initramfs (basically started the VM w/o specifying root=/dev/vda as the command line argument). Then I mounted /dev/vda and chroot'd to it. Then I could change the root password to something that I knew, and setup the keys / config. So, instead of tweaking the setup script, I ended up making the changes manually, but it should be scriptable. I also was able to use the Ubuntu kernel/initrd to load a Debian 10 image as well. The default Debian 10 cloud image doesn't include the necessary kernel modules (virtio_console might be the only one necessary to add). I've spent a few days testing out this and the linked vftool (https://github.com/evansm7/vftool https://github.com/evansm7/vftool) to try to get a Debian VM. It's not an easy thing, but it did eventually work. I ended up corrupting the disk image though, so that wasn't fun. In order to do this on my Mac w/o needing a Linux machine, I installed ext4fuse so that I could mount raw disk images and mount partitions. I followed instructions from this GH issue, which was a great help. https://github.com/evansm7/vftool/issues/2 https://github.com/evansm7/vftool/issues/2

ulzeraj 6y ago

This is cool. Definitely going to test it. Years ago I’ve looked into something like this but found only find xhyve which looked a bit crude. Ended using Parallels Desktop Lite which to my understanding is a GUI to VF. Used to be free for Linux VMs then they’ve changed to yearly subscriptions. And they charge premium for a year of their glorified virt-manager. To make things worse their 1 year subscription is blocking me from changing countries on app store. I hate that company to guts now.

ajconway 6y ago
> Parallels Desktop Lite which to my understanding is a GUI to VF No, Parallels has its own hypervisor as a kernel module (mainly used pre-Big Sur) and can use Hypervisor Framework (mainly used with Big Sur). It also emulates a lot of hardware. Virtualization Framework only supports booting Linux VMs with minimal hardware emulation at the moment. It is only available on Big Sur.
- ulzeraj 6y ago
  You are right. I was thinking Hypervisor Framework and its usage by Parallels Desktop Lite from the App Store. Lite version doesn’t require additional drivers and bridge adapters. It runs on top of Hypervisor Framework.

boris 6y ago

Here is a previous discussion of a similar tool that also lists some of the limitations (like inability to use bridged networking without Apple's permission) of using Virtualization.framework: https://news.ycombinator.com/item?id=25382529 https://news.ycombinator.com/item?id=25382529 I am personally more interested in tools that are based on Hypervisor.framework (which as I understand offers lower-level support for virtualization). Anyone knows what is the current state of Qemu support for M1 (last time I heard there were some out-of-tree patches)?

gorgoiler 6y ago
I won’t repeat any more outrage on the matter, but the requirement for permission seems so unlike the way things used to be with everyone’s favourite translucent NextStepish FreeBSDlike OS. Oh how I long for the days when the most controversial thing about Mac OS X was that it included some, gasp, GNU tools.
- ogre_codes 6y ago
  > the requirement for permission seems so unlike the way things used to be It's not 1997 anymore. Security needs have changed over time. With modern systems, running unknown code is vastly riskier than it was 20 years ago.
  - gorgoiler 6y ago
    I see what you mean but I don’t know it’s that simple. There have always been unspoken dividing lines between software based on reputation. Systems software. Utilities. Trusted apps. Speculative apps from less well know authors. Web Browsers that run the most dangerous code of all: webpages. Sliding down the loss-of-permissions slope seems to mean that everything is untrusted these days and nothing counts as part of the club anymore, unless it’s shipped by the vendor. What made Mac OS X so wonderful was that the vendor seemed humble. They shipped their glitter but it was on top of an open system. Third parties were kept at arms length if they were apps but underneath — if you knew what you were doing enough to know there was an underneath — the choices were yours to make. Those choices were taken away. I use Linux now.

6a74 6y ago

Wow. This is fantastic. I had no idea Virtualization.framework was a thing. I've installed and tested vmctl/vmcli out on my M1 MacBook Air. Your instructions worked like a charm. Edit: I don't see the option to list running VMs though. I was hoping to check that powering off the VM worked. The inability to ping the VM tells me it is off, but if I didn't have the IP, I don't know how I would check this.

jbverschoor 6y ago
It has been a thing for 3 releases. Unfortunately, only veertu used it, and xhyve. Veertu is not what it was anymore. Sorry.. that was HVK. VK is newer

amedvednikov 6y ago

Amazing, thank you! Looks much cleaner than virtualbox. I assume the performance is much better as well. Any way to build it without installing xcode, just with xcode command line tools?

e12e 6y ago

So looking at: https://developer.apple.com/documentation/virtualization https://developer.apple.com/documentation/virtualization I see that the host supports virtio for providing devices - but this is a hypervisor, not an emulator? You can't run amd64 VMs on m1 and vice-versa? Is there a "standard" 64bit arm target that means pre-built VMs are available? It should also be possible to run windows arm VMs?

bonzini 6y ago
There is no support for Windows VMs in Virtualization.framework and therefore in vmcli. Hopefully QEMU 6.0 will include ARM support for the low-level Hypervisor.framework and you should be able to run both Linux and Windows with it.
- e12e 6y ago
  Isn't that the other way around - there aren't windows arm builds that run under virtualization.framework? I mean, there are windows builds for 64bit arm, just as there are Linux builds?
  - bonzini 6y ago
    Virtualization.framework has its own boot loader and it can only boot Linux. With Hypervisor.framework QEMU can boot to a bog standard UEFI firmware image, which then loads either Windows or GRUB.

GEBBL 6y ago

Thank you so much. I bit the bullet and bought an M1 Mac but I’ve been really missing a Linux VM for hackthebox etc and was having to route through a Dropbox droplet. This should solve my issue

proyb2 6y ago

A quick benchmark, Vmctl has fare slight lower/better latencies than Parallels.

prime31 6y ago

For those looking for a UI and full desktop environment UTM has full support for the M1 now: https://github.com/utmapp/UTM https://github.com/utmapp/UTM

stephenr 6y ago

I wonder how practical it would be to implement a vagrant provider on top of this. Has anyone tried running this on an intel Mac?