15 ms·
Running Intel Binaries in Linux VMs with Rosetta
- a-dub 4y agointeresting. so this fixes the major performance issue by allowing users to install a user installable rosetta inside user created vms. could still end up with some "works for me" but still broken on prod issues resulting from the underlying architecture being different (for intel backing infra), and also some questions around how it would work in practice in terms of integrating with production container workflows, but seems like a boon for anyone who is struggling today with intel vm or container performance issues on apple silicon. nice!
- MBCook 4y agoThere is a specific piece of x86-64 software I need for my job, a back end service. It meant there was no way for me to use an M1 Mac for my job. I wish this was available last year, maybe I could’ve upgraded to an M1 based Mac.
- a-dub 4y agowhen the m1 initially shipped, i was watching this issue like a hawk. rosetta was really cool technology, but they seemed to really miss the mark in terms of many developer workflows that involve intel vms and containers. it was particularly interesting to me as i always got the sense that a lot of the success of the intel era macs was carried by their popularity and (and recommendability) amongst the internet development and engineering crowd. it seemed to me like a mistake to potentially alienate that group.
- MBCook 4y agoI figured it was largely a time management thing. They could only do so much (I mean they moved the whole base OS architecture… again). Glad to see it. Hope that by the time I get my next work laptop it’s still there, or better yet we’ve moved to a newer version of the software where this isn’t a problem.
- dilyevsky 4y agoIme podman (or docker) + qemu-user-static covers nearly everything. There are some issues with older go binaries emulation but most of c/c++ x86 code works as is
- a-dub 4y agowhat's the performance cost vs. intel code on the host under rosetta2 and native arm code on the host?
- dilyevsky 4y agoI think it depends greatly on what instructions are being used (sse, avx, etc vs neon). Some benches I’ve seen boast rosetta2 is only 30% hit but that’s doubtful to me based on my own experience running factorio =)
- deleted 4y ago[deleted]
- _ph_ 4y agoI am in the same boat and this news makes me very happy. I hope it is reliable when I have to replace my x86-MB Pro one day.
- cpuguy83 4y agoI don't know about "production container workflows" on a Mac, however the only thing that needs to be done is to use the new Rosetta binfmt handlers instead of qemu (which Docker Desktop already sets up for you). I imagine Docker will have an option to use Rosetta instead of qemu.
- shp0ngle 4y agoI don’t understand this part > The remaining steps required to activate Rosetta in a Linux guest aren’t commands that your app can execute or that you can script from inside your application to a Linux VM; the user must perform them either interactively or as part of a script while logged in to the Linux guest. You must communicate these requirements to the user of your app. That means Docker cannot do it magically for me?
- cpuguy83 4y agoMost likely Docker Desktop would have an option to toggle between Rosetta and qemu. Docker Desktop manages the vm, so it would be up to it to mount the Rosetta share into the vm and register it with binfmt.
- shp0ngle 4y agothey specifically say the user must do it himself, for some reason? but let’s see how it works actually
- jellyksong 4y agoBasic question: Why is this faster than running Intel Linux apps in an emulated Intel Linux VM? Because Rosetta is faster than QEMU, and you only need to emulate one application rather than the entire kernel?
- nicoburns 4y agoIt’s because Rosetta is able to do AOT compilation for the most part. So it actually converts the binary rather than emulating it at run time.
- deleted 4y ago[deleted]
- a-dub 4y agoa lot of the performance gains in rosetta 2 come from load time translation of executables and libraries. so when you run a binary on the host mac, rosetta jumps in, does a one time translation to something that can run natively on the mx processor and then (probably) caches the result on the filesystem. next time you run it, it runs the cached translation. if you're running a vm without support inside the guest for this, then you're just running a big intel blob on the mx processor and having to do realtime translation which is really slow. (worst case, you have an interrupt for every instruction, as you have to translate it... although i assume it must be better than that. either way you're constantly interrupting and context switching between the target code you're trying to run and the translation environment, context switches are expensive in and of themselves, and they also defeat a lot of caching)
- saagarjha 4y agoCorrect, plus Rosetta is substantially faster than QEMU because of AOT (as others mentioned) as well as a greater focus on performance.
- brigade 4y agoEmulation of an x86 kernel level means you lose the hardware-assisted virtualization support you'd get with an ARM kernel, and emulating an MMU is slow (among other things.) Technically this would be replacing QEMU user-mode emulation. Which isn't fast in a large part because QEMU being portable to all host architectures was more important than speed.
- Retr0id 4y agoDo any hypervisors support this yet? (i.e. can I try it out right now?)
- foodstances 4y agoYou're asking about something that was just released in a beta version of an OS announced a couple hours ago.
- Retr0id 4y agoCorrect!
- easton 4y agoThere will be a session posted tomorrow from Apple on how to use their sample to run a Linux VM with Rosetta support. I wonder if the sample on this page will do the trick by itself? https://developer.apple.com/documentation/virtualization/running_gui_linux_in_a_virtual_machine_on_a_mac https://developer.apple.com/documentation/virtualization/run...
- garaetjjte 4y agoVirtualization.framework is the hypervisor itself, you only need simple tool to launch it. You probably could just copy-paste provided code into eg. https://github.com/gyf304/vmcli https://github.com/gyf304/vmcli. However macOS 13 beta currently seems to be only available for registered developers.
- saagarjha 4y agoNo, this requires changes to use new APIs. I expect the major VM developers to adopt it in the coming days and weeks.
- morpheuskafka 4y agoIt seems that very little on the part of the hypervisor is required, so it shouldn't take them long to add it. They basically just have to call a function to create the virtual filesystem device for the guest.
- rudolph9 4y agoHow would this affect running an x86 container on a M1/M2 Mac?
- olliej 4y agoIt explicitly states it doesn't support bootstrapping - you still need your OS to actually run on arm64
- speedgoose 4y agoI think it’s already the case. The kernel is arm64 and the amd64 containers are running with qemu. Most containers are fine but I experienced some issues, with maintainers refusing to support this setup. I understand that you can replace qemu by rosetta now.
- deleted 4y ago[deleted]
- saagarjha 4y agoContainers are virtualized on macOS. x86-64 containers have used qemu to run the whole system under emulation, which is pretty slow. This workflow continues to require QEMU; however, now you can create an arm64 container and run x86-64 code in that using this technology.
- dilyevsky 4y agoWhat is arm64 container? Container is just a namespaced process so if your binary is x86 it is still technically x86 container just run using emulation. And qemu user static already does translation a la rosetta 2 so there probably wont be much difference in speed
- saagarjha 4y agoDocker contains a Linux kernel, does it not?
- eckza 4y agoSoooo does this mean that Docker will suck less on macOS, Real Soon Now?
- saagarjha 4y agoThis can't boot x86-64 containers, but you can create an arm64 container and run x86-64 code in it.
- duskwuff 4y agoI have to admit to a little bit of trepidation about the Rosetta mount point. Will this work for Docker without explicitly passing that mount point through to the containers?
- saagarjha 4y agoYeah I'm not entirely sure how they're going to make this work, but I kind of understand why it's so weird, because there's really not any very good way to communicate between macOS userspace and some arbitrary kernel running inside the VM…
- cpuguy83 4y agoThe way this would work is Docker would be setting this up for you (maybe through some option to switch between qemu and Rosetta). The containers should not need the mount. What happens is Rosetta gets registered with the kernel (binfmt_misc) to execute x86 binaries. The is the same mechanism that allows seamless qemu support.
- WatchDog 4y agoDocker on macOS already has some kind of qemu based solution for running x86_64 containers on m1. Potentially this could be used as an alternative?
- zamalek 4y agoIt's QEMU emulation. It's slow. It also segfaults a lot. I am able to run a select few containers on it, but I needed to sort out native arm64 for the vast majority.
- taejavu 4y agoDoes this mean I can run MSSQL Server on an M1 Mac?
- cassandratt 4y agoI mean, you can do that now. So, yes, but not because of this.
- draebek 4y agoI know of Azure SQL Edge, which has significant limitations, but I don't know how to run MS SQL Server on an M1 at anything more than glacial speeds (i.e. full system emulation in Qemu). Is there something I don't know about? Please share!
- my123 4y agothe Drawbridge environment has some peculiar needs that most JITted envs don't provide. So unlikely to work but will test that out soon...
- draebek 4y agoIf you're speaking about the existing technique MS has developed to run MS SQL on Linux, it doesn't work on M1s in Docker under Qemu user emulation: https://github.com/microsoft/mssql-docker/issues/668 https://github.com/microsoft/mssql-docker/issues/668 That's 2019. I haven't tested 2017 or 2022, but I assume they don't work either. AFAIK the official advice is to use Azure SQL Edge instead.
- my123 4y agoqemu-user is indeed hopelessly unsuitable for any kind of heavy use. Use anything else. :) But that doesn't answer the question of "does Rosetta deal with it properly?".
- 4y ago
- PeopleB4Cars 4y agoCan Apple please just make a simple Virtualization front end ala libvirts Virtual Machine Manager. They seem to have all the bits, just need to bring them all together.
- zaptrem 4y agoSeems like they're leaving that part up to the dev community (which I'm fine with). They'll never be able to make every person and every use case happy, so better to just give us the tools to do so ourselves. Check out UTM if you want a GUI for hypervisor.framework powered VMs.
- vbezhenar 4y agoI investigated virtualization framework recently and I'm kind of impressed. They really implemented a lot of things. And coming updates makes it even better. Here's what I've learned: 1. You can run console Linux. But you need to download ARM ISO, extract kernel and initrd and find out exact kernel parameters. Not a big issue, but probably can't be done automatically. You can't just boot ISO. 2. Disk support is bad. No snapshots, etc. Just single file. I think that you can emulate snapshots with APFS CoW support, but that's not as good as something like qcow2. 3. No GUI support (for Linux, for macOS there's GUI support but I have no idea how it works). 4. Network support is limited. You can run NAT but then you can't communicate with host. You can run bridge, but I don't like this mode and didn't even try it. You can implement userspace networking completely by yourself, but that's a lot of work and I didn't try it either. But I think that if someone would want to build a virtualization frontend, he had to do it. Otherwise basic features like port mapping will not be available. Now from what I read at this topic: ISO boot is coming, UEFI support is coming, Linux GUI support is coming. And I guess with enough tinkering it should be possible to run other BSDs, as they just leverage standard virtio protocols. Also Windows virtualization theoretically should be possible. I mean it's just another UEFI OS. You would need to put virtio drivers, but Redhat wrote virtio drivers (for x86, but should be possible to compile for ARM). I just wish that this virtualization framework was more extensible. Right now it's like use it or drop it. For example I didn't find any way to implement qcow2 support while using it for other things.
- siggen 4y agoIf I cannot use Apple Boot Camp to boot into another OS and avoid macOS, I'm not really interested in non-Intel macs.
- bfgoodrich 4y ago
- 58028641 4y agoThe effort to support Linux has made decent progress. https://asahilinux.org https://asahilinux.org Apple said Microsoft would have to cooperate to get Window working. https://www.macrumors.com/2020/11/20/craig-federighi-on-windows-for-m1-macs/ https://www.macrumors.com/2020/11/20/craig-federighi-on-wind...
- moron4hire 4y agoMicrosoft has a version of Windows (and even Visual Studio Pro) that runs on aarch64, specifically for Qualcomm SOCs, so it wouldn't be completely outside of possible. But I suspect they won't do it because they're probably be more interested in pushing Windows on ARM enthusiasts to Microsoft's and their partners' own hardware.
- _ph_ 4y agoWindows 11 for ARM apparently runs nicely in a VM on ARM Macs, a coworker of mine is using it for small projects. Including x86 emultion - his project requires some x86 tools.
- jeroenhd 4y agoFrom what I've heard, Microsoft and Qualcomm signed an exclusive deal for Windows on ARM for a number of years. Microsoft could theoretically release a Windows version for the M1 but they'd be breaking their contract with Qualcomm which can only go down badly. However, in a few years' time the deal runs out and it might be possible again, unless they can think of a reason to renew the deal (i.e. inclusion of Qualcomm's optimised x86/amd64 translation layer).
- Wowfunhappy 4y agoDoes anyone understand (or have any theories on) how this actually works? I don't understand how it's possible. Surely they didn't write a Linux version of Rosetta, it must be talking to the host OS somehow—but how? Where is the boundary?
- qeternity 4y agoIt seems pretty clear from TFA - there’s a directory share with the host, and given that Rosetta isn’t an emulator, but rather a translation layer, they don’t need a Linux version: x86 instructions go in, arm64 come out.
- Wowfunhappy 4y agoBut surely that would be too slow? Although Rosetta is great at caching instructions ahead of time, it does need to emulate a lot of code (ie, anything generated at runtime).
- anyfoo 4y agoHave you tried Rosetta? It can be pretty impressive.
- Wowfunhappy 4y agoRosetta is very impressive! I just don't see how they could maintain that by passing instructions back and fourth over a shared drive, that would be a ridiculous amount of latency!
- my123 4y agoThe shared drive is just a licensing trick. They do an ioctl over /proc/self/exe as the licensing mechanism. (and that's routed over with virtio-fs to the host)
- deleted 4y ago
- phendrenad2 4y agoSemi-related question for those who might know: Is it possible to run, say, Debian for x86 in a VM on M1 Macs? If so, I assume the performance is low because it has to translate every x86 instruction to ARM?
- roblabla 4y agoUTM[0] does this, which uses qemu under the hood. It's pretty slow, but for some non-GUI workloads, the performance is acceptable. [0]: https://mac.getutm.app/ https://mac.getutm.app/
- up6w6 4y agoDoes anyone know some work in the opposite direction, that is, running OSX applications with x86 CPUs? I'm seeing a lot of effort with making the Apple's hardware to be universal and run everything else (like Asahi Linux and Rosetta), while I can only imagine how impossible it is becoming to emulate the M1/ARM and run a hackintosh.
- Syonyk 4y ago> ...while I can only imagine how impossible it is to emulate a hackintosh considering the M1/ARM factor. Simply not possible. Apple Silicon is ARMv8, but with a number of their own custom extensions on top of it.
- deleted 4y ago[deleted]
- ccrush 4y agoThey still make Intel macs, right? So it's not impossible.
- steve_taylor 4y agoSure. Just reverse the polarity.
- ktta 4y agohttps://www.darlinghq.org https://www.darlinghq.org
- jaimex2 4y agoWhat for? There's just Apple software and a few niche things.
- shp0ngle 4y agoIf I am not wrong, this will allow much faster Docker with x86 base images? Docker is “just” a binary with cgroups magic.
- ec109685 4y agoIt’s a whole Linux kernel on the Mac that must run through the virtualization system, so it’s heavier weight than Linux docker. Sounds like x86 binaries within an already running ARM Linux base image can run much faster.
- mhio 4y ago> Sounds like x86 binaries within an already running ARM Linux base image can run much faster. At the moment the Docker desktop VM is setup to use qemu when running amd64 containers (processes) on the arm64 VM. Rosetta would replace that qemu binfmt setup, or maybe qemu-static wraps rosetta when available.
- threatripper 4y ago"heavier" means more footprint in memory and probably more startup time but not necessarily slower once it's running.
- vbezhenar 4y agoHopefully! It's still a major roadblock for me. I have to rent a VPS with x86 Linux just for docker.
- ComradePhil 4y agoFor performance reasons? Docker Desktop for macOS does run x86 docker on a VM. Has been doing it for a very long time.
- vbezhenar 4y agoYes, I tried emulation and it basically doesn't work for me, I guess because of Java.
- jordemort 4y agoThis in combination with the 3D acceleration work going into QEMU [1] could end up being a pretty compelling solution for running the x86 Linux version of Steam on macOS. [1] https://gist.github.com/akihikodaki/87df4149e7ca87f18dc56807ec5a1bc5 https://gist.github.com/akihikodaki/87df4149e7ca87f18dc56807...
- CraftThatBlock 4y agoGames on Proton (WINE) on Linux on Rosetta on macOS?
- viraptor 4y agoIf it's stupid and it works, it's not stupid :-)
- weberer 4y agoYou would need Apple to support Vulkan in their GPU drivers first.
- CraftThatBlock 4y agoUsing MoltenV maybe, similarly to vkd3d?
- xrisk 4y agoWouldn’t people working on Asahi right now write their own GPU drivers?
- ekianjo 4y agoDepends however on what the 3D acceleration supports in terms of features.
- 58028641 4y agoI'm hoping Alyssa Rosenzweig and Hector Martin's work will allow Linux to access the GPU directly.
- dankle 4y agoWill this enable tensorflow in docker on apple silicon?
- ComradePhil 4y agotensorflow is available natively on Apple silicon as of version 2.8.0.
- pachico 4y agoSomehow, I wish there was an easy way to do the opposite: creating and running arm images in intel laptops. This would allow me to run images easily and safely in Graviton based ec2 instances.
- ndesaulniers 4y agohttps://azeria-labs.com/arm-on-x86-qemu-user/ https://azeria-labs.com/arm-on-x86-qemu-user/
- ccrush 4y agolibhoudini is how Atom tablets ran ARM apps for Android on Intel ISA chips. It can be done. Rosetta is just a ripoff of libhoudini. Nothing new.
- KMag 4y agoSee DEC's FX!32 (running x86 WinNT 3.51 binaries on WinNT 3.51 workstations with Alpha AXP CPUs) and HP's Project Dynamo. There are probably even earlier examples, but FX!32 was roughly equivalent to Rosetta, and shipping production code in the 1990s.
- _kbh_ 4y agoIsn't Rosetta 2 the opposite of libhoundini? (translating from x86 to ARM, instead of ARM to x86), I also fail to see how an application that has its roots in a PPC to x86 translator (Rosetta 1) from 2006, is somehow a rip off of an application from around 5 years ago. If anything libhoudini is a rip off of Rosetta.
- rowanG077 4y agoYou realize that Rosetta is over 16 years old right?
- sofixa 4y agoIf you're talking about Docker images, you can already do that with buildx multiarch builds.
- ngcc_hk 4y agoTry to simplify my understanding- can we position of apple on a) doing wsl1 or b) even less just provide tool to do wsl1 or c) … wsl2 … tool only to wsl2 … Or we have a simpler mental model ? From a non-system land programmer point of view ?
- tarasglek 4y agoIs this the first time (at-least this decade) that apple is shipping Linux binaries?
- mxey 4y agoThis is probably the first time macOS includes a Linux binary, yes. Outside of macOS, Apple has been providing Linux binaries of the Swift toolchain for years: https://www.swift.org/download/ https://www.swift.org/download/
- Sporktacular 4y agoSo this lets me run windows steam games on apple silicon, with a low performance penalty? x86/Win game -> Steam for Windows(?) -> Linux VM Guest OS -> MacOS 13 VM Host OS -> Mac HW
- nikanj 4y agoFor some values of "low" anyway.
- Sporktacular 4y agoI guess what I don't get is the: Steam for Windows -> Linux VM Guest OS You can double click the Steam for Windows installer, in a Linux GUI and it installs right now? Using WINE? What magic is this?
- olliej 4y agoHow would this help windows specifically, or are you thinking steam+wine? In which case presumably steam+wine on Mac would have lower perf overhead. If your thinking involves a windows VM under linux you’re SOOL, as that has the same problem as an win/x86 vm on Mac: it’s a full system emulation - eg qemu, etc Rosetta can translate user mode code, because every can either be translated to native code /or/ is a system call, and apparently Rosetta knows how to translate linux syscalls.
- lisper 4y agoIs there any way to capture or view Rosetta's output, i.e. to use Rosetta as a tool for porting x86 code to the M1?
- olliej 4y agoYou’d surely only need that if you had significant raw assembly to translate, in which case you’d have to assume that there were technical reasons for exact memory layout in x86, or perf reasons where you’d presumably want handcoded arm64 assembly?
- lisper 4y agoYes, that is exactly the case. I am involved in the effort to port Clozure Common Lisp to the M1 and it has a lot of bespoke assembly code. (The x86_64 version of CCL actually works under Rosetta (more or less) which absolutely blows my mind since it's a compiler that emits x86 code at run-time!)
- olliej 4y agoOne issue you'd hit is that Rosetta is going to do fairly linear translation, and isn't going to improve calling conventions to the native one (in practice not terrible as x86_64 mercifully has enough registers for the all arguments in the general case). In a sense if you were to go through the hand assembly one at a time and do 1:1 substitution with ARMv8 equivalent you'd probably be able to do a bit better, just because you don't have to consider behaviour of generic x86 code.