4 ms·

The thing is, such an emulator exists, MacOS itself uses it to run old x86 MacOS apps transparently. Why is that as soon as that app happens to be, say, VMWare,

by TylerE 3y ago

The thing is, such an emulator exists, MacOS itself uses it to run old x86 MacOS apps transparently. Why is that as soon as that app happens to be, say, VMWare, the OS suddenly says nuh uh, not gonna do it? The technology works, I run AAA x86 games on my Mac Studio via Crossover. Sure, the performance is a bit limited, but it's limited by the nature of an integrated, albeit fairly powerful, GPU. It works surprisingly well considering many of these games are targeted at, say, 1000-series NVidia cards. But wanting to run a 8GB Linux instance so I can run a local dev environment is an impossible ask? (Before anyone asks, no, ARM linux isn't really a viable solution for...reasons I don't feel like going into but are mostly boring and technical-debty).

andrewaylett 3y ago
The canonical mechanism for running amd64 Linux processes appears to be to virtualise aarch64 and use binfmt-misc with Rosetta 2 to get emulation working. It does make a certain amount of sense that Apple would have hardware virtualisation support for native VMs but not for emulated VMs. I can imagine (but I've not checked) that support for emulation of the VT extensions is lacking. As a random person on the Internet, I'm obviously overqualified to suggest that you use native virtualisation to run aarch64 Linux, then use Rosetta within that Linux VM to run whatever amd64 software virtualisation tool you prefer. This is quite similar to what containerisation tooling does -- Docker (and similar) on aarch64 runs an ARM VM, then uses Rosetta inside that VM to run containers. You don't get a native amd64 kernel that way, but even without nested virtualisation you get a complete (namespaced) amd64 userspace.
aseipp 3y ago
You can do it with QEMU using its emulation backend to emulate the entire x86 boot chain. It will be dozens (hundreds?) of times slower. Hypervisors work efficiently because they allow you to "nest" certain functions inside of other userspace processes, which are typically only available to the operating system (with a higher privilege level), things like delivering interrupts efficiently, or managing page tables. The nesting means that the nested operating system is running on the same hardware in practice, and so it inherits many of the constraints of the host architecture. So, virtualization extensions can be seen as a kind of generalization of existing features, designed so that nesting them is efficient and secure. For example, different architectures have different rules about how page tables are set up and how virtual memory page faults are handled in that environment. The entire memory mapped layout of the system is completely different. The entire memory model (TSO vs weak ordering) is different. There are strict correctness requirements. A linux kernel for x86 has specific x86 code to manage that and perform those sequences correctly. You cannot just translate x86 code to ARM code and hope the same sequence works; you have to emulate the entire processor environment, so that x86 code works as it should. Rosetta does not emulate code; it is a binary translator. It translates x86 code to ARM code up front, then runs that. The only reason this works is because normal userspace programs have an ABI that dictates how they interoperate with each other across process and address space and function boundaries. When CrossOver runs an x86 game for example, it translates the x86 to ARM. That program then calls Metal APIs, libmetal.dylib or something, but that .dylib itself isn't x86-to-ARM translated. It is simply "pass through" shim to your system-native Metal APIs. So the graphics stack achieves native performance; the application overhead comes from the x86-to-ARM translation, which needs to preserve the semantics of the original code. Rosetta-for-Linux works the same way, because there is a stable ABI that exists between processes and function calls, and the Linux kernel ABI is considered stable (though not between architectures in some cases). It translates the x86-to-ARM, and then that binary is run and it gets to call into the native Linux kernel, which is not translated, etc. This basically works well in practice. It is also how Windows' x86-on-ARM emulation works. If you want to emulate an entire x86 processor, including an x86 Linux kernel, you have to do exactly that, emulate it. Which includes the entire boot process, the memory model, CPU instructions that may not have efficient 1-to-1 translations, etc. Unfortunately, what you are asking is not actually reasonably possible, in a technical sense. Your options are either to use Rosetta-on-Linux to translate your binary, or get an actual x86 Linux machine.
- astrange 3y ago
  > Rosetta does not emulate code; it is a binary translator. It translates x86 code to ARM code up front, then runs that. The only reason this works is because normal userspace programs have an ABI that dictates how they interoperate with each other across process and address space and function boundaries. When CrossOver runs an x86 game for example, it translates the x86 to ARM. I don't think this is true. I think code run under WINE is always JITted because it's too much unlike a Mac binary.
  - olliej 3y ago
    Honestly I feel like ([edit: removed personal detail that in hindsight isn't really relevant in this specific discussion]) translation vs emulation is very close to a purely marketing distinction. I feel like the only thing that makes it a semantically valid distinction is if you take an interpretation of translation being explicitly the "translated" code is run as native code on the cpu, rather than as a loop written in native code that manually evaluating each original instruction. But that's still iffy, as many things that people would absolutely call emulators nowadays are generating native code as well. JIT vs non-JIT also isn't really sufficient. Rosetta necessarily does runtime code generation as well[1], because Rosetta needs to support running code generated by x86_64 JITs at runtime (note that this is actually often slower than simply not using a JIT in the first place, to the extent that JavaScriptCore at least has a mechanism to ensure it does not use the JITs if its running under rosetta for this reason). Then as you say programs like WINE are definitionally not using platform binaries, so any code that they load and start to execute looks basically identical to "this program generated code at runtime". I do wonder if there's something that could be done either as some kind of Rosetta API to register a block of code as being logically static to allow some kind of caching, or by wine doing some fairly butchered rewriting to convert the windows objects into something that looks like "normal" machos to essentially trick rosetta into the AOT and/or caching path. [1] Until the 2023 rosetta update the linux VM support only supported JIT translation, that update added support for a more Mac like model where there's a central daemon to handle loading binaries and libraries so they can be cached and shared
olliej 3y ago
Ah, that's not quite accurate. Rosetta is an emulator (sorry <marketing>translator</marketing>) for userspace. Happily Apple actually provides direct support for linux in rosetta (as in it can parse linux binaries and libraries, etc can remap syscalls correctly, etc in linux VMs), and there's even a WWDC video on how to integrate it, so that you can install an arm linux kernel, and then run x86_64 linux binaries via rosetta. Windows has its own translator so doesn't need rosetta. At a basic level the semantics of usermode x86 and usermode arm are identical, so you can basically say "I just care about translating the x86 instructions to arm equivalents", and all the rest of the "make this code run correctly" semantics of the hardware (memory protection, etc) then just work. That basically breaks down completely for kernel mode code, for a variety of reasons, at one level a lot of the things rosetta style translators can do for speed break down because they can't just leverage memory protection any more. But at a more fundamental level your translator now needs to translate completely different mechanisms for how memory is accessed, how page tables are set up, how interrupts, are handled, etc. It's hard to say exactly which bit would be the worst part, but likely memory protection - you could very easily end up in a state where your translator has to added non-hardware supported memory protection checks on every memory access, and that is slow. Significant silicon resources are expended (the extremely high performance and physically colocated TLBs, caches, etc) to make paged memory efficient, and suddenly you have actual code running and walking what (to the cpu) is just regular ram. So now what was previously a single memory access for the CPU involves a code loop with multiple (again, for the cpu) random memory accesses. Those problems are why CPUs added explicit support for virtualization in the early 2000s. VMWare, etc started making virtual machines an actual viable product, because they were able to set things up so that the majority of code in the client OS was running directly on the host CPU. The problem they had for performance was essentially what I described above, only they didn't have to also translate all of the instructions (I believe their implementation at the time did translate some instructions, especially kernel mode code, as part of their "tolerable" performance was being very clever, and of course the client os kernel is running in the host os's user mode so definitely has to handle client kernel use of kernel mode only instructions). A lot of virtualization support in CPUs basically boils down to things like nested page tables, which lets your VM just say "these are the page tables you should also be looking at while I'm running". Now for cross architecture emulation that's just not an option, as the problem is your client kernel is trying to construct the data structures used by its target cpu, and those data structures don't match, and may or may not even have an equivalent, so there is no "fix" that doesn't eventually boil down to the host architecture supporting the client architecture directly, and at that point you're essentially adding multiple different implementations of the same features to the silicon which is astronomically expensive. The far better solution is to say, make the child OS run a native to the host architecture kernel, and then have that do usermode translation like rosetta for application compatibility.

by TylerE 3y ago

andrewaylett 3y ago

The canonical mechanism for running amd64 Linux processes appears to be to virtualise aarch64 and use binfmt-misc with Rosetta 2 to get emulation working. It does make a certain amount of sense that Apple would have hardware virtualisation support for native VMs but not for emulated VMs. I can imagine (but I've not checked) that support for emulation of the VT extensions is lacking. As a random person on the Internet, I'm obviously overqualified to suggest that you use native virtualisation to run aarch64 Linux, then use Rosetta within that Linux VM to run whatever amd64 software virtualisation tool you prefer. This is quite similar to what containerisation tooling does -- Docker (and similar) on aarch64 runs an ARM VM, then uses Rosetta inside that VM to run containers. You don't get a native amd64 kernel that way, but even without nested virtualisation you get a complete (namespaced) amd64 userspace.

aseipp 3y ago

You can do it with QEMU using its emulation backend to emulate the entire x86 boot chain. It will be dozens (hundreds?) of times slower. Hypervisors work efficiently because they allow you to "nest" certain functions inside of other userspace processes, which are typically only available to the operating system (with a higher privilege level), things like delivering interrupts efficiently, or managing page tables. The nesting means that the nested operating system is running on the same hardware in practice, and so it inherits many of the constraints of the host architecture. So, virtualization extensions can be seen as a kind of generalization of existing features, designed so that nesting them is efficient and secure. For example, different architectures have different rules about how page tables are set up and how virtual memory page faults are handled in that environment. The entire memory mapped layout of the system is completely different. The entire memory model (TSO vs weak ordering) is different. There are strict correctness requirements. A linux kernel for x86 has specific x86 code to manage that and perform those sequences correctly. You cannot just translate x86 code to ARM code and hope the same sequence works; you have to emulate the entire processor environment, so that x86 code works as it should. Rosetta does not emulate code; it is a binary translator. It translates x86 code to ARM code up front, then runs that. The only reason this works is because normal userspace programs have an ABI that dictates how they interoperate with each other across process and address space and function boundaries. When CrossOver runs an x86 game for example, it translates the x86 to ARM. That program then calls Metal APIs, libmetal.dylib or something, but that .dylib itself isn't x86-to-ARM translated. It is simply "pass through" shim to your system-native Metal APIs. So the graphics stack achieves native performance; the application overhead comes from the x86-to-ARM translation, which needs to preserve the semantics of the original code. Rosetta-for-Linux works the same way, because there is a stable ABI that exists between processes and function calls, and the Linux kernel ABI is considered stable (though not between architectures in some cases). It translates the x86-to-ARM, and then that binary is run and it gets to call into the native Linux kernel, which is not translated, etc. This basically works well in practice. It is also how Windows' x86-on-ARM emulation works. If you want to emulate an entire x86 processor, including an x86 Linux kernel, you have to do exactly that, emulate it. Which includes the entire boot process, the memory model, CPU instructions that may not have efficient 1-to-1 translations, etc. Unfortunately, what you are asking is not actually reasonably possible, in a technical sense. Your options are either to use Rosetta-on-Linux to translate your binary, or get an actual x86 Linux machine.

astrange 3y ago
> Rosetta does not emulate code; it is a binary translator. It translates x86 code to ARM code up front, then runs that. The only reason this works is because normal userspace programs have an ABI that dictates how they interoperate with each other across process and address space and function boundaries. When CrossOver runs an x86 game for example, it translates the x86 to ARM. I don't think this is true. I think code run under WINE is always JITted because it's too much unlike a Mac binary.
- olliej 3y ago
  Honestly I feel like ([edit: removed personal detail that in hindsight isn't really relevant in this specific discussion]) translation vs emulation is very close to a purely marketing distinction. I feel like the only thing that makes it a semantically valid distinction is if you take an interpretation of translation being explicitly the "translated" code is run as native code on the cpu, rather than as a loop written in native code that manually evaluating each original instruction. But that's still iffy, as many things that people would absolutely call emulators nowadays are generating native code as well. JIT vs non-JIT also isn't really sufficient. Rosetta necessarily does runtime code generation as well[1], because Rosetta needs to support running code generated by x86_64 JITs at runtime (note that this is actually often slower than simply not using a JIT in the first place, to the extent that JavaScriptCore at least has a mechanism to ensure it does not use the JITs if its running under rosetta for this reason). Then as you say programs like WINE are definitionally not using platform binaries, so any code that they load and start to execute looks basically identical to "this program generated code at runtime". I do wonder if there's something that could be done either as some kind of Rosetta API to register a block of code as being logically static to allow some kind of caching, or by wine doing some fairly butchered rewriting to convert the windows objects into something that looks like "normal" machos to essentially trick rosetta into the AOT and/or caching path. [1] Until the 2023 rosetta update the linux VM support only supported JIT translation, that update added support for a more Mac like model where there's a central daemon to handle loading binaries and libraries so they can be cached and shared

olliej 3y ago

Ah, that's not quite accurate. Rosetta is an emulator (sorry <marketing>translator</marketing>) for userspace. Happily Apple actually provides direct support for linux in rosetta (as in it can parse linux binaries and libraries, etc can remap syscalls correctly, etc in linux VMs), and there's even a WWDC video on how to integrate it, so that you can install an arm linux kernel, and then run x86_64 linux binaries via rosetta. Windows has its own translator so doesn't need rosetta. At a basic level the semantics of usermode x86 and usermode arm are identical, so you can basically say "I just care about translating the x86 instructions to arm equivalents", and all the rest of the "make this code run correctly" semantics of the hardware (memory protection, etc) then just work. That basically breaks down completely for kernel mode code, for a variety of reasons, at one level a lot of the things rosetta style translators can do for speed break down because they can't just leverage memory protection any more. But at a more fundamental level your translator now needs to translate completely different mechanisms for how memory is accessed, how page tables are set up, how interrupts, are handled, etc. It's hard to say exactly which bit would be the worst part, but likely memory protection - you could very easily end up in a state where your translator has to added non-hardware supported memory protection checks on every memory access, and that is slow. Significant silicon resources are expended (the extremely high performance and physically colocated TLBs, caches, etc) to make paged memory efficient, and suddenly you have actual code running and walking what (to the cpu) is just regular ram. So now what was previously a single memory access for the CPU involves a code loop with multiple (again, for the cpu) random memory accesses. Those problems are why CPUs added explicit support for virtualization in the early 2000s. VMWare, etc started making virtual machines an actual viable product, because they were able to set things up so that the majority of code in the client OS was running directly on the host CPU. The problem they had for performance was essentially what I described above, only they didn't have to also translate all of the instructions (I believe their implementation at the time did translate some instructions, especially kernel mode code, as part of their "tolerable" performance was being very clever, and of course the client os kernel is running in the host os's user mode so definitely has to handle client kernel use of kernel mode only instructions). A lot of virtualization support in CPUs basically boils down to things like nested page tables, which lets your VM just say "these are the page tables you should also be looking at while I'm running". Now for cross architecture emulation that's just not an option, as the problem is your client kernel is trying to construct the data structures used by its target cpu, and those data structures don't match, and may or may not even have an equivalent, so there is no "fix" that doesn't eventually boil down to the host architecture supporting the client architecture directly, and at that point you're essentially adding multiple different implementations of the same features to the silicon which is astronomically expensive. The far better solution is to say, make the child OS run a native to the host architecture kernel, and then have that do usermode translation like rosetta for application compatibility.