5 ms·

How would this affect running an x86 container on a M1/M2 Mac?

by rudolph9 4y ago

olliej 4y ago
It explicitly states it doesn't support bootstrapping - you still need your OS to actually run on arm64
- speedgoose 4y ago
  I think it’s already the case. The kernel is arm64 and the amd64 containers are running with qemu. Most containers are fine but I experienced some issues, with maintainers refusing to support this setup. I understand that you can replace qemu by rosetta now.
  - deleted 4y ago
    [deleted]
  - saagarjha 4y ago
    Containers are virtualized on macOS. x86-64 containers have used qemu to run the whole system under emulation, which is pretty slow. This workflow continues to require QEMU; however, now you can create an arm64 container and run x86-64 code in that using this technology.
    - dilyevsky 4y ago
      What is arm64 container? Container is just a namespaced process so if your binary is x86 it is still technically x86 container just run using emulation. And qemu user static already does translation a la rosetta 2 so there probably wont be much difference in speed
      - saagarjha 4y ago
        Docker contains a Linux kernel, does it not?
        dilyevsky 4y ago
        Nope, docker just sets up linux cgroups/namespaces and spawns a new linux process, no kernel of its own.
        cpuguy83 4y ago
        Docker Desktop (ie Docker on Mac) absolutely sets up a VM.
        dilyevsky 4y ago
        Correct, on macs it spawns a (usually aarch64) vm using apples hypervisor framework api (hw virt) and runs docker daemon inside of it. Then it’s what I described above - just an process running arm instructions. If you’re using qemu binfmt_misc runner which uses “user mode emulation” not to be confused with system emulation which does full blown hw emulation. So it’s aready doing similar things to rosetta 2
        cpuguy83 4y ago
        Yes qemu and Rosetta perform the same basic role here (bintfmt handler for x86 bins), but Rosetta should perform significantly better (otherwise there would be virtually no reason at all to build this).
        mr_toad 4y ago
        > Docker Desktop (ie Docker on Mac) absolutely sets up a VM. Docker runs inside the VM, not the other way around.
        nirvdrum 4y ago
        Something needs to set up the VM that Docker runs in. Docker Desktop is that something. The GUI even lets you tweak the VM resources. I suppose you could do it all on your own, but most macOS users probably just install Docker Desktop and let it handle the VM.
      - cpuguy83 4y ago
        The whole point in providing Rosetta is because it (should) be significantly faster than plain qemu.
        dilyevsky 4y ago
        If by plain qemu you mean its user mode emulation then i think it remains to be seen what the performance improvement will be. User mode already can be ok if it’s not doing endianness translation
        dekhn 4y ago
        I'm gonna guess that if Apple invested this level of effort into it, it's not a "remains to be seen what the performance improvement will be". You can take their graphs and charts and sort of de-reality-distortion-field them to know that it will be much faster than qemu for the workloads Apple knows people run frequently, but still slower than if you had a new x86_64 machine.
        flurie 4y ago
        I've done a lot of work with running userland qemu for x86_64 in aarch64 vms on macOS 12. The experience is serviceable for simple things, but if I build anything really heavyweight, I find I need a native arch builder. I have little doubt that Rosetta will be an improvement.
cpuguy83 4y ago
Instead of running in qemu (as it currently would), it would run under Rosetta.
therealmarv 4y ago
really good question... is a x86 container already bootstrapping or can we put it through Rosetta (instead through QEMU) to get it run?
- olliej 4y ago
  The container’s kernel has to run on hardware (that’s what kernels do). If you have an x86 container that means an x86 kernel. Rosetta only runs user mode processes (and presumably in this environment knows how to translate linux system calls?). If you /have/ to run an x86 kernel (I’m not sure what the reason would be?) you’re stuck with full system emulation a la qemu. In reality what you probably have is a few x86 binaries that you need, and it sounds like this will let docker, VMware, qemu, whatever, leverage Rosetta w/o having to implement it themselves. Now, if you’re running something that does codegen at runtime, then Rosetta is pretty much just as screwed as any other tech you might want to use.
rudolph9 4y ago
Op of thread here: there seems to be a lot of back and forth on running Arm64 containers and executing Rosetta in them to run something x86 but as far as I can tell (correct me if I’m mistaken and I think I might be) that doesn’t really help because my production containers are x86 so I would need to compile my container for Arm64 then run Rosetta in which is a totally divergent image and kind of useless for running production images locally.

How would this affect running an x86 container on a M1/M2 Mac?

olliej 4y ago

It explicitly states it doesn't support bootstrapping - you still need your OS to actually run on arm64

speedgoose 4y ago
I think it’s already the case. The kernel is arm64 and the amd64 containers are running with qemu. Most containers are fine but I experienced some issues, with maintainers refusing to support this setup. I understand that you can replace qemu by rosetta now.
- deleted 4y ago
  [deleted]
- saagarjha 4y ago
  Containers are virtualized on macOS. x86-64 containers have used qemu to run the whole system under emulation, which is pretty slow. This workflow continues to require QEMU; however, now you can create an arm64 container and run x86-64 code in that using this technology.
  - dilyevsky 4y ago
    What is arm64 container? Container is just a namespaced process so if your binary is x86 it is still technically x86 container just run using emulation. And qemu user static already does translation a la rosetta 2 so there probably wont be much difference in speed
    - saagarjha 4y ago
      Docker contains a Linux kernel, does it not?
      - dilyevsky 4y ago
        Nope, docker just sets up linux cgroups/namespaces and spawns a new linux process, no kernel of its own.
        cpuguy83 4y ago
        Docker Desktop (ie Docker on Mac) absolutely sets up a VM.
        dilyevsky 4y ago
        Correct, on macs it spawns a (usually aarch64) vm using apples hypervisor framework api (hw virt) and runs docker daemon inside of it. Then it’s what I described above - just an process running arm instructions. If you’re using qemu binfmt_misc runner which uses “user mode emulation” not to be confused with system emulation which does full blown hw emulation. So it’s aready doing similar things to rosetta 2
        cpuguy83 4y ago
        Yes qemu and Rosetta perform the same basic role here (bintfmt handler for x86 bins), but Rosetta should perform significantly better (otherwise there would be virtually no reason at all to build this).
        mr_toad 4y ago
        > Docker Desktop (ie Docker on Mac) absolutely sets up a VM. Docker runs inside the VM, not the other way around.
        nirvdrum 4y ago
        Something needs to set up the VM that Docker runs in. Docker Desktop is that something. The GUI even lets you tweak the VM resources. I suppose you could do it all on your own, but most macOS users probably just install Docker Desktop and let it handle the VM.
    - cpuguy83 4y ago
      The whole point in providing Rosetta is because it (should) be significantly faster than plain qemu.
      - dilyevsky 4y ago
        If by plain qemu you mean its user mode emulation then i think it remains to be seen what the performance improvement will be. User mode already can be ok if it’s not doing endianness translation
        dekhn 4y ago
        I'm gonna guess that if Apple invested this level of effort into it, it's not a "remains to be seen what the performance improvement will be". You can take their graphs and charts and sort of de-reality-distortion-field them to know that it will be much faster than qemu for the workloads Apple knows people run frequently, but still slower than if you had a new x86_64 machine.
        flurie 4y ago
        I've done a lot of work with running userland qemu for x86_64 in aarch64 vms on macOS 12. The experience is serviceable for simple things, but if I build anything really heavyweight, I find I need a native arch builder. I have little doubt that Rosetta will be an improvement.

cpuguy83 4y ago

Instead of running in qemu (as it currently would), it would run under Rosetta.

therealmarv 4y ago

really good question... is a x86 container already bootstrapping or can we put it through Rosetta (instead through QEMU) to get it run?

olliej 4y ago
The container’s kernel has to run on hardware (that’s what kernels do). If you have an x86 container that means an x86 kernel. Rosetta only runs user mode processes (and presumably in this environment knows how to translate linux system calls?). If you /have/ to run an x86 kernel (I’m not sure what the reason would be?) you’re stuck with full system emulation a la qemu. In reality what you probably have is a few x86 binaries that you need, and it sounds like this will let docker, VMware, qemu, whatever, leverage Rosetta w/o having to implement it themselves. Now, if you’re running something that does codegen at runtime, then Rosetta is pretty much just as screwed as any other tech you might want to use.

rudolph9 4y ago

Op of thread here: there seems to be a lot of back and forth on running Arm64 containers and executing Rosetta in them to run something x86 but as far as I can tell (correct me if I’m mistaken and I think I might be) that doesn’t really help because my production containers are x86 so I would need to compile my container for Arm64 then run Rosetta in which is a totally divergent image and kind of useless for running production images locally.