5 ms·

I recently (yesterday) ran into emulation issues when I tried to run my dedicated game server build for Linux (that I produced from Unity) inside an amazon Linu

I recently (yesterday) ran into emulation issues when I tried to run my dedicated game server build for Linux (that I produced from Unity) inside an amazon Linux docker container on my M2 Mac. I use docker for apple silicon that supposedly allows you to run amd64 images on arm (which M1/M2 macs are). Yes even specifying the correct —-platform and upgrading the emulation software. It was only when I opened up all logging in a dev build did I discover there was some syscall from the mono side of things that wasn’t emulated properly enough. I thought it ironic recalling how many people had sold containers as lighter weight VMs that remove “works on my machine” hassles. Anyways, I just spun up an AWS EC2 instance to test this nothing part of the process. I know podman uses qemu and maybe it could run this Unity based workload and maybe I could uninstall docker but eesh the boring image tech is starting to become a bit of a hassle. I’ll probably just keep using EC2 instances or write up a vagrant file to create a local VM. I’m sure eventually my workload will be fully supported eventually so I can optionally just wait. I love and hate knowing so much.

ksala_ 3y ago
Isn't Docker for Mac just using VMs in the background?
- fathyb 3y ago
  It is, and Docker for Mac uses QEMU for `--platform`, just like Podman AFAIK. The issue here is that it's QEMU user-space emulation, which does not support some syscalls, like `ioctl`. The fix here would be to spin an amd64 VM and run the binary or install Docker there, instead of relying on the more convenient - but less compatible - user-space emulation of Docker. Not very related to this very interesting presentation on virtualizing ARM macOS guests on QEMU hosts..
  - zbentley 3y ago
    > Docker for Mac uses QEMU for `--platform` Only on older setups and on the default config. Updated MacOS (13+) and updated docker provides an option to use Rosetta and the new virtualization system. That, I believe, replaces QEMU for x86 emulation with the Apple emulator. I've had substantial success with that method; not only does it seem a bit faster, but some containerized apps that very frequently failed to start (e.g. Apache Pulsar) work consistently in that mode. I suspect this difference in experience speaks to bugs in QEMU that trigger in the presence of certain application behavior (in Pulsar's case, something weird happening with the JVM on startup). QEMU's still an incredible feat of engineering and something I frequently use on Linux, though. It just seems that the Apple emulation on Macs is better (which makes sense, given that it's built by the people who made the semi-proprietary silicon it's emulating from).
    - xp84 3y ago
      Can I ask a question? It seems like your post is combining two separate things. As I understand it and based on earlier comments in the thread: 1. To run a Linux container you need to be running Linux 2. This is why Docker for Max has always run a VM under the covers. It uses qemu. 3. This was true even before Apple Silicon when everything was x86_64 and nothing was emulated Now you are bringing up Rosetta and x86 emulation. While I understand sometimes it’s necessary to run a container that hasn’t been compiled for aarch64 (or whatever they call Apple) so I can see why the actual cpu emulation capabilities of qemu or Rosetta would be necessary there, this seems to be beside the point. if you have no Linux system involved because you have no VM, how would that work? What would the base system “under” the container be and what would respond to Linux’s OS-level calls?
      - zbentley 3y ago
        This is the core of my challenges: > sometimes it’s necessary to run a container that hasn’t been compiled for aarch64 I have several containers in that boat, for which no ARM versions exist at all. Those containers were previously using QEMU to emulate x86 on ARM linux (in Docker's VM on an ARM Mac). That emulation encountered the failures I described, and was slow. Switching to use Rosetta 2 in Docker solved the problems; the process for enabling it is described here: https://levelup.gitconnected.com/docker-on-apple-silicon-mac-how-to-run-x86-containers-with-rosetta-2-4a679913a0d5 https://levelup.gitconnected.com/docker-on-apple-silicon-mac... Something I don't know and am curious about is how Docker-for-Mac is actually using Rosetta 2. Is it running an additional Linux VM containing an x86 Linux OS, and running that VM through a Rosetta-2-enabled hypervisor? Or is Rosetta 2 distributed as a Linux program that is being invoked inside Docker's pre-existing aarch64 Linux VM instead of (or inside of?) QEMU? Edit: as for your question: > if you have no Linux system involved because you have no VM, how would that work? That's not my situation, so I'm not sure.
        sgerenser 3y ago
        Yes, Rosetta 2 is being used by Docker’s pre-existing aarch64 Linux VM. So it’s only translating userspace x86 code to Arm, then running it on the ARM Linux VM. Syscalls, etc. are still handled by the (aarch64) Linux VM.
- deleted 3y ago
  [deleted]
magic_hamster 3y ago
Containers can and do remove the "it works on my machine" problem when you're on the same platform, and preferably the same OS exactly. The big deal here at least in my experience is the ability to completely replicate your environment, however you still need the underlying OS, because docker is not a VM. It sounds to me like you're running a container on a massively different architecture and I think it's still pretty cool that it runs at all. Obviously when trying to run a game, which needs access to hardware and system calls beyond the use case of a simple CLI tool, running the container on a different architecture becomes more of a long shot. I think given the circumstances it's actually kind of amazing you got this far, and if you're able to go all the way with your game with emulation, that'd be pretty cool.
- saagarjha 3y ago
  Docker is a VM on macOS
  - verst 3y ago
    And on M1/M2 Macs that means an arm64 Linux VM. So whatever issue came up here probably would come up attempting to run amd64 containers on regular arm64 linux (assuming the right compatibility tooling is installed).
duped 3y ago
> I thought it ironic recalling how many people had sold containers as lighter weight VMs that remove “works on my machine” hassles. Containers aren't VMs, they're Linux namespaces. If you want to run a container you need Linux. If you want to run Linux on MacOS you need virtualization. There's no "lightweight" VM about it - it's still a VM.
- 2OEH8eoCRo0 3y ago
  Containers are virtualization without the machine part.
  - monocasa 3y ago
    When running Linux containers on Mac, the containers run within a full virtual machine.
    - hiatus 3y ago
      Docker Desktop on Linux works this way, too.
  - sleepybrett 3y ago
    While the machine part may not matter, the OS certainly does. Linux containers depend on the Linux kernel which is external to the container.
  - packetlost 3y ago
    No, they are not. Containers are namespaced process groups (CGroups) with a sandboxed filesystem and (optionally) sandboxed network layer via bridge interfaces and virtual devices.
    - 2OEH8eoCRo0 3y ago
      What is virtualization to you? https://en.wikipedia.org/wiki/Docker_(software) https://en.wikipedia.org/wiki/Docker_(software) > Docker is a set of platform as a service products that use OS-level virtualization https://www.netapp.com/devops-solutions/what-are-containers https://www.netapp.com/devops-solutions/what-are-containers > Containers are a form of operating system virtualization.
Steltek 3y ago
Even when the arch matched, macOS was a bad platform for running containers. I'm not even sure you can call them "edge cases" because it was basic things like volumes not working right. macOS is engineered for users and desktops, not servers and development. Apple's long history of blocking virtualizing macOS is a testament to this.
- mannyv 3y ago
  MacOS has been virtualizable in x86 since mac-on-x86 was released. It also is engineered for development. Maybe not your particular type of development. Apple silicon isn't technically development-unfriendly, it's just apple silicon makes it less easy to do x86 development. But so does mips, sparc, ppc, and other architectures.
  - smoldesu 3y ago
    MacOS can be simultaneously "engineered for development" and "a worse container host than a chromebook". They aren't mutually exclusive.
  - Steltek 3y ago
    Do you understand the limits apple put on virtualization? Are you aware of how it's handled for cloud providers? I didn't say it was technically impossible. I said apple has a history of blocking it.
    - naikrovek 3y ago
      maximum of two virtual machines running MacOS on a single Apple device, no matter how it is configured. this is in the EULA and is enforced by Virtualization.Framework. cloud providers get no exception to this.
worksonmine 3y ago
> I thought it ironic recalling how many people had sold containers as lighter weight VMs That's your problem there, containers are by design not VMs. Anyone who sold them as such aren't understanding the tech and you shouldn't have blindly trusted them.
Thoeu388 3y ago
Why not just use native Amd64 Linux machine for development? Using emulators and syscall debugger is not very productive experience!
- sovietmudkipz 3y ago
  That’s kinda the conclusion I came to. Being apart of the bleeding edge is fun in some senses but I need to get moving quicker. That’s why I just said I’d spin up an EC2 instance that will 100% run on amd64 cpu arch. I’ll be exploring vagrant but if it too fails I have workarounds. That said I’m not really planning on hauling two computers with me just in case I want to test something on an intel processor.
saagarjha 3y ago
This should work, since it spins up a full VM to run your container.

ksala_ 3y ago

Isn't Docker for Mac just using VMs in the background?

fathyb 3y ago
It is, and Docker for Mac uses QEMU for `--platform`, just like Podman AFAIK. The issue here is that it's QEMU user-space emulation, which does not support some syscalls, like `ioctl`. The fix here would be to spin an amd64 VM and run the binary or install Docker there, instead of relying on the more convenient - but less compatible - user-space emulation of Docker. Not very related to this very interesting presentation on virtualizing ARM macOS guests on QEMU hosts..
- zbentley 3y ago
  > Docker for Mac uses QEMU for `--platform` Only on older setups and on the default config. Updated MacOS (13+) and updated docker provides an option to use Rosetta and the new virtualization system. That, I believe, replaces QEMU for x86 emulation with the Apple emulator. I've had substantial success with that method; not only does it seem a bit faster, but some containerized apps that very frequently failed to start (e.g. Apache Pulsar) work consistently in that mode. I suspect this difference in experience speaks to bugs in QEMU that trigger in the presence of certain application behavior (in Pulsar's case, something weird happening with the JVM on startup). QEMU's still an incredible feat of engineering and something I frequently use on Linux, though. It just seems that the Apple emulation on Macs is better (which makes sense, given that it's built by the people who made the semi-proprietary silicon it's emulating from).
  - xp84 3y ago
    Can I ask a question? It seems like your post is combining two separate things. As I understand it and based on earlier comments in the thread: 1. To run a Linux container you need to be running Linux 2. This is why Docker for Max has always run a VM under the covers. It uses qemu. 3. This was true even before Apple Silicon when everything was x86_64 and nothing was emulated Now you are bringing up Rosetta and x86 emulation. While I understand sometimes it’s necessary to run a container that hasn’t been compiled for aarch64 (or whatever they call Apple) so I can see why the actual cpu emulation capabilities of qemu or Rosetta would be necessary there, this seems to be beside the point. if you have no Linux system involved because you have no VM, how would that work? What would the base system “under” the container be and what would respond to Linux’s OS-level calls?
    - zbentley 3y ago
      This is the core of my challenges: > sometimes it’s necessary to run a container that hasn’t been compiled for aarch64 I have several containers in that boat, for which no ARM versions exist at all. Those containers were previously using QEMU to emulate x86 on ARM linux (in Docker's VM on an ARM Mac). That emulation encountered the failures I described, and was slow. Switching to use Rosetta 2 in Docker solved the problems; the process for enabling it is described here: https://levelup.gitconnected.com/docker-on-apple-silicon-mac-how-to-run-x86-containers-with-rosetta-2-4a679913a0d5 https://levelup.gitconnected.com/docker-on-apple-silicon-mac... Something I don't know and am curious about is how Docker-for-Mac is actually using Rosetta 2. Is it running an additional Linux VM containing an x86 Linux OS, and running that VM through a Rosetta-2-enabled hypervisor? Or is Rosetta 2 distributed as a Linux program that is being invoked inside Docker's pre-existing aarch64 Linux VM instead of (or inside of?) QEMU? Edit: as for your question: > if you have no Linux system involved because you have no VM, how would that work? That's not my situation, so I'm not sure.
      - sgerenser 3y ago
        Yes, Rosetta 2 is being used by Docker’s pre-existing aarch64 Linux VM. So it’s only translating userspace x86 code to Arm, then running it on the ARM Linux VM. Syscalls, etc. are still handled by the (aarch64) Linux VM.
deleted 3y ago
[deleted]

magic_hamster 3y ago

Containers can and do remove the "it works on my machine" problem when you're on the same platform, and preferably the same OS exactly. The big deal here at least in my experience is the ability to completely replicate your environment, however you still need the underlying OS, because docker is not a VM. It sounds to me like you're running a container on a massively different architecture and I think it's still pretty cool that it runs at all. Obviously when trying to run a game, which needs access to hardware and system calls beyond the use case of a simple CLI tool, running the container on a different architecture becomes more of a long shot. I think given the circumstances it's actually kind of amazing you got this far, and if you're able to go all the way with your game with emulation, that'd be pretty cool.

saagarjha 3y ago
Docker is a VM on macOS
- verst 3y ago
  And on M1/M2 Macs that means an arm64 Linux VM. So whatever issue came up here probably would come up attempting to run amd64 containers on regular arm64 linux (assuming the right compatibility tooling is installed).

duped 3y ago

> I thought it ironic recalling how many people had sold containers as lighter weight VMs that remove “works on my machine” hassles. Containers aren't VMs, they're Linux namespaces. If you want to run a container you need Linux. If you want to run Linux on MacOS you need virtualization. There's no "lightweight" VM about it - it's still a VM.

2OEH8eoCRo0 3y ago
Containers are virtualization without the machine part.
- monocasa 3y ago
  When running Linux containers on Mac, the containers run within a full virtual machine.
  - hiatus 3y ago
    Docker Desktop on Linux works this way, too.
- sleepybrett 3y ago
  While the machine part may not matter, the OS certainly does. Linux containers depend on the Linux kernel which is external to the container.
- packetlost 3y ago
  No, they are not. Containers are namespaced process groups (CGroups) with a sandboxed filesystem and (optionally) sandboxed network layer via bridge interfaces and virtual devices.
  - 2OEH8eoCRo0 3y ago
    What is virtualization to you? https://en.wikipedia.org/wiki/Docker_(software) https://en.wikipedia.org/wiki/Docker_(software) > Docker is a set of platform as a service products that use OS-level virtualization https://www.netapp.com/devops-solutions/what-are-containers https://www.netapp.com/devops-solutions/what-are-containers > Containers are a form of operating system virtualization.

Steltek 3y ago

Even when the arch matched, macOS was a bad platform for running containers. I'm not even sure you can call them "edge cases" because it was basic things like volumes not working right. macOS is engineered for users and desktops, not servers and development. Apple's long history of blocking virtualizing macOS is a testament to this.

mannyv 3y ago
MacOS has been virtualizable in x86 since mac-on-x86 was released. It also is engineered for development. Maybe not your particular type of development. Apple silicon isn't technically development-unfriendly, it's just apple silicon makes it less easy to do x86 development. But so does mips, sparc, ppc, and other architectures.
- smoldesu 3y ago
  MacOS can be simultaneously "engineered for development" and "a worse container host than a chromebook". They aren't mutually exclusive.
- Steltek 3y ago
  Do you understand the limits apple put on virtualization? Are you aware of how it's handled for cloud providers? I didn't say it was technically impossible. I said apple has a history of blocking it.
  - naikrovek 3y ago
    maximum of two virtual machines running MacOS on a single Apple device, no matter how it is configured. this is in the EULA and is enforced by Virtualization.Framework. cloud providers get no exception to this.

worksonmine 3y ago

> I thought it ironic recalling how many people had sold containers as lighter weight VMs That's your problem there, containers are by design not VMs. Anyone who sold them as such aren't understanding the tech and you shouldn't have blindly trusted them.

Thoeu388 3y ago

Why not just use native Amd64 Linux machine for development? Using emulators and syscall debugger is not very productive experience!

sovietmudkipz 3y ago
That’s kinda the conclusion I came to. Being apart of the bleeding edge is fun in some senses but I need to get moving quicker. That’s why I just said I’d spin up an EC2 instance that will 100% run on amd64 cpu arch. I’ll be exploring vagrant but if it too fails I have workarounds. That said I’m not really planning on hauling two computers with me just in case I want to test something on an intel processor.

saagarjha 3y ago

This should work, since it spins up a full VM to run your container.