5 ms·

I ended up just getting a System 76 Meerkat computer for doing anything locally with LXC/Docker. Linux emulation in Windows or macOS is unbearably bad if you r

I ended up just getting a System 76 Meerkat computer for doing anything locally with LXC/Docker. Linux emulation in Windows or macOS is unbearably bad if you really need to do serious work.

LoganDark 4y ago
Even systemd will run in Docker on macOS. Can you clarify what makes macOS unacceptable for emulating Linux?
- cmeacham98 4y ago
  Using a VM for Docker gives you the worst of both worlds. You get all the downsides of a VM (persistent wasted RAM/disk space, awful disk IO performance, etc) AND all the downsides of containers (difficult to introspect when your minimal image has no shell, janky networking, etc). On recent M1/M2 Macs you take the additional pain (and thus performance hit) of translation to x86 (because that's almost certainly what your containers are in.
  - inkyoto 4y ago
    > On recent M1/M2 Macs you take the additional pain (and thus performance hit) of translation to x86 (because that's almost certainly what your containers are in. What on Earth are you talking about? arm64 Docker images predate the emergence of M1, and their number has proliferated in the last two years like there was no tomorrow. Here is an excerpt from my own tiny, non-representative collection of Docker images I have had to deal with lately: $ (for i in `docker images | tail -n +2 | awk '{ printf "%s:%s\n", $1, $2 }'`; do docker inspect $i | jq '.[].Architecture'; done) | grep -c amd64 7 $ (for i in `docker images | tail -n +2 | awk '{ printf "%s:%s\n", $1, $2 }'`; do docker inspect $i | jq '.[].Architecture'; done) | grep -c arm64 23 Most of the 23 arm64 Docker images is pretty obscure stuff, with one or two manually rebuilt for the arm64 architecture. Yet, I have downloaded all of my obscure images directly from the Docker Hub – not traded them for a pinch of spice from a shady Docker image dealer at a drench inducing corner of the dark web. All of the arm64 images run at the native speed, locally and in the cloud. The only reason I store x86 (i.e. «amd64») images locally is my laptop being a transient transit point between their source and the destination. I would have rebuilt them as well but the build process is either broken for each of them or is too convoluted to justify spending a time trying to fix it.
    - nxrabl 4y ago
      But then it's not the same container, right? Either you'd need to switch your whole cloud infrastructure to run on ARM (which I see is possible now, and I'd be curious to hear if people have done that successfully) or we're back to "works on my machine ^W architecture".
      - inkyoto 4y ago
        > Either you'd need to switch your whole cloud infrastructure to run on ARM … Nope, the whole cloud infrastructure does not need to switch. Since containers are fully self-contained, self-sufficient and isolated from each other, they can coexist peacefully and run alongside each other in a mixed setup in AWS ECS/Fargate clusters. And they do. AWS Lambda also supports aarch64, and the degree of isolation for the lambdas is even higher and they can also run out of a x86 or an aarch64 Docker image (if required), and 30-40% cheaper with performance loss incurred for the workloads I am interested in.
        inkyoto 4y ago
        > […] and 30-40% cheaper with performance loss incurred for the workloads I am interested in. Should read: «… with no performance loss incurred for the workloads I am interested in». A typo crept in.
    - cmeacham98 4y ago
      I was exaggerating slightly - you will get some images in ARM. But it's hard to not see this as evidence for my point when you were only able to rebuild "one or two" of the 8-9 amd64 images you're running without significant time investment. By your own stats roughly one third of the images you use don't support ARM. And if you want to run one of those on ARM - you're likely screwed! Even some _official_ images don't support ARM, random example: https://hub.docker.com/_/percona https://hub.docker.com/_/percona Docker ARM support is improving, but acting like it's ready for prime time is laughable. Try doing your day job with no amd64 containers for any reason for a month and come back to me.
  - msbarnett 4y ago
    At this point I’ve only got one non-arm64 container, and it’s solely optional support for testing some obscure cases that our CI infrastructure handles more robustly anyways. Your information is pretty out of date.
  - notjoemama 4y ago
    > persistent wasted RAM/disk space, awful disk IO performance I'm getting cranky about arguments against VMs. 1) I have had more RAM than I ever use for about 25 years. I upgrade about every 4 to 6 years. Is everyone else writing code on Amigas? 2) I keep getting told by everyone (IT, devops, DBA, my boss, his boss) that disk space is cheap. 3) VirtualBox has been consistently ok for about 10 years and I almost always can outrun scripts in a Linux VM on my Windows machine compared to our production AWS instance running K8s. Of course my machine would never keep up with a high user count but this is about disk performance, and locally I can prove both my monolith and my VM are faster. I don't see how arguments like these are good anymore with the modern hardware of a local dev system. That makes containers an over complication. The only reason to do them is to save devops the headache of having to create a docker file. I mean the guys read books and watch YouTube while I'm writing code, so...they can write the damn docker file.
    - lstodd 4y ago
      Disk space is cheap; IOPS aren't. Same stuff with RAM: it's cheap, but fast access isn't.
      - tracker1 4y ago
        Even then, access is still significantly better on a local machine in a VM than it is on a shared server... You are effectively the only real user, generally speaking. The Gen4 NVME drive and 64gb+ ram on my local system is still better than shared server resources are generally going to be, short of dedicated (ie: expensive) server environments to do the same. I'm not a huge fan of VM development, generally speaking... that said, my current work setup is Windows + WSL + Rancher Desktop + VS Code (remoting extensions) for nearly everything. My windows terminal default is set to WSL, so it's pretty close to in the box... I can launch windows apps from the wsl environment... the integration is pretty seamless. In terms of performance, if you're stuck on an older Windows version (I am), then you can limit the max memory that the WSL environment gets, similar to a VM... I usually reserve 8-12gb for my host environment and give the rest to my dev environment. My personal desktop is Linux (Ubuntu-Budgie). My previous job the laptop was an M1 Max, which had faster ram/disk than the windows laptop for the current job, or my personal desktop (which is impressive to me). In the end, for most things, it's not significant enough to worry about... The aarch macs and i/o have been more problematic than wsl and other x86 environments... I think as rosetta support improves with qemu and docker, it will change and get significantly better. That said, with VS Code's remoting extensions, I can be working on a remote system nearly as effectively as local anyway... which is how I typically use my laptop. Wireguard to home, vs code w/ ssh to my personal desktop.
  - LoganDark 4y ago
    I don't use a VM for Docker. I ran systemd directly inside Docker for macOS. https://github.com/LoganDark/initbuntu https://github.com/LoganDark/initbuntu
    - lloeki 4y ago
      > I don't use a VM for Docker. Yes you are. https://github.com/moby/hyperkit https://github.com/moby/hyperkit (Or maybe you mean a non-Docker For Mac VM)
      - LoganDark 4y ago
        Interesting. Docker for Mac uses that library for virtualization? I suppose that would explain how I was able to get systemd to run, then.

I ended up just getting a System 76 Meerkat computer for doing anything locally with LXC/Docker. Linux emulation in Windows or macOS is unbearably bad if you really need to do serious work.

LoganDark 4y ago

Even systemd will run in Docker on macOS. Can you clarify what makes macOS unacceptable for emulating Linux?

cmeacham98 4y ago
Using a VM for Docker gives you the worst of both worlds. You get all the downsides of a VM (persistent wasted RAM/disk space, awful disk IO performance, etc) AND all the downsides of containers (difficult to introspect when your minimal image has no shell, janky networking, etc). On recent M1/M2 Macs you take the additional pain (and thus performance hit) of translation to x86 (because that's almost certainly what your containers are in.
- inkyoto 4y ago
  > On recent M1/M2 Macs you take the additional pain (and thus performance hit) of translation to x86 (because that's almost certainly what your containers are in. What on Earth are you talking about? arm64 Docker images predate the emergence of M1, and their number has proliferated in the last two years like there was no tomorrow. Here is an excerpt from my own tiny, non-representative collection of Docker images I have had to deal with lately: $ (for i in `docker images | tail -n +2 | awk '{ printf "%s:%s\n", $1, $2 }'`; do docker inspect $i | jq '.[].Architecture'; done) | grep -c amd64 7 $ (for i in `docker images | tail -n +2 | awk '{ printf "%s:%s\n", $1, $2 }'`; do docker inspect $i | jq '.[].Architecture'; done) | grep -c arm64 23 Most of the 23 arm64 Docker images is pretty obscure stuff, with one or two manually rebuilt for the arm64 architecture. Yet, I have downloaded all of my obscure images directly from the Docker Hub – not traded them for a pinch of spice from a shady Docker image dealer at a drench inducing corner of the dark web. All of the arm64 images run at the native speed, locally and in the cloud. The only reason I store x86 (i.e. «amd64») images locally is my laptop being a transient transit point between their source and the destination. I would have rebuilt them as well but the build process is either broken for each of them or is too convoluted to justify spending a time trying to fix it.
  - nxrabl 4y ago
    But then it's not the same container, right? Either you'd need to switch your whole cloud infrastructure to run on ARM (which I see is possible now, and I'd be curious to hear if people have done that successfully) or we're back to "works on my machine ^W architecture".
    - inkyoto 4y ago
      > Either you'd need to switch your whole cloud infrastructure to run on ARM … Nope, the whole cloud infrastructure does not need to switch. Since containers are fully self-contained, self-sufficient and isolated from each other, they can coexist peacefully and run alongside each other in a mixed setup in AWS ECS/Fargate clusters. And they do. AWS Lambda also supports aarch64, and the degree of isolation for the lambdas is even higher and they can also run out of a x86 or an aarch64 Docker image (if required), and 30-40% cheaper with performance loss incurred for the workloads I am interested in.
      - inkyoto 4y ago
        > […] and 30-40% cheaper with performance loss incurred for the workloads I am interested in. Should read: «… with no performance loss incurred for the workloads I am interested in». A typo crept in.
  - cmeacham98 4y ago
    I was exaggerating slightly - you will get some images in ARM. But it's hard to not see this as evidence for my point when you were only able to rebuild "one or two" of the 8-9 amd64 images you're running without significant time investment. By your own stats roughly one third of the images you use don't support ARM. And if you want to run one of those on ARM - you're likely screwed! Even some _official_ images don't support ARM, random example: https://hub.docker.com/_/percona https://hub.docker.com/_/percona Docker ARM support is improving, but acting like it's ready for prime time is laughable. Try doing your day job with no amd64 containers for any reason for a month and come back to me.
- msbarnett 4y ago
  At this point I’ve only got one non-arm64 container, and it’s solely optional support for testing some obscure cases that our CI infrastructure handles more robustly anyways. Your information is pretty out of date.
- notjoemama 4y ago
  > persistent wasted RAM/disk space, awful disk IO performance I'm getting cranky about arguments against VMs. 1) I have had more RAM than I ever use for about 25 years. I upgrade about every 4 to 6 years. Is everyone else writing code on Amigas? 2) I keep getting told by everyone (IT, devops, DBA, my boss, his boss) that disk space is cheap. 3) VirtualBox has been consistently ok for about 10 years and I almost always can outrun scripts in a Linux VM on my Windows machine compared to our production AWS instance running K8s. Of course my machine would never keep up with a high user count but this is about disk performance, and locally I can prove both my monolith and my VM are faster. I don't see how arguments like these are good anymore with the modern hardware of a local dev system. That makes containers an over complication. The only reason to do them is to save devops the headache of having to create a docker file. I mean the guys read books and watch YouTube while I'm writing code, so...they can write the damn docker file.
  - lstodd 4y ago
    Disk space is cheap; IOPS aren't. Same stuff with RAM: it's cheap, but fast access isn't.
    - tracker1 4y ago
      Even then, access is still significantly better on a local machine in a VM than it is on a shared server... You are effectively the only real user, generally speaking. The Gen4 NVME drive and 64gb+ ram on my local system is still better than shared server resources are generally going to be, short of dedicated (ie: expensive) server environments to do the same. I'm not a huge fan of VM development, generally speaking... that said, my current work setup is Windows + WSL + Rancher Desktop + VS Code (remoting extensions) for nearly everything. My windows terminal default is set to WSL, so it's pretty close to in the box... I can launch windows apps from the wsl environment... the integration is pretty seamless. In terms of performance, if you're stuck on an older Windows version (I am), then you can limit the max memory that the WSL environment gets, similar to a VM... I usually reserve 8-12gb for my host environment and give the rest to my dev environment. My personal desktop is Linux (Ubuntu-Budgie). My previous job the laptop was an M1 Max, which had faster ram/disk than the windows laptop for the current job, or my personal desktop (which is impressive to me). In the end, for most things, it's not significant enough to worry about... The aarch macs and i/o have been more problematic than wsl and other x86 environments... I think as rosetta support improves with qemu and docker, it will change and get significantly better. That said, with VS Code's remoting extensions, I can be working on a remote system nearly as effectively as local anyway... which is how I typically use my laptop. Wireguard to home, vs code w/ ssh to my personal desktop.
- LoganDark 4y ago
  I don't use a VM for Docker. I ran systemd directly inside Docker for macOS. https://github.com/LoganDark/initbuntu https://github.com/LoganDark/initbuntu
  - lloeki 4y ago
    > I don't use a VM for Docker. Yes you are. https://github.com/moby/hyperkit https://github.com/moby/hyperkit (Or maybe you mean a non-Docker For Mac VM)
    - LoganDark 4y ago
      Interesting. Docker for Mac uses that library for virtualization? I suppose that would explain how I was able to get systemd to run, then.