5 ms·

I'm excited about arm in more places but my experience with arm and docker isn't as easy as i expected Is it just me? When I've started using arm more, I've no

by xrd 3y ago

I'm excited about arm in more places but my experience with arm and docker isn't as easy as i expected Is it just me? When I've started using arm more, I've noticed that docker images are often incomplete or behind the x86 release cycle. I love the ease of wiring docker images together for all my services (corollary: never having to understand the myriad packaging issues with whatever language the service is written in, python, nodejs, etc). But when I'm using an arm image, often it is not the same version as the latest on x86, or even worse, is packaged by someone random on the internet. If I were to install the JavaScript service myself, I could audit it (not that I ever do!) by looking into the package.json file, or reviewing the source code, or whatever. There is a clear path to reviewing it. But with a docker image from the Internet, I'm not sure how I would assert it is a well behaved service. Docker itself gives me some guarantees, but it still feels less straightforward. I've packaged things for an arm container myself and it isn't always exactly the same as for x86. Is this just me? Am I doing it wrong on arm?

shepherdjerred 3y ago
I felt this a year or two back, but today I've had as good of an experience on Docker w/ arm64 as I do w/ x86_64. I use arm64 Docker a lot since I work on a M1 MacBook. I usually stick to the common base images, e.g. ubuntu, alpine, nodejs, golang, etc. and install based off of that. Also, I rarely write Dockerfiles these days and instead use Earthly [0], which is a tool that really shines as a CI/make alternative, but it incidentally also has a nicer syntax which makes it easier to write multi-platform Docker images. What images or other problems have you ran into on arm64? [0]: https://earthly.dev/ https://earthly.dev/
- xrd 3y ago
  For example gitlab. The latest arm image, as far as I could tell, isn't the same as the most recent x86. And, iirc, it was from some other person, not gitlab. It's often hard to tell what you are getting when you run an image, because docker pull can pull an image that isn't a multi platform build. I've had issues where the SSL certificates don't work, and I'm assuming it is because the stack could listen on 443, but the full ssl when running on arm didn't work. I'm not sure if that was because it is emulating using Rosetta or whether the software inside the container built correctly but isn't actually running on the arm platform correctly or what. It just feels like the wild west with arm images right now. I'm sure it will get better but it is still a minority platform and that comes with those issues. And, this might just be exposing my ignorance. Until recently I hadn't needed to use arm but now with macos it's gotten more interesting and more complicated.
  - yjftsjthsd-h 3y ago
    > And, iirc, it was from some other person, not gitlab That would have to be a different image, then?
    - xrd 3y ago
      Yes, my memory is a bit foggy, but it was difficult to get any of the images to work, so I started playing with other contributors. But, you are right.
  - shepherdjerred 3y ago
    > The latest arm image, as far as I could tell, isn't the same as the most recent x86. And, iirc, it was from some other person, not gitlab. Ah, that's fair. If you're running software packaged by others, that might be less well-covered because you'll have to wait until all of the vendors you care about add in that supprort. If you're developing software in arm64 Docker, I think that case is pretty good today.
morrbo 3y ago
You're not wrong...but it will get there and get better. I believe asahi will be a driving force behind it, and arm in general being more widely used for non mobile device stuff....however (despite using fedora on arm64 as a daily driver) I firmly believe we're going to be 6-12 months absolute minimum until arm docker is "alright" (I'm also broadly including fully user transparent x86 emulation into this sweeping statement with no basis lol)
bsnnkv 3y ago
The friction with using Docker across arm and x86 was one of the big reasons that I ended up learning NixOS. Now, all the services on my personal remote box and all my one-man-SaaS services run on NixOS + systemd services and my life is so much easier and less stressful.
acdha 3y ago
I don’t have too many gaps but also don’t use that many different base containers for security and reliability reasons. As you mentioned, I feel like in a decade the current experience of running random code from strangers all over the internet with no more protection than Docker Desktop provides is gong to sound similar to 1970s swingers’ accounts of unprotected orgies sound to all of us who grew up after HIV, etc. where people will kind of accept that it happened but be amazed that everyone was so reckless.
znpy 3y ago
I’m not excited that much yet, because except for dumb single board computers, i still can’t get a proper arm system to run at home. My home server (a repurposed fujitsu esprimo q920) is still intel based and it doesn’t seem to be anything available with comparable performance and connectivity. And I’m not even considering the price point. Basically: arm cpus don’t play any significant role in my everyday computing life. At work, I’ve been migrating all of our infra to graviton and we realised substantial savings… but then again, I don’t pay the cloud bills and my salary is still the same, so meh.
jeroenhd 3y ago
You can inspect the layers of a Docker image. Tools like dive[0] provide a quick and easy way to navigate through the different components your image of choice is made up of. In terms of functionality once the container is running, you'll have to put some amount of trust into the project maintainers, no more or less than the trust you need om amd64. For containers repackaged by third parties that's quite a pain, but in most cases you can get by just fine with the official container. If your container of choice has been made by someone real fancy, you may be able to get reproducible builds for all the files inside the container. That would verify that the source and the binary match (though container metadata may not, so a direct image compare would be challenging). [0]: https://github.com/wagoodman/dive https://github.com/wagoodman/dive
- pjmlp 3y ago
  Dive seems to have been abandoned though. I used it a few times in the past.
  - rendaw 3y ago
    Does it no longer work? I thought I used it just fine a couple weeks ago.
    - pjmlp 3y ago
      It still works, until there are OCI updates that it can't handle, and there are a couple of occasional bugs, depending on the image.
jrockway 3y ago
This sounds about right to me. At work, we make a rather complex stack that uses quite a few third-party containers. When we wanted to do arm64 support a couple years ago, most of these dependencies did not support arm64, so we had to build and publish the containers ourselves. (We already sort of had to do this anyway, because customers ran into Docker rate limiting issues, and images from our account aren't rate limited because we pay them not to. But when we only supported amd64, we just re-tagged and pushed.) As an aside, some comments in this thread say "just look at the layers", but that's the wrong level of abstraction for multi-arch images. In the past, when you ran "docker pull ..." you were looking for an Image Manifest: https://github.com/opencontainers/image-spec/blob/main/manifest.md https://github.com/opencontainers/image-spec/blob/main/manif.... But now in the world of multi-arch, you are getting an Image Index first: https://github.com/opencontainers/image-spec/blob/main/image-index.md https://github.com/opencontainers/image-spec/blob/main/image...
harha_ 3y ago
I haven't had problems because I just build images myself, using images such as alpine as a base.
fzeindl 3y ago
> never having to understand the myriad packaging issues with whatever language the service is written in, python, nodejs, etc) How do you fix issues with the docker images if you don't understand them?
arjvik 3y ago
I face similar problems running a docker-based NAS on a Raspberry Pi. But I end up just building the official images myself on the Pi (or on my dev machine with qemu) from the open source Dockerfile of the official image.
Plasmoid 3y ago
One thing that made arm on docker much easier was by using the kubernetes builder for docker. Spin up an arm nodes in kubernetes, create the docker builder pod, and it'll build/push your docker image easy as can be.
predictabl3 3y ago
Sounds like a docker issue, not an ARM issue. My full desktop NixOS config builds for x86_64-linux and aarch64-linux. It even about 90% cross compiles, possibly just one "external" package that isn't setup right for it. And actually that might even be fixed, I just saw a cross-compilation fix go in today.

by xrd 3y ago

shepherdjerred 3y ago

I felt this a year or two back, but today I've had as good of an experience on Docker w/ arm64 as I do w/ x86_64. I use arm64 Docker a lot since I work on a M1 MacBook. I usually stick to the common base images, e.g. ubuntu, alpine, nodejs, golang, etc. and install based off of that. Also, I rarely write Dockerfiles these days and instead use Earthly [0], which is a tool that really shines as a CI/make alternative, but it incidentally also has a nicer syntax which makes it easier to write multi-platform Docker images. What images or other problems have you ran into on arm64? [0]: https://earthly.dev/ https://earthly.dev/

xrd 3y ago
For example gitlab. The latest arm image, as far as I could tell, isn't the same as the most recent x86. And, iirc, it was from some other person, not gitlab. It's often hard to tell what you are getting when you run an image, because docker pull can pull an image that isn't a multi platform build. I've had issues where the SSL certificates don't work, and I'm assuming it is because the stack could listen on 443, but the full ssl when running on arm didn't work. I'm not sure if that was because it is emulating using Rosetta or whether the software inside the container built correctly but isn't actually running on the arm platform correctly or what. It just feels like the wild west with arm images right now. I'm sure it will get better but it is still a minority platform and that comes with those issues. And, this might just be exposing my ignorance. Until recently I hadn't needed to use arm but now with macos it's gotten more interesting and more complicated.
- yjftsjthsd-h 3y ago
  > And, iirc, it was from some other person, not gitlab That would have to be a different image, then?
  - xrd 3y ago
    Yes, my memory is a bit foggy, but it was difficult to get any of the images to work, so I started playing with other contributors. But, you are right.
- shepherdjerred 3y ago
  > The latest arm image, as far as I could tell, isn't the same as the most recent x86. And, iirc, it was from some other person, not gitlab. Ah, that's fair. If you're running software packaged by others, that might be less well-covered because you'll have to wait until all of the vendors you care about add in that supprort. If you're developing software in arm64 Docker, I think that case is pretty good today.

morrbo 3y ago

You're not wrong...but it will get there and get better. I believe asahi will be a driving force behind it, and arm in general being more widely used for non mobile device stuff....however (despite using fedora on arm64 as a daily driver) I firmly believe we're going to be 6-12 months absolute minimum until arm docker is "alright" (I'm also broadly including fully user transparent x86 emulation into this sweeping statement with no basis lol)

bsnnkv 3y ago

The friction with using Docker across arm and x86 was one of the big reasons that I ended up learning NixOS. Now, all the services on my personal remote box and all my one-man-SaaS services run on NixOS + systemd services and my life is so much easier and less stressful.

acdha 3y ago

I don’t have too many gaps but also don’t use that many different base containers for security and reliability reasons. As you mentioned, I feel like in a decade the current experience of running random code from strangers all over the internet with no more protection than Docker Desktop provides is gong to sound similar to 1970s swingers’ accounts of unprotected orgies sound to all of us who grew up after HIV, etc. where people will kind of accept that it happened but be amazed that everyone was so reckless.

znpy 3y ago

I’m not excited that much yet, because except for dumb single board computers, i still can’t get a proper arm system to run at home. My home server (a repurposed fujitsu esprimo q920) is still intel based and it doesn’t seem to be anything available with comparable performance and connectivity. And I’m not even considering the price point. Basically: arm cpus don’t play any significant role in my everyday computing life. At work, I’ve been migrating all of our infra to graviton and we realised substantial savings… but then again, I don’t pay the cloud bills and my salary is still the same, so meh.

jeroenhd 3y ago

You can inspect the layers of a Docker image. Tools like dive[0] provide a quick and easy way to navigate through the different components your image of choice is made up of. In terms of functionality once the container is running, you'll have to put some amount of trust into the project maintainers, no more or less than the trust you need om amd64. For containers repackaged by third parties that's quite a pain, but in most cases you can get by just fine with the official container. If your container of choice has been made by someone real fancy, you may be able to get reproducible builds for all the files inside the container. That would verify that the source and the binary match (though container metadata may not, so a direct image compare would be challenging). [0]: https://github.com/wagoodman/dive https://github.com/wagoodman/dive

pjmlp 3y ago
Dive seems to have been abandoned though. I used it a few times in the past.
- rendaw 3y ago
  Does it no longer work? I thought I used it just fine a couple weeks ago.
  - pjmlp 3y ago
    It still works, until there are OCI updates that it can't handle, and there are a couple of occasional bugs, depending on the image.

jrockway 3y ago

This sounds about right to me. At work, we make a rather complex stack that uses quite a few third-party containers. When we wanted to do arm64 support a couple years ago, most of these dependencies did not support arm64, so we had to build and publish the containers ourselves. (We already sort of had to do this anyway, because customers ran into Docker rate limiting issues, and images from our account aren't rate limited because we pay them not to. But when we only supported amd64, we just re-tagged and pushed.) As an aside, some comments in this thread say "just look at the layers", but that's the wrong level of abstraction for multi-arch images. In the past, when you ran "docker pull ..." you were looking for an Image Manifest: https://github.com/opencontainers/image-spec/blob/main/manifest.md https://github.com/opencontainers/image-spec/blob/main/manif.... But now in the world of multi-arch, you are getting an Image Index first: https://github.com/opencontainers/image-spec/blob/main/image-index.md https://github.com/opencontainers/image-spec/blob/main/image...

harha_ 3y ago

I haven't had problems because I just build images myself, using images such as alpine as a base.

fzeindl 3y ago

> never having to understand the myriad packaging issues with whatever language the service is written in, python, nodejs, etc) How do you fix issues with the docker images if you don't understand them?

arjvik 3y ago

I face similar problems running a docker-based NAS on a Raspberry Pi. But I end up just building the official images myself on the Pi (or on my dev machine with qemu) from the open source Dockerfile of the official image.

Plasmoid 3y ago

One thing that made arm on docker much easier was by using the kubernetes builder for docker. Spin up an arm nodes in kubernetes, create the docker builder pod, and it'll build/push your docker image easy as can be.

predictabl3 3y ago

Sounds like a docker issue, not an ARM issue. My full desktop NixOS config builds for x86_64-linux and aarch64-linux. It even about 90% cross compiles, possibly just one "external" package that isn't setup right for it. And actually that might even be fixed, I just saw a cross-compilation fix go in today.