5 ms·

Buildx also has a completely different build output and it's completely useless for anything. Use `--output=plain`. Also, buildx won't commit the intermediate

Buildx also has a completely different build output and it's completely useless for anything. Use `--output=plain`. Also, buildx won't commit the intermediate layers during the build. So if something fails, you can't just grab the previous intermediate layer and do `docker run --entrypoint /bin/bash` on that layer to poke around. I've had cases where BuildKit will get stuck and `docker buildx prune -a` will leave things around. The way to fix it is with `systemctl restart containerd.io`. This seems to happen a lot when I interrupt the build (eg, I know it will fail and need to restart it with a fix), but it's also happened on successful builds. All of that is on top of docker itself having extremely poor error messages. I really do not like docker or buildx.

adamgordonbell 3y ago
If you build your dockerfile with earthly, we have a -i flag that will pop you into a shell on a failure, and we are using buildkit, so it's totally possible.
4oo4 3y ago
Well said, these are the reasons I use podman/buildah now.
- EdwardDiego 3y ago
  Was it a drop in replacement or did you need to give things a jiggle to get it going?
  - Narushia 3y ago
    I've had a good experience with Podman as a Docker alternative. The Podman CLI is designed to be API-compatible with Docker's CLI, so most things should just work. The only thing I've had to change with my Dockerfiles (or Containerfiles) is the `SHELL` instruction, since Podman by default builds OCI images, and that instruction is not supported by the OCI standard. But you can still specify the shell individually for the `RUN` instructions, or even just use `--format=docker` if you don't want OCI images.
  - abound 3y ago
    Similar to sibling comment, mostly just works out of the box. Things I've noticed that are different: - There's no docker.sock API socket by default, which makes sense because Podman doesn't have Docker's daemon-based architecture. You can run one with `podman system service [...]` if you have non-CLI clients expecting to connect to some $DOCKER_HOST - Mounts/volumes behave subtly differently IIRC, I had a bunch of scripts that didn't work with podman-as-docker out of the box, I think the issue was that Docker will create some directories automatically that Podman won't? Aside from that, rootless Podman is great and I've had far fewer issues with it than I did for rootless Docker, only real caveat is that -p <some port> doesn't actually connect to the host, it's still namespaced. You have to use --net=host for actual host networking.
    - nicce 3y ago
      Only reason I have used buildx is the easy multi-arch for the same tag (big manifest). How to do this with Podman/buildah? The mount/volume issues are probably caused by SELinux / other security decisions
  - floating-io 3y ago
    I recently took the plunge in a couple of my CI pipelines and had little trouble. The only thing that actually broke was that the `--cache-from` option doesn't accept a tag, where it does for docker. Everything else pretty much Just Worked. That said, my pipelines are fairly simple, as are my Dockerfiles, so... It shaved significant time off my pipelines, though, not having to wait for a docker-in-docker service to spin up (I use Gitlab with Kubernetes runners).
- throwaway894345 3y ago
  Honestly when I dug into the podman/buildah code it was just differently cursed. It was reallly disappointing.
cpuguy83 3y ago
"BUILDX_EXPERIMENTAL=1 docker buildx build <args> --invoke /bin/sh" There's actually a whole debugging client available.
- radus 3y ago
  This is great! I’ve been waiting a long time for this and it seems like more debug features are in the pipeline too: https://github.com/docker/buildx/issues/1104 https://github.com/docker/buildx/issues/1104 Docs: https://docs.docker.com/engine/reference/commandline/buildx_debug/ https://docs.docker.com/engine/reference/commandline/buildx_...
mdaniel 3y ago
and its "oh, you want multi-arch, do you?" friend. While prosecuting this <https://gitlab.com/gitlab-org/gitlab/-/issues/339567 https://gitlab.com/gitlab-org/gitlab/-/issues/339567> I learned that they bundle binfmt into the buildkit docker image <https://github.com/moby/buildkit/blob/v0.12.5/Dockerfile#L110 https://github.com/moby/buildkit/blob/v0.12.5/Dockerfile#L11...> which itself is compiled against a static copy of qemu <https://github.com/tonistiigi/binfmt/blob/deploy/v6.2.0-26/Dockerfile#L8 https://github.com/tonistiigi/binfmt/blob/deploy/v6.2.0-26/D...> with various /proc/sys/fs/binfmt_misc wizardry <https://github.com/tonistiigi/binfmt/blob/deploy/v6.2.0-26/cmd/binfmt/main.go https://github.com/tonistiigi/binfmt/blob/deploy/v6.2.0-26/c...> We ended up switching to using multiarch/qemu-user-static:7.2.0-1 <https://hub.docker.com/layers/multiarch/qemu-user-static/7.2.0-1/images/sha256-fe60359c92e86a43cc87b3d906006245f77bfc0565676b80004cc666e4feb9f0?context=explore https://hub.docker.com/layers/multiarch/qemu-user-static/7.2...> which also mutates binfmt_misc in buildx's context in order to exec the static copy of qemu in it but I find their shell script much more legible about what's going on: <https://github.com/multiarch/qemu-user-static/blob/v7.2.0-1/containers/latest/register.sh#L14 https://github.com/multiarch/qemu-user-static/blob/v7.2.0-1/...> not to mention the fact that it more clearly represents the recent qemu version versus secreting it away inside the moby/buildkit:heheh-good-luck-friend They have an entire GH label for that: https://github.com/docker/buildx/labels/area%2Fqemu https://github.com/docker/buildx/labels/area%2Fqemu and my journey into the buildx sewers started from https://github.com/docker/buildx/issues/1170 https://github.com/docker/buildx/issues/1170
- zamalek 3y ago
  I see you're trying to do multi-arch builds. At reasonable (on bare metal) speeds you say? Delete the entire docker stack. It's utter shite. Then install Podman and Buildah, and rejoice! For all the evil buildx magic can now be done explicitly. This solution was born when trying to do native builds on CircleCI - where you do not have a buildx cluster with multiple architectures. You have to build the archs as independent build steps, then pull them together afterwards. This is something that Docker never considered, and Docker only functions on the happy path. Buildah has multiarch manipulation commands: https://danmanners.com/posts/2022-01-buildah-multi-arch/ https://danmanners.com/posts/2022-01-buildah-multi-arch/ (not a plug, first blog on search) We migrated to GHA, but still use separate builders to avoid the Docker trite. We also ditched DCT (Docker signing) for Cosign because - again - DCT is home-grown garbage.
deleted 3y ago
[deleted]
AkihiroSuda 3y ago
> Also, buildx won't commit the intermediate layers during the build. So if something fails, you can't just grab the previous intermediate layer and do `docker run --entrypoint /bin/bash` on that layer to poke around. `buildg debug` (Dockerfile debugger based on BuildKit) to rescue: https://github.com/ktock/buildg https://github.com/ktock/buildg

adamgordonbell 3y ago

If you build your dockerfile with earthly, we have a -i flag that will pop you into a shell on a failure, and we are using buildkit, so it's totally possible.

4oo4 3y ago

Well said, these are the reasons I use podman/buildah now.

EdwardDiego 3y ago
Was it a drop in replacement or did you need to give things a jiggle to get it going?
- Narushia 3y ago
  I've had a good experience with Podman as a Docker alternative. The Podman CLI is designed to be API-compatible with Docker's CLI, so most things should just work. The only thing I've had to change with my Dockerfiles (or Containerfiles) is the `SHELL` instruction, since Podman by default builds OCI images, and that instruction is not supported by the OCI standard. But you can still specify the shell individually for the `RUN` instructions, or even just use `--format=docker` if you don't want OCI images.
- abound 3y ago
  Similar to sibling comment, mostly just works out of the box. Things I've noticed that are different: - There's no docker.sock API socket by default, which makes sense because Podman doesn't have Docker's daemon-based architecture. You can run one with `podman system service [...]` if you have non-CLI clients expecting to connect to some $DOCKER_HOST - Mounts/volumes behave subtly differently IIRC, I had a bunch of scripts that didn't work with podman-as-docker out of the box, I think the issue was that Docker will create some directories automatically that Podman won't? Aside from that, rootless Podman is great and I've had far fewer issues with it than I did for rootless Docker, only real caveat is that -p <some port> doesn't actually connect to the host, it's still namespaced. You have to use --net=host for actual host networking.
  - nicce 3y ago
    Only reason I have used buildx is the easy multi-arch for the same tag (big manifest). How to do this with Podman/buildah? The mount/volume issues are probably caused by SELinux / other security decisions
- floating-io 3y ago
  I recently took the plunge in a couple of my CI pipelines and had little trouble. The only thing that actually broke was that the `--cache-from` option doesn't accept a tag, where it does for docker. Everything else pretty much Just Worked. That said, my pipelines are fairly simple, as are my Dockerfiles, so... It shaved significant time off my pipelines, though, not having to wait for a docker-in-docker service to spin up (I use Gitlab with Kubernetes runners).
throwaway894345 3y ago
Honestly when I dug into the podman/buildah code it was just differently cursed. It was reallly disappointing.

cpuguy83 3y ago

"BUILDX_EXPERIMENTAL=1 docker buildx build <args> --invoke /bin/sh" There's actually a whole debugging client available.

radus 3y ago
This is great! I’ve been waiting a long time for this and it seems like more debug features are in the pipeline too: https://github.com/docker/buildx/issues/1104 https://github.com/docker/buildx/issues/1104 Docs: https://docs.docker.com/engine/reference/commandline/buildx_debug/ https://docs.docker.com/engine/reference/commandline/buildx_...

mdaniel 3y ago

and its "oh, you want multi-arch, do you?" friend. While prosecuting this <https://gitlab.com/gitlab-org/gitlab/-/issues/339567 https://gitlab.com/gitlab-org/gitlab/-/issues/339567> I learned that they bundle binfmt into the buildkit docker image <https://github.com/moby/buildkit/blob/v0.12.5/Dockerfile#L110 https://github.com/moby/buildkit/blob/v0.12.5/Dockerfile#L11...> which itself is compiled against a static copy of qemu <https://github.com/tonistiigi/binfmt/blob/deploy/v6.2.0-26/Dockerfile#L8 https://github.com/tonistiigi/binfmt/blob/deploy/v6.2.0-26/D...> with various /proc/sys/fs/binfmt_misc wizardry <https://github.com/tonistiigi/binfmt/blob/deploy/v6.2.0-26/cmd/binfmt/main.go https://github.com/tonistiigi/binfmt/blob/deploy/v6.2.0-26/c...> We ended up switching to using multiarch/qemu-user-static:7.2.0-1 <https://hub.docker.com/layers/multiarch/qemu-user-static/7.2.0-1/images/sha256-fe60359c92e86a43cc87b3d906006245f77bfc0565676b80004cc666e4feb9f0?context=explore https://hub.docker.com/layers/multiarch/qemu-user-static/7.2...> which also mutates binfmt_misc in buildx's context in order to exec the static copy of qemu in it but I find their shell script much more legible about what's going on: <https://github.com/multiarch/qemu-user-static/blob/v7.2.0-1/containers/latest/register.sh#L14 https://github.com/multiarch/qemu-user-static/blob/v7.2.0-1/...> not to mention the fact that it more clearly represents the recent qemu version versus secreting it away inside the moby/buildkit:heheh-good-luck-friend They have an entire GH label for that: https://github.com/docker/buildx/labels/area%2Fqemu https://github.com/docker/buildx/labels/area%2Fqemu and my journey into the buildx sewers started from https://github.com/docker/buildx/issues/1170 https://github.com/docker/buildx/issues/1170

zamalek 3y ago
I see you're trying to do multi-arch builds. At reasonable (on bare metal) speeds you say? Delete the entire docker stack. It's utter shite. Then install Podman and Buildah, and rejoice! For all the evil buildx magic can now be done explicitly. This solution was born when trying to do native builds on CircleCI - where you do not have a buildx cluster with multiple architectures. You have to build the archs as independent build steps, then pull them together afterwards. This is something that Docker never considered, and Docker only functions on the happy path. Buildah has multiarch manipulation commands: https://danmanners.com/posts/2022-01-buildah-multi-arch/ https://danmanners.com/posts/2022-01-buildah-multi-arch/ (not a plug, first blog on search) We migrated to GHA, but still use separate builders to avoid the Docker trite. We also ditched DCT (Docker signing) for Cosign because - again - DCT is home-grown garbage.

deleted 3y ago

[deleted]

AkihiroSuda 3y ago

> Also, buildx won't commit the intermediate layers during the build. So if something fails, you can't just grab the previous intermediate layer and do `docker run --entrypoint /bin/bash` on that layer to poke around. `buildg debug` (Dockerfile debugger based on BuildKit) to rescue: https://github.com/ktock/buildg https://github.com/ktock/buildg