4 ms·

POSIX sh has “set -u” to handle this category of errors — scripts that attempt to expand an unset parameter will exit with an error message. Perl has “use warn

POSIX sh has “set -u” to handle this category of errors — scripts that attempt to expand an unset parameter will exit with an error message. Perl has “use warnings ‘uninitialized’” for the same reason. It sounds like this feature would be worth considering for the Dockerfile spec.

diarrhea 2y ago
Empty/unset environment variables not, by default, being cause for immediate error and abort is the billion dollar mistake of the DevOps/infra world.
- gorgoiler 2y ago
  I agree but let’s hear the counter arguments so as to not sound like we aren’t listening. Environment variables can be useful. The first version of some code can be oblivious to environment variables and the second version can use them for feature flags. Because it is entirely optional to provide these, the second version of the code can be deployed to the first version’s infrastructure, and the first version will happily ignore environment variables set in the second version’s infrastructure. As a tool for incrementally moving heterogeneous parts of a system forward without needing atomic leaps, the use of environment variables as optional flags is very useful. I just wouldn’t even start from there. Instead I configure everything in code, in one repo, with all new deployments appearing as new clusters where the only thing they have in common is that they take production traffic. This kind of production infra is very useful but there are downsides for sure. It makes database schema changes, for example, something between very hard and anathema to execute.
vbezhenar 2y ago
For this use-case you don't need anything in Dockerfile spec. ARG is not magically expanded inside RUN blocks, it's just an env available during build stage. In other words you can use this code to catch this error: ARG DEBVER="10" ARG CAPVER="7.8" FROM debian:${DEBVER} RUN <<EOF set -eu printf "DEB=${DEBVER}\nCAP=${CAPVER}\n" EOF This code fails with expected error.
- iainmerrick 2y ago
  So now you need to wrap all your RUN lines in here docs? Better defaults and/or an easier way to change the defaults would be good! The previous commenter is right -- it's all very well to say "just read all the docs" or "it's consistent with the way other tools work", but sometimes both the docs and the tools can be made less error-prone for users.
- gorgoiler 2y ago
  That seems like a perfectly good idiom and it’s fantastic to hear that Dockerfiles support heredocs — I had no idea, thank you! https://www.docker.com/blog/introduction-to-heredocs-in-dockerfiles/ https://www.docker.com/blog/introduction-to-heredocs-in-dock... It would still be possible to accidentally reference uninitialized environment variables in contexts other than RUN though. Having those be treated as errors would be useful.
- kmarc 2y ago
  This is unnecessarily complicated. Just use `SHELL` SHELL ["/bin/sh", "-o", "nounset", "-c"] RUN printf "DEB=${DEBVER}\nCAP=${CAPVER}\n"
  - MereInterest 2y ago
    Follow-up question: Does the SHELL need to be set within each stage of a multistage build? Ideally, it could be set once to remove this footgun, but I’m guessing the footgun gets reloaded with each stage.
    - kmarc 2y ago
      Yes. In general, I think of multi-stage builds as if multiple docker files were concatenated; if ARG or SHELL doesn't exist in the theoretical split Dockerfile, then it wouldn't exist in the multistage build. Docker(file) is a thing which brought in some quite neat new ideas, and a lot of these "footguns". I did read the docs, and I am comfortable now building even complicated "Dockerfile-architectures" with proper linting, watching out for good caching, etc. I wish there were better alternatives; I saw images being built with Bazel, and thanks, but that is even more convoluted, with presumably a different set of footguns. I am not saying that Dockerfiles are good. But I also think that knowing the tool you use differentiates you from being a rookie to an experienced professional.
steve_rambo 2y ago
I wish we would rather get rid of Dockerfile in favor of what something like buildah does: https://github.com/containers/buildah/blob/main/examples/lighttpd.sh https://github.com/containers/buildah/blob/main/examples/lig... Since Dockerfile is a rather limited and (IMHO) poorly executed re-implementation of a shell script, why not use shell directly? Not even bash with coreutils is necessary: even posix sh with busybox can do much more than Dockerfile, and you can use something else (like Python) and take it very far indeed.
- dvhh 2y ago
  That look quite the same as running a container in docker then commiting it into an image. But this does not seem to allow to set entrypoint or some image configuration values.
  - akdev1l 2y ago
    https://github.com/containers/buildah/blob/main/docs/buildah-config.1.md https://github.com/containers/buildah/blob/main/docs/buildah...
- mass_and_energy 2y ago
  That's like saying "why do we bother with makefiles when we can just make a shell script that invokes the toolchain as needed based on positional arguments?". Well, we certainly could do that but it's over complicated compared to the existing solution and would represent a shift away from what most Docker devs have grown to use efficiently. What's so bad about Dockerfile anyway?
  - MereInterest 2y ago
    > What's so bad about Dockerfile anyway? Things I've run into: * Cannot compose together. Suppose I have three packages, A/B/C. I would like to build each package in an image, and also build an image with all three packages installed. I cannot extract functionality into a subroutine. Instead, I need to make a separate build script, add it to the image, and run it in the build. * Easy to have unintentional image bloat. The obvious way to install a package in a debian-based container is with `RUN apt-get update` followed by `RUN apt-get install FOO`. However, this causes the `/var/lib/apt/lists` directory to be included in the downloaded images, even if `RUN rm -rf /var/lib/apt/lists/` is included in the Dockerfile. In order to avoid bloating the image, the all three steps of update/install/rm must be in a single RUN command. Cannot mark commands as order-independent. If I am installing N different packages * Cannot do a dry run. There is no command that will tell you if an image is up-to-date with the current Dockerfile, and what stages must be rebuilt to bring it up to date. * Must be sequestered away in a subdirectory. Anything that is in the directory of the dockerfile is treated as part of the build context, and is copied to the docker server. Having a Dockerfile in a top-level source directory will cause all docker commands to grind to a halt. (Gee, if only there were an explicit ADD command indicating which files are actually needed.) * Must NOT be sequestered away in a subdirectory. The dockerfile may only add files to the image if they are contained in the dockerfile's directory. * No support for symlinks. Symlinks are the obvious way to avoid the contradiction in the previous two bullet points, but are not allowed. Instead, you must re-structure your entire project based on whether docker requires a file. (The documented reason for this is that the target of a symlink can change. If applied consistently, I might accept this reasoning, but the ADD command can download from a URL. Pretending that symlinks are somehow less consistent than a remote resource is ridiculous.) * Requires periodic cleanup. A failed build command results in a container left in an exited state. This occurs even if the build occurred in a command that explicitly tries to avoid leaving containers running. (e.g. "docker run --rm", where the image must be built before running.)

diarrhea 2y ago

Empty/unset environment variables not, by default, being cause for immediate error and abort is the billion dollar mistake of the DevOps/infra world.

gorgoiler 2y ago
I agree but let’s hear the counter arguments so as to not sound like we aren’t listening. Environment variables can be useful. The first version of some code can be oblivious to environment variables and the second version can use them for feature flags. Because it is entirely optional to provide these, the second version of the code can be deployed to the first version’s infrastructure, and the first version will happily ignore environment variables set in the second version’s infrastructure. As a tool for incrementally moving heterogeneous parts of a system forward without needing atomic leaps, the use of environment variables as optional flags is very useful. I just wouldn’t even start from there. Instead I configure everything in code, in one repo, with all new deployments appearing as new clusters where the only thing they have in common is that they take production traffic. This kind of production infra is very useful but there are downsides for sure. It makes database schema changes, for example, something between very hard and anathema to execute.

vbezhenar 2y ago

For this use-case you don't need anything in Dockerfile spec. ARG is not magically expanded inside RUN blocks, it's just an env available during build stage. In other words you can use this code to catch this error: ARG DEBVER="10" ARG CAPVER="7.8" FROM debian:${DEBVER} RUN <<EOF set -eu printf "DEB=${DEBVER}\nCAP=${CAPVER}\n" EOF This code fails with expected error.

iainmerrick 2y ago
So now you need to wrap all your RUN lines in here docs? Better defaults and/or an easier way to change the defaults would be good! The previous commenter is right -- it's all very well to say "just read all the docs" or "it's consistent with the way other tools work", but sometimes both the docs and the tools can be made less error-prone for users.
gorgoiler 2y ago
That seems like a perfectly good idiom and it’s fantastic to hear that Dockerfiles support heredocs — I had no idea, thank you! https://www.docker.com/blog/introduction-to-heredocs-in-dockerfiles/ https://www.docker.com/blog/introduction-to-heredocs-in-dock... It would still be possible to accidentally reference uninitialized environment variables in contexts other than RUN though. Having those be treated as errors would be useful.
kmarc 2y ago
This is unnecessarily complicated. Just use `SHELL` SHELL ["/bin/sh", "-o", "nounset", "-c"] RUN printf "DEB=${DEBVER}\nCAP=${CAPVER}\n"
- MereInterest 2y ago
  Follow-up question: Does the SHELL need to be set within each stage of a multistage build? Ideally, it could be set once to remove this footgun, but I’m guessing the footgun gets reloaded with each stage.
  - kmarc 2y ago
    Yes. In general, I think of multi-stage builds as if multiple docker files were concatenated; if ARG or SHELL doesn't exist in the theoretical split Dockerfile, then it wouldn't exist in the multistage build. Docker(file) is a thing which brought in some quite neat new ideas, and a lot of these "footguns". I did read the docs, and I am comfortable now building even complicated "Dockerfile-architectures" with proper linting, watching out for good caching, etc. I wish there were better alternatives; I saw images being built with Bazel, and thanks, but that is even more convoluted, with presumably a different set of footguns. I am not saying that Dockerfiles are good. But I also think that knowing the tool you use differentiates you from being a rookie to an experienced professional.

steve_rambo 2y ago

I wish we would rather get rid of Dockerfile in favor of what something like buildah does: https://github.com/containers/buildah/blob/main/examples/lighttpd.sh https://github.com/containers/buildah/blob/main/examples/lig... Since Dockerfile is a rather limited and (IMHO) poorly executed re-implementation of a shell script, why not use shell directly? Not even bash with coreutils is necessary: even posix sh with busybox can do much more than Dockerfile, and you can use something else (like Python) and take it very far indeed.

dvhh 2y ago
That look quite the same as running a container in docker then commiting it into an image. But this does not seem to allow to set entrypoint or some image configuration values.
- akdev1l 2y ago
  https://github.com/containers/buildah/blob/main/docs/buildah-config.1.md https://github.com/containers/buildah/blob/main/docs/buildah...
mass_and_energy 2y ago
That's like saying "why do we bother with makefiles when we can just make a shell script that invokes the toolchain as needed based on positional arguments?". Well, we certainly could do that but it's over complicated compared to the existing solution and would represent a shift away from what most Docker devs have grown to use efficiently. What's so bad about Dockerfile anyway?
- MereInterest 2y ago
  > What's so bad about Dockerfile anyway? Things I've run into: * Cannot compose together. Suppose I have three packages, A/B/C. I would like to build each package in an image, and also build an image with all three packages installed. I cannot extract functionality into a subroutine. Instead, I need to make a separate build script, add it to the image, and run it in the build. * Easy to have unintentional image bloat. The obvious way to install a package in a debian-based container is with `RUN apt-get update` followed by `RUN apt-get install FOO`. However, this causes the `/var/lib/apt/lists` directory to be included in the downloaded images, even if `RUN rm -rf /var/lib/apt/lists/` is included in the Dockerfile. In order to avoid bloating the image, the all three steps of update/install/rm must be in a single RUN command. Cannot mark commands as order-independent. If I am installing N different packages * Cannot do a dry run. There is no command that will tell you if an image is up-to-date with the current Dockerfile, and what stages must be rebuilt to bring it up to date. * Must be sequestered away in a subdirectory. Anything that is in the directory of the dockerfile is treated as part of the build context, and is copied to the docker server. Having a Dockerfile in a top-level source directory will cause all docker commands to grind to a halt. (Gee, if only there were an explicit ADD command indicating which files are actually needed.) * Must NOT be sequestered away in a subdirectory. The dockerfile may only add files to the image if they are contained in the dockerfile's directory. * No support for symlinks. Symlinks are the obvious way to avoid the contradiction in the previous two bullet points, but are not allowed. Instead, you must re-structure your entire project based on whether docker requires a file. (The documented reason for this is that the target of a symlink can change. If applied consistently, I might accept this reasoning, but the ADD command can download from a URL. Pretending that symlinks are somehow less consistent than a remote resource is ridiculous.) * Requires periodic cleanup. A failed build command results in a container left in an exited state. This occurs even if the build occurred in a command that explicitly tries to avoid leaving containers running. (e.g. "docker run --rm", where the image must be built before running.)