38 ms·

If all you want to do is run a series of commands while only creating a single layer, heredocs are probably the simplest / most readable approach: https://www.

If all you want to do is run a series of commands while only creating a single layer, heredocs are probably the simplest / most readable approach: https://www.docker.com/blog/introduction-to-heredocs-in-dockerfiles/ https://www.docker.com/blog/introduction-to-heredocs-in-dock...

e12e 4y ago
I like here-docs, but frequently I think just making small shell scripts to be invoked by RUN is better, eg putting apt invocations in something like: buildscripts/install_deps and simply RUN that from the Dockerfile.
tomca32 4y ago
Why didn't I ever think of this. This is great and really makes things way simpler. Thanks!
- jbott 4y ago
  Unfortunately this syntax is not generally supported yet - it's only supported with the buildkit backend and only landed in the 1.3 "labs" release. It was moved to stable in early 2022 (see https://github.com/moby/buildkit/issues/2574 https://github.com/moby/buildkit/issues/2574), so that seems to be better, but I think may still require a syntax directive to enable. Many other dockerfile build tools still don't support it, e.g. buildah (see https://github.com/containers/buildah/issues/3474 https://github.com/containers/buildah/issues/3474) Useful now if you have control over the environment your images are being built in, but I'm excited to the future where it's commonplace!
  - Groxx 4y ago
    And AFAIK buildkit is still a real pain to troubleshoot, as you can't use intermediate stages :/ You can put stuff in a script file and just run that script too.
    - cpuguy83 4y ago
      You can! It's experimental but https://github.com/docker/buildx/pull/1168 https://github.com/docker/buildx/pull/1168 Tracking for this: https://github.com/docker/buildx/issues/1104 https://github.com/docker/buildx/issues/1104
      - Groxx 4y ago
        excellent, thanks! this has been a serious pain in my side for a while, both for my own debugging and for telling people I try to help "you're gonna have to start over and do X or this will take hours longer".
gitgud 4y ago
Nice syntax, but I like the caching that comes with creating each layer. If you want to reduce layer/image size, then I think [1] "multi-stage builds" is a good option [1] https://docs.docker.com/build/building/multi-stage/ https://docs.docker.com/build/building/multi-stage/
- DanHulton 4y ago
  I mean, you can do both. Or, technically with that link you mentioned, all three. You can use HEREDOCS to combo together commands that make sense in a layer, ensure your layers are ordered such that the more-frequently changing ones are further on in your Dockerfile when possible (this will also speed up your builds, ensuring as many caches as possible are more likely to be valid), and use mutli-stage builds on top of that to really pare it down to the bare necessities.
- klabb3 4y ago
  > Nice syntax Is it though? From the post: RUN <<EOF apt-get update apt-get upgrade -y apt-get install -y ... EOF It may be due to my ninja level abilities to dodge learning more advanced shell mastery for decades, but to me it looks haphazard and error prone. Are the line breaks semantic, or is it all a multiline string? Is EOF a special end-of-file token, or a variable, if so what’s it’s type? Where is it documented? Is the first EOF sent to stdin, if so why is that needed? What is the second EOF doing? I can usually pick up a new imperative language quickly, but I still feel like an idiot when looking at shell.
  - awwaiid 4y ago
    The <<XYZ ... XYZ syntax for multi-line strings is worth learning since it is used in shell, ruby, php, and others. See https://en.m.wikipedia.org/wiki/Here_document https://en.m.wikipedia.org/wiki/Here_document . You get to pick the "EOF" delimiter.
  - hddqsb 4y ago
    I know those questions are rhetorical, but to answer them anyway: > > Nice syntax > Is it though? Before the heredoc syntax was added, the usual approach was to use a backslash at the end of each line, creating a line continuation. This has several issues: The backslash swallows the newline, so one must also insert a semicolon* to mark the end of each command. Forgetting the semicolon leads to weird errors. Also, while Docker supports line continuations interspersed with comments, sh doesn't, so if such a command contains comments it can't be copied into sh. The new heredoc syntax doesn't have any of these issues. I think it is infinitely better :) (There is also JSON-style syntax, but it requires all backslashes to be doubled, and is less popular.) *In practice "&&" is normally used rather than ";" in order to stop the build if any command fails (otherwise sh only propagates the exit status of the last command). This actually leads to a small footgun with the heredoc syntax: it allows the programmer to use just a newline, which is equivalent to a semicolon and means the exit status will be ignored for all but the last command. The programmer must remember to insert "&&" after each command, or use `set -e` at the start of the RUN command, or use `SHELL ["/bin/sh", "-e", "-c"]` at the top of the Dockerfile. But this footgun is due to sh's error handling quirks, not the heredoc syntax itself. > Are the line breaks semantic, or is it all a multiline string? The line breaks are preserved ("what you see is what you get"). > Is EOF a special end-of-file token You can choose which token to use (EOF is a common convention, but any token can be used). The text right after the "<<" indicates which token you've chosen, and the heredoc is terminated by the first line that contains just that token. This allows you to easily create a heredoc containing other heredocs. Can you think of any other quoting syntax that allows that? (Lisp's quote form comes to mind.) > Where is it documented? The introduction blog post has already been linked. The reference documentation (https://docs.docker.com/engine/reference/builder/ https://docs.docker.com/engine/reference/builder/, https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#here-documents https://github.com/moby/buildkit/blob/master/frontend/docker...) explains the syntax using examples. It doesn't have a formal specification; unfortunately this is a wider problem with the Dockerfile syntax (see https://supercontainers.github.io/containers-wg/ideas/dockerfile_A6/ https://supercontainers.github.io/containers-wg/ideas/docker...). Instead, the reference links to the sh syntax specification (https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04 https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...), on which the Dockerfile heredoc syntax is based.

e12e 4y ago

I like here-docs, but frequently I think just making small shell scripts to be invoked by RUN is better, eg putting apt invocations in something like: buildscripts/install_deps and simply RUN that from the Dockerfile.

tomca32 4y ago

Why didn't I ever think of this. This is great and really makes things way simpler. Thanks!

jbott 4y ago
Unfortunately this syntax is not generally supported yet - it's only supported with the buildkit backend and only landed in the 1.3 "labs" release. It was moved to stable in early 2022 (see https://github.com/moby/buildkit/issues/2574 https://github.com/moby/buildkit/issues/2574), so that seems to be better, but I think may still require a syntax directive to enable. Many other dockerfile build tools still don't support it, e.g. buildah (see https://github.com/containers/buildah/issues/3474 https://github.com/containers/buildah/issues/3474) Useful now if you have control over the environment your images are being built in, but I'm excited to the future where it's commonplace!
- Groxx 4y ago
  And AFAIK buildkit is still a real pain to troubleshoot, as you can't use intermediate stages :/ You can put stuff in a script file and just run that script too.
  - cpuguy83 4y ago
    You can! It's experimental but https://github.com/docker/buildx/pull/1168 https://github.com/docker/buildx/pull/1168 Tracking for this: https://github.com/docker/buildx/issues/1104 https://github.com/docker/buildx/issues/1104
    - Groxx 4y ago
      excellent, thanks! this has been a serious pain in my side for a while, both for my own debugging and for telling people I try to help "you're gonna have to start over and do X or this will take hours longer".

gitgud 4y ago

Nice syntax, but I like the caching that comes with creating each layer. If you want to reduce layer/image size, then I think [1] "multi-stage builds" is a good option [1] https://docs.docker.com/build/building/multi-stage/ https://docs.docker.com/build/building/multi-stage/

DanHulton 4y ago
I mean, you can do both. Or, technically with that link you mentioned, all three. You can use HEREDOCS to combo together commands that make sense in a layer, ensure your layers are ordered such that the more-frequently changing ones are further on in your Dockerfile when possible (this will also speed up your builds, ensuring as many caches as possible are more likely to be valid), and use mutli-stage builds on top of that to really pare it down to the bare necessities.
klabb3 4y ago
> Nice syntax Is it though? From the post: RUN <<EOF apt-get update apt-get upgrade -y apt-get install -y ... EOF It may be due to my ninja level abilities to dodge learning more advanced shell mastery for decades, but to me it looks haphazard and error prone. Are the line breaks semantic, or is it all a multiline string? Is EOF a special end-of-file token, or a variable, if so what’s it’s type? Where is it documented? Is the first EOF sent to stdin, if so why is that needed? What is the second EOF doing? I can usually pick up a new imperative language quickly, but I still feel like an idiot when looking at shell.
- awwaiid 4y ago
  The <<XYZ ... XYZ syntax for multi-line strings is worth learning since it is used in shell, ruby, php, and others. See https://en.m.wikipedia.org/wiki/Here_document https://en.m.wikipedia.org/wiki/Here_document . You get to pick the "EOF" delimiter.
- hddqsb 4y ago
  I know those questions are rhetorical, but to answer them anyway: > > Nice syntax > Is it though? Before the heredoc syntax was added, the usual approach was to use a backslash at the end of each line, creating a line continuation. This has several issues: The backslash swallows the newline, so one must also insert a semicolon* to mark the end of each command. Forgetting the semicolon leads to weird errors. Also, while Docker supports line continuations interspersed with comments, sh doesn't, so if such a command contains comments it can't be copied into sh. The new heredoc syntax doesn't have any of these issues. I think it is infinitely better :) (There is also JSON-style syntax, but it requires all backslashes to be doubled, and is less popular.) *In practice "&&" is normally used rather than ";" in order to stop the build if any command fails (otherwise sh only propagates the exit status of the last command). This actually leads to a small footgun with the heredoc syntax: it allows the programmer to use just a newline, which is equivalent to a semicolon and means the exit status will be ignored for all but the last command. The programmer must remember to insert "&&" after each command, or use `set -e` at the start of the RUN command, or use `SHELL ["/bin/sh", "-e", "-c"]` at the top of the Dockerfile. But this footgun is due to sh's error handling quirks, not the heredoc syntax itself. > Are the line breaks semantic, or is it all a multiline string? The line breaks are preserved ("what you see is what you get"). > Is EOF a special end-of-file token You can choose which token to use (EOF is a common convention, but any token can be used). The text right after the "<<" indicates which token you've chosen, and the heredoc is terminated by the first line that contains just that token. This allows you to easily create a heredoc containing other heredocs. Can you think of any other quoting syntax that allows that? (Lisp's quote form comes to mind.) > Where is it documented? The introduction blog post has already been linked. The reference documentation (https://docs.docker.com/engine/reference/builder/ https://docs.docker.com/engine/reference/builder/, https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#here-documents https://github.com/moby/buildkit/blob/master/frontend/docker...) explains the syntax using examples. It doesn't have a formal specification; unfortunately this is a wider problem with the Dockerfile syntax (see https://supercontainers.github.io/containers-wg/ideas/dockerfile_A6/ https://supercontainers.github.io/containers-wg/ideas/docker...). Instead, the reference links to the sh syntax specification (https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04 https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...), on which the Dockerfile heredoc syntax is based.