5 ms·

Hopefully native squashing will allow some people to avoid writing Dockerfiles that look like RUN this && that && thistoo && thattoo && rm this && rm that i

Hopefully native squashing will allow some people to avoid writing Dockerfiles that look like RUN this && that && thistoo && thattoo && rm this && rm that in order to avoid generating extraneous layers. There have been some third party tools and registries that implement this feature, but it's nice to finally have it upstream.

deleted 10y ago
[deleted]
alexlarsson 10y ago
Finally they added this! I submitted a PR for this over two years ago (https://github.com/docker/docker/pull/4232 https://github.com/docker/docker/pull/4232).
donw 10y ago
Honestly, I see this as an antipattern, not because it generates extra layers, but because a Dockerfile isn't the right place to write a provisioning script. When building apps, I keep a small number of shell scripts that live in a `bin` directory, that perform essential operational tasks. One script installs all the dependencies required to either run or develop the app; another runs all tests and exits with either success or failure; and the final script just runs the app. Docker then just runs those scripts, which are written deliberately to be easily read, and thus function as living documentation for the project. These same scripts get used, both by developers on a daily basis, as well as by our CI/CD system when prepping containers. This also makes onboarding a snap: you run `bin/setup`, and your Mac or Linux box is good to go. And, because that script gets used every time the CI system spins up a build, it will never go out of sync with reality.
- yladiz 10y ago
  The problem is that if you're building images a lot, Docker doesn't cache COPY calls because it doesn't know if the file has changed or not when calling the COPY command. So if you constantly build images and there are a lot of computationally intensive tasks within those scripts, but you've only changed one small item near the end of the script, your build time may be significantly longer by using the script than by putting it directly in the Dockerfile using the RUN command. I build a lot of modules within one of my Docker images for work, and while it doesn't change often, I would really not like to wait the 20 minutes if I really needed to push to production when I only change some certificate population segment at the end of the Dockerfile or something after a particularly intensive module build RUN step.
  - shawabawa3 10y ago
    Docker does cache COPY calls AFAIK. If you're copying your whole directory though the cache breaks if any file changes. The normal way of doing it is to copy stuff like requirements.txt, package.json, Gemfile etc first and install stuff, then copy everything else at the end
    - yladiz 10y ago
      Ah, maybe that's true, I always figured it didn't cache it since it wasn't text. In this case it doesn't matter since the script would change, but if the scripts were broken apart and ran using RUN commands it would be a better middle-ground solution than wholly putting it all in a script with a single RUN command.
  - deleted 10y ago
    [deleted]
- solatic 10y ago
  It never gets out of sync, until somebody gets tripped up by some idiosyncracy of running the server directly on their machine instead of through Docker. One of the original benefits of containers is in its potential to reduce the disparity between dev and prod. If you're running containers based on the exact same image as dev and prod, then there are fewer reasons why something would only work on a dev machine. If your devs are using containers correctly, and building candidate images on their own machines, then there's no need for separate bin scripts. If your devs need separate bin scripts so that they can avoid installing and using Docker on their own workstations, then you're throwing out a lot of the benefit which containers give you.
alexellisuk 10y ago
Yes, but bear in mind that layers are what enable differential push/pulls from the Hub or a registry. So you may now want to have separate dependent Dockerfiles - a base for sys deps and app deps squashing it and then an application image deriving from it - probably also squashed. If you squash a single image I guess you will always push/pull the entire thing even if the system dependencies haven't changed.
- cpuguy83 10y ago
  Squashing is down up to the `FROM` statement in the Dockerfile. Any layers from the parent image are preserved. Where this does mess things up is with content-addressable storage where if you have two layers from completely separate images that produced the same content, the layers would be shared, but not if you squash (because there is no layer).
dewiz 10y ago
I would prefer the ability to squash a set of adjacent layers. Squashing the entire history doesn't seem like a great idea, you lose the benefit of common layers shared by different images. As for that multiple "&&" it's not a big deal, you can escape a new line and keep the file readable.
- cpuguy83 10y ago
  It doesn't squash the whole history, just up to the `FROM` command. But yes, it defeats one of the optimizations that content addressable storage introduced.
majewsky 10y ago
I don't like that they're doing it as a flag to docker build, though. I would have liked something like RUN echo "this gets its own layer" BEGIN RUN echo "this will be squashed" RUN echo "and this, too" SQUASH RUN echo "this gets its own layer again" Similar to how BEGIN..COMMIT works in SQL.
- CSDude 10y ago
  Yeah, I like that. Because even a chmod of a directory causes another layer with all the data duplicated, it would be better if I can just squash the selected lines.

deleted 10y ago

[deleted]

alexlarsson 10y ago

Finally they added this! I submitted a PR for this over two years ago (https://github.com/docker/docker/pull/4232 https://github.com/docker/docker/pull/4232).

donw 10y ago

Honestly, I see this as an antipattern, not because it generates extra layers, but because a Dockerfile isn't the right place to write a provisioning script. When building apps, I keep a small number of shell scripts that live in a `bin` directory, that perform essential operational tasks. One script installs all the dependencies required to either run or develop the app; another runs all tests and exits with either success or failure; and the final script just runs the app. Docker then just runs those scripts, which are written deliberately to be easily read, and thus function as living documentation for the project. These same scripts get used, both by developers on a daily basis, as well as by our CI/CD system when prepping containers. This also makes onboarding a snap: you run `bin/setup`, and your Mac or Linux box is good to go. And, because that script gets used every time the CI system spins up a build, it will never go out of sync with reality.

yladiz 10y ago
The problem is that if you're building images a lot, Docker doesn't cache COPY calls because it doesn't know if the file has changed or not when calling the COPY command. So if you constantly build images and there are a lot of computationally intensive tasks within those scripts, but you've only changed one small item near the end of the script, your build time may be significantly longer by using the script than by putting it directly in the Dockerfile using the RUN command. I build a lot of modules within one of my Docker images for work, and while it doesn't change often, I would really not like to wait the 20 minutes if I really needed to push to production when I only change some certificate population segment at the end of the Dockerfile or something after a particularly intensive module build RUN step.
- shawabawa3 10y ago
  Docker does cache COPY calls AFAIK. If you're copying your whole directory though the cache breaks if any file changes. The normal way of doing it is to copy stuff like requirements.txt, package.json, Gemfile etc first and install stuff, then copy everything else at the end
  - yladiz 10y ago
    Ah, maybe that's true, I always figured it didn't cache it since it wasn't text. In this case it doesn't matter since the script would change, but if the scripts were broken apart and ran using RUN commands it would be a better middle-ground solution than wholly putting it all in a script with a single RUN command.
- deleted 10y ago
  [deleted]
solatic 10y ago
It never gets out of sync, until somebody gets tripped up by some idiosyncracy of running the server directly on their machine instead of through Docker. One of the original benefits of containers is in its potential to reduce the disparity between dev and prod. If you're running containers based on the exact same image as dev and prod, then there are fewer reasons why something would only work on a dev machine. If your devs are using containers correctly, and building candidate images on their own machines, then there's no need for separate bin scripts. If your devs need separate bin scripts so that they can avoid installing and using Docker on their own workstations, then you're throwing out a lot of the benefit which containers give you.

alexellisuk 10y ago

Yes, but bear in mind that layers are what enable differential push/pulls from the Hub or a registry. So you may now want to have separate dependent Dockerfiles - a base for sys deps and app deps squashing it and then an application image deriving from it - probably also squashed. If you squash a single image I guess you will always push/pull the entire thing even if the system dependencies haven't changed.

cpuguy83 10y ago
Squashing is down up to the `FROM` statement in the Dockerfile. Any layers from the parent image are preserved. Where this does mess things up is with content-addressable storage where if you have two layers from completely separate images that produced the same content, the layers would be shared, but not if you squash (because there is no layer).

dewiz 10y ago

I would prefer the ability to squash a set of adjacent layers. Squashing the entire history doesn't seem like a great idea, you lose the benefit of common layers shared by different images. As for that multiple "&&" it's not a big deal, you can escape a new line and keep the file readable.

cpuguy83 10y ago
It doesn't squash the whole history, just up to the `FROM` command. But yes, it defeats one of the optimizations that content addressable storage introduced.

majewsky 10y ago

I don't like that they're doing it as a flag to docker build, though. I would have liked something like RUN echo "this gets its own layer" BEGIN RUN echo "this will be squashed" RUN echo "and this, too" SQUASH RUN echo "this gets its own layer again" Similar to how BEGIN..COMMIT works in SQL.

CSDude 10y ago
Yeah, I like that. Because even a chmod of a directory causes another layer with all the data duplicated, it would be better if I can just squash the selected lines.