3 ms·

I'm pretty new to Docker, so I'm curious about "a project with 75 gems/packages and a couple of native extensions that need to be compiled"... Is the common pr

I'm pretty new to Docker, so I'm curious about "a project with 75 gems/packages and a couple of native extensions that need to be compiled"... Is the common procedure in the Docker world to build an application image that includes all the build tools that were used to build native dependencies? That seems like it does generate a pretty large image. I figured I'd take a three-step approach to my first node.js app in Docker: 1. Build an image to build my dependencies. This uses the same base image as step #2 will, but installs all the development tools and libraries (eg. build-essentials, libpq5-dev), and then outputs a .tar.gz to a shared volume containing my node_modules folder. 2. Build an image with my dependencies; imports the runtime versions of any libraries (eg. libpq5), imports & expands the .tar.gz generated by #1. 3. Build an image with my application, FROM the image in #2. The process is optimized by having the automation check for the existence of #2 by hashing the contents of the relevant Dockerfiles, and the package.json list of dependencies, and doing a `docker pull` with that hash to see if I've already built #2. If so, my build just needs to build #3. It's a bit more complex (Hello, everything in Docker-land), but ends up being pretty powerful. But your post makes me think I've over-complexified it a bit.

kgilpin 11y ago
My suggestion is to build a package installer for your app and use that to build the final image. For example, we use fpm (running in a container) to build .deb packages, then we push those to an apt repository (artifactory) and then build images downstream using apt-get. Initially we did a lot of cloning from source and compiling/installing dependencies, but it's very slow, there's a lot of wasted time in rebuilding identical code, and it's hard to provide patches and upgrades to customers.
rcarmo 11y ago
I simply build all required packages externally to the container and then bake the resulting binaries into the image by adding the requisite file trees. Fairly easy to script after the first two or three attempts, really.
nickjj 11y ago
Yours isn't overly complex, it's one way to trim down an image. However, it is a lot more complicated than just defining 1 Dockerfile that at least copies in your package.json file separately to speed up future deploys that don't touch your packages. I guess I just don't see the time vs. effort value in optimizing most smaller projects. For example, that 75 gem project may take 5 minutes to build once but after that it takes 10 seconds to build and push a version that updates the app code. I'm ok with this pattern for most of my projects because you can easily get by with 1 host to serve tens of thousands of requests a month on most typical projects. It's not like I'm spinning up and destroying dozens of instances a day where the network overhead is a legit concern (if I were, then I would optimize).

kgilpin 11y ago

My suggestion is to build a package installer for your app and use that to build the final image. For example, we use fpm (running in a container) to build .deb packages, then we push those to an apt repository (artifactory) and then build images downstream using apt-get. Initially we did a lot of cloning from source and compiling/installing dependencies, but it's very slow, there's a lot of wasted time in rebuilding identical code, and it's hard to provide patches and upgrades to customers.

rcarmo 11y ago

I simply build all required packages externally to the container and then bake the resulting binaries into the image by adding the requisite file trees. Fairly easy to script after the first two or three attempts, really.

nickjj 11y ago

Yours isn't overly complex, it's one way to trim down an image. However, it is a lot more complicated than just defining 1 Dockerfile that at least copies in your package.json file separately to speed up future deploys that don't touch your packages. I guess I just don't see the time vs. effort value in optimizing most smaller projects. For example, that 75 gem project may take 5 minutes to build once but after that it takes 10 seconds to build and push a version that updates the app code. I'm ok with this pattern for most of my projects because you can easily get by with 1 host to serve tens of thousands of requests a month on most typical projects. It's not like I'm spinning up and destroying dozens of instances a day where the network overhead is a legit concern (if I were, then I would optimize).