5 ms·

Do you recommend using the same Docker image that passes tests on CircleCI in production? I have an open source Rails app <https://github.com/coreinfrastructur

Do you recommend using the same Docker image that passes tests on CircleCI in production? I have an open source Rails app <https://github.com/coreinfrastructure/best-practices-badge> https://github.com/coreinfrastructure/best-practices-badge> that self-certifies that open source projects are following best practices. It currently uses CircleCI 1.0 and then deploys to Heroku <https://circleci.com/gh/coreinfrastructure/best-practices-badge> https://circleci.com/gh/coreinfrastructure/best-practices-ba.... If I configure it for development and test, it will contain all the test and development gems in addition to the production ones <https://github.com/coreinfrastructure/best-practices-badge/blob/master/Gemfile> https://github.com/coreinfrastructure/best-practices-badge/b.... I would also need to include gcc and other build tools. But I would have immutability in that my dev, test and production images would be identical. Or, should I use one image for dev and test, and then build a different image for production, in order to have the smallest possible size? CircleCI supports building <https://circleci.com/docs/2.0/building-docker-images/> https://circleci.com/docs/2.0/building-docker-images/> further Docker images, but then there's a chance of losing dev/prod parity if, for example, a software dependency gets updated in the meantime.

147 9y ago
I'm not sure if Circle supports it yet but you should definitely look into docker multistage builds as that should solve this problem for you. https://docs.docker.com/engine/userguide/eng-image/multistage-build/ https://docs.docker.com/engine/userguide/eng-image/multistag...
- dankohn1 9y ago
  I've been experimenting with multi-stage builds, and they do let me build a leaner image with clearer syntax by using one the first image to build my gems, and then copying the gems to the final image. <https://github.com/coreinfrastructure/best-practices-badge/blob/dan-docker-updated/Dockerfile#L48> https://github.com/coreinfrastructure/best-practices-badge/b... However, that doesn't fix the problem of having lots of development and test gems installed in test that I don't want for production. I don't know of a way of copying just my production gems to a production image.
  - lstamour 9y ago
    Well, you could load only the gems required for the environment: http://bundler.io/groups.html http://bundler.io/groups.html Or maybe have more than one path for gems and change load paths between environments? Or keep tests in a separate image from the production image under test? Similarly, build tools could live in a different image/layer.
    - dankohn1 9y ago
      It's not a crazy idea to first install production only gems, then copy them (cache them), and then install the remaining gems. But I'm asking for info on best practices.
      - lstamour 9y ago
        You can bundle into vendor folders, one for each env and one for the rest.
        dankohn1 9y ago
        Do you have a URL demonstrating this, please?
        147 9y ago
        Here’s a node example that demonstrates the same concept. https://codefresh.io/blog/node_docker_multistage/ https://codefresh.io/blog/node_docker_multistage/
        lstamour 9y ago
        No URL right this second (maybe I should write a blog post–), but I would set GEM_HOME or BUNDLE_PATH env variables as I ran bundler to have it install in a different folder, and add the folders as necessary to GEM_PATH at runtime. I would also consider using a local cache folders for the source of the gems bundler uses. This is the purpose of http://bundler.io/v1.6/bundle_package.html http://bundler.io/v1.6/bundle_package.html which assumes that gems are installed separately rather than loaded directly by path because of platform differences (packaging on Mac to run on Linux, for example). But you don’t need the overhead of bundler’s packaging and install process if your binaries are built in the same environment they run in, as could be the case with Docker or VMs. You could also point BUNDLE_PATH at the last place bundler ran for that environment (don’t delete the old gems) and that would speed up the install, but a better approach might be to package to a shared folder and then install to env specific bundle path folders in your docker image/build workspace, if you want bundler to manage the shared caching while keeping gem output separate and unique to each build. It’s important to remember that gems are just folders of ruby files placed in the load path, sometimes with compiled bits to load at runtime, that could in turn be dynamically compiled to rely on system libraries or non-Ruby packages. So you have similar constraints with Ruby that you would have with Node.js – the potential for issues from the portability or lack thereof for native code dependencies. That’s why there’s a distinction between “install” and “package” – install will make sure the binary output can run on the current system unless flagged not to.
- felicianotech 9y ago
  You can optionally use a newer version of Docker CE that supports multistage builds.
stingraycharles 9y ago
Typically it's a good idea to separate build and testing phases strictly, exactly for these types of reasons. You deploy the same image to dev, testing and production, and it becomes easy to reason about an artifact without confusing it with the environment it runs in. I hope this helps.
- dankohn1 9y ago
  Right, but the point of my question is that deploying the same image to test, staging and production means that I'm necessarily using a less lean image, since I'm including gems only needed in development and test.
  - kingbirdy 9y ago
    Why do you have gems only needed in those envs?
    - dankohn1 9y ago
      I'm trying to balance two conflicting goals in Docker: identical images in different stages and the leanest possible production images. My question is whether development and testing gems should be included in production or not.
damagednoob 9y ago
I think you might be missing an environment: staging. I would install whatever you want for development. Then once you're happy, push that to CircleCI and have it build the image. Push that image to staging for final testing. Once you're happy with that, push it to production.
- dankohn1 9y ago
  I currently push all passing builds on master to master.bestpractices.coreinfrastructure.org, and also run staging and production environments. See https://github.com/coreinfrastructure/best-practices-badge/blob/master/circle.yml#L7-L26 https://github.com/coreinfrastructure/best-practices-badge/b... My question is when switching to a Docker workflow, does it make sense to use the same image for test on CircleCI as for deployment on staging and production.
zemanel 9y ago
The "on-the-knee" setup i have been using for my (Django) side project is: - base image that installs OS Updates + OS package requirements + production dependencies - dev image inherits from base image, installs dev dependencies + source code volume mounted on '/app/src' - test image inherits from base image, installs test dependencies + source code volume mounted on '/app/src' (image used both for local tests and CI) - prod image inherits from base image, adds source code to '/app/src' (this image will eventually be tagged and pushed to a production registry after builds) Only adding source code to the image in prod image for not breaking cache during development.
dankohn1 9y ago
That makes sense, thanks. Wim not clear how to break up the gems for different environmenta into different directories.
thom_nic 9y ago
If we're honest we have to admit that any difference between dev/test (where you're running tests) and prod (what you're deploying) can introduce subtle bugs either due to configuration errors or something like a missing dependency that you accidentally put in the development group but it was actually needed during production. I've seen this happen and to a large extent it's unavoidable unless you're actually able to run your integration tests with production environment settings (I don't know anyone who really does this.) On one hand, having dev/test gems in the image should be innocuous, if I recall correctly bundler will not activate them when the environment is production. So other than some additional image size (is this really a problem?) it doesn't have any material effect. But I empathize with wanting to have a completely minimalist prod image, regardless of whether it has measurable benefit. iron.io has for quite some time advocated having two parallel base images, one dev which has the build tools, and then copying the built production gems to a prod image. [1] If you do this, then the build tools (gcc and the like) are never in the docker image which is used for test or prod. But yes you still need test gems in an image that runs rspec/capybara/etc. Maybe the real solution then is to take your prod image, and deploy it to some staging environment which then gets treated to a functional test suite. The approach would necessitate running the suite in one container, targeting the application running in another container however. (This is naturally how you would do it if you were using webdriver/selenium-based testing tools.) [1] https://github.com/iron-io/dockers/tree/master/ruby https://github.com/iron-io/dockers/tree/master/ruby
felicianotech 9y ago
Ideally, yes you would want a single Docker image. That's somewhat the point of Docker, to isolate your app and have it run the same, everywhere. If you use multiple images, then you're circumventing that benefit. You could install all the dev gems, build tools, etc in the CircleCI build container and run unit testing. Once everything looks good and you can build your file app that passes, then take the app and throw it in a new Docker image. Then, integration test that Docker image itself. That final piece could be done on some sort of staging server of in a new build environment using CircleCI 2.0 Workflows.

147 9y ago

I'm not sure if Circle supports it yet but you should definitely look into docker multistage builds as that should solve this problem for you. https://docs.docker.com/engine/userguide/eng-image/multistage-build/ https://docs.docker.com/engine/userguide/eng-image/multistag...

dankohn1 9y ago
I've been experimenting with multi-stage builds, and they do let me build a leaner image with clearer syntax by using one the first image to build my gems, and then copying the gems to the final image. <https://github.com/coreinfrastructure/best-practices-badge/blob/dan-docker-updated/Dockerfile#L48> https://github.com/coreinfrastructure/best-practices-badge/b... However, that doesn't fix the problem of having lots of development and test gems installed in test that I don't want for production. I don't know of a way of copying just my production gems to a production image.
- lstamour 9y ago
  Well, you could load only the gems required for the environment: http://bundler.io/groups.html http://bundler.io/groups.html Or maybe have more than one path for gems and change load paths between environments? Or keep tests in a separate image from the production image under test? Similarly, build tools could live in a different image/layer.
  - dankohn1 9y ago
    It's not a crazy idea to first install production only gems, then copy them (cache them), and then install the remaining gems. But I'm asking for info on best practices.
    - lstamour 9y ago
      You can bundle into vendor folders, one for each env and one for the rest.
      - dankohn1 9y ago
        Do you have a URL demonstrating this, please?
        147 9y ago
        Here’s a node example that demonstrates the same concept. https://codefresh.io/blog/node_docker_multistage/ https://codefresh.io/blog/node_docker_multistage/
        lstamour 9y ago
        No URL right this second (maybe I should write a blog post–), but I would set GEM_HOME or BUNDLE_PATH env variables as I ran bundler to have it install in a different folder, and add the folders as necessary to GEM_PATH at runtime. I would also consider using a local cache folders for the source of the gems bundler uses. This is the purpose of http://bundler.io/v1.6/bundle_package.html http://bundler.io/v1.6/bundle_package.html which assumes that gems are installed separately rather than loaded directly by path because of platform differences (packaging on Mac to run on Linux, for example). But you don’t need the overhead of bundler’s packaging and install process if your binaries are built in the same environment they run in, as could be the case with Docker or VMs. You could also point BUNDLE_PATH at the last place bundler ran for that environment (don’t delete the old gems) and that would speed up the install, but a better approach might be to package to a shared folder and then install to env specific bundle path folders in your docker image/build workspace, if you want bundler to manage the shared caching while keeping gem output separate and unique to each build. It’s important to remember that gems are just folders of ruby files placed in the load path, sometimes with compiled bits to load at runtime, that could in turn be dynamically compiled to rely on system libraries or non-Ruby packages. So you have similar constraints with Ruby that you would have with Node.js – the potential for issues from the portability or lack thereof for native code dependencies. That’s why there’s a distinction between “install” and “package” – install will make sure the binary output can run on the current system unless flagged not to.
felicianotech 9y ago
You can optionally use a newer version of Docker CE that supports multistage builds.

stingraycharles 9y ago

Typically it's a good idea to separate build and testing phases strictly, exactly for these types of reasons. You deploy the same image to dev, testing and production, and it becomes easy to reason about an artifact without confusing it with the environment it runs in. I hope this helps.

dankohn1 9y ago
Right, but the point of my question is that deploying the same image to test, staging and production means that I'm necessarily using a less lean image, since I'm including gems only needed in development and test.
- kingbirdy 9y ago
  Why do you have gems only needed in those envs?
  - dankohn1 9y ago
    I'm trying to balance two conflicting goals in Docker: identical images in different stages and the leanest possible production images. My question is whether development and testing gems should be included in production or not.

damagednoob 9y ago

I think you might be missing an environment: staging. I would install whatever you want for development. Then once you're happy, push that to CircleCI and have it build the image. Push that image to staging for final testing. Once you're happy with that, push it to production.

dankohn1 9y ago
I currently push all passing builds on master to master.bestpractices.coreinfrastructure.org, and also run staging and production environments. See https://github.com/coreinfrastructure/best-practices-badge/blob/master/circle.yml#L7-L26 https://github.com/coreinfrastructure/best-practices-badge/b... My question is when switching to a Docker workflow, does it make sense to use the same image for test on CircleCI as for deployment on staging and production.

zemanel 9y ago

The "on-the-knee" setup i have been using for my (Django) side project is: - base image that installs OS Updates + OS package requirements + production dependencies - dev image inherits from base image, installs dev dependencies + source code volume mounted on '/app/src' - test image inherits from base image, installs test dependencies + source code volume mounted on '/app/src' (image used both for local tests and CI) - prod image inherits from base image, adds source code to '/app/src' (this image will eventually be tagged and pushed to a production registry after builds) Only adding source code to the image in prod image for not breaking cache during development.

dankohn1 9y ago

That makes sense, thanks. Wim not clear how to break up the gems for different environmenta into different directories.

thom_nic 9y ago

If we're honest we have to admit that any difference between dev/test (where you're running tests) and prod (what you're deploying) can introduce subtle bugs either due to configuration errors or something like a missing dependency that you accidentally put in the development group but it was actually needed during production. I've seen this happen and to a large extent it's unavoidable unless you're actually able to run your integration tests with production environment settings (I don't know anyone who really does this.) On one hand, having dev/test gems in the image should be innocuous, if I recall correctly bundler will not activate them when the environment is production. So other than some additional image size (is this really a problem?) it doesn't have any material effect. But I empathize with wanting to have a completely minimalist prod image, regardless of whether it has measurable benefit. iron.io has for quite some time advocated having two parallel base images, one dev which has the build tools, and then copying the built production gems to a prod image. [1] If you do this, then the build tools (gcc and the like) are never in the docker image which is used for test or prod. But yes you still need test gems in an image that runs rspec/capybara/etc. Maybe the real solution then is to take your prod image, and deploy it to some staging environment which then gets treated to a functional test suite. The approach would necessitate running the suite in one container, targeting the application running in another container however. (This is naturally how you would do it if you were using webdriver/selenium-based testing tools.) [1] https://github.com/iron-io/dockers/tree/master/ruby https://github.com/iron-io/dockers/tree/master/ruby

felicianotech 9y ago

Ideally, yes you would want a single Docker image. That's somewhat the point of Docker, to isolate your app and have it run the same, everywhere. If you use multiple images, then you're circumventing that benefit. You could install all the dev gems, build tools, etc in the CircleCI build container and run unit testing. Once everything looks good and you can build your file app that passes, then take the app and throw it in a new Docker image. Then, integration test that Docker image itself. That final piece could be done on some sort of staging server of in a new build environment using CircleCI 2.0 Workflows.