4 ms·

Interesting, the side snipe about Docker > One of the mostly-false promises of Docker, as it was sold to me by the true believers who introduced me to it, was

Interesting, the side snipe about Docker > One of the mostly-false promises of Docker, as it was sold to me by the true believers who introduced me to it, was that, if you do it right, you can run the same docker image, and therefore have basically the same environment in production, in CI, and on your local development machine. I literally have never worked anywhere that hasn't used the same container in dev/test/prod. People who have had to deal with processes that rebuilt images for each environment, can you tell me what was different in each image?

dijksterhuis 3y ago
> People who have had to deal with processes that rebuilt images for each environment, can you tell me what was different in each image? Tensorflow used to generate multiple images for different use cases: cpu only, GPU, cpu with Jupyterhub, GPU with Jupyterhub. So each release would generate minimum 4x container builds. CPU only images were useful for local testing of ml code at CLI (does it run) when a GPU might no be available. Jupyterhub variants for simple GUI for development work. GPU for obvious reasons (run the damn thing). I can’t remember if they used to do builds for their release against multiple CUDA / CuDNN versions… that might have been PyTorch.
AdrienPoupa 3y ago
PHP dev here, we have extensions for development that make no sense in production, xdebug for example. You need it for breakpoints and debugging in general but it should not be installed in production. So we extend our production image and install it on top of it. Similarly, we include Composer (package manager) in the dev image only as we only need the installed dependencies in production but not the package manager. Our dev image is a flavor of the production one, really.
- comprev 3y ago
  Would multi-stage Docker builds not help here? Composer executes in one step and the result artefacts are copied into a "clean" PHP image without Composer installed.
  - twunde 3y ago
    Based on the description they are doing a multi stage build, but using the prod container as a base and then building the dev container atop that. But yes you could easily go the other way with dev building an artifact and adding it to a secure locked down container. This is less typical with dynamic languages that don't typically create a single binary, but still comes up. The downsides are that your prod container is now significantly different and for dynamic languages the fast feedback loop now has a slowish build step
  - AdrienPoupa 3y ago
    This what we are doing for the prod container that does not have Composer installed yes. But in development it's much easier to have it in the image. Additionally we do not bundle the code in the dev image but bind mount it in Docker Compose, which is much faster than rebuilding the image to test changes in development; PHP not being compiled allows us to do that to reduce the feedback loop duration.
- nullify88 3y ago
  In my experience, Xdebug absolutely made sense in production just not enabled by default for all requests. A lot of its functionality can be enabled via a cookie for a single session, and its made debugging production much easier as well as identifying bottlenecks in code or production infrastructure.
  - AdrienPoupa 3y ago
    That's certainly possible but we have other tools for that such as NewRelic that served us well.
DylanSp 3y ago
At a recent job, we had slightly different containers for local dev; our backend containers (for a Go app) had Air [1] installed for live reloading, plus Delve [2] running inside the container for VS Code's debugger to connect to. We also had a frontend container for local dev, which didn't get deployed as a container, just as static files. [1] https://github.com/cosmtrek/air https://github.com/cosmtrek/air [2] https://github.com/go-delve/delve/ https://github.com/go-delve/delve/
twunde 3y ago
I'm general, it's an anti pattern since it's makes everything much more complicated. The typical reason to have it is so that there are dev tools in the dev container (autoformatter, linter etc) or support for hot reloading and then the prod container is locked down. The other pattern you'll sometimes see is that the prod container will include an agent or a certificate bundle, although it's more common to use sidecars for this. It becomes problematic because it then becomes easy for engineers to have a completely different container for dev then is used for prod. I recently found an issue where a dev container was using a completely different base and had a different version of node installed compared to the prod container
- hedora 3y ago
  Multi-stage docker files mostly solve this problem: https://docs.docker.com/build/building/multi-stage/ https://docs.docker.com/build/building/multi-stage/
  - opello 3y ago
    I agree that multi-stage builds help address this, but that also means there are different images.
- never_inline 3y ago
  > It becomes problematic because it then becomes easy for engineers to have a completely different container for dev then is used for prod If the differences between dev and prod are also programmatic in nature (eg based on flag, and mostly configuration-like values), it should be fine.
anschwa 3y ago
I think it's common (and important!) to build your "production" image once and deploy it into a dev/staging environment first. This means there are no re-builds between deploys to each environment. However, I think it's less common to run automated testing within the same Docker image you build and deploy to these environments. This is challenging because you probably don't want to include any build/test related packages in your production image (or layer) but you still want some level of confidence that your CI and Prod environments are the same. I have often see builds pass automated testing but fail after deployment because our production Dockerfile/layer was missing packages or shared libraries that were present in the CI environment.
- mkesper 3y ago
  But then there have to be rebuilds, no?
- mmcnl 3y ago
  Why don't you want to include build/test related packages in your production image? Exactly because you mention you can't test the Docker image without these tools in the image, you cannot guarantee the production image works as you would expect it to work. Precisely because of that I think any packages that are required to run the tests should be part of the production image.
- happymellon 3y ago
  > However, I think it's less common to run automated testing within the same Docker image you build and deploy to these environments. When you say "automated tests" do you mean unit or integration tests, or both? Because unit should generally be in your compilation process, not part of any image, and integration shouldn't require anything different in the image compared to any other environment. Build it, which includes unit tests, and then deploy and integration tests should be run against your stable API.
  - anschwa 3y ago
    Great question! In my experience with ruby on rails, the application runtime is so dynamic, that even unit tests may not be reliable if run outside the production image. I think normally this isn't much of an issue because other automated/manual integration testing in the staging environments will catch any major problems, to your point. Another example would be for browser testing via chromedriver. I've usually seen this implemented along side unit tests (i.e., prior to the build phase) but since it generally serves as an integration level test for many applications, this has lead to issues due to the testing and production environments being out of sync. I think multi-layer Docker images are a compelling solution to this, but it's not usually how I've seen it implemented. Instead, I've typically seen that test and prod environments are manually maintained. Sometimes these are two separate Dockerfiles or some shared CI environment managed by different teams.
chromatin 3y ago
I understand staging and prod running the same executable, but wouldn't dev typically be a debug build?

dijksterhuis 3y ago

> People who have had to deal with processes that rebuilt images for each environment, can you tell me what was different in each image? Tensorflow used to generate multiple images for different use cases: cpu only, GPU, cpu with Jupyterhub, GPU with Jupyterhub. So each release would generate minimum 4x container builds. CPU only images were useful for local testing of ml code at CLI (does it run) when a GPU might no be available. Jupyterhub variants for simple GUI for development work. GPU for obvious reasons (run the damn thing). I can’t remember if they used to do builds for their release against multiple CUDA / CuDNN versions… that might have been PyTorch.

AdrienPoupa 3y ago

PHP dev here, we have extensions for development that make no sense in production, xdebug for example. You need it for breakpoints and debugging in general but it should not be installed in production. So we extend our production image and install it on top of it. Similarly, we include Composer (package manager) in the dev image only as we only need the installed dependencies in production but not the package manager. Our dev image is a flavor of the production one, really.

comprev 3y ago
Would multi-stage Docker builds not help here? Composer executes in one step and the result artefacts are copied into a "clean" PHP image without Composer installed.
- twunde 3y ago
  Based on the description they are doing a multi stage build, but using the prod container as a base and then building the dev container atop that. But yes you could easily go the other way with dev building an artifact and adding it to a secure locked down container. This is less typical with dynamic languages that don't typically create a single binary, but still comes up. The downsides are that your prod container is now significantly different and for dynamic languages the fast feedback loop now has a slowish build step
- AdrienPoupa 3y ago
  This what we are doing for the prod container that does not have Composer installed yes. But in development it's much easier to have it in the image. Additionally we do not bundle the code in the dev image but bind mount it in Docker Compose, which is much faster than rebuilding the image to test changes in development; PHP not being compiled allows us to do that to reduce the feedback loop duration.
nullify88 3y ago
In my experience, Xdebug absolutely made sense in production just not enabled by default for all requests. A lot of its functionality can be enabled via a cookie for a single session, and its made debugging production much easier as well as identifying bottlenecks in code or production infrastructure.
- AdrienPoupa 3y ago
  That's certainly possible but we have other tools for that such as NewRelic that served us well.

DylanSp 3y ago

At a recent job, we had slightly different containers for local dev; our backend containers (for a Go app) had Air [1] installed for live reloading, plus Delve [2] running inside the container for VS Code's debugger to connect to. We also had a frontend container for local dev, which didn't get deployed as a container, just as static files. [1] https://github.com/cosmtrek/air https://github.com/cosmtrek/air [2] https://github.com/go-delve/delve/ https://github.com/go-delve/delve/

twunde 3y ago

I'm general, it's an anti pattern since it's makes everything much more complicated. The typical reason to have it is so that there are dev tools in the dev container (autoformatter, linter etc) or support for hot reloading and then the prod container is locked down. The other pattern you'll sometimes see is that the prod container will include an agent or a certificate bundle, although it's more common to use sidecars for this. It becomes problematic because it then becomes easy for engineers to have a completely different container for dev then is used for prod. I recently found an issue where a dev container was using a completely different base and had a different version of node installed compared to the prod container

hedora 3y ago
Multi-stage docker files mostly solve this problem: https://docs.docker.com/build/building/multi-stage/ https://docs.docker.com/build/building/multi-stage/
- opello 3y ago
  I agree that multi-stage builds help address this, but that also means there are different images.
never_inline 3y ago
> It becomes problematic because it then becomes easy for engineers to have a completely different container for dev then is used for prod If the differences between dev and prod are also programmatic in nature (eg based on flag, and mostly configuration-like values), it should be fine.

anschwa 3y ago

I think it's common (and important!) to build your "production" image once and deploy it into a dev/staging environment first. This means there are no re-builds between deploys to each environment. However, I think it's less common to run automated testing within the same Docker image you build and deploy to these environments. This is challenging because you probably don't want to include any build/test related packages in your production image (or layer) but you still want some level of confidence that your CI and Prod environments are the same. I have often see builds pass automated testing but fail after deployment because our production Dockerfile/layer was missing packages or shared libraries that were present in the CI environment.

mkesper 3y ago
But then there have to be rebuilds, no?
mmcnl 3y ago
Why don't you want to include build/test related packages in your production image? Exactly because you mention you can't test the Docker image without these tools in the image, you cannot guarantee the production image works as you would expect it to work. Precisely because of that I think any packages that are required to run the tests should be part of the production image.
happymellon 3y ago
> However, I think it's less common to run automated testing within the same Docker image you build and deploy to these environments. When you say "automated tests" do you mean unit or integration tests, or both? Because unit should generally be in your compilation process, not part of any image, and integration shouldn't require anything different in the image compared to any other environment. Build it, which includes unit tests, and then deploy and integration tests should be run against your stable API.
- anschwa 3y ago
  Great question! In my experience with ruby on rails, the application runtime is so dynamic, that even unit tests may not be reliable if run outside the production image. I think normally this isn't much of an issue because other automated/manual integration testing in the staging environments will catch any major problems, to your point. Another example would be for browser testing via chromedriver. I've usually seen this implemented along side unit tests (i.e., prior to the build phase) but since it generally serves as an integration level test for many applications, this has lead to issues due to the testing and production environments being out of sync. I think multi-layer Docker images are a compelling solution to this, but it's not usually how I've seen it implemented. Instead, I've typically seen that test and prod environments are manually maintained. Sometimes these are two separate Dockerfiles or some shared CI environment managed by different teams.

chromatin 3y ago

I understand staging and prod running the same executable, but wouldn't dev typically be a debug build?