5 ms·

Immutable Infrastructure as Code. Docker containers are the modern gold standard. You can get fairly far on a stock OS with a combination of vendored binaries/l

Immutable Infrastructure as Code. Docker containers are the modern gold standard. You can get fairly far on a stock OS with a combination of vendored binaries/libraries and virtualenv/pip, but containers are much simpler.

Waterluvian 6y ago
I dunno if your plan is containers all the way down but for us it was literally docker-compose that was broken. More important than immutable containers is repeatable builds. I need to be able to go back to a version in 12 months and build the exact same output avec a few changes. Containers don’t help with that.
- zelphirkalt 6y ago
  I am quite fed up with people thinking: "Oh, I'll use docker compose, it'll ve simpler." adding another layerof abstraction on top of something that does not need another layer and the having issues, but not documenting their precise docker run commmands, so that I need to reverse engineer them from the docker compose yaml files.
- peterwwillis 6y ago
  Literally any change in your system can break. It matters what you do once it breaks, and what I describe makes the whole thing much more repeatable and easier to recover from. Docker containers builds are extremely repeatable. You run some software called Artifactory, which keeps the built versions of your artifacts, including base image tags. You store your applications in there, as well as the Docker containers that you built. If you need to go back a version, you pull it out of Artifactory. This is industry standard at this point.
mmmpetrichor 6y ago
If your dockerfile is using something like pip install commands, but doesn't pin all versions, you'll still run into stuff like this.
- peterwwillis 6y ago
  Correct, you have to do both.
rumanator 6y ago
> Immutable Infrastructure as Code. Docker containers are the modern gold standard. Your comment makes no sense once you account for the fact that, as part of containerization security best practices, containers are continuously rebuilt to use the latest version of their base images and dependencies, and those images are then pushed to overwrite the latest release.
- peterwwillis 6y ago
  Uh, no, you don't ever overwrite a release. You build new versions, test them, deploy them, and if you find a bug along the way, you roll back. You can pin versions of base images and update them when they have security patches, or you can rebuild a base yourself and patch it yourself. This isn't containerization best practice, this is just plain old Ops best practice that we've been doing for decades. The new part is doing it with containers and building immutable images rather than continuously muting state.
  - rumanator 6y ago
    > Uh, no, you don't ever overwrite a release. You build new versions, test them, deploy them, and if you find a bug along the way, you roll back. No, not really. You pull from image:latest, or debian, or debian:stable, or debian:stretch, or debian:stretch-slim. That's what you use as base images. The reason you do that is security. You need to incorporate security updates in your Docker images, and in order to ensure your images always incorporate the latest bug fixes and security updates you simply pull from the default base images expecting them to be continuously updated.
    - peterwwillis 6y ago
      You absolutely should not ship updates by pulling from an unversioned tag; it's bad for a number of reasons. There's some simple ways to avoid it: # In your CI pipeline, something like this: $ cat Dockerfile ARG release_tag FROM baseimg:${release_tag} # etc; i think the container metadata keeps the base version, or you can store it in the container with a RUN line $ base_version_tag=$(./get-latest-versioned-tag.sh some-docker-repo:my_prefix/my_image:stable) $ new_container_version=$(./make-new-version.sh) $ docker build -t my_img:${new_container_version} --build-arg release_tag=${base_version_tag} Then you run your tests on the new image, maybe do a gradual deployment, and can quickly roll back to the last base image or skip broken versions. Your own container version is always rolling-forward, and you can revert the base image version at any time. If you do this with a specific stable tree (like debian:8-stable or something) you should only get security updates, but if you use :latest, that's a total nightmare.
peterwwillis 6y ago
I can't edit my old comment, but it's extremely disturbing that I got downvoted so much. What I described is literally the best practice. If you don't use Docker containers, use some other form of Immutable Infrastructure like Packer VM images. But those are huge and clunky, so you should really get used to Docker containers. You could instead make Linux packages, but they don't make for as Immutable a system as the OS state constantly changes, and packaging is more work in general.

Waterluvian 6y ago

I dunno if your plan is containers all the way down but for us it was literally docker-compose that was broken. More important than immutable containers is repeatable builds. I need to be able to go back to a version in 12 months and build the exact same output avec a few changes. Containers don’t help with that.

zelphirkalt 6y ago
I am quite fed up with people thinking: "Oh, I'll use docker compose, it'll ve simpler." adding another layerof abstraction on top of something that does not need another layer and the having issues, but not documenting their precise docker run commmands, so that I need to reverse engineer them from the docker compose yaml files.
peterwwillis 6y ago
Literally any change in your system can break. It matters what you do once it breaks, and what I describe makes the whole thing much more repeatable and easier to recover from. Docker containers builds are extremely repeatable. You run some software called Artifactory, which keeps the built versions of your artifacts, including base image tags. You store your applications in there, as well as the Docker containers that you built. If you need to go back a version, you pull it out of Artifactory. This is industry standard at this point.

mmmpetrichor 6y ago

If your dockerfile is using something like pip install commands, but doesn't pin all versions, you'll still run into stuff like this.

peterwwillis 6y ago
Correct, you have to do both.

rumanator 6y ago

> Immutable Infrastructure as Code. Docker containers are the modern gold standard. Your comment makes no sense once you account for the fact that, as part of containerization security best practices, containers are continuously rebuilt to use the latest version of their base images and dependencies, and those images are then pushed to overwrite the latest release.

peterwwillis 6y ago
Uh, no, you don't ever overwrite a release. You build new versions, test them, deploy them, and if you find a bug along the way, you roll back. You can pin versions of base images and update them when they have security patches, or you can rebuild a base yourself and patch it yourself. This isn't containerization best practice, this is just plain old Ops best practice that we've been doing for decades. The new part is doing it with containers and building immutable images rather than continuously muting state.
- rumanator 6y ago
  > Uh, no, you don't ever overwrite a release. You build new versions, test them, deploy them, and if you find a bug along the way, you roll back. No, not really. You pull from image:latest, or debian, or debian:stable, or debian:stretch, or debian:stretch-slim. That's what you use as base images. The reason you do that is security. You need to incorporate security updates in your Docker images, and in order to ensure your images always incorporate the latest bug fixes and security updates you simply pull from the default base images expecting them to be continuously updated.
  - peterwwillis 6y ago
    You absolutely should not ship updates by pulling from an unversioned tag; it's bad for a number of reasons. There's some simple ways to avoid it: # In your CI pipeline, something like this: $ cat Dockerfile ARG release_tag FROM baseimg:${release_tag} # etc; i think the container metadata keeps the base version, or you can store it in the container with a RUN line $ base_version_tag=$(./get-latest-versioned-tag.sh some-docker-repo:my_prefix/my_image:stable) $ new_container_version=$(./make-new-version.sh) $ docker build -t my_img:${new_container_version} --build-arg release_tag=${base_version_tag} Then you run your tests on the new image, maybe do a gradual deployment, and can quickly roll back to the last base image or skip broken versions. Your own container version is always rolling-forward, and you can revert the base image version at any time. If you do this with a specific stable tree (like debian:8-stable or something) you should only get security updates, but if you use :latest, that's a total nightmare.

peterwwillis 6y ago

I can't edit my old comment, but it's extremely disturbing that I got downvoted so much. What I described is literally the best practice. If you don't use Docker containers, use some other form of Immutable Infrastructure like Packer VM images. But those are huge and clunky, so you should really get used to Docker containers. You could instead make Linux packages, but they don't make for as Immutable a system as the OS state constantly changes, and packaging is more work in general.