11 ms·

Just say no to :latest

229 points by kiyanwang 5y ago

synergy20 5y ago
I agree. Or provide something in parallel such as: :lts -- newest long term stable release :release -- newest regular stable release :x.y.z -- supported but you need find out the exact version each time!
keymone 5y ago
# GOOD: image: "nginx:1.21.6" nothing "good" about it, literally no different from ":latest". only full hash reference.
- fuzzy2 5y ago
  Sure version tags could also move, but by convention they do not. Unlike the latest tag, which by convention does move a lot.
  - isbvhodnvemrwvn 5y ago
    By convention npm packages are not deleted or hijacked.
    - fuzzy2 5y ago
      You do realize that NPM package versions already are immutable? Dunno since when though. Deletion is possible within limits but it also is with Docker. All the hashes in the world won't bring back a deleted image.
      - keymone 5y ago
        version numbers are never immutable. they are arbitrary labels that are created by some centralized authority and can be changed by that same authority. artifacts being removed is much less of a problem as artifacts being spoofed with malicious content.
      - isbvhodnvemrwvn 5y ago
        They were made immutable after the issue was already widespread. Which is also why I'd heavily encourage to use hashes to pin container versions, even though people might not see the immediate need to do so.
- delusional 5y ago
  From the original article > This brings up an interesting side point, in that Docker Hub and most other registries allow mutable tags by default. So nginx:1.21.6 might not be the same image today as it was yesterday. In reality, you probably need a mechanism to enforce tag immutability: e.g., your own registry mirror, or referring to images by SHA)
  - keymone 5y ago
    yeah, well, i stopped reading when i saw that "GOOD" example. it's not. and that isn't a "side point", that's the most critical point for preserving security and reproducibility of builds.
    - delusional 5y ago
      It's literally the line that follows that example. Why do you think it's useful to comment when you haven't even read the topic of discussion. You're like a person not reading the book at a voluntary book club. If you don't want to read it, just don't show up.
      - keymone 5y ago
        i suggest you to read my comment again, because so far what you've said applies to you more than to me.
    - zufallsheld 5y ago
      Well then you'd vendor your docker images anyway and not pull them from the internet, right?
      - keymone 5y ago
        Sure, and many do that, but you don't have to if you refer by digest of image's contents.
- hughrr 5y ago
  Stuff like this makes me miss pinned RPM packages.
saulr 5y ago
Worth noting that Hadolint[1] raises warnings the issues mentioned in the article. Some examples of warnings: - https://github.com/hadolint/hadolint/wiki/DL3007 https://github.com/hadolint/hadolint/wiki/DL3007: Using latest is prone to errors if the image will ever update. Pin the version explicitly to a release tag. - https://github.com/hadolint/hadolint/wiki/DL3013 https://github.com/hadolint/hadolint/wiki/DL3013: Pin versions in pip. - https://github.com/hadolint/hadolint/wiki/DL3018 https://github.com/hadolint/hadolint/wiki/DL3018: Pin versions in apk add. [1] https://github.com/hadolint/hadolint https://github.com/hadolint/hadolint
- ievans 5y ago
  Hadolint is great! If you want to customize your lint logic beyond the checks in it, I recently wrote a Semgrep rule to require all our Dockerfiles to pin images with a sha256 hash that could be a good starting point: https://github.com/returntocorp/semgrep-rules/pull/1861/files https://github.com/returntocorp/semgrep-rules/pull/1861/file...
deleted 5y ago
[deleted]
oxfordmale 5y ago
This is always a balance. The moment you pin to a specific version you need to have a process in place to ensure you regularly upgrade to avoid introducing vulnerabilities in your production system. Throughout my career I have seen many cases where certain software still runs on ancient versions as the team originally maintaining it is no longer around (e.g. reorganisations or lay-offs). It is always hard to convince senior management to invest any resources to upgrade (If it works don't fix it) If I have project that has good unit test coverage, I prefer to use :latest, as this results in a gradual update over time. If something breaks due to a version discrepancy, it is a lot easier to convince management to fix this, as the breakage would only be noticed as part of a feature request, and often would only require a small amount of work.
- pojzon 5y ago
  Its hard to convince management that them adapting devops means they actually have to follow devops principles. Too many times Ive seen companies throwing this buzzword around without understanding how much work is actually required to have real devops.
  - cube00 5y ago
    :s/devops/agile/g
- yardstick 5y ago
  Doesn’t the same logic apply to using :latest? If you are always pulling the latest, unvetted, code, shouldn’t you also have a vulnerability management process in place? You need vulnerability & risk management for any dependency, full stop.
  - oxfordmale 5y ago
    Ideally you only use Docker official images,or their equivalent to avoid using unvetted code. It is always a trade off, however it is far more likely that a hacker will use a ten year old well exploited CVE, rather than a recent one
jbverschoor 5y ago
:latest sounds like npm without lockfiles.
erwincoumans 5y ago
Same with pip/Python. Recucing dependencies to a bare minimum helps long term survival of software.
- paskozdilar 5y ago
  It also helps with reducing dependency distribution issues - I've written single-file Python programs using just the stdlib, sometimes of a really old Python3 version. It's really nice to just `scp` the script to a bunch of remote devices and have everything work out-of-the-box. Of course, PyInstaller can solve dependency distribution issues - but I've sometimes ran into packages that don't play well with PyInstaller (uwsgi, for example, because it's basically a C program distributed using pip, so PyInstaller can't figure out how to bundle it).
- ComradePhil 5y ago
  But it may hinder short-term survival (because you're spending time reinventing the wheel instead of building competitive product). Lockfiles (poetry.lock, Pipfile.lock etc.) solve the survivality problem for the most part.
  - izacus 5y ago
    > But it may hinder short-term survival (because you're spending time reinventing the wheel instead of building competitive product). That's very rare. It's a common trope though and causes amazing examples of startup companies self-destructing themselves when they end up being bogged down in dependency maintenance wars where they're spending a lot of time just fighting all the issues in 100s of alpha libraries they've put on to be "faster". The argument for that was always the one you used. Adding a dependency isn't just "free code", it's also "free code that will probably break at some time and will need to be updated and deconflicted." In my experience, a good set of older, tested and well maintained core depdenencies is usually much better for company, even if that means that they themselves maintain a piece of functionality that exists as a library.
    - ComradePhil 5y ago
hpaavola 5y ago
Use latest during development, but push to production using image in your private Docker registry which has proper names and tags. git push -> Docker build (bonus points for building only if needed [for example only if Dockerfile, Jenkinsfile or requirements.txt has changed], otherwise use latest from Artifactory) -> run all automated tests -> if pass, push the Docker image to Artifactory with reasonable name and tag. When doing a release, push the image from Artifactory to production. This way you don't have to have a process to update all the things in Dockerfile on reqular basis, but you still only push to production the actual binaries that are really tested and proven to be ok. And can re-use the actual binary when doing a hotfix of something similar.
- peakaboo 5y ago
  So what actually always happens with this approach is that companies stay on ancient docker images, because there is never time for updates as long as they work. It's the same with locking python pip packages to specific versions. Nobody ever looks at it again and you run code that is 5 years old in production. People only look at it when it breaks.
  - yardstick 5y ago
    At least they would have never upgraded to log4j 2…
  - hpaavola 5y ago
    I just said that use latest exactly to avoid this issue.
    - peakaboo 5y ago
      You said to use it for development. But I don't use Artifactory so maybe you meant to use the latest for production also, if it passes the build.
      - hpaavola 5y ago
        Having robust process requires some tools. Artifactory (or any other private Docker registry) is one such tool.
bee_rider 5y ago
It seems kind of weird that we don't do "stable repositories" anymore.
charcircuit 5y ago
>It breaks one of the core requirements of continuous delivery: reproducible, idempotent builds As long as you are able to look up what latest evaluated to during the build you retain most of the benefits while getting automatic updates.
solatic 5y ago
Forget for a second the security concerns - If you lose :latest, and the build breaks, you need to spend unexpected time fixing the build, for whatever reason that :latest broke it, in order to continue with the business objective that you were actually tasked to do. Essentially, you have InfoSec muscling in and demanding, by the most coercive way possible, that Thou Shalt Not Do Your Job Without Bowing Before Me. Who cares what the business wanted? Who cares about predictable schedules? Who cares about delivering for a customer by a point in time which was promised to the customer? This isn't the way to promote a healthy culture in the company. Nobody should get immediate and total veto rights. In the real world, healthy organizations balance between different stakeholders, and once a path is set, everybody shuts up and executes. The right place to make a stand for your interests is in the stakeholder meeting, not the deployment pipeline.
- ashildr 5y ago
  > you have InfoSec muscling in and demanding, by the most coercive way possible, that Thou Shalt Not Do Your Job Without Bowing Before Me. InfoSec is not optional.
  - solatic 5y ago
    Of course it's not. But it's not King, either.
    - jimmydorry 5y ago
      I could see it argued that in $CurrentYear, any information oriented company that doesn't put InfoSec as their #1 priority, is just asking to be pwned. It's not an if, but a when and to what extent.
      - izacus 5y ago
        No, #1 priority is always doing core business - the job that gets money in, satisfies users and keeps the company running. Everything else comes after - without core business, infosec is pointless and can't sustain itself. InfoSec is critically important, but it's important just like IT people, janitors and server maintainers - business breaks without them, but they aren't actually earning money and prioritizing them over core business is the tail wagging the dog. (And yes, I've seen way too much entitled "InfoSec" experts explicitly undermining their own company because they forgot that. Read The Phoenix Project or similar for concrete examples.)
sascha_sl 5y ago
> In reality, you probably need a mechanism to enforce tag immutability: e.g., your own registry mirror. Ah, that sounds so trivial. And yet it is very much not. Aside from the hot mess that is the docker registry daemon, some tags are meant to be rolling. You absolutely do not want them cached.
jahirul247 5y ago
[flagged]
ashildr 5y ago
There are containers that I want ASAP to be updated to the latest version which likely contains all known security fixes. I consider the risk of missing an update for a - then public - security problem greater than the risk of getting a new, still unknown security risk. While I am trying to, I will not have the time to be really on top of all updates at all times. And I am more comfortable with a container breaking than with a container missing an important update. There are aspects where reproducibility is key, but thats only a subset of all containers on our network. And on my home network it’s none at all.
throwaway984393 5y ago
Eh. It depends. There are exceptions to every rule. latest is useful for builds where you want to know when something will break. If you only stick to pinned versions, it's common for nobody to ever update the pinned versions for years. Then you're way behind the latest versions and suddenly upgrading becomes a huge pain. You may be stuck in a situation where you're forced to upgrade because of a security hole in your pinned version, but to upgrade to a patched version breaks everything. ".....But that's terrible!", you say. "My build won't be repeatable! I won't be able to perform a roll-back build! My build will constantly break!" All of that is true, but there are workarounds. For example, if you download, version, and store all artifacts used for each build, you can reuse them later. This isn't hard if you take the time to write some scripts. You can pull a container and export it to a tarball and store it. System packages and language dependencies can be downloaded similarly, and repos mirrored. You can do this once a week and version all artifacts with a datestamp, and make your own app builds pinned to a particular datestamp. If you always build from those versioned archived datestamps, you can always rebuild or revert to an old working latest build. As far as builds breaking, they certainly will! You need good testing to catch bugs and regressions. But would you rather learn to adapt quickly to broken builds, or have a sleeping tiger waiting to bite you the one time you finally have to upgrade quickly? Using a stable branch/tag is the safest hedge against frequently breaking builds while still getting security patches. But stable branches still introduce problems. You will eventually need to revert a stable change, and eventually the stable branch will be End Of Life. So even if you use stable, you should still use the practice of installing from downloaded versioned artifacts. Since EOL will come eventually, you also have to commit to upgrading to a new stable branch and its breaking changes. You must therefore plan to sunset your own code. Find out when your dependencies will EOL, and plan to completely rebuild your apps using latest before then. It has to be a real commitment and not just a "nice to have", because you will end up being forced to do it by a security vuln or repos that stop carrying an old branch. Ultimately you need to decide how much risk to take and how much planning is needed to avoid sticky situations. Be aware of the consequences of your design and have contingency plans.
jacobr 5y ago
People are mentioning that pinning versions lead to overhead in updates. These people have probably not used Renovate https://docs.renovatebot.com/ https://docs.renovatebot.com/ Renovate is smart enough to understand that a task such as “Update Node to v16” means updating .nvmrc files, updating FROM in Dockerfiles and engines in package.json in a single PR it creates for you.
- cesnja 5y ago
  Renovate is great, far better than relying on somebody remembering to upgrade dependencies in 12 different places. But if you want full coverage, there will be a need to write a few regex managers. And testing can only be done in production on the mainline branch. The best way I found to do that was forking the repo to configure and try out renovate separately.
- onedr0p 5y ago
  Renovate is amazing, and a tool most Developer and DevOps engineers should learn. It's FOSS, and does a great job at dependency management across a very wide range of software like NPM, Terraform, Docker, Ansible, Maven, Golang, Rust and many others. https://docs.renovatebot.com/modules/manager/ https://docs.renovatebot.com/modules/manager/
ComradePhil 5y ago
Terrible advice. It doesn't solve the "immutability" problem and may give you a false sense that it does, which is a much bigger problem. Here's the "good" example from the article: # GOOD: image: "nginx:1.21.6" Here is the header of the source of the Dockerfile for this image: FROM debian:bullseye-slim LABEL maintainer="NGINX Docker Maintainers <docker-maint@nginx.com>" ENV NGINX_VERSION 1.21.6 ENV NJS_VERSION 0.7.2 ENV PKG_RELEASE 1~bullseye ... ... ... Notice the first line? `debian:bullseye-slim` is not "immutable". Should the `nginx:1.21.6` image auto-update when new `debian:bullseye-slim` is available? Or should it keep it what it was during the build time? I am not exactly sure how nginx does it but it would not be incorrect to do either (both address different issues and are completely valid depending on what you want to do). If you are serious about using docker, you need a private registry and you need to make a tagging policy that is best for YOUR use-case.
- badLiveware 5y ago
  The easier solution rather than managing your own OCI registry is likely to just pin the digest and have dependency update automation e.g. renovate update the digest while targeting a tag. For example FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:70b890cd12f73f8ad80061d242081b61da666bda7ec2d729113855a8b9410e1e AS build Where the tag is used by humans for targeting a tag while the digest locks it to a certain image version
deleted 5y ago
[deleted]
timost 5y ago
Maintaining docker containers up to date can quickly become time consumming compared to good old distribution packages. I use both and I'm still trying to find a good combination of tools to reproduce the "package" and unattended upgrades UX where things are safely kept up to date automatically.
ratherbefuddled 5y ago
In practise using latest or just the major version has caused far less wasted effort for our team than scrambling to update versions because of some newly discovered CVE. It's a problem in frontend builds because js libs are quite poorly behaved so we fix more there but any backend stuff, be it alpine / debian base images, jvm libs, toolchains so on we've not had an issue with specifying only major versions. Dogmatic rules are wrong 100% of the time.
KaiserPro 5y ago
Yes, and no. This does mean that you have to actively manage your versions for any security fixes. Which you should be doing anyway. However when upgrading or patching you have to check in a bunch of places to make sure the update was effective. As someone mentioned else where, its better to pin to major versions rather than specific. You'll need to make sure you have decent integration tests and monitoring though. I mean you have that already right?
Havoc 5y ago
Depends on usage case. At home I :latest everything. The work of constant adjusting tags to bring it up to date is more hassle than the risk of something breaking. Prod stuff....yeah best to pin it for supervised upgrades
3np 5y ago
Call me stuffy but I argue for writing your own Dockerfiles, all from one or a small number of common base images (e.g. scratch+debian, or what have you), served from your self-hosted registry. If you want to use some provided official image for vendored software, just port it over. Then you tie the build+push into your pipeline however suits your org, be it fully manual or end-to-end CI with regular rebuilds. This solves several issues mentioned elsewhere in the thread. As a bonus you won't be affected should docker.io or quay have issues or make breaking changes.
thenerdhead 5y ago
The age old question is "How quickly do you update?" Too slow - Exposed to known vulnerabilities. Too fast - Susceptible to active supply chain attacks. There's tradeoffs, and there's no solution here. It's a hard problem.
- lbriner 5y ago
  Exactly this. The OP reads like a nice perfect world scenario but... 1) Just because there is a vulnerability in a package of library doesn't mean that you are susceptible to that vulnerability 2) If you don't update immediately when will you? Most companies can't afford people to understand all posted vulnerabilities to work out whether updating is better than not updating The truth is there are easier ways to reduce risk than pretending that a pinned version somehow solves a lot of problems: Reduce your codebase/dependencies and apply defence-in-depth. Reproducability is something for some companies but, again, I doubt there would be many times that I would care about what exactly was deployed 6 months ago. If something is broken now, it gets fixed now.
  - bigfudge 5y ago
    I really like Microsoft’s mirror of CRAN because it’s versioned by date- you can install everything as it was on a particular date, so that kind of reproducibility is easy. I wonder if this could be added to pypi
- ziml77 5y ago
  Not just from a vulnerability poerspective but also a dev and testing perspective. Constant updates makes testing far more involved for every release since you can't just focus on the parts the dev work would have affected, you also need to thoroughly test everything that was calling into the updated dependency. But if you don't update enough, once you really do need to update either for security or feature reasons, you may have a nightmarish process ahead of you of dealing with breaking changes and transitive dependency incompatibilities.
jayzalowitz 5y ago
Speaking as potentially one of the most prolific editors in tech history, your wife was right for the wrong reason. You needed your article to state an example where latest dependency would kill a puppy. Then your title is valid.
bombcar 5y ago
With a bit of work you can run with pinned versions that act like :latest - in the equivalent gradle scripts we have explicit dependencies but we have another script that updates them to whatever is actually the latest at the time. So it’s like master-SNAPSHOT when we need it to be, but still with reproducibility and regular updates.
bearjaws 5y ago
The security questionaires we have to perform for our larger customers now include that we pin the exact hash of a container or mirror any image locally. This is in addition to pinning versions of NPM packages and using our package lock for installation. We now have a basic rule that :latest doesn't make it past proof of concept stage. This has made automation of our package & container maintenance crucial, since we are a lean shop. It's pretty much a weekly occurrence that a low-medium vulnerability pops up somewhere once your application is large enough.
crucialfelix 5y ago
Agree. I think it's better to explicitly update major versions and know to keep an eye on the deployment in case it's something that tests can't cover. I had a CloudBuild fail to deploy due to a bug in one of the builders :latest. Ironically the bug was due to a change in error logging, so it failed silently. So it's not just docker or dependencies. In this case I could've used a GCP notify that the builder failed.
cedws 5y ago
Pinning image tags is only useful if the upstream doesn't keep retagging images. I've had a case in the past where a vendor has retagged an image and broken something. Maybe we should even be pinning to a specific layer/checksum like so: FROM ubuntu@sha256:8ae9bafbb64f63a50caab98fd3a5e37b3eb837a3e0780b78e5218e63193961f9
- addingnumbers 5y ago
  The article covered that five paragraphs in: > In reality, you probably need a mechanism to enforce tag immutability: e.g., your own registry mirror, or referring to images by SHA).
ipsocannibal 5y ago
I think this is an X/Y problem. The problem isn't pinning to latest. The problem is lack of automation that makes builds reproducable. In at least one FANG company I've worked for if you aren't building from latest no one will listen to your issue. Too much software changes to quickly to be pinning to specific versions. However, the build system keeps track of the build audit details and can rollback any build to any state. Teams are required to add the necessary layers of unit, integration, stress, crush, and chaos testing to validate each build. Its not cheap but when you need to do a monthly firedrill of 'emergency update this dep because of Z vulnerability' its worth it.
snicker7 5y ago
Dockerfiles are generally bad for reproducibility. This is mostly because Linux distros generally don’t care about it. Nix and Guix both support Docker export.
raffraffraff 5y ago
I somewhat agree. But it's not that simple in practice. For example, I pin every single version and the container fails a security scan because it's not using latest versions of everything. Or I pin to arbitrary versions that happened to be latest when pip/npm/terraform first created the precious word-of-god lock file, leaving future engineers afraid to upgrade those packages because "Chesterton's Fence". Pinning versions creates work in the future. Using latest versions can result in ad-hoc breakages and security risk. So instead of saying "v1.22=good, latest=bad" and talking generally about the fact that lock files exist in most decent systems, I'd like an article that contains example strategies for paying back the tech debt of a thousand arbitrarily pinned versions across your code projects. I like the idea of deleting the lock file on a regular cadence, accepting whatever new shit comes in, perhaps bumping a major version number, and then testing the crap out of everything and leaving this new artifact in integration/staging for a few extra days. If things go wrong between builds, then you have the lock files in git (and in your artifacts) to let you know what changed. If it turns out that you now have a real reason to pin a specific package (ie: it broke something), then you have to find some way to note that outside of the usual lock files, since they arbitrarily lock everything.
- mr_tristan 5y ago
  Pinning can be far less painful with tooling to review and manage all the changes. One example would be the Maven versions plugin: you can use it to query "what's been updated", update to next release, etc. The list of goals is pretty explanatory, and a good starting point: https://www.mojohaus.org/versions-maven-plugin/ https://www.mojohaus.org/versions-maven-plugin/ When evaluating new toolchains, I often ask myself "what is the versions plugin equivalent here". There's often commercial offerings that help this process out. Dependency management is one of these topics that rarely gets attention until much later in the ecosystem's lifetime. So chances are you'll need to find or create tooling to fill gaps. I mean, you can do the whole "damn the torpedos and see what breaks" approach, but, I'd prefer a way to review changes. And I'd prefer doing those updates _separate_ from other changes.
- 88913527 5y ago
  Needlessly upgrading all of your packages leaves you open to supply chain attacks. If one of those NPM packages got quietly compromised (and my lockfile likely has 4-to-5 digits worth of dependencies in it), you invested time in making what amounts to an unneeded change to expose yourself.
Jhsto 5y ago
In a perfect one would use CRIU [0] to suspend the old container, then download the new one and start it. If the service comes up with the new container, then remove the suspended one and otherwise, fallback to the suspended container. [0]: https://criu.org/Docker https://criu.org/Docker
neonate 5y ago
https://archive.ph/XeoiH https://archive.ph/XeoiH http://web.archive.org/web/20220306110425/https://platformers.dev/log/2022-03-02-latest-literally-kills-puppies/ http://web.archive.org/web/20220306110425/https://platformer...
deleted 5y ago
[deleted]
tlarkworthy 5y ago
If you don't use latest, your ecosystem fragments and risks diamond dependencies. If you do use latest, some of your ecosystem risks randomly breaking. I think it's best to use latest because the combinatorics of the ecosystem as a whole is much smaller. With everything on latest, you have to play whack-mole on individual services to get them to catch up. But you are doing this just when the community at large is most invested in understanding the process of upgrading. The alternative is that ancient binary A depends on ancient binary B's legacy behaviour, and B has no need to upgrade but A have a very serious need and now you have a serious problem. Maybe upgrading B breaks C for instance. It's almost uncoordinateable at that point once versions drift significantly. You get true technical debt.
outside1234 5y ago
Going even further with this - it has always struck me that the ALWAYS should be just be a hash of the docker image such that it is essentially content addressed and can be verified on the receiver end. Is there any reason that this isn't a good idea?

229 points by kiyanwang 5y ago

I agree. Or provide something in parallel such as: :lts -- newest long term stable release :release -- newest regular stable release :x.y.z -- supported but you need find out the exact version each time!

keymone 5y ago

# GOOD: image: "nginx:1.21.6" nothing "good" about it, literally no different from ":latest". only full hash reference.

fuzzy2 5y ago
Sure version tags could also move, but by convention they do not. Unlike the latest tag, which by convention does move a lot.
- isbvhodnvemrwvn 5y ago
  By convention npm packages are not deleted or hijacked.
  - fuzzy2 5y ago
    You do realize that NPM package versions already are immutable? Dunno since when though. Deletion is possible within limits but it also is with Docker. All the hashes in the world won't bring back a deleted image.
    - keymone 5y ago
      version numbers are never immutable. they are arbitrary labels that are created by some centralized authority and can be changed by that same authority. artifacts being removed is much less of a problem as artifacts being spoofed with malicious content.
    - isbvhodnvemrwvn 5y ago
      They were made immutable after the issue was already widespread. Which is also why I'd heavily encourage to use hashes to pin container versions, even though people might not see the immediate need to do so.
delusional 5y ago
From the original article > This brings up an interesting side point, in that Docker Hub and most other registries allow mutable tags by default. So nginx:1.21.6 might not be the same image today as it was yesterday. In reality, you probably need a mechanism to enforce tag immutability: e.g., your own registry mirror, or referring to images by SHA)
- keymone 5y ago
  yeah, well, i stopped reading when i saw that "GOOD" example. it's not. and that isn't a "side point", that's the most critical point for preserving security and reproducibility of builds.
  - delusional 5y ago
    It's literally the line that follows that example. Why do you think it's useful to comment when you haven't even read the topic of discussion. You're like a person not reading the book at a voluntary book club. If you don't want to read it, just don't show up.
    - keymone 5y ago
      i suggest you to read my comment again, because so far what you've said applies to you more than to me.
  - zufallsheld 5y ago
    Well then you'd vendor your docker images anyway and not pull them from the internet, right?
    - keymone 5y ago
      Sure, and many do that, but you don't have to if you refer by digest of image's contents.
hughrr 5y ago
Stuff like this makes me miss pinned RPM packages.

saulr 5y ago

Worth noting that Hadolint[1] raises warnings the issues mentioned in the article. Some examples of warnings: - https://github.com/hadolint/hadolint/wiki/DL3007 https://github.com/hadolint/hadolint/wiki/DL3007: Using latest is prone to errors if the image will ever update. Pin the version explicitly to a release tag. - https://github.com/hadolint/hadolint/wiki/DL3013 https://github.com/hadolint/hadolint/wiki/DL3013: Pin versions in pip. - https://github.com/hadolint/hadolint/wiki/DL3018 https://github.com/hadolint/hadolint/wiki/DL3018: Pin versions in apk add. [1] https://github.com/hadolint/hadolint https://github.com/hadolint/hadolint

ievans 5y ago
Hadolint is great! If you want to customize your lint logic beyond the checks in it, I recently wrote a Semgrep rule to require all our Dockerfiles to pin images with a sha256 hash that could be a good starting point: https://github.com/returntocorp/semgrep-rules/pull/1861/files https://github.com/returntocorp/semgrep-rules/pull/1861/file...

deleted 5y ago

[deleted]

oxfordmale 5y ago

This is always a balance. The moment you pin to a specific version you need to have a process in place to ensure you regularly upgrade to avoid introducing vulnerabilities in your production system. Throughout my career I have seen many cases where certain software still runs on ancient versions as the team originally maintaining it is no longer around (e.g. reorganisations or lay-offs). It is always hard to convince senior management to invest any resources to upgrade (If it works don't fix it) If I have project that has good unit test coverage, I prefer to use :latest, as this results in a gradual update over time. If something breaks due to a version discrepancy, it is a lot easier to convince management to fix this, as the breakage would only be noticed as part of a feature request, and often would only require a small amount of work.

pojzon 5y ago
Its hard to convince management that them adapting devops means they actually have to follow devops principles. Too many times Ive seen companies throwing this buzzword around without understanding how much work is actually required to have real devops.
- cube00 5y ago
  :s/devops/agile/g
yardstick 5y ago
Doesn’t the same logic apply to using :latest? If you are always pulling the latest, unvetted, code, shouldn’t you also have a vulnerability management process in place? You need vulnerability & risk management for any dependency, full stop.
- oxfordmale 5y ago
  Ideally you only use Docker official images,or their equivalent to avoid using unvetted code. It is always a trade off, however it is far more likely that a hacker will use a ten year old well exploited CVE, rather than a recent one

jbverschoor 5y ago

:latest sounds like npm without lockfiles.

erwincoumans 5y ago

Same with pip/Python. Recucing dependencies to a bare minimum helps long term survival of software.

paskozdilar 5y ago
It also helps with reducing dependency distribution issues - I've written single-file Python programs using just the stdlib, sometimes of a really old Python3 version. It's really nice to just `scp` the script to a bunch of remote devices and have everything work out-of-the-box. Of course, PyInstaller can solve dependency distribution issues - but I've sometimes ran into packages that don't play well with PyInstaller (uwsgi, for example, because it's basically a C program distributed using pip, so PyInstaller can't figure out how to bundle it).
ComradePhil 5y ago
But it may hinder short-term survival (because you're spending time reinventing the wheel instead of building competitive product). Lockfiles (poetry.lock, Pipfile.lock etc.) solve the survivality problem for the most part.
- izacus 5y ago
  > But it may hinder short-term survival (because you're spending time reinventing the wheel instead of building competitive product). That's very rare. It's a common trope though and causes amazing examples of startup companies self-destructing themselves when they end up being bogged down in dependency maintenance wars where they're spending a lot of time just fighting all the issues in 100s of alpha libraries they've put on to be "faster". The argument for that was always the one you used. Adding a dependency isn't just "free code", it's also "free code that will probably break at some time and will need to be updated and deconflicted." In my experience, a good set of older, tested and well maintained core depdenencies is usually much better for company, even if that means that they themselves maintain a piece of functionality that exists as a library.
  - ComradePhil 5y ago

hpaavola 5y ago

Use latest during development, but push to production using image in your private Docker registry which has proper names and tags. git push -> Docker build (bonus points for building only if needed [for example only if Dockerfile, Jenkinsfile or requirements.txt has changed], otherwise use latest from Artifactory) -> run all automated tests -> if pass, push the Docker image to Artifactory with reasonable name and tag. When doing a release, push the image from Artifactory to production. This way you don't have to have a process to update all the things in Dockerfile on reqular basis, but you still only push to production the actual binaries that are really tested and proven to be ok. And can re-use the actual binary when doing a hotfix of something similar.

peakaboo 5y ago
So what actually always happens with this approach is that companies stay on ancient docker images, because there is never time for updates as long as they work. It's the same with locking python pip packages to specific versions. Nobody ever looks at it again and you run code that is 5 years old in production. People only look at it when it breaks.
- yardstick 5y ago
  At least they would have never upgraded to log4j 2…
- hpaavola 5y ago
  I just said that use latest exactly to avoid this issue.
  - peakaboo 5y ago
    You said to use it for development. But I don't use Artifactory so maybe you meant to use the latest for production also, if it passes the build.
    - hpaavola 5y ago
      Having robust process requires some tools. Artifactory (or any other private Docker registry) is one such tool.

bee_rider 5y ago

It seems kind of weird that we don't do "stable repositories" anymore.

charcircuit 5y ago

>It breaks one of the core requirements of continuous delivery: reproducible, idempotent builds As long as you are able to look up what latest evaluated to during the build you retain most of the benefits while getting automatic updates.

solatic 5y ago

Forget for a second the security concerns - If you lose :latest, and the build breaks, you need to spend unexpected time fixing the build, for whatever reason that :latest broke it, in order to continue with the business objective that you were actually tasked to do. Essentially, you have InfoSec muscling in and demanding, by the most coercive way possible, that Thou Shalt Not Do Your Job Without Bowing Before Me. Who cares what the business wanted? Who cares about predictable schedules? Who cares about delivering for a customer by a point in time which was promised to the customer? This isn't the way to promote a healthy culture in the company. Nobody should get immediate and total veto rights. In the real world, healthy organizations balance between different stakeholders, and once a path is set, everybody shuts up and executes. The right place to make a stand for your interests is in the stakeholder meeting, not the deployment pipeline.

ashildr 5y ago
> you have InfoSec muscling in and demanding, by the most coercive way possible, that Thou Shalt Not Do Your Job Without Bowing Before Me. InfoSec is not optional.
- solatic 5y ago
  Of course it's not. But it's not King, either.
  - jimmydorry 5y ago
    I could see it argued that in $CurrentYear, any information oriented company that doesn't put InfoSec as their #1 priority, is just asking to be pwned. It's not an if, but a when and to what extent.
    - izacus 5y ago
      No, #1 priority is always doing core business - the job that gets money in, satisfies users and keeps the company running. Everything else comes after - without core business, infosec is pointless and can't sustain itself. InfoSec is critically important, but it's important just like IT people, janitors and server maintainers - business breaks without them, but they aren't actually earning money and prioritizing them over core business is the tail wagging the dog. (And yes, I've seen way too much entitled "InfoSec" experts explicitly undermining their own company because they forgot that. Read The Phoenix Project or similar for concrete examples.)

sascha_sl 5y ago

> In reality, you probably need a mechanism to enforce tag immutability: e.g., your own registry mirror. Ah, that sounds so trivial. And yet it is very much not. Aside from the hot mess that is the docker registry daemon, some tags are meant to be rolling. You absolutely do not want them cached.

jahirul247 5y ago

[flagged]

ashildr 5y ago

There are containers that I want ASAP to be updated to the latest version which likely contains all known security fixes. I consider the risk of missing an update for a - then public - security problem greater than the risk of getting a new, still unknown security risk. While I am trying to, I will not have the time to be really on top of all updates at all times. And I am more comfortable with a container breaking than with a container missing an important update. There are aspects where reproducibility is key, but thats only a subset of all containers on our network. And on my home network it’s none at all.

throwaway984393 5y ago

Eh. It depends. There are exceptions to every rule. latest is useful for builds where you want to know when something will break. If you only stick to pinned versions, it's common for nobody to ever update the pinned versions for years. Then you're way behind the latest versions and suddenly upgrading becomes a huge pain. You may be stuck in a situation where you're forced to upgrade because of a security hole in your pinned version, but to upgrade to a patched version breaks everything. ".....But that's terrible!", you say. "My build won't be repeatable! I won't be able to perform a roll-back build! My build will constantly break!" All of that is true, but there are workarounds. For example, if you download, version, and store all artifacts used for each build, you can reuse them later. This isn't hard if you take the time to write some scripts. You can pull a container and export it to a tarball and store it. System packages and language dependencies can be downloaded similarly, and repos mirrored. You can do this once a week and version all artifacts with a datestamp, and make your own app builds pinned to a particular datestamp. If you always build from those versioned archived datestamps, you can always rebuild or revert to an old working latest build. As far as builds breaking, they certainly will! You need good testing to catch bugs and regressions. But would you rather learn to adapt quickly to broken builds, or have a sleeping tiger waiting to bite you the one time you finally have to upgrade quickly? Using a stable branch/tag is the safest hedge against frequently breaking builds while still getting security patches. But stable branches still introduce problems. You will eventually need to revert a stable change, and eventually the stable branch will be End Of Life. So even if you use stable, you should still use the practice of installing from downloaded versioned artifacts. Since EOL will come eventually, you also have to commit to upgrading to a new stable branch and its breaking changes. You must therefore plan to sunset your own code. Find out when your dependencies will EOL, and plan to completely rebuild your apps using latest before then. It has to be a real commitment and not just a "nice to have", because you will end up being forced to do it by a security vuln or repos that stop carrying an old branch. Ultimately you need to decide how much risk to take and how much planning is needed to avoid sticky situations. Be aware of the consequences of your design and have contingency plans.

jacobr 5y ago

People are mentioning that pinning versions lead to overhead in updates. These people have probably not used Renovate https://docs.renovatebot.com/ https://docs.renovatebot.com/ Renovate is smart enough to understand that a task such as “Update Node to v16” means updating .nvmrc files, updating FROM in Dockerfiles and engines in package.json in a single PR it creates for you.

cesnja 5y ago
Renovate is great, far better than relying on somebody remembering to upgrade dependencies in 12 different places. But if you want full coverage, there will be a need to write a few regex managers. And testing can only be done in production on the mainline branch. The best way I found to do that was forking the repo to configure and try out renovate separately.
onedr0p 5y ago
Renovate is amazing, and a tool most Developer and DevOps engineers should learn. It's FOSS, and does a great job at dependency management across a very wide range of software like NPM, Terraform, Docker, Ansible, Maven, Golang, Rust and many others. https://docs.renovatebot.com/modules/manager/ https://docs.renovatebot.com/modules/manager/

ComradePhil 5y ago

Terrible advice. It doesn't solve the "immutability" problem and may give you a false sense that it does, which is a much bigger problem. Here's the "good" example from the article: # GOOD: image: "nginx:1.21.6" Here is the header of the source of the Dockerfile for this image: FROM debian:bullseye-slim LABEL maintainer="NGINX Docker Maintainers <docker-maint@nginx.com>" ENV NGINX_VERSION 1.21.6 ENV NJS_VERSION 0.7.2 ENV PKG_RELEASE 1~bullseye ... ... ... Notice the first line? `debian:bullseye-slim` is not "immutable". Should the `nginx:1.21.6` image auto-update when new `debian:bullseye-slim` is available? Or should it keep it what it was during the build time? I am not exactly sure how nginx does it but it would not be incorrect to do either (both address different issues and are completely valid depending on what you want to do). If you are serious about using docker, you need a private registry and you need to make a tagging policy that is best for YOUR use-case.

badLiveware 5y ago
The easier solution rather than managing your own OCI registry is likely to just pin the digest and have dependency update automation e.g. renovate update the digest while targeting a tag. For example FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:70b890cd12f73f8ad80061d242081b61da666bda7ec2d729113855a8b9410e1e AS build Where the tag is used by humans for targeting a tag while the digest locks it to a certain image version

deleted 5y ago

[deleted]

timost 5y ago

Maintaining docker containers up to date can quickly become time consumming compared to good old distribution packages. I use both and I'm still trying to find a good combination of tools to reproduce the "package" and unattended upgrades UX where things are safely kept up to date automatically.

ratherbefuddled 5y ago

In practise using latest or just the major version has caused far less wasted effort for our team than scrambling to update versions because of some newly discovered CVE. It's a problem in frontend builds because js libs are quite poorly behaved so we fix more there but any backend stuff, be it alpine / debian base images, jvm libs, toolchains so on we've not had an issue with specifying only major versions. Dogmatic rules are wrong 100% of the time.

KaiserPro 5y ago

Yes, and no. This does mean that you have to actively manage your versions for any security fixes. Which you should be doing anyway. However when upgrading or patching you have to check in a bunch of places to make sure the update was effective. As someone mentioned else where, its better to pin to major versions rather than specific. You'll need to make sure you have decent integration tests and monitoring though. I mean you have that already right?

Havoc 5y ago

Depends on usage case. At home I :latest everything. The work of constant adjusting tags to bring it up to date is more hassle than the risk of something breaking. Prod stuff....yeah best to pin it for supervised upgrades

3np 5y ago

Call me stuffy but I argue for writing your own Dockerfiles, all from one or a small number of common base images (e.g. scratch+debian, or what have you), served from your self-hosted registry. If you want to use some provided official image for vendored software, just port it over. Then you tie the build+push into your pipeline however suits your org, be it fully manual or end-to-end CI with regular rebuilds. This solves several issues mentioned elsewhere in the thread. As a bonus you won't be affected should docker.io or quay have issues or make breaking changes.

thenerdhead 5y ago

The age old question is "How quickly do you update?" Too slow - Exposed to known vulnerabilities. Too fast - Susceptible to active supply chain attacks. There's tradeoffs, and there's no solution here. It's a hard problem.

lbriner 5y ago
Exactly this. The OP reads like a nice perfect world scenario but... 1) Just because there is a vulnerability in a package of library doesn't mean that you are susceptible to that vulnerability 2) If you don't update immediately when will you? Most companies can't afford people to understand all posted vulnerabilities to work out whether updating is better than not updating The truth is there are easier ways to reduce risk than pretending that a pinned version somehow solves a lot of problems: Reduce your codebase/dependencies and apply defence-in-depth. Reproducability is something for some companies but, again, I doubt there would be many times that I would care about what exactly was deployed 6 months ago. If something is broken now, it gets fixed now.
- bigfudge 5y ago
  I really like Microsoft’s mirror of CRAN because it’s versioned by date- you can install everything as it was on a particular date, so that kind of reproducibility is easy. I wonder if this could be added to pypi
ziml77 5y ago
Not just from a vulnerability poerspective but also a dev and testing perspective. Constant updates makes testing far more involved for every release since you can't just focus on the parts the dev work would have affected, you also need to thoroughly test everything that was calling into the updated dependency. But if you don't update enough, once you really do need to update either for security or feature reasons, you may have a nightmarish process ahead of you of dealing with breaking changes and transitive dependency incompatibilities.

jayzalowitz 5y ago

Speaking as potentially one of the most prolific editors in tech history, your wife was right for the wrong reason. You needed your article to state an example where latest dependency would kill a puppy. Then your title is valid.

bombcar 5y ago

With a bit of work you can run with pinned versions that act like :latest - in the equivalent gradle scripts we have explicit dependencies but we have another script that updates them to whatever is actually the latest at the time. So it’s like master-SNAPSHOT when we need it to be, but still with reproducibility and regular updates.

bearjaws 5y ago

The security questionaires we have to perform for our larger customers now include that we pin the exact hash of a container or mirror any image locally. This is in addition to pinning versions of NPM packages and using our package lock for installation. We now have a basic rule that :latest doesn't make it past proof of concept stage. This has made automation of our package & container maintenance crucial, since we are a lean shop. It's pretty much a weekly occurrence that a low-medium vulnerability pops up somewhere once your application is large enough.

crucialfelix 5y ago

Agree. I think it's better to explicitly update major versions and know to keep an eye on the deployment in case it's something that tests can't cover. I had a CloudBuild fail to deploy due to a bug in one of the builders :latest. Ironically the bug was due to a change in error logging, so it failed silently. So it's not just docker or dependencies. In this case I could've used a GCP notify that the builder failed.

cedws 5y ago

Pinning image tags is only useful if the upstream doesn't keep retagging images. I've had a case in the past where a vendor has retagged an image and broken something. Maybe we should even be pinning to a specific layer/checksum like so: FROM ubuntu@sha256:8ae9bafbb64f63a50caab98fd3a5e37b3eb837a3e0780b78e5218e63193961f9

addingnumbers 5y ago
The article covered that five paragraphs in: > In reality, you probably need a mechanism to enforce tag immutability: e.g., your own registry mirror, or referring to images by SHA).

ipsocannibal 5y ago

I think this is an X/Y problem. The problem isn't pinning to latest. The problem is lack of automation that makes builds reproducable. In at least one FANG company I've worked for if you aren't building from latest no one will listen to your issue. Too much software changes to quickly to be pinning to specific versions. However, the build system keeps track of the build audit details and can rollback any build to any state. Teams are required to add the necessary layers of unit, integration, stress, crush, and chaos testing to validate each build. Its not cheap but when you need to do a monthly firedrill of 'emergency update this dep because of Z vulnerability' its worth it.

snicker7 5y ago

Dockerfiles are generally bad for reproducibility. This is mostly because Linux distros generally don’t care about it. Nix and Guix both support Docker export.

raffraffraff 5y ago

I somewhat agree. But it's not that simple in practice. For example, I pin every single version and the container fails a security scan because it's not using latest versions of everything. Or I pin to arbitrary versions that happened to be latest when pip/npm/terraform first created the precious word-of-god lock file, leaving future engineers afraid to upgrade those packages because "Chesterton's Fence". Pinning versions creates work in the future. Using latest versions can result in ad-hoc breakages and security risk. So instead of saying "v1.22=good, latest=bad" and talking generally about the fact that lock files exist in most decent systems, I'd like an article that contains example strategies for paying back the tech debt of a thousand arbitrarily pinned versions across your code projects. I like the idea of deleting the lock file on a regular cadence, accepting whatever new shit comes in, perhaps bumping a major version number, and then testing the crap out of everything and leaving this new artifact in integration/staging for a few extra days. If things go wrong between builds, then you have the lock files in git (and in your artifacts) to let you know what changed. If it turns out that you now have a real reason to pin a specific package (ie: it broke something), then you have to find some way to note that outside of the usual lock files, since they arbitrarily lock everything.

mr_tristan 5y ago
Pinning can be far less painful with tooling to review and manage all the changes. One example would be the Maven versions plugin: you can use it to query "what's been updated", update to next release, etc. The list of goals is pretty explanatory, and a good starting point: https://www.mojohaus.org/versions-maven-plugin/ https://www.mojohaus.org/versions-maven-plugin/ When evaluating new toolchains, I often ask myself "what is the versions plugin equivalent here". There's often commercial offerings that help this process out. Dependency management is one of these topics that rarely gets attention until much later in the ecosystem's lifetime. So chances are you'll need to find or create tooling to fill gaps. I mean, you can do the whole "damn the torpedos and see what breaks" approach, but, I'd prefer a way to review changes. And I'd prefer doing those updates _separate_ from other changes.
88913527 5y ago
Needlessly upgrading all of your packages leaves you open to supply chain attacks. If one of those NPM packages got quietly compromised (and my lockfile likely has 4-to-5 digits worth of dependencies in it), you invested time in making what amounts to an unneeded change to expose yourself.

Jhsto 5y ago

In a perfect one would use CRIU [0] to suspend the old container, then download the new one and start it. If the service comes up with the new container, then remove the suspended one and otherwise, fallback to the suspended container. [0]: https://criu.org/Docker https://criu.org/Docker

neonate 5y ago

https://archive.ph/XeoiH https://archive.ph/XeoiH http://web.archive.org/web/20220306110425/https://platformers.dev/log/2022-03-02-latest-literally-kills-puppies/ http://web.archive.org/web/20220306110425/https://platformer...

deleted 5y ago

[deleted]

tlarkworthy 5y ago

If you don't use latest, your ecosystem fragments and risks diamond dependencies. If you do use latest, some of your ecosystem risks randomly breaking. I think it's best to use latest because the combinatorics of the ecosystem as a whole is much smaller. With everything on latest, you have to play whack-mole on individual services to get them to catch up. But you are doing this just when the community at large is most invested in understanding the process of upgrading. The alternative is that ancient binary A depends on ancient binary B's legacy behaviour, and B has no need to upgrade but A have a very serious need and now you have a serious problem. Maybe upgrading B breaks C for instance. It's almost uncoordinateable at that point once versions drift significantly. You get true technical debt.

outside1234 5y ago

Going even further with this - it has always struck me that the ALWAYS should be just be a hash of the docker image such that it is essentially content addressed and can be verified on the receiver end. Is there any reason that this isn't a good idea?