16 ms·

Dockerfile Security Best Practices

407 points by gbrindisi 6y ago

tigrezno 6y ago
About the "apt upgrade". In major distributions, that's safe. They won't introduce major package versions, only minor (and security fixes). It should be totally OK to upgrade the packages. Also I don't get that it's advised to use "apt update" when you can't do "apt upgrade". What's the point??
- kortex 6y ago
  Apt update just updates the cache of the package lists - /var/lib/apt/lists/ - based on your lists - /etc/apt/sources.list.d/. "upgrade" actually upgrades the code on the system. `rm -rf /var/lib/apt/lists/*` in the same RUN decreases bloat and I think possibly decreases cache misses as well.
  - Nullabillity 6y ago
    > and I think possibly decreases cache misses as well. Unlikely, you'd still have misses because of stuff like file metadata mismatches (think modification times).
  - richardwhiuk 6y ago
    Base images usually ship with that done. `apt-get update` just undoes the good work of removing irrelevant lists.
    - acdha 6y ago
      Some base images do, but some don’t, and they update on their schedule rather than yours. Doing the update on your schedule is trivial, has no meaningful downside, and means it’s done and tested as quickly as you need.
    - kortex 6y ago
      Huh, TIL. I'll give that a try. Usually, that's the least of my problems. In my line of work, I'm often doing things which would likely horrify most webapp devs, like building a container with multiple conda envs in it. OTOH, most of these containers run on airgapped systems with petabytes of storage, so shipping a bit of bloat barely hurts the end user. But every time I do, I die a bit on the inside.
- diggan 6y ago
  It's not entirely clear from the article but my guess would be that you should really update the repository + base dependencies in the base image, not the "application" image or whatever you wanna call it. But not the end of the world if you're not reusing that base image. Feels weird that an article on "Security Best Practices" would advise _against_ latest versions of your software, especially using apt as an example which is usually used in stable distributions (major version upgrades happens seldom) like Ubuntu/Debian. But yeah, that last part doesn't make any sense. If you're not running `apt-get upgrade`, it doesn't make sense to run `apt-get update` as nothing is using the newly fetched data anyways...
- shaded-enmity 6y ago
  If you upgrade your packages during a build then inadvertently if 2 people build your Dockerfile at two distinct times they can end up with 2 different images. So rather than focusing on "we're using base-image:1.0.2" you need to start asking questions like "list all packages and versions and start comparing those". If there's a security issue you need to rebuild your images anyway and optimally you have a system in place that represents images and their dependencies as some sort of dependency graph structure, so when you upgrade your base image all dependent images get rebuilt automatically.
3pt14159 6y ago
Overall great post. > If you rely on latest you might silently inherit updated packages that in the best worst case might impact your application reliability, in the worst worst case might introduce a vulnerability. On this one I disagree. Your CI should handle reliability (and if not, you have a bigger problem) and you're more likely to patch a vulnerability than to introduce a new one and it's unlikely that by the time a PR hits production that the version is compromised. I understand that updating cuts both ways when it comes to security, but I agree with Matt Tait's ultimate conclusion when he spoke on this issue a few years ago: For most medium size and smaller companies constantly updating is safer than delaying. He had real world data and graphs of compromise windows, etc. Short answer was that attackers are more time motivated than defenders.
- dbrgn 6y ago
  Operating system updates are the responsibility of the base image. Use a base image that is regularly updated. For your own images, ensure that they are also being rebuilt regularly. There are a lot of semi-official Docker images on Docker Hub that are published once and never updated until the next software release. That is a huge anti-pattern in my view, those images should not be relied on for production.
- cooljacob204 6y ago
  Would you have a link to the talk? I couldn't find it with a google search.
  - 3pt14159 6y ago
    I don't have time to track down a link to a video at the moment, but it was at Infiltrate, an offensive-minded cybersecurity conference out of Miami.
    - kronin 6y ago
      Is it this one? https://vimeo.com/267445424 https://vimeo.com/267445424
- 6y ago
lysium 6y ago
Great post. Two points: ..If you don’t inspect the wget script you might as while pipe it into bash. .. How to distribute secrets if not by env? (which I agree! Honest question)
- nicolast 6y ago
  > How to distribute secrets if not by env? (which I agree! Honest question) You'll want to use BuildKit (`docker buildx`), see https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information https://docs.docker.com/develop/develop-images/build_enhance... [edit] My bad, that works for secrets needed at build time, not at runtime of course.
- freedomben 6y ago
  Disclaimer: I work for Red Hat as an OpenShift consultant so I'm biased There are competing pieces of advice for secure distribution of secrets, but my current preference comes down to one of these ways, depending on the organization: 1. OpenShift/Kubernetes Secrets mounted into the Pod at runtime. 2. Hashicorp Vault (has a really well designed API. It's very usable just with curl, which makes using it a joy) 3. Sealed Secrets (less experience here but it's looking positive right now) - https://github.com/bitnami-labs/sealed-secrets https://github.com/bitnami-labs/sealed-secrets If you're using a different PaaS besides OpenShift, it may also offer options worth considering (although do think about portability. These days apps move platforms every few years on average, though I think that may be changing now that K8s is becoming the standard).
  - csunbird 6y ago
    > 1. OpenShift/Kubernetes Secrets mounted into the Pod at runtime. Do you recommend mounting secrets as environment variables to the kubernetes pods instead of files?
    - freedomben 6y ago
      Yes, that is by far my preference. Much more 12 factor app-ish and framework independent. A lot of Java apps will want files though, so sometimes it isn't possible.
freedomben 6y ago
Really solid post. I've helped a lot of people with Dockerfiles ranging from horrendous security issues to simple bad practice making lives hell, and much of this is solid advice. A lot of what I tell people boils down to: keep your container lean and clean. Don't do things in the container that you wouldn't do on the host (like curl-ing from the internet into bash as root :-D, or using questionable base images). My deployment life has been vastly improved by shipping in containers, but I have seen a lot of security regressions because people feel safe to be reckless (like running the app as root) due to the container guard rails. Don't think this way.
xaduha 6y ago
> Using ENV to store secrets is bad practice because Dockerfiles are usually distributed with the application Nevermind Dockerfiles, env variables are preserved in stopped containers that are hanging around even when you use docker-machine AFAIK, you can easily `docker inspect` them.
- tsumnia 6y ago
  Is this referring to ENVs inside the Docker image or on the system running it as a whole?
  - wadkar 6y ago
    Aren’t they the same thing (sort of)? The ENV in Dockerfile (or .env if you’re doing docker-compose) will be available during the build as well as runtime.
- move-on-by 6y ago
  Always use `--rm` flag to automatically remove containers when it exits could probably be another best practice.
sergioisidoro 6y ago
A note about not running as root: In certain systems if you break out of the application and have access to a shell, it might already be game over. An attacker probably already has access to secrets, to the database, and all the assets worth protecting. On the other hand, changing the docker user to non root might introduce some failure scenarios (eg file ownership) which might lead to other problems like availability incidents. Security should start with threat modelling, and taking a risk-based approach. You can spend hours fearing that someone might break out of the Docker virtual machine through a zero day, instead of using that time to fix much more likely and plausible threat scenarios. Pick your battles.
- felipelemos 6y ago
  > On the other hand, changing the docker user to non root might introduce some failure scenarios (eg file ownership) If you application needs root to execute, with very few exceptions, it is already wrong.
  - freedomben 6y ago
    If you are are writing the app, I agree with you. Unfortunately in some cases the person/team that wrote the app has been gone for a long time. I've even seen a case where the source code was missing and nobody knew where it was, yet the service had to continue running. If you're in that boat, there isn't much you can do except work with it.
  - sergioisidoro 6y ago
    Agreed. But when copying files to docker and building the image, you will have to take care that files are not written with root ownership in any stage of the build, which would make them inaccessible to the application running as non root. That's the case I had in mind when writing that quote.
    - junon 6y ago
      > Agreed. But when copying files to docker and building the image, you will have to take care that files are not written with root ownership in any stage of the build, which would make them inaccessible to the application running as non root. That's not the case, either. And root inside the container != root outside the container. A completely new user:group namespace is created inside the container. This is, in very large part, what Linux namespaces are for. Further, you can certainly have a root-owned file accessible to non-root users, via chmod bits. There are only a handful of excuses, ever, to run a privileged container. If you're not 100% sure, then it is not one of those excuses.
PhantomBKB 6y ago
You can also use talisman to make sure you are not checking in secrets in dockerfiles https://github.com/thoughtworks/talisman https://github.com/thoughtworks/talisman
pella 6y ago
imho: always check what is upgrading - before make a decisions .. the ubuntu:20.10 now want to upgrade the "libssl1.1"! docker run --rm -it ubuntu:20.10 bash -c "apt update && apt upgrade" ... The following packages will be upgraded: debianutils diffutils findutils gcc-10-base libgcc-s1 libgnutls30 libprocps8 libssl1.1 libstdc++6 libsystemd0 libudev1 procps sed zlib1g the ubuntu:20.04 is better docker run --rm -it ubuntu:20.04 bash -c "apt update && apt upgrade" ... The following packages will be upgraded: gcc-10-base libgcc-s1 libstdc++6 zlib1g
- mfontani 6y ago
  One could do worse than using (more) stable distribution for their base images. 20.10 is not a LTS, so it's more likely to get semi-spurious updates than a LTS version like 20.04 or, say, debian stable. For most of my non-alpine-based images, I use debian:buster-slim as base, as it's got a fairly stable base and gets quite routinely updated: $ docker run --rm -it debian:buster-slim bash -c "apt update >dev/null 2>&1 && apt upgrade" Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
  - pella 6y ago
    > I use debian:buster-slim as base this image has been upgraded "39 hours ago".. so you have to check 1-2 month later debian buster-slim f49666103347 39 hours ago 69.2MB
    - mfontani 6y ago
      That's part of my point ;) It's updated very often. I check for base image updates every day, and it's one of the most oft-updated ones -- hence my preference for using it as "the" base of all others.
      - TimWolla 6y ago
        Monthly typically (and when important security updates are released): https://github.com/docker-library/official-images/pulls?q=is%3Apr+label%3Alibrary%2Fdebian+is%3Aclosed https://github.com/docker-library/official-images/pulls?q=is...
sgnnseven 6y ago
Solid post though there are a couple of things I would disagree with: > Do not upgrade your system packages Most distros will have smooth upgrades and provide you with patched libs that your app may need and the latest image may not provide. It's slightly more prone to breaks but it creates a less vulnerable runtime app env. > Do not use ‘latest’ tag for base image Depends on the org but sometimes pinning means that you will likely end up using and end-of-life image because it requires proactive work to maintain. If you leave it as 'latest' this won't happen but you will get out-of-band breaks to keep that working. Choose wisely. A few things I would add too: - Don't mount Docker socket into any container unless absolutely necessary - Your biggest security threat will be from your app's dependencies, not the container's setup - Do not run a full init system unless absolutely necessary as this is just a security disaster waiting to happen. There are valid use cases for it but they're rare.
- TheGallopedHigh 6y ago
  Can you explain your last point further please?
  - daxelrod 6y ago
    “Full” init systems tend to need to do things that are hard to secure in a container. Many must run as root, and the reasons not to do that are discussed in the article this HN thread is discussing. Systemd is particularly tricky because it needs to be able to control the cgroups of its child processes, which means the container needs to be granted that capability. See https://developers.redhat.com/blog/2019/04/24/how-to-run-systemd-in-a-container/ https://developers.redhat.com/blog/2019/04/24/how-to-run-sys... about how to run systemd in a container via Podman, and is a follow up to https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-non-privileged-container/ https://developers.redhat.com/blog/2016/09/13/running-system... which discusses why the Docker case is even more difficult. That said, if you just want a process supervisor for a multi process container, there are several more minimal init systems that will work well, for example, supervisord.
    - TheGallopedHigh 6y ago
itamarst 6y ago
1. "Don't update system packages" is bad advice. There are base Docker images with out-of-date packages that need security updates. CentOS for example doesn't update their base image for months on end. 2. Given you do want system package updates, you need to deal with Docker caching breaking your security updates. So you need to rebuild your image from scratch, without caching, either every time there is a security update or on a weekly basis. https://pythonspeed.com/articles/docker-cache-insecure-images/ https://pythonspeed.com/articles/docker-cache-insecure-image... 3. Not mentioned: don't use build secrets via ARG. Some approaches to secure build secrets: 1. BuildKit supports them: https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information https://docs.docker.com/develop/develop-images/build_enhance... 2. Via the network, which is a hack, but it works: https://pythonspeed.com/articles/docker-build-secrets/ https://pythonspeed.com/articles/docker-build-secrets/ 3. Via multi-stage builds, but this destroys caching.
- BossingAround 6y ago
  > "Don't update system packages" is bad advice. There are base Docker images with out-of-date packages that need security updates. CentOS for example doesn't update their base image for months on end. Generally, if you want to update system packages, rebuild the container make it the new base for you. Updating with every build provides a potentially non-reproducible build.
  - itamarst 6y ago
    It's a tradeoff, yes. You can do more work to rebuild a base image, or you can say "technically have slightly newer version of glibc isn't reproducible but in practice I don't expect that to break anything so I'll live with the risk".
  - jsabo 6y ago
    If you're using a tag like centos:8 that gets updated periodically, you already do not have reproducible builds. This just ensures you get updated packages as soon as possible
usr1106 6y ago
I think he mixes 2 aspects. There is security and there is reproducibility/traceability/reliability. For security using the latest versions of both base images and packages is typically a good thing. The cases that the newest package is more vulnerable since something 1, 2, 3 years old are not that common. However, if your process requires reproducibility/traceability (medical and other regulated domains) you cannot just deploy the latest and greatest. You need to pass it through some release process first. That should not be an excuse to run outdated, vulnerable software though. The same holds if you require high availability. Even if you might not need to document what you are using, you want to test whether it causes performance issues (zero performance is the worst one...).
tetha 6y ago
Besides the Docker practices - does someone have experience with OPA? This is the first time I've heard of that tool, but an extensible policy tool like that might solve a lot of challenges we have at work.
- andyroid 6y ago
  We use OPA for use cases ranging from kubernetes admission control, to microservice authorization and CI/CD pipeline policies. It's one of those tools you can't realize you could have lived without once you start using it.
blago 6y ago
I would add one more general security tip. Always restrict your mapped ports to localhost (as in 127.0.0.1:3306:3306) unless you really want to expose it to the world. I think it's counterintuitive, but I learned the hard way that 3306:3306 will automatically add a rule to open the firewall on linux and make MySQL publicly accessible.
- BossingAround 6y ago
  > I learned the hard way that 3306:3306 will automatically add a rule to open the firewall on linux and make MySQL publicly accessible. Is that true? 3306:3306 would bind the port on all interfaces, but I was under the assumption you'd have to explicitly enable firewall port 3306 for the machine to accept traffic to port 3306 from outside of your machine. I'll have to test that.
  - blago 6y ago
    https://github.com/docker/for-linux/issues/690 https://github.com/docker/for-linux/issues/690
  - isbvhodnvemrwvn 6y ago
    From memory and a quick glance at one of my servers: When an IP packet related to your container arrives (<host ip>:<host port>): - docker rewrites it to target <your container's ip address>:<your container's port> (NAT table, chain PREROUTING delegates to DOCKER chain with the relevant DNAT entry) - since the IP does not match your host, the packet is forwarded (there's a relevant routing entry pointing to docker's virtual interface) - the first thing you encounter in the FORWARD chain of the FILTER table is a few jumps to the docker-related chains, DOCKER chain in particular accepts all packets destined to <your container's ip address>:<your container's port> So a few takeaways: - your standard firewall might not be involved because its chains are plugged in after the docker chains in the FORWARD chain (e.g. ufw under Ubuntu) - if the above is true and you want your firewall to matter, you have to add stuff to DOCKER-USER chain in the FILTER table - at that point the host port and IP doesn't matter since it's already been mapped in the NAT table's PREROUTE chain at the beginning of processing - write your firewall rules to address specific containers
azangru 6y ago
A very nice web page, too! Snappy, no javascript apart from analytics, tastefully designed, mobile-friendly.
dkarp 6y ago
Something really useful I recently discovered is multi-stage Dockerfiles. Using FROM and then COPY --from to copy from a previous stage to prevent unwanted intermediate build steps that might expose secrets or just bloat your images. Two ways this was useful for us. Firstly, we needed a private key in the image to pull some private git repos. By doing this in a previous stage, they're not included in the final image layers. Secondly, we have a python backend and small react app served from the same image. By splitting their build steps into a backend and frontend stage, changes to frontend code don't break caching for the later backend steps or vice versa. E.g. FROM python:3.8.3-slim-buster AS frontend # Do frontend build steps FROM python:3.8.3-slim-buster AS backend # Do backend build steps FROM python:3.8.3-slim-buster AS final COPY --from=frontend /app /app/frontend COPY --from=backend /app /app/backend
- junon 6y ago
  This is how C/C++ services are done in production, too. You have a build step that does `apk add --update alpine-sdk git cmake` etc., builds the service, and then you start again fresh with a FROM (new stage) and `COPY --from` over the build artifacts. Reading this thread I'm surprised this isn't common knowledge by now given how it's so incredibly paramount to efficient production releases.
  - dkarp 6y ago
    It may well be common knowledge now. I discovered it when writing our Dockerfiles earlier this year and before that hadn't seen it. It looks like multi-stage builds were added to the best practices in 2018 [https://github.com/docker/docker.github.io/blob/master/develop/develop-images/multistage-build.md https://github.com/docker/docker.github.io/blob/master/devel...] which is probably why I missed it before
- nullsense 6y ago
  Docker has official support for secrets using Buildkit.
  - dkarp 6y ago
surfer7837 6y ago
I also use Red Hat UBI (Universal Base Images). If you pull down any package from DockerHub (even official images) you'll be surprised about how many vulnerabilities they have.
- freedomben 6y ago
  Second this. The UBI images are meticulously maintained by Red Hat and are freely available and redistributable without a subscription. Some of the largest companies in the world are using these in production and putting dollars behind them, so you can be pretty confident that they will work and be maintained.
lipanski 6y ago
One of the most common mistakes I see is not using a .dockerignore file or, better said, relying on .gitignore when calling `COPY` on entire directories. Without a .dockerignore file in place, you could be copying over your local .env files and other unwanted things into the final image. On top of that, you might also want to add `.git/` to your .dockerignore file, as it could significantly reduce the size of your image when calling `COPY`. A more subtle issue I've noticed is the fact that `COPY` operations don't honour the user set when calling `USER`. The `COPY` command has its own `--chown` argument, which needs to be set if you'd like to override the default user (which is root or a root-enabled user in most cases). I wrote up a similar article a while back, though it's focused on general best practices: https://lipanski.com/posts/dockerfile-ruby-best-practices https://lipanski.com/posts/dockerfile-ruby-best-practices
- silverwind 6y ago
  Better yet, only COPY the actually needed needed files instead of the whole working directory. That way, there's no need for a `.dockerignore`.
  - unleashit 6y ago
    Wondering why you think this is better. Not sure the trade off of a messy dockerfile and/or adding a bunch of layers (possibly bloating the image size) is worth the trade off if the concern is just about forgetting to update the dockerignore. The same could be said about gitignore.
    - kinghajj 6y ago
      Not the person you replied to, but personally, I like having control over what exactly gets into the final image, and (IME) have found that devs aren't great about remembering to update .dockerignore files. Re: extra layers, if you use multi-stage builds to separate the builder and final app images, you can avoid that.
    - ric2b 6y ago
      It's nice to know what files you need to build the image. Sort of like importing libraries at the top of a source code file.
rzimmerman 6y ago
> Do not store secrets in environment variables Yes, definitely don't put secrets in the Dockerfile itself. I'm curious if there are reasons not to use a .env file though? > Only use trusted base images This is a good sentiment, but docker hub also has plenty of images that are built directly from a github repo. You can inspect the Dockerfile and (as long as you trust Docker) trust that it was built as written. In this case I recommend pinning to a specific container SHA (image_name@sha256:...) in case the source repo gets compromised. For official images, you can pin to a tag IMO. > Do not use ‘latest’ tag for base image Regardless of the security concerns, pinning to latest will probably bite you when there's a major version bump (or even maybe a minor one). Imagine you built a container on ubuntu:latest when latest was 20.04 and some new employee gets hit with 22.04. That's a bad surprise. > Avoid curl bashing Evergreen fight here, but I agree with the author that you should at least validate checksums on external files before executing or using them. > Do not upgrade your system packages This is where the throne of lies about Docker idempotency crumbles. You have to apt-get update because Debian/Ubuntu repositories remove old versions when there are security fixes (not 100% sure on this, feel free to correct me). So if the ubuntu:20.04 image is released and there's a security update in openssh-client, running "apt-get install openssh-client" without "apt-get update" will fail. So we all run "apt-get update" and pretend that the containers we build are time-invariant. They're not, and in fact we occasionally get security updates snuck in there. Luckily Debian and Ubuntu do a good job not breaking things and no one complains. But if you build a container on Tuesday it's not guaranteed to be the same as on Wednesday and there's nothing you can do about that with a Debian distro. But it's actually fine in practice so we pretend not to notice. The point about not running "apt-get upgrade" is kind of moot - you're going to effectively be upgrading whatever you "apt-get install", so it's probably worth taking the same trade on the built-in packages. Importantly, there's no security risk - just a configuration one. > Do not use ADD if possible Same as above - you should be checksumming external dependencies or self-hosting them. > Do not root > Do not sudo I haven't thought deeply about these. I'd prefer to trust the container system isolation rather than playing with users and permissions. I don't understand this risk well enough to have a well-formed opinion.
- kelnos 6y ago
  > Yes, definitely don't put secrets in the Dockerfile itself. I'm curious if there are reasons not to use a .env file though? Because often that .env file will get checked into source control (sometimes intentionally, sometimes accidentally), and then you have secrets in your git history that you need to rotate. In general the best thing to do is store your secrets in a place/service specifically designed for secrets, and either fetch them directly into the container in your entrypoint, or pull them into a location that you mount as a volume in the container. > I'd prefer to trust the container system isolation rather than playing with users and permissions. Don't. Containers don't give you full isolation. The 'root' user inside the container is the same 'root' user as outside, and container escapes may be possible. A good defense-in-depth strategy suggests that you should run things using the least privileges possible, and that doesn't change just because something is running in a container.
nomadiccoder 6y ago
This article is interesting but it would be far more helpful to link to a solution for each point.
andmarios 6y ago
Dockerfile security best practices: treat is as you would treat any linux server. :)
vorticalbox 6y ago
Quick question of you're not meant to use env for secrets then how are you meant to get secrets into your application? What's the best way to handle this?
- keikun17 6y ago
  using buildkit https://pythonspeed.com/articles/docker-build-secrets/ https://pythonspeed.com/articles/docker-build-secrets/
- tynorf 6y ago
  Probably either via a third party service (such as AWS secrets manager), or mounted as files scoped to the user your process is running as (which is not root, right? :) ).
- raxor53 6y ago
  Typically your host will have a service specifically for secrets. For example: https://docs.github.com/en/free-pro-team@latest/rest/reference/actions#secrets https://docs.github.com/en/free-pro-team@latest/rest/referen...
- CGamesPlay 6y ago
  Two kinds of secrets: build-time (I need an API key to access a download needed to set up the container) and run-time (I need an API key to talk to the database). The post argues against using ENV for build-time secrets, but it's still recommended for run-time secrets. Sibling comments address alternatives for getting build-time secrets in.
jtchang 6y ago
How do you get around sometimes needing root inside the container to build things? For example building a container with buildroot inside.
- vitalysh 6y ago
  Using intermediate container to build and copy over resulting binaries should work.
tofflos 6y ago
The number one advice should be to use a linter. The number two advice should be to use an image security scanner. These tools combined will prevent most issues. Integrate them with CI to enforce a common set of best practices across an organization and to prevent security bike shedding.
jose_zap 6y ago
Hadolint, a docker file linter checks for most of those best practices https://github.com/hadolint/hadolint https://github.com/hadolint/hadolint
- bentolor 6y ago
  It looks really great at first to have a hassle-free alternative with sensible defaults. Unfortunately installing seems non-trivial. Would love simple binary or .deb/.rpm install.
  - jose_zap 6y ago
    Looks like it is packaged as .rpm at least https://pkgs.org/download/hadolint https://pkgs.org/download/hadolint
naranha 6y ago
> Using ENV to store secrets is bad practice because Dockerfiles are usually distributed with the application, so there is no difference from hard coding secrets in code. This sounds wrong. If secrets are in the environment they are not in the Dockerfile, so they are NOT distributed with the application.
- LawnGnome 6y ago
  What they're referring to is more specific than environment variables in general: you can use the ENV command in a Dockerfile to bake in a secret at build time, and that's what you generally shouldn't do. Injecting environment variables at runtime, however (through docker run -e or whatever orchestration system you're using), is good.
  - naranha 6y ago
    Ah yes, that makes sense. I didn't understand they were talking about the ENV command at build time. It was the heading that got me on the wrong path, I think that should be clarified further: > Do not store secrets in environment variables
    - pvtmert 6y ago
      I'm bit aganist having env-secrets inside container. Because PID 1 has that env, all processes spawned from that can read all of those. I prefer mounting them to /run/secrets via tmpfs. Which can also have selinux policy attached. This way, someone else cannot read them by spawning shell inside container
- sethhochberg 6y ago
  I don't think the author is talking about loading secrets from the environment - I think they're specifically talking about hardcoding secrets into the Dockerfile and using the Dockerfile ENV directive to set secrets for the processes running in the container (baking them into the image), instead of passing them at runtime, which sounds just horrifying enough that I'm sure people do it in real codebases.
- 6y ago
chias 6y ago
> Do not upgrade your system packages Ah, the joys of working somewhere that isn't required to document, answer for, and ultimately remediate every CVE that is present in any package installed on any of your containers within your production application. Sadly, compliance and regulatory oversight don't leave this option open to everyone.
- cmwelsh 6y ago
  Is this a good argument for building containers as “bare metal” as possible? You don’t have to remedy CVEs (and rebuild your containers) for anything that isn’t actually your application.
  - chias 6y ago
    That is pretty much the only thing possible in these scenarios. Anything Debian or Ubuntu or pretty much any "normal" distribution is right out: external vulnerability scanners always seem to go by package version, and `packagename-12.5.1-debian-security-fixes.b` is still the vulnerable version 12.5.1 as far as any scanner is concerned. At this point, we `FROM scratch` when possible, and deploy on AL2 when not. There's good reasoning against the concept of barebones containers, but unfortunately everything from bricks, knives, and well-reasoned arguments all bounce harmlessly off of regulations and external compliance requirements.
minimaxir 6y ago
Google's distroless containers are an interesting approach for both security and performance as well, albeit with limited language support: https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless
- _nhynes 6y ago
  Relatedly, rust-musl-builder [0] is useful for getting Rust binaries to run on the `static` instead of `cc` base image. [0] https://github.com/emk/rust-musl-builder https://github.com/emk/rust-musl-builder
  - minimaxir 6y ago
    What's the advantage of doing that?
    - acomar 6y ago
      you can drop the resulting binary in a scratch container that has no shell, etc.
EdSchouten 6y ago
If you want to create container images for an application you wrote yourself in a commonly used programming language (Go, Python), consider using Bazel with rules_docker: https://github.com/bazelbuild/rules_docker https://github.com/bazelbuild/rules_docker rules_docker allows you to create byte-for-byte reproducible container images on your system, without even having a Docker daemon installed. So much cleaner to use than 'docker build' once you get the hang of it!
- ImJasonH 6y ago
  If you're building images from go binaries you can do that without Bazel using github.com/google/ko :)
m0zg 6y ago
Why would you write such an article and not show what the actual _recommended_ practices look like?
kelnos 6y ago
Regarding using USER somenonrootuser ... how do people deal with the need to write things out on container start? Some of my services require config files, and I need to interpolate the values of environment variables into those config files, write them out, and then start the application. (Also assume the application itself doesn't have options for dropping privileges.) I'd rather not make the filesystem locations in question writable by `somenonrootuser`. My current strategy is to let the container entrypoint start as root, but then I have a wrapper program installed that drops privileges before it exec()s the actual service. It works, but is there a better / more accepted way of doing this?
- Propaganda_ 6y ago
  If the services can't be modified to load their config directly from env vars, write the config to an off-root scratch volume (e.g. mounted to /tmp/) and have them load from that. The root volume should be mounted read-only either way to prevent modification of your services should something get RCE.
- CGamesPlay 6y ago
  [deleted] this idea doesn't work on attempting
- piaste 6y ago
  If you control the services, make them read from user-writable files instead of root ones. (Ideally you could skip the middleman and just read from env vars directly!) If you don't control the services, maybe in the Dockerfile you could write a dummy config file and chmod it?
zmmmmm 6y ago
Something that I would love to see some specific guidance on is TLS certificates. It seems like commonly used webservers (eg: nginx) pretty much insist on the private certificate being available effectively unencrypted to the web server process. On a traditional server that gets protected by user permissions, but inside Docker I am not sure how to achieve this. Is there a pattern people use to deploy certificates inside docker containers, or is the pattern "don't do that"?
- GauntletWizard 6y ago
  Don’t deploy your certificates with docker. That’s not to say: Don’t give your web servers running in docker certificates, but that you shouldn’t distribute certificates with your docker images. The Private Key/Certificate pair should be added at runtime. How it gets there is specific to your management framework. To give a specific example, on Kubernetes you store your Private Key and Certificate as a secret (there’s a special “type” of secret for this, but K8s doesn’t actually treat it any differently). You then mount that certificate as a volume (in /var/run/secrets/tls, or wherever you define it), and it shows up as a normal file, accessible to whichever user your container runs as.
  - zmmmmm 6y ago
    thanks! So for clarity then, if somebody shells into the container - at that point it's still sitting there mounted and they can read the file? Or does k8s somehow manage this in a way that only the web server process can see it? Or do we just accept that you have to treat anyone with access to kubectl as authorised to know your private cert ?
    - FridgeSeal 6y ago
      > at that point it's still sitting there mounted and they can read the file? Yes, if someone has exec access into your container, they'll be able to see the secret, unless you do something like making the K8s secret an already encrypted blob, and then in process decrypting it again and reading it. If someone's got exec access to your container though, you've got bigger problems. > Or do we just accept that you have to treat anyone with access to kubectl as authorised to know your private cert ? You can set up access so that someone can login with Kubectl but still only be readonly, and you can preclude read ("get") access to secrets to the readonly role, so they won't be able to view the contents of the secret.
atticusCr 6y ago
How about using a hardening guide such as CIS as part of the build process?
gautamdivgi 6y ago
I have one more that I have seen used within corporate networks. If a proxy is used to build docker a lot of engineers add ENV http_proxy, https_proxy in the Dockerfile. This causes the proxy to be on by default. Use ARG instead if you just want the proxies at build time.
schemescape 6y ago
Is it just me or does this article highlight bad practices (and ways to detect them), but then not really address the “correct” way to avoid many of the bad practices?
- dandigangi 6y ago
  +1... was just thinking this.
josep2 6y ago
Great post, love the use of Open Policy Agent and Conftest!
creztoe 6y ago
Do not pass sensitive data to docker build via --build-arg. When you access this with "ARG" you will log the information in the docker history, visible to all. Use "--secret" or use the ARG in an intermediate build stage which doesn't have it's history preserved, then copy any necessary files form the intermediate image to your file image manually. A perfect example of this would be passing your NPM_TOKEN to install company scope packages.

407 points by gbrindisi 6y ago

About the "apt upgrade". In major distributions, that's safe. They won't introduce major package versions, only minor (and security fixes). It should be totally OK to upgrade the packages. Also I don't get that it's advised to use "apt update" when you can't do "apt upgrade". What's the point??

kortex 6y ago
Apt update just updates the cache of the package lists - /var/lib/apt/lists/ - based on your lists - /etc/apt/sources.list.d/. "upgrade" actually upgrades the code on the system. `rm -rf /var/lib/apt/lists/*` in the same RUN decreases bloat and I think possibly decreases cache misses as well.
- Nullabillity 6y ago
  > and I think possibly decreases cache misses as well. Unlikely, you'd still have misses because of stuff like file metadata mismatches (think modification times).
- richardwhiuk 6y ago
  Base images usually ship with that done. `apt-get update` just undoes the good work of removing irrelevant lists.
  - acdha 6y ago
    Some base images do, but some don’t, and they update on their schedule rather than yours. Doing the update on your schedule is trivial, has no meaningful downside, and means it’s done and tested as quickly as you need.
  - kortex 6y ago
    Huh, TIL. I'll give that a try. Usually, that's the least of my problems. In my line of work, I'm often doing things which would likely horrify most webapp devs, like building a container with multiple conda envs in it. OTOH, most of these containers run on airgapped systems with petabytes of storage, so shipping a bit of bloat barely hurts the end user. But every time I do, I die a bit on the inside.
diggan 6y ago
It's not entirely clear from the article but my guess would be that you should really update the repository + base dependencies in the base image, not the "application" image or whatever you wanna call it. But not the end of the world if you're not reusing that base image. Feels weird that an article on "Security Best Practices" would advise _against_ latest versions of your software, especially using apt as an example which is usually used in stable distributions (major version upgrades happens seldom) like Ubuntu/Debian. But yeah, that last part doesn't make any sense. If you're not running `apt-get upgrade`, it doesn't make sense to run `apt-get update` as nothing is using the newly fetched data anyways...
shaded-enmity 6y ago
If you upgrade your packages during a build then inadvertently if 2 people build your Dockerfile at two distinct times they can end up with 2 different images. So rather than focusing on "we're using base-image:1.0.2" you need to start asking questions like "list all packages and versions and start comparing those". If there's a security issue you need to rebuild your images anyway and optimally you have a system in place that represents images and their dependencies as some sort of dependency graph structure, so when you upgrade your base image all dependent images get rebuilt automatically.

3pt14159 6y ago

Overall great post. > If you rely on latest you might silently inherit updated packages that in the best worst case might impact your application reliability, in the worst worst case might introduce a vulnerability. On this one I disagree. Your CI should handle reliability (and if not, you have a bigger problem) and you're more likely to patch a vulnerability than to introduce a new one and it's unlikely that by the time a PR hits production that the version is compromised. I understand that updating cuts both ways when it comes to security, but I agree with Matt Tait's ultimate conclusion when he spoke on this issue a few years ago: For most medium size and smaller companies constantly updating is safer than delaying. He had real world data and graphs of compromise windows, etc. Short answer was that attackers are more time motivated than defenders.

dbrgn 6y ago
Operating system updates are the responsibility of the base image. Use a base image that is regularly updated. For your own images, ensure that they are also being rebuilt regularly. There are a lot of semi-official Docker images on Docker Hub that are published once and never updated until the next software release. That is a huge anti-pattern in my view, those images should not be relied on for production.
cooljacob204 6y ago
Would you have a link to the talk? I couldn't find it with a google search.
- 3pt14159 6y ago
  I don't have time to track down a link to a video at the moment, but it was at Infiltrate, an offensive-minded cybersecurity conference out of Miami.
  - kronin 6y ago
    Is it this one? https://vimeo.com/267445424 https://vimeo.com/267445424
6y ago

lysium 6y ago

Great post. Two points: ..If you don’t inspect the wget script you might as while pipe it into bash. .. How to distribute secrets if not by env? (which I agree! Honest question)

nicolast 6y ago
> How to distribute secrets if not by env? (which I agree! Honest question) You'll want to use BuildKit (`docker buildx`), see https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information https://docs.docker.com/develop/develop-images/build_enhance... [edit] My bad, that works for secrets needed at build time, not at runtime of course.
freedomben 6y ago
Disclaimer: I work for Red Hat as an OpenShift consultant so I'm biased There are competing pieces of advice for secure distribution of secrets, but my current preference comes down to one of these ways, depending on the organization: 1. OpenShift/Kubernetes Secrets mounted into the Pod at runtime. 2. Hashicorp Vault (has a really well designed API. It's very usable just with curl, which makes using it a joy) 3. Sealed Secrets (less experience here but it's looking positive right now) - https://github.com/bitnami-labs/sealed-secrets https://github.com/bitnami-labs/sealed-secrets If you're using a different PaaS besides OpenShift, it may also offer options worth considering (although do think about portability. These days apps move platforms every few years on average, though I think that may be changing now that K8s is becoming the standard).
- csunbird 6y ago
  > 1. OpenShift/Kubernetes Secrets mounted into the Pod at runtime. Do you recommend mounting secrets as environment variables to the kubernetes pods instead of files?
  - freedomben 6y ago
    Yes, that is by far my preference. Much more 12 factor app-ish and framework independent. A lot of Java apps will want files though, so sometimes it isn't possible.

freedomben 6y ago

Really solid post. I've helped a lot of people with Dockerfiles ranging from horrendous security issues to simple bad practice making lives hell, and much of this is solid advice. A lot of what I tell people boils down to: keep your container lean and clean. Don't do things in the container that you wouldn't do on the host (like curl-ing from the internet into bash as root :-D, or using questionable base images). My deployment life has been vastly improved by shipping in containers, but I have seen a lot of security regressions because people feel safe to be reckless (like running the app as root) due to the container guard rails. Don't think this way.

xaduha 6y ago

> Using ENV to store secrets is bad practice because Dockerfiles are usually distributed with the application Nevermind Dockerfiles, env variables are preserved in stopped containers that are hanging around even when you use docker-machine AFAIK, you can easily `docker inspect` them.

tsumnia 6y ago
Is this referring to ENVs inside the Docker image or on the system running it as a whole?
- wadkar 6y ago
  Aren’t they the same thing (sort of)? The ENV in Dockerfile (or .env if you’re doing docker-compose) will be available during the build as well as runtime.
move-on-by 6y ago
Always use `--rm` flag to automatically remove containers when it exits could probably be another best practice.

sergioisidoro 6y ago

A note about not running as root: In certain systems if you break out of the application and have access to a shell, it might already be game over. An attacker probably already has access to secrets, to the database, and all the assets worth protecting. On the other hand, changing the docker user to non root might introduce some failure scenarios (eg file ownership) which might lead to other problems like availability incidents. Security should start with threat modelling, and taking a risk-based approach. You can spend hours fearing that someone might break out of the Docker virtual machine through a zero day, instead of using that time to fix much more likely and plausible threat scenarios. Pick your battles.

felipelemos 6y ago
> On the other hand, changing the docker user to non root might introduce some failure scenarios (eg file ownership) If you application needs root to execute, with very few exceptions, it is already wrong.
- freedomben 6y ago
  If you are are writing the app, I agree with you. Unfortunately in some cases the person/team that wrote the app has been gone for a long time. I've even seen a case where the source code was missing and nobody knew where it was, yet the service had to continue running. If you're in that boat, there isn't much you can do except work with it.
- sergioisidoro 6y ago
  Agreed. But when copying files to docker and building the image, you will have to take care that files are not written with root ownership in any stage of the build, which would make them inaccessible to the application running as non root. That's the case I had in mind when writing that quote.
  - junon 6y ago
    > Agreed. But when copying files to docker and building the image, you will have to take care that files are not written with root ownership in any stage of the build, which would make them inaccessible to the application running as non root. That's not the case, either. And root inside the container != root outside the container. A completely new user:group namespace is created inside the container. This is, in very large part, what Linux namespaces are for. Further, you can certainly have a root-owned file accessible to non-root users, via chmod bits. There are only a handful of excuses, ever, to run a privileged container. If you're not 100% sure, then it is not one of those excuses.

PhantomBKB 6y ago

You can also use talisman to make sure you are not checking in secrets in dockerfiles https://github.com/thoughtworks/talisman https://github.com/thoughtworks/talisman

pella 6y ago

imho: always check what is upgrading - before make a decisions .. the ubuntu:20.10 now want to upgrade the "libssl1.1"! docker run --rm -it ubuntu:20.10 bash -c "apt update && apt upgrade" ... The following packages will be upgraded: debianutils diffutils findutils gcc-10-base libgcc-s1 libgnutls30 libprocps8 libssl1.1 libstdc++6 libsystemd0 libudev1 procps sed zlib1g the ubuntu:20.04 is better docker run --rm -it ubuntu:20.04 bash -c "apt update && apt upgrade" ... The following packages will be upgraded: gcc-10-base libgcc-s1 libstdc++6 zlib1g

mfontani 6y ago
One could do worse than using (more) stable distribution for their base images. 20.10 is not a LTS, so it's more likely to get semi-spurious updates than a LTS version like 20.04 or, say, debian stable. For most of my non-alpine-based images, I use debian:buster-slim as base, as it's got a fairly stable base and gets quite routinely updated: $ docker run --rm -it debian:buster-slim bash -c "apt update >dev/null 2>&1 && apt upgrade" Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
- pella 6y ago
  > I use debian:buster-slim as base this image has been upgraded "39 hours ago".. so you have to check 1-2 month later debian buster-slim f49666103347 39 hours ago 69.2MB
  - mfontani 6y ago
    That's part of my point ;) It's updated very often. I check for base image updates every day, and it's one of the most oft-updated ones -- hence my preference for using it as "the" base of all others.
    - TimWolla 6y ago
      Monthly typically (and when important security updates are released): https://github.com/docker-library/official-images/pulls?q=is%3Apr+label%3Alibrary%2Fdebian+is%3Aclosed https://github.com/docker-library/official-images/pulls?q=is...

sgnnseven 6y ago

Solid post though there are a couple of things I would disagree with: > Do not upgrade your system packages Most distros will have smooth upgrades and provide you with patched libs that your app may need and the latest image may not provide. It's slightly more prone to breaks but it creates a less vulnerable runtime app env. > Do not use ‘latest’ tag for base image Depends on the org but sometimes pinning means that you will likely end up using and end-of-life image because it requires proactive work to maintain. If you leave it as 'latest' this won't happen but you will get out-of-band breaks to keep that working. Choose wisely. A few things I would add too: - Don't mount Docker socket into any container unless absolutely necessary - Your biggest security threat will be from your app's dependencies, not the container's setup - Do not run a full init system unless absolutely necessary as this is just a security disaster waiting to happen. There are valid use cases for it but they're rare.

TheGallopedHigh 6y ago
Can you explain your last point further please?
- daxelrod 6y ago
  “Full” init systems tend to need to do things that are hard to secure in a container. Many must run as root, and the reasons not to do that are discussed in the article this HN thread is discussing. Systemd is particularly tricky because it needs to be able to control the cgroups of its child processes, which means the container needs to be granted that capability. See https://developers.redhat.com/blog/2019/04/24/how-to-run-systemd-in-a-container/ https://developers.redhat.com/blog/2019/04/24/how-to-run-sys... about how to run systemd in a container via Podman, and is a follow up to https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-non-privileged-container/ https://developers.redhat.com/blog/2016/09/13/running-system... which discusses why the Docker case is even more difficult. That said, if you just want a process supervisor for a multi process container, there are several more minimal init systems that will work well, for example, supervisord.
  - TheGallopedHigh 6y ago

itamarst 6y ago

1. "Don't update system packages" is bad advice. There are base Docker images with out-of-date packages that need security updates. CentOS for example doesn't update their base image for months on end. 2. Given you do want system package updates, you need to deal with Docker caching breaking your security updates. So you need to rebuild your image from scratch, without caching, either every time there is a security update or on a weekly basis. https://pythonspeed.com/articles/docker-cache-insecure-images/ https://pythonspeed.com/articles/docker-cache-insecure-image... 3. Not mentioned: don't use build secrets via ARG. Some approaches to secure build secrets: 1. BuildKit supports them: https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information https://docs.docker.com/develop/develop-images/build_enhance... 2. Via the network, which is a hack, but it works: https://pythonspeed.com/articles/docker-build-secrets/ https://pythonspeed.com/articles/docker-build-secrets/ 3. Via multi-stage builds, but this destroys caching.

BossingAround 6y ago
> "Don't update system packages" is bad advice. There are base Docker images with out-of-date packages that need security updates. CentOS for example doesn't update their base image for months on end. Generally, if you want to update system packages, rebuild the container make it the new base for you. Updating with every build provides a potentially non-reproducible build.
- itamarst 6y ago
  It's a tradeoff, yes. You can do more work to rebuild a base image, or you can say "technically have slightly newer version of glibc isn't reproducible but in practice I don't expect that to break anything so I'll live with the risk".
- jsabo 6y ago
  If you're using a tag like centos:8 that gets updated periodically, you already do not have reproducible builds. This just ensures you get updated packages as soon as possible

usr1106 6y ago

I think he mixes 2 aspects. There is security and there is reproducibility/traceability/reliability. For security using the latest versions of both base images and packages is typically a good thing. The cases that the newest package is more vulnerable since something 1, 2, 3 years old are not that common. However, if your process requires reproducibility/traceability (medical and other regulated domains) you cannot just deploy the latest and greatest. You need to pass it through some release process first. That should not be an excuse to run outdated, vulnerable software though. The same holds if you require high availability. Even if you might not need to document what you are using, you want to test whether it causes performance issues (zero performance is the worst one...).

tetha 6y ago

Besides the Docker practices - does someone have experience with OPA? This is the first time I've heard of that tool, but an extensible policy tool like that might solve a lot of challenges we have at work.

andyroid 6y ago
We use OPA for use cases ranging from kubernetes admission control, to microservice authorization and CI/CD pipeline policies. It's one of those tools you can't realize you could have lived without once you start using it.

blago 6y ago

I would add one more general security tip. Always restrict your mapped ports to localhost (as in 127.0.0.1:3306:3306) unless you really want to expose it to the world. I think it's counterintuitive, but I learned the hard way that 3306:3306 will automatically add a rule to open the firewall on linux and make MySQL publicly accessible.

BossingAround 6y ago
> I learned the hard way that 3306:3306 will automatically add a rule to open the firewall on linux and make MySQL publicly accessible. Is that true? 3306:3306 would bind the port on all interfaces, but I was under the assumption you'd have to explicitly enable firewall port 3306 for the machine to accept traffic to port 3306 from outside of your machine. I'll have to test that.
- blago 6y ago
  https://github.com/docker/for-linux/issues/690 https://github.com/docker/for-linux/issues/690
- isbvhodnvemrwvn 6y ago
  From memory and a quick glance at one of my servers: When an IP packet related to your container arrives (<host ip>:<host port>): - docker rewrites it to target <your container's ip address>:<your container's port> (NAT table, chain PREROUTING delegates to DOCKER chain with the relevant DNAT entry) - since the IP does not match your host, the packet is forwarded (there's a relevant routing entry pointing to docker's virtual interface) - the first thing you encounter in the FORWARD chain of the FILTER table is a few jumps to the docker-related chains, DOCKER chain in particular accepts all packets destined to <your container's ip address>:<your container's port> So a few takeaways: - your standard firewall might not be involved because its chains are plugged in after the docker chains in the FORWARD chain (e.g. ufw under Ubuntu) - if the above is true and you want your firewall to matter, you have to add stuff to DOCKER-USER chain in the FILTER table - at that point the host port and IP doesn't matter since it's already been mapped in the NAT table's PREROUTE chain at the beginning of processing - write your firewall rules to address specific containers

azangru 6y ago

A very nice web page, too! Snappy, no javascript apart from analytics, tastefully designed, mobile-friendly.

dkarp 6y ago

Something really useful I recently discovered is multi-stage Dockerfiles. Using FROM and then COPY --from to copy from a previous stage to prevent unwanted intermediate build steps that might expose secrets or just bloat your images. Two ways this was useful for us. Firstly, we needed a private key in the image to pull some private git repos. By doing this in a previous stage, they're not included in the final image layers. Secondly, we have a python backend and small react app served from the same image. By splitting their build steps into a backend and frontend stage, changes to frontend code don't break caching for the later backend steps or vice versa. E.g. FROM python:3.8.3-slim-buster AS frontend # Do frontend build steps FROM python:3.8.3-slim-buster AS backend # Do backend build steps FROM python:3.8.3-slim-buster AS final COPY --from=frontend /app /app/frontend COPY --from=backend /app /app/backend

junon 6y ago
This is how C/C++ services are done in production, too. You have a build step that does `apk add --update alpine-sdk git cmake` etc., builds the service, and then you start again fresh with a FROM (new stage) and `COPY --from` over the build artifacts. Reading this thread I'm surprised this isn't common knowledge by now given how it's so incredibly paramount to efficient production releases.
- dkarp 6y ago
  It may well be common knowledge now. I discovered it when writing our Dockerfiles earlier this year and before that hadn't seen it. It looks like multi-stage builds were added to the best practices in 2018 [https://github.com/docker/docker.github.io/blob/master/develop/develop-images/multistage-build.md https://github.com/docker/docker.github.io/blob/master/devel...] which is probably why I missed it before
nullsense 6y ago
Docker has official support for secrets using Buildkit.
- dkarp 6y ago

surfer7837 6y ago

I also use Red Hat UBI (Universal Base Images). If you pull down any package from DockerHub (even official images) you'll be surprised about how many vulnerabilities they have.

freedomben 6y ago
Second this. The UBI images are meticulously maintained by Red Hat and are freely available and redistributable without a subscription. Some of the largest companies in the world are using these in production and putting dollars behind them, so you can be pretty confident that they will work and be maintained.

lipanski 6y ago

One of the most common mistakes I see is not using a .dockerignore file or, better said, relying on .gitignore when calling `COPY` on entire directories. Without a .dockerignore file in place, you could be copying over your local .env files and other unwanted things into the final image. On top of that, you might also want to add `.git/` to your .dockerignore file, as it could significantly reduce the size of your image when calling `COPY`. A more subtle issue I've noticed is the fact that `COPY` operations don't honour the user set when calling `USER`. The `COPY` command has its own `--chown` argument, which needs to be set if you'd like to override the default user (which is root or a root-enabled user in most cases). I wrote up a similar article a while back, though it's focused on general best practices: https://lipanski.com/posts/dockerfile-ruby-best-practices https://lipanski.com/posts/dockerfile-ruby-best-practices

silverwind 6y ago
Better yet, only COPY the actually needed needed files instead of the whole working directory. That way, there's no need for a `.dockerignore`.
- unleashit 6y ago
  Wondering why you think this is better. Not sure the trade off of a messy dockerfile and/or adding a bunch of layers (possibly bloating the image size) is worth the trade off if the concern is just about forgetting to update the dockerignore. The same could be said about gitignore.
  - kinghajj 6y ago
    Not the person you replied to, but personally, I like having control over what exactly gets into the final image, and (IME) have found that devs aren't great about remembering to update .dockerignore files. Re: extra layers, if you use multi-stage builds to separate the builder and final app images, you can avoid that.
  - ric2b 6y ago
    It's nice to know what files you need to build the image. Sort of like importing libraries at the top of a source code file.

rzimmerman 6y ago

> Do not store secrets in environment variables Yes, definitely don't put secrets in the Dockerfile itself. I'm curious if there are reasons not to use a .env file though? > Only use trusted base images This is a good sentiment, but docker hub also has plenty of images that are built directly from a github repo. You can inspect the Dockerfile and (as long as you trust Docker) trust that it was built as written. In this case I recommend pinning to a specific container SHA (image_name@sha256:...) in case the source repo gets compromised. For official images, you can pin to a tag IMO. > Do not use ‘latest’ tag for base image Regardless of the security concerns, pinning to latest will probably bite you when there's a major version bump (or even maybe a minor one). Imagine you built a container on ubuntu:latest when latest was 20.04 and some new employee gets hit with 22.04. That's a bad surprise. > Avoid curl bashing Evergreen fight here, but I agree with the author that you should at least validate checksums on external files before executing or using them. > Do not upgrade your system packages This is where the throne of lies about Docker idempotency crumbles. You have to apt-get update because Debian/Ubuntu repositories remove old versions when there are security fixes (not 100% sure on this, feel free to correct me). So if the ubuntu:20.04 image is released and there's a security update in openssh-client, running "apt-get install openssh-client" without "apt-get update" will fail. So we all run "apt-get update" and pretend that the containers we build are time-invariant. They're not, and in fact we occasionally get security updates snuck in there. Luckily Debian and Ubuntu do a good job not breaking things and no one complains. But if you build a container on Tuesday it's not guaranteed to be the same as on Wednesday and there's nothing you can do about that with a Debian distro. But it's actually fine in practice so we pretend not to notice. The point about not running "apt-get upgrade" is kind of moot - you're going to effectively be upgrading whatever you "apt-get install", so it's probably worth taking the same trade on the built-in packages. Importantly, there's no security risk - just a configuration one. > Do not use ADD if possible Same as above - you should be checksumming external dependencies or self-hosting them. > Do not root > Do not sudo I haven't thought deeply about these. I'd prefer to trust the container system isolation rather than playing with users and permissions. I don't understand this risk well enough to have a well-formed opinion.

kelnos 6y ago
> Yes, definitely don't put secrets in the Dockerfile itself. I'm curious if there are reasons not to use a .env file though? Because often that .env file will get checked into source control (sometimes intentionally, sometimes accidentally), and then you have secrets in your git history that you need to rotate. In general the best thing to do is store your secrets in a place/service specifically designed for secrets, and either fetch them directly into the container in your entrypoint, or pull them into a location that you mount as a volume in the container. > I'd prefer to trust the container system isolation rather than playing with users and permissions. Don't. Containers don't give you full isolation. The 'root' user inside the container is the same 'root' user as outside, and container escapes may be possible. A good defense-in-depth strategy suggests that you should run things using the least privileges possible, and that doesn't change just because something is running in a container.

nomadiccoder 6y ago

This article is interesting but it would be far more helpful to link to a solution for each point.

andmarios 6y ago

Dockerfile security best practices: treat is as you would treat any linux server. :)

vorticalbox 6y ago

Quick question of you're not meant to use env for secrets then how are you meant to get secrets into your application? What's the best way to handle this?

keikun17 6y ago
using buildkit https://pythonspeed.com/articles/docker-build-secrets/ https://pythonspeed.com/articles/docker-build-secrets/
tynorf 6y ago
Probably either via a third party service (such as AWS secrets manager), or mounted as files scoped to the user your process is running as (which is not root, right? :) ).
raxor53 6y ago
Typically your host will have a service specifically for secrets. For example: https://docs.github.com/en/free-pro-team@latest/rest/reference/actions#secrets https://docs.github.com/en/free-pro-team@latest/rest/referen...
CGamesPlay 6y ago
Two kinds of secrets: build-time (I need an API key to access a download needed to set up the container) and run-time (I need an API key to talk to the database). The post argues against using ENV for build-time secrets, but it's still recommended for run-time secrets. Sibling comments address alternatives for getting build-time secrets in.

jtchang 6y ago

How do you get around sometimes needing root inside the container to build things? For example building a container with buildroot inside.

vitalysh 6y ago
Using intermediate container to build and copy over resulting binaries should work.

tofflos 6y ago

The number one advice should be to use a linter. The number two advice should be to use an image security scanner. These tools combined will prevent most issues. Integrate them with CI to enforce a common set of best practices across an organization and to prevent security bike shedding.

jose_zap 6y ago

Hadolint, a docker file linter checks for most of those best practices https://github.com/hadolint/hadolint https://github.com/hadolint/hadolint

bentolor 6y ago
It looks really great at first to have a hassle-free alternative with sensible defaults. Unfortunately installing seems non-trivial. Would love simple binary or .deb/.rpm install.
- jose_zap 6y ago
  Looks like it is packaged as .rpm at least https://pkgs.org/download/hadolint https://pkgs.org/download/hadolint

naranha 6y ago

> Using ENV to store secrets is bad practice because Dockerfiles are usually distributed with the application, so there is no difference from hard coding secrets in code. This sounds wrong. If secrets are in the environment they are not in the Dockerfile, so they are NOT distributed with the application.

LawnGnome 6y ago
What they're referring to is more specific than environment variables in general: you can use the ENV command in a Dockerfile to bake in a secret at build time, and that's what you generally shouldn't do. Injecting environment variables at runtime, however (through docker run -e or whatever orchestration system you're using), is good.
- naranha 6y ago
  Ah yes, that makes sense. I didn't understand they were talking about the ENV command at build time. It was the heading that got me on the wrong path, I think that should be clarified further: > Do not store secrets in environment variables
  - pvtmert 6y ago
    I'm bit aganist having env-secrets inside container. Because PID 1 has that env, all processes spawned from that can read all of those. I prefer mounting them to /run/secrets via tmpfs. Which can also have selinux policy attached. This way, someone else cannot read them by spawning shell inside container
sethhochberg 6y ago
I don't think the author is talking about loading secrets from the environment - I think they're specifically talking about hardcoding secrets into the Dockerfile and using the Dockerfile ENV directive to set secrets for the processes running in the container (baking them into the image), instead of passing them at runtime, which sounds just horrifying enough that I'm sure people do it in real codebases.
6y ago

chias 6y ago

> Do not upgrade your system packages Ah, the joys of working somewhere that isn't required to document, answer for, and ultimately remediate every CVE that is present in any package installed on any of your containers within your production application. Sadly, compliance and regulatory oversight don't leave this option open to everyone.

cmwelsh 6y ago
Is this a good argument for building containers as “bare metal” as possible? You don’t have to remedy CVEs (and rebuild your containers) for anything that isn’t actually your application.
- chias 6y ago
  That is pretty much the only thing possible in these scenarios. Anything Debian or Ubuntu or pretty much any "normal" distribution is right out: external vulnerability scanners always seem to go by package version, and `packagename-12.5.1-debian-security-fixes.b` is still the vulnerable version 12.5.1 as far as any scanner is concerned. At this point, we `FROM scratch` when possible, and deploy on AL2 when not. There's good reasoning against the concept of barebones containers, but unfortunately everything from bricks, knives, and well-reasoned arguments all bounce harmlessly off of regulations and external compliance requirements.

minimaxir 6y ago

Google's distroless containers are an interesting approach for both security and performance as well, albeit with limited language support: https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless

_nhynes 6y ago
Relatedly, rust-musl-builder [0] is useful for getting Rust binaries to run on the `static` instead of `cc` base image. [0] https://github.com/emk/rust-musl-builder https://github.com/emk/rust-musl-builder
- minimaxir 6y ago
  What's the advantage of doing that?
  - acomar 6y ago
    you can drop the resulting binary in a scratch container that has no shell, etc.

EdSchouten 6y ago

If you want to create container images for an application you wrote yourself in a commonly used programming language (Go, Python), consider using Bazel with rules_docker: https://github.com/bazelbuild/rules_docker https://github.com/bazelbuild/rules_docker rules_docker allows you to create byte-for-byte reproducible container images on your system, without even having a Docker daemon installed. So much cleaner to use than 'docker build' once you get the hang of it!

ImJasonH 6y ago
If you're building images from go binaries you can do that without Bazel using github.com/google/ko :)

m0zg 6y ago

Why would you write such an article and not show what the actual _recommended_ practices look like?

kelnos 6y ago

Regarding using USER somenonrootuser ... how do people deal with the need to write things out on container start? Some of my services require config files, and I need to interpolate the values of environment variables into those config files, write them out, and then start the application. (Also assume the application itself doesn't have options for dropping privileges.) I'd rather not make the filesystem locations in question writable by `somenonrootuser`. My current strategy is to let the container entrypoint start as root, but then I have a wrapper program installed that drops privileges before it exec()s the actual service. It works, but is there a better / more accepted way of doing this?

Propaganda_ 6y ago
If the services can't be modified to load their config directly from env vars, write the config to an off-root scratch volume (e.g. mounted to /tmp/) and have them load from that. The root volume should be mounted read-only either way to prevent modification of your services should something get RCE.
CGamesPlay 6y ago
[deleted] this idea doesn't work on attempting
piaste 6y ago
If you control the services, make them read from user-writable files instead of root ones. (Ideally you could skip the middleman and just read from env vars directly!) If you don't control the services, maybe in the Dockerfile you could write a dummy config file and chmod it?

zmmmmm 6y ago

Something that I would love to see some specific guidance on is TLS certificates. It seems like commonly used webservers (eg: nginx) pretty much insist on the private certificate being available effectively unencrypted to the web server process. On a traditional server that gets protected by user permissions, but inside Docker I am not sure how to achieve this. Is there a pattern people use to deploy certificates inside docker containers, or is the pattern "don't do that"?

GauntletWizard 6y ago
Don’t deploy your certificates with docker. That’s not to say: Don’t give your web servers running in docker certificates, but that you shouldn’t distribute certificates with your docker images. The Private Key/Certificate pair should be added at runtime. How it gets there is specific to your management framework. To give a specific example, on Kubernetes you store your Private Key and Certificate as a secret (there’s a special “type” of secret for this, but K8s doesn’t actually treat it any differently). You then mount that certificate as a volume (in /var/run/secrets/tls, or wherever you define it), and it shows up as a normal file, accessible to whichever user your container runs as.
- zmmmmm 6y ago
  thanks! So for clarity then, if somebody shells into the container - at that point it's still sitting there mounted and they can read the file? Or does k8s somehow manage this in a way that only the web server process can see it? Or do we just accept that you have to treat anyone with access to kubectl as authorised to know your private cert ?
  - FridgeSeal 6y ago
    > at that point it's still sitting there mounted and they can read the file? Yes, if someone has exec access into your container, they'll be able to see the secret, unless you do something like making the K8s secret an already encrypted blob, and then in process decrypting it again and reading it. If someone's got exec access to your container though, you've got bigger problems. > Or do we just accept that you have to treat anyone with access to kubectl as authorised to know your private cert ? You can set up access so that someone can login with Kubectl but still only be readonly, and you can preclude read ("get") access to secrets to the readonly role, so they won't be able to view the contents of the secret.

atticusCr 6y ago

How about using a hardening guide such as CIS as part of the build process?

gautamdivgi 6y ago

I have one more that I have seen used within corporate networks. If a proxy is used to build docker a lot of engineers add ENV http_proxy, https_proxy in the Dockerfile. This causes the proxy to be on by default. Use ARG instead if you just want the proxies at build time.

schemescape 6y ago

Is it just me or does this article highlight bad practices (and ways to detect them), but then not really address the “correct” way to avoid many of the bad practices?

dandigangi 6y ago
+1... was just thinking this.

josep2 6y ago

Great post, love the use of Open Policy Agent and Conftest!

creztoe 6y ago

Do not pass sensitive data to docker build via --build-arg. When you access this with "ARG" you will log the information in the docker history, visible to all. Use "--secret" or use the ARG in an intermediate build stage which doesn't have it's history preserved, then copy any necessary files form the intermediate image to your file image manually. A perfect example of this would be passing your NPM_TOKEN to install company scope packages.