13 ms·

Things to avoid in Docker containers

271 points by 0xmohit 11y ago

api 11y ago
Don't run more than one process needs nuance. Apache and Postgres work fine in containers and are fork based, and helper apps or threads that are purely supportive don't matter either. No more than one job or "thing" makes sense. I hate ham fisted advice like this that will always end up being dogmatized and multiplying complexity needlessly.
- conradev 11y ago
  The author does make a valid point in that logs are harder to collect the more processes you add. For example, I run gunicorn inside of a container with worker processes, and I had to do some trickery (writing to /proc/1/fd/1) to redirect the logs of the worker processes to the stdout of the master process so that they would show up in `docker logs`.
  - stephenr 11y ago
    It's almost like the idea of relying on stdout from processes in a container to collect logs is fucking stupid. With all the hoops people jump through so they can say "we use Docker, we're cool" I think I shall refer to Docker fan-boys as Olympic Gymnasts from now on.
    - XorNot 11y ago
      If you drill through the docker source code, it becomes apparent that their are some rather serious issues to doing something other then this. I really really want a LOG directive for docker where I can just setup some pipes which the daemon will collect and tag into it's log files (would make dealing with old Java processes a lot easier). But actually finding somewhere to hook this logic in - huge task. A lot harder then grabbing stdout/stderr of the init process in the container.
      - stephenr 11y ago
        The whole point of getting logs out of containers is to centralise and store them somehow. This exact job is what Syslog (over a network) does.
      - Gigablah 11y ago
        You can use different logging drivers: https://docs.docker.com/engine/admin/logging/overview/ https://docs.docker.com/engine/admin/logging/overview/
        ibotty 11y ago
        That does not answer the question on how to capture _different_ application logs.
        elbear 11y ago
        Why not have the application log to a file on a data container?
        DanielDent 11y ago
        Well, now you've just created a new problem to deal with...
        lisivka 11y ago
        Who will manage these volumes, directories and log files? Why not just use existing infrastructure? I use containers for 10 years (they are called VServer), and I newer had a problem with standard setup: some less important logs are stored directly in container and rotated with logrotate, some more important logs (and error messages from less important logs) are forwarded to central server for analyzing, some are just disabled or discarded.
      - DanielDent 11y ago
        I'm a heavy docker (ab)user, but I haven't gone through the source code - i.e. perhaps there is some complexity I'm overlooking. But couldn't Docker provide additional file handles for extra logging streams? Programs which understand they live in a Docker world could hook into an special system call directly. Non-Docker-Logging-Enabled apps could have a helper program (conceptually similar to 'ip netns exec') remap STDOUT & STDERR.
    - Gigablah 11y ago
      Or maybe we can refrain from calling each other perjorative names? Wouldn't that be grand?
    - aprdm 11y ago
      Why is it stupid ? Is the same approach that heroku advocates. I have loads of containers logging to stdout and another container which collects all stdouts and send to a logging-as-a-service. Why is it a bad idea ?
      - stephenr 11y ago
        > Why is it stupid Let's compare logging via stdout, to say syslog, with a simple test. How do you identify the difference between a warning and an error or an emergency, in a string sent to stdout? What happens if the message contains a newline. How do you know if what follows is a continuation of the log entry, or is a new log entry? If your application dares to have two processes in the same container (i know, crazy concept) how do you separate them. The "literally one process per container" idea is what both causes the issue, and causes the obvious solution to the issue to be even harder to solve for the cases where software doesn't natively support the syslog protocol.
        aprdm 11y ago
        Were you the one who downvotted me because you disagree? o.O Anyways, this is dealt with in the logging-as-a-service (LAAS) that receives the logs... say you are using the python logger, you have different types of logging (DEBUG, INFO, WARN, ERR). If you just start the line with one of those statuses, your LAAS will know it should email you or something like that. The print, or file, or whatever, it's just how your data is getting somewhere for the proper analysis. You know if it's a new line or not because every log message starts with the same format, something like {APP, TIME, SEVERITY} and then the message...
        stephenr 11y ago
        So instead of having actually structured information in defined fields, you're relying on a specific service's format. So presumably you then have to try to get all the services you use (e.g. Apache / NGinx / HAproxy / Varnish / (my|postgre)sql / redis / memcache etc) to format logs in the same way, and do it to stdout... > Were you the one who downvotted me because you disagree? o.O Down voting on HN is broken[1], so I don't use it, ever[2]. 1. In the sense of, it does the wrong thing, not in the sense of it doesn't do anything. 2. Except when i hit the wrong thing because of the complete lack of usability on a mobile device.
  - derFunk 11y ago
    Normally, if you run Docker you won't have a single container anyway, normally you want to scale out. Hence you'd be better off using a centralized logging infrastructure like ELK or one of the many commercial solutions like Splunk etc.
    - LoSboccacc 11y ago
      I don't follow. If you go with one service one machine model, not using a container is more or less the same than just scripting a repeatable install. I'd say they are for isolation before they're for scaling out
    - renke1 11y ago
      True, but it's not uncommon to only route the stdout/stderr of each container to fluentd, logstash and the like such that the containers do not have to care about where the logs go (other than just writing to stdout/stderr). And now that we have logging driver I definitely would want to avoid custom in-container log routing.
      - derFunk 11y ago
        Yes, it's good practice to have a separated logging container (eg logstash forwarder, filebeat etc), which accesses the shared log folder of the other container you want to actually forward the logs from. I'm recommending using file forwarder because this way you can simply categorize your logs by filename. Pushing all to stdout is a pain.
        fatherlinnux 11y ago
        Yeah, the problem is getting them from each process in the container, then get them to the centralized logging. It can be done either way though. Especially with the syslog driver in Docker...
  - deleted 11y ago
    [deleted]
- audidude 11y ago
  If you put Apache and Postgres into the same cgroup/pid namespace/mount namespace/network namespace, an attack on Apache could result in access to the mount namespace of your Postgres. (Presumably that is something you are trying to prevent).
  - _delirium 11y ago
    I think what the comment you're replying to was saying is that even putting just one of them in a container would violate the rules here, because Apache and Postgres are multi-process applications (at least in many common configurations), and the rules say flat-out no more than one process.
    - lwf 11y ago
      Referring to these as "rules" when they aren't is exactly what great-grandparent was warning against: > […] will always end up being dogmatized and multiplying complexity needlessly. If more than one process helps you get your job done, great. If it's all one self-contained application, go for it. You start feeling the pain when you have unrelated or loosely-coupled applications in the same container.
gizi 11y ago
> Don’t store data in containers You could manage your own data volumes or else let docker do it for you. I don't see the difference. In the end, you will still need a backup policy. I snapshot all data every 30 minutes for storage on another device. Nothing would change to that policy when using external data containers. > Don’t ship your application in two pieces Internally -and externally provided software have different dynamics. Issues are different. I update my own software with bug fixes and new features quite frequently. I don't need to do that for externally provided software. If that happens, I will indeed rebuild the container. In all practical terms, my own software sits in a host folder where I can update it, without rebuilding the container. > Don’t use only the “latest” tag I use debian:latest. I don't see a problem with that. It may lead to trouble some day, but then I just change the tag to the latest but one version before rebuilding. The problem has not occurred up till now. > Don’t run more than one process in a single container I have containers that internally queue their tasks. The queue processor is then a second process, besides the main network listener process. There are many reasons why it could be meaningful for a program to use more than one process. A categorical imperative in this respect is misguided. Other engineering concerns will take precedence. You see, if I can reasonably split a container into two, I will. It's just like with a function. If it is possible to split it in two, I will most likely do so. But then again, such design recommendation should never be phrased as a categorical imperative. > Don’t store credentials in the image. Use environment variables Both are pretty much the same problem. If the attacker can read files containing credentials, he will also be able to read environment variables with them. > Containers are ephemeral A network-based service uses a long-running listener to process requests. Why shut it down? That would just disrupt the service. Containers may very well be long-lived. They are not necessarily ephemeral. This subject is not part of a domain such as morality where categorical imperatives are the norm. There are pretty much no categorical imperatives in software engineering. Software is mostly subject to just an Aristotelian non-contradiction policy. If your choices are non-contradictory, feel free to go with them. Furthermore, unmotivated, categorical imperatives simply have no place in this field.
- jarfil 11y ago
  > Containers are ephemeral It doesn't matter how long they are up, it means you should expect them to go down at any moment and lose all data stored inside. Plan for it by storing your data and config somewhere else. * Custom volumes and system mounts are persistent. * Images are persistent-ish, they get replaced by updates. * Containers are ephemeral, intended to scale up, down, sideways or whatever. > Don't store data in containers As stated above, you should expect your containers to puff out and into thin air at any moment. Also remember that docker auto-created volumes get auto-deleted when no more containers are using them. Use custom created volumes or system mounts if you don't want to wake up to a nasty surprise. > Don’t use only the “latest” tag It will bite you in the ass if you do. It may not today, nor tomorrow, but it will some day, the exact day that a bigger version upgrade happens. It's not a "hasn't happened to me yet, so it may not happen" thing, it will happen to you if you keep using the "latest" tag, no doubts about that. So just plan accordingly and don't use the "latest" tag (for deployment). > Don’t run more than one process in a single container This could be more of a "really try not to" than a "don't", but the truth is the more stuff you put in a single container, the harder it becomes to manage and scale. It is much better to have self-contained containers (see what I did there?) that you can match and mix with stuff like docker-compose. > Don’t store credentials in the image. Use environment variables No, they are not the same problem, and it has nothing to do with security. When you want to scale and spin up a new container... now you can't because your credentials are baked into the image. Don't store credentials in the image, period. If you don't like environment variables, you could mount a per-container config dir, but it's usually much easier to manage config passed as environment variables to the container.
cardoni 11y ago
Great article. Very well written! Nice.
dcw303 11y ago
I'm a docker newbie and I had a hare-brained idea to install postfix and dovecot in a container. I thought it would be neat to version/deploy the various config necessary on top of a clean OS install. As I'll probably mess it up the first time, a quick way to blast the system clean is appealing. If I can keep credentials and data outside of the container, is this still a bad idea? Would I run into trouble with other containers needing to use SMTP?
- piqufoh 11y ago
  I've run simple SMTP from a container and not had any issues. I've not tried a more complex stack (but I have thought about it... ) and I was looking at this project https://github.com/tomav/docker-mailserver https://github.com/tomav/docker-mailserver I'm afraid I can't account for it - having never tried it, but it looks like the sort of thing you might be interested in! PS If you are a newbie, do try something simple first - it took a while to get my head around the differences in the docker tooling (for example this project uses docker-compose which is another tool on top of the docker cl tool). EDIT: Ooh - a tutorial https://tvi.al/simple-mail-server-with-docker/ https://tvi.al/simple-mail-server-with-docker/
- contingencies 11y ago
  The biggest problem you will have is that a lot of critical SMTP functionality (SPF, DKIM, DMARC, antispam blackhole lists, SSL certificates, email addresses themselves) rely on DNS. So instead of testing with just a container, to get any meaningfully useful notion of reliability and/or applicability to the real world for your container you will have to test with entire fake DNS infrastructure and multiple hosts. While you could technically do it, the reality is you're entering a world of pain. Currently setting up a mail-server myself, the majority of clients are in China but I want reliable sending to foreign servers: it's quite difficult. Like finding a host, when so many Chinese IP ranges are blanket blacklisted by western mailservers, and the China/foreign link is often slow/overloaded/lossy. Taking the time to update Gentoo postfix documentation en-route, for the other DIY'ers with an NCH (not compiled here) problem. Not using docker.
derFunk 11y ago
All is quite valid, yet I disagree with "7) Don’t run more than one process in a single container". This is perfectly doable with supervisors like ... supervisord, which ensure that the multiple processes you want to run stay alive. In some environments it's just virtually impossible to only let one process run (eg webserver/proxy, log forwarder, fastcgi etc).
- derFunk 11y ago
  Oh, I want to add why supervisord makes so much sense: If your "main process" (PID 1) dies, your container would stop. Even if you only have a single process to run, using supervisord makes sense to monitor that single process, because restarting a stopped container can be a pain because it mostly involves some manual work. Supervisord itself is written in python and is very stable (because simple) itself.
  - azylman 11y ago
    Most container orchestration systems (Kubernetes, Mesos, ECS, etc.) handle restarting failed containers - that's generally something you want to be outside of your container, not inside. If you don't use one of those, you can still have supervisor run outside of your containers, and manage a set of "docker run" commands.
    - derFunk 11y ago
      > Most container orchestration systems (Kubernetes, Mesos, ECS, etc.) handle restarting failed containers - that's generally something you want to be outside of your container, not inside. Not sure if I understand correctly. I'm using ECS. How would you provision the Host with supervisord (automatedly)? ECS and the other systems are taking care of restarting the containers anyway, but they don't necessarily have any logic attached (like restart it 5 times, then stop and notifify if the container doesn't come up again). From my POV it's definitely better to directly have it deployed within the container. Maybe it's depending on the use case. The problem is that it often happens that the container cannot restart because the main process (invoked by ENTRYPOINT or CMD) cannot start up anymore, e.g. because of corrupted (config) data which has to be loaded on startup or other suddenly unsatisfied dependencies.
XxFurCornxX 11y ago
The article is very well written, and I mostly agree with it. Upvoted. Some other people will disagree with me.
- philjackson 11y ago
  Apart from this one, your comment was the most pointless in this thread. Just use the up arrow!
Perceptes 11y ago
Any security folks have some specific practical examples of how running processes as root inside the container is dangerous? I understand how root in the container mapping to root on the host is risky (this is what user namespaces are for) but I don't have enough experience to know why it matters inside the container.
- darkr 11y ago
  Though things have improved considerably a while back with docker dropping most capabilities before starting a container, but without user namespacing on a shared kernel, ultimately root in the container == root in the host machine. More specifically, uid 0 on the container == uid 0 on the host, so if container breakout occurs then you have root on the host. As you mention, user namespaces and remapping the root user should solve this issue, but initial support for this was only released 7 days ago in docker 1.10.2 > https://github.com/docker/docker/commit/ed9434c5bb64f49db442027e5e748bfbb46d0a0a https://github.com/docker/docker/commit/ed9434c5bb64f49db442... [edit]: Preliminary support for user namespaces was added in 1.10, above commit is addressing a bug in userns support.
- xorcist 11y ago
  Apart from that Docker doesn't really utilize user namespaces until recently, user namespaces are tricky. There have historically been buts and ifs involved. Don't rely on it unless you really understand it! But why would you want to run things as root anyway? You probably didn't before Docker, so why do it now? With Docker, there's even less reason to muck about with capabilities for binding low ports etc. It's absolutely an unnecessary risk to take.
  - fpoling 11y ago
    If one use host volumes, Docker has no way to setup them in a way that are usable for non-root container. There are workarounds, but running as root is a simplest thing to do that I think many follow.
    - xorcist 11y ago
fatherlinnux 11y ago
The large yum update is a symptom, not the problem: 1. The upstream base image maintainers aren’t doing their job of keeping the image up to date, so it is huge. If the upstream maintainer had this base image updated, there would be no updates. Not the developer’s fault. Also, if the developer puts it into production, it’s their responsibility to make sure there are no CVE’s in their code, and hence their responsibility to do a yum update. You touched it, you own it. 2. The Docker tools don’t have a way of flattening the image easily (without doing a save/load). This can be fixed down the road. Telling developers not to do yum updates terrifies me as a security person….
- ahoka 11y ago
  Maybe the conclusion here is to build your own base images?
  - justingood 11y ago
    This is exactly the conclusion we arrived at. We re-build our base images nightly (including all available updates) and then base the application images on those.
  - fatherlinnux 11y ago
    That solves part of the problem. The problem with base images, are they are the root - I call them Adam and Eve images. They do not solve all of the supply chain problems. Layered images such as Apache and Nginx still may be maintained by operations or even Middlware teams. Each team is responsible for making sure they update their piece of the supply chain. If the operations team (or image supplier) doesn't, the "yum upgrade" (or apt-get) downstream (by the developer or middlware maintainer) will bloat the environment. The problem is, it's a symptom, not the problem. Telling people to NOT do yum updates, just promotes security problems because people upstream aren't keeping up with updates. All of that said, the ONLY sane way to do all of this is with Automation. Hence, the reason for ImageStreams [1], BuildConfigs, DeployConfigs, and Triggers inside of OpenShift. When any piece of the spider web detects a change, all of the dependent layers kick off jobs to rebuild themselves. Also, having good tests is critical. Yeah, most people on the planet haven't thought through this. Yes, Docker images make your life more convenient, at the deploy stage, with the downside that you need to do a LOT more thinking (automation and testing) at build phase. Like shipping containers, we pack them at the factory, but can move them around at will after that. Instead of having dock hands load the ship at the port (which took 45 days in 1906). The supply chain problems are the same either way, but at least Docker gives you a good format to solve them. [1]: https://docs.openshift.com/enterprise/3.0/architecture/core_concepts/builds_and_image_streams.html https://docs.openshift.com/enterprise/3.0/architecture/core_...
anentropic 11y ago
Does anyone have any advice on using "init" scripts as CMD for the container, vs something like Ansible for post-provisioning? Example: a Django web app... after deploying the new version container you need to run migrations etc Similarly for say RabbitMQ you may need to ensure your vhosts are defined etc... commands that need to be executed against a running server, can't be done in the Dockerfile itself. So far I've ended up with a bash script in each container that does the necessary init tasks before exec-ing the main process. Or in some cases (Rabbit, Mongo) they first run the server in a limited state, do the init tasks against it, then stop it and exec again as the container process. Have started to feel it might be cleaner to instead just start the main process and then provision it with Ansible afterwards. But then I lose the ability to have a system up and running just via docker-compose...
- goldbrick 11y ago
  You should think of all these operations in terms of the state they persist. For example, a database migration doesn't modify the application at all, but only the data directory wherever your database is running (whether mounted in another container or across a network). Whatever process is orchestrating your containers should take responsibility for running setup chores in a separate, ephemeral container from the eventual application container. docker-compose is often not enough, unfortunately.
fpoling 11y ago
I do no think "Don’t create large images" is a good one. I prefer to have a single image that I use to run multiple containers with different services that forms a pod. This greatly simplifies distribution as I need just single image file. It also greatly simplifies updates. But with such image it is useful to add few tools so docker exec -ti container /bin/bash creates a shell that one can use for debugging or verification.
grantlmiller 11y ago
we open-sourced www.fromlatest.io as a way to share docker best practices in a really easy to use way

271 points by 0xmohit 11y ago

api 11y ago

Don't run more than one process needs nuance. Apache and Postgres work fine in containers and are fork based, and helper apps or threads that are purely supportive don't matter either. No more than one job or "thing" makes sense. I hate ham fisted advice like this that will always end up being dogmatized and multiplying complexity needlessly.

conradev 11y ago
The author does make a valid point in that logs are harder to collect the more processes you add. For example, I run gunicorn inside of a container with worker processes, and I had to do some trickery (writing to /proc/1/fd/1) to redirect the logs of the worker processes to the stdout of the master process so that they would show up in `docker logs`.
- stephenr 11y ago
  It's almost like the idea of relying on stdout from processes in a container to collect logs is fucking stupid. With all the hoops people jump through so they can say "we use Docker, we're cool" I think I shall refer to Docker fan-boys as Olympic Gymnasts from now on.
  - XorNot 11y ago
    If you drill through the docker source code, it becomes apparent that their are some rather serious issues to doing something other then this. I really really want a LOG directive for docker where I can just setup some pipes which the daemon will collect and tag into it's log files (would make dealing with old Java processes a lot easier). But actually finding somewhere to hook this logic in - huge task. A lot harder then grabbing stdout/stderr of the init process in the container.
    - stephenr 11y ago
      The whole point of getting logs out of containers is to centralise and store them somehow. This exact job is what Syslog (over a network) does.
    - Gigablah 11y ago
      You can use different logging drivers: https://docs.docker.com/engine/admin/logging/overview/ https://docs.docker.com/engine/admin/logging/overview/
      - ibotty 11y ago
        That does not answer the question on how to capture _different_ application logs.
        elbear 11y ago
        Why not have the application log to a file on a data container?
        DanielDent 11y ago
        Well, now you've just created a new problem to deal with...
        lisivka 11y ago
        Who will manage these volumes, directories and log files? Why not just use existing infrastructure? I use containers for 10 years (they are called VServer), and I newer had a problem with standard setup: some less important logs are stored directly in container and rotated with logrotate, some more important logs (and error messages from less important logs) are forwarded to central server for analyzing, some are just disabled or discarded.
    - DanielDent 11y ago
      I'm a heavy docker (ab)user, but I haven't gone through the source code - i.e. perhaps there is some complexity I'm overlooking. But couldn't Docker provide additional file handles for extra logging streams? Programs which understand they live in a Docker world could hook into an special system call directly. Non-Docker-Logging-Enabled apps could have a helper program (conceptually similar to 'ip netns exec') remap STDOUT & STDERR.
  - Gigablah 11y ago
    Or maybe we can refrain from calling each other perjorative names? Wouldn't that be grand?
  - aprdm 11y ago
    Why is it stupid ? Is the same approach that heroku advocates. I have loads of containers logging to stdout and another container which collects all stdouts and send to a logging-as-a-service. Why is it a bad idea ?
    - stephenr 11y ago
      > Why is it stupid Let's compare logging via stdout, to say syslog, with a simple test. How do you identify the difference between a warning and an error or an emergency, in a string sent to stdout? What happens if the message contains a newline. How do you know if what follows is a continuation of the log entry, or is a new log entry? If your application dares to have two processes in the same container (i know, crazy concept) how do you separate them. The "literally one process per container" idea is what both causes the issue, and causes the obvious solution to the issue to be even harder to solve for the cases where software doesn't natively support the syslog protocol.
      - aprdm 11y ago
        Were you the one who downvotted me because you disagree? o.O Anyways, this is dealt with in the logging-as-a-service (LAAS) that receives the logs... say you are using the python logger, you have different types of logging (DEBUG, INFO, WARN, ERR). If you just start the line with one of those statuses, your LAAS will know it should email you or something like that. The print, or file, or whatever, it's just how your data is getting somewhere for the proper analysis. You know if it's a new line or not because every log message starts with the same format, something like {APP, TIME, SEVERITY} and then the message...
        stephenr 11y ago
        So instead of having actually structured information in defined fields, you're relying on a specific service's format. So presumably you then have to try to get all the services you use (e.g. Apache / NGinx / HAproxy / Varnish / (my|postgre)sql / redis / memcache etc) to format logs in the same way, and do it to stdout... > Were you the one who downvotted me because you disagree? o.O Down voting on HN is broken[1], so I don't use it, ever[2]. 1. In the sense of, it does the wrong thing, not in the sense of it doesn't do anything. 2. Except when i hit the wrong thing because of the complete lack of usability on a mobile device.
- derFunk 11y ago
  Normally, if you run Docker you won't have a single container anyway, normally you want to scale out. Hence you'd be better off using a centralized logging infrastructure like ELK or one of the many commercial solutions like Splunk etc.
  - LoSboccacc 11y ago
    I don't follow. If you go with one service one machine model, not using a container is more or less the same than just scripting a repeatable install. I'd say they are for isolation before they're for scaling out
  - renke1 11y ago
    True, but it's not uncommon to only route the stdout/stderr of each container to fluentd, logstash and the like such that the containers do not have to care about where the logs go (other than just writing to stdout/stderr). And now that we have logging driver I definitely would want to avoid custom in-container log routing.
    - derFunk 11y ago
      Yes, it's good practice to have a separated logging container (eg logstash forwarder, filebeat etc), which accesses the shared log folder of the other container you want to actually forward the logs from. I'm recommending using file forwarder because this way you can simply categorize your logs by filename. Pushing all to stdout is a pain.
      - fatherlinnux 11y ago
        Yeah, the problem is getting them from each process in the container, then get them to the centralized logging. It can be done either way though. Especially with the syslog driver in Docker...
- deleted 11y ago
  [deleted]
audidude 11y ago
If you put Apache and Postgres into the same cgroup/pid namespace/mount namespace/network namespace, an attack on Apache could result in access to the mount namespace of your Postgres. (Presumably that is something you are trying to prevent).
- _delirium 11y ago
  I think what the comment you're replying to was saying is that even putting just one of them in a container would violate the rules here, because Apache and Postgres are multi-process applications (at least in many common configurations), and the rules say flat-out no more than one process.
  - lwf 11y ago
    Referring to these as "rules" when they aren't is exactly what great-grandparent was warning against: > […] will always end up being dogmatized and multiplying complexity needlessly. If more than one process helps you get your job done, great. If it's all one self-contained application, go for it. You start feeling the pain when you have unrelated or loosely-coupled applications in the same container.

gizi 11y ago

> Don’t store data in containers You could manage your own data volumes or else let docker do it for you. I don't see the difference. In the end, you will still need a backup policy. I snapshot all data every 30 minutes for storage on another device. Nothing would change to that policy when using external data containers. > Don’t ship your application in two pieces Internally -and externally provided software have different dynamics. Issues are different. I update my own software with bug fixes and new features quite frequently. I don't need to do that for externally provided software. If that happens, I will indeed rebuild the container. In all practical terms, my own software sits in a host folder where I can update it, without rebuilding the container. > Don’t use only the “latest” tag I use debian:latest. I don't see a problem with that. It may lead to trouble some day, but then I just change the tag to the latest but one version before rebuilding. The problem has not occurred up till now. > Don’t run more than one process in a single container I have containers that internally queue their tasks. The queue processor is then a second process, besides the main network listener process. There are many reasons why it could be meaningful for a program to use more than one process. A categorical imperative in this respect is misguided. Other engineering concerns will take precedence. You see, if I can reasonably split a container into two, I will. It's just like with a function. If it is possible to split it in two, I will most likely do so. But then again, such design recommendation should never be phrased as a categorical imperative. > Don’t store credentials in the image. Use environment variables Both are pretty much the same problem. If the attacker can read files containing credentials, he will also be able to read environment variables with them. > Containers are ephemeral A network-based service uses a long-running listener to process requests. Why shut it down? That would just disrupt the service. Containers may very well be long-lived. They are not necessarily ephemeral. This subject is not part of a domain such as morality where categorical imperatives are the norm. There are pretty much no categorical imperatives in software engineering. Software is mostly subject to just an Aristotelian non-contradiction policy. If your choices are non-contradictory, feel free to go with them. Furthermore, unmotivated, categorical imperatives simply have no place in this field.

jarfil 11y ago
> Containers are ephemeral It doesn't matter how long they are up, it means you should expect them to go down at any moment and lose all data stored inside. Plan for it by storing your data and config somewhere else. * Custom volumes and system mounts are persistent. * Images are persistent-ish, they get replaced by updates. * Containers are ephemeral, intended to scale up, down, sideways or whatever. > Don't store data in containers As stated above, you should expect your containers to puff out and into thin air at any moment. Also remember that docker auto-created volumes get auto-deleted when no more containers are using them. Use custom created volumes or system mounts if you don't want to wake up to a nasty surprise. > Don’t use only the “latest” tag It will bite you in the ass if you do. It may not today, nor tomorrow, but it will some day, the exact day that a bigger version upgrade happens. It's not a "hasn't happened to me yet, so it may not happen" thing, it will happen to you if you keep using the "latest" tag, no doubts about that. So just plan accordingly and don't use the "latest" tag (for deployment). > Don’t run more than one process in a single container This could be more of a "really try not to" than a "don't", but the truth is the more stuff you put in a single container, the harder it becomes to manage and scale. It is much better to have self-contained containers (see what I did there?) that you can match and mix with stuff like docker-compose. > Don’t store credentials in the image. Use environment variables No, they are not the same problem, and it has nothing to do with security. When you want to scale and spin up a new container... now you can't because your credentials are baked into the image. Don't store credentials in the image, period. If you don't like environment variables, you could mount a per-container config dir, but it's usually much easier to manage config passed as environment variables to the container.

cardoni 11y ago

Great article. Very well written! Nice.

dcw303 11y ago

I'm a docker newbie and I had a hare-brained idea to install postfix and dovecot in a container. I thought it would be neat to version/deploy the various config necessary on top of a clean OS install. As I'll probably mess it up the first time, a quick way to blast the system clean is appealing. If I can keep credentials and data outside of the container, is this still a bad idea? Would I run into trouble with other containers needing to use SMTP?

piqufoh 11y ago
I've run simple SMTP from a container and not had any issues. I've not tried a more complex stack (but I have thought about it... ) and I was looking at this project https://github.com/tomav/docker-mailserver https://github.com/tomav/docker-mailserver I'm afraid I can't account for it - having never tried it, but it looks like the sort of thing you might be interested in! PS If you are a newbie, do try something simple first - it took a while to get my head around the differences in the docker tooling (for example this project uses docker-compose which is another tool on top of the docker cl tool). EDIT: Ooh - a tutorial https://tvi.al/simple-mail-server-with-docker/ https://tvi.al/simple-mail-server-with-docker/
contingencies 11y ago
The biggest problem you will have is that a lot of critical SMTP functionality (SPF, DKIM, DMARC, antispam blackhole lists, SSL certificates, email addresses themselves) rely on DNS. So instead of testing with just a container, to get any meaningfully useful notion of reliability and/or applicability to the real world for your container you will have to test with entire fake DNS infrastructure and multiple hosts. While you could technically do it, the reality is you're entering a world of pain. Currently setting up a mail-server myself, the majority of clients are in China but I want reliable sending to foreign servers: it's quite difficult. Like finding a host, when so many Chinese IP ranges are blanket blacklisted by western mailservers, and the China/foreign link is often slow/overloaded/lossy. Taking the time to update Gentoo postfix documentation en-route, for the other DIY'ers with an NCH (not compiled here) problem. Not using docker.

derFunk 11y ago

All is quite valid, yet I disagree with "7) Don’t run more than one process in a single container". This is perfectly doable with supervisors like ... supervisord, which ensure that the multiple processes you want to run stay alive. In some environments it's just virtually impossible to only let one process run (eg webserver/proxy, log forwarder, fastcgi etc).

derFunk 11y ago
Oh, I want to add why supervisord makes so much sense: If your "main process" (PID 1) dies, your container would stop. Even if you only have a single process to run, using supervisord makes sense to monitor that single process, because restarting a stopped container can be a pain because it mostly involves some manual work. Supervisord itself is written in python and is very stable (because simple) itself.
- azylman 11y ago
  Most container orchestration systems (Kubernetes, Mesos, ECS, etc.) handle restarting failed containers - that's generally something you want to be outside of your container, not inside. If you don't use one of those, you can still have supervisor run outside of your containers, and manage a set of "docker run" commands.
  - derFunk 11y ago
    > Most container orchestration systems (Kubernetes, Mesos, ECS, etc.) handle restarting failed containers - that's generally something you want to be outside of your container, not inside. Not sure if I understand correctly. I'm using ECS. How would you provision the Host with supervisord (automatedly)? ECS and the other systems are taking care of restarting the containers anyway, but they don't necessarily have any logic attached (like restart it 5 times, then stop and notifify if the container doesn't come up again). From my POV it's definitely better to directly have it deployed within the container. Maybe it's depending on the use case. The problem is that it often happens that the container cannot restart because the main process (invoked by ENTRYPOINT or CMD) cannot start up anymore, e.g. because of corrupted (config) data which has to be loaded on startup or other suddenly unsatisfied dependencies.

XxFurCornxX 11y ago

The article is very well written, and I mostly agree with it. Upvoted. Some other people will disagree with me.

philjackson 11y ago
Apart from this one, your comment was the most pointless in this thread. Just use the up arrow!

Perceptes 11y ago

Any security folks have some specific practical examples of how running processes as root inside the container is dangerous? I understand how root in the container mapping to root on the host is risky (this is what user namespaces are for) but I don't have enough experience to know why it matters inside the container.

darkr 11y ago
Though things have improved considerably a while back with docker dropping most capabilities before starting a container, but without user namespacing on a shared kernel, ultimately root in the container == root in the host machine. More specifically, uid 0 on the container == uid 0 on the host, so if container breakout occurs then you have root on the host. As you mention, user namespaces and remapping the root user should solve this issue, but initial support for this was only released 7 days ago in docker 1.10.2 > https://github.com/docker/docker/commit/ed9434c5bb64f49db442027e5e748bfbb46d0a0a https://github.com/docker/docker/commit/ed9434c5bb64f49db442... [edit]: Preliminary support for user namespaces was added in 1.10, above commit is addressing a bug in userns support.
xorcist 11y ago
Apart from that Docker doesn't really utilize user namespaces until recently, user namespaces are tricky. There have historically been buts and ifs involved. Don't rely on it unless you really understand it! But why would you want to run things as root anyway? You probably didn't before Docker, so why do it now? With Docker, there's even less reason to muck about with capabilities for binding low ports etc. It's absolutely an unnecessary risk to take.
- fpoling 11y ago
  If one use host volumes, Docker has no way to setup them in a way that are usable for non-root container. There are workarounds, but running as root is a simplest thing to do that I think many follow.
  - xorcist 11y ago

fatherlinnux 11y ago

The large yum update is a symptom, not the problem: 1. The upstream base image maintainers aren’t doing their job of keeping the image up to date, so it is huge. If the upstream maintainer had this base image updated, there would be no updates. Not the developer’s fault. Also, if the developer puts it into production, it’s their responsibility to make sure there are no CVE’s in their code, and hence their responsibility to do a yum update. You touched it, you own it. 2. The Docker tools don’t have a way of flattening the image easily (without doing a save/load). This can be fixed down the road. Telling developers not to do yum updates terrifies me as a security person….

ahoka 11y ago
Maybe the conclusion here is to build your own base images?
- justingood 11y ago
  This is exactly the conclusion we arrived at. We re-build our base images nightly (including all available updates) and then base the application images on those.
- fatherlinnux 11y ago
  That solves part of the problem. The problem with base images, are they are the root - I call them Adam and Eve images. They do not solve all of the supply chain problems. Layered images such as Apache and Nginx still may be maintained by operations or even Middlware teams. Each team is responsible for making sure they update their piece of the supply chain. If the operations team (or image supplier) doesn't, the "yum upgrade" (or apt-get) downstream (by the developer or middlware maintainer) will bloat the environment. The problem is, it's a symptom, not the problem. Telling people to NOT do yum updates, just promotes security problems because people upstream aren't keeping up with updates. All of that said, the ONLY sane way to do all of this is with Automation. Hence, the reason for ImageStreams [1], BuildConfigs, DeployConfigs, and Triggers inside of OpenShift. When any piece of the spider web detects a change, all of the dependent layers kick off jobs to rebuild themselves. Also, having good tests is critical. Yeah, most people on the planet haven't thought through this. Yes, Docker images make your life more convenient, at the deploy stage, with the downside that you need to do a LOT more thinking (automation and testing) at build phase. Like shipping containers, we pack them at the factory, but can move them around at will after that. Instead of having dock hands load the ship at the port (which took 45 days in 1906). The supply chain problems are the same either way, but at least Docker gives you a good format to solve them. [1]: https://docs.openshift.com/enterprise/3.0/architecture/core_concepts/builds_and_image_streams.html https://docs.openshift.com/enterprise/3.0/architecture/core_...

anentropic 11y ago

Does anyone have any advice on using "init" scripts as CMD for the container, vs something like Ansible for post-provisioning? Example: a Django web app... after deploying the new version container you need to run migrations etc Similarly for say RabbitMQ you may need to ensure your vhosts are defined etc... commands that need to be executed against a running server, can't be done in the Dockerfile itself. So far I've ended up with a bash script in each container that does the necessary init tasks before exec-ing the main process. Or in some cases (Rabbit, Mongo) they first run the server in a limited state, do the init tasks against it, then stop it and exec again as the container process. Have started to feel it might be cleaner to instead just start the main process and then provision it with Ansible afterwards. But then I lose the ability to have a system up and running just via docker-compose...

goldbrick 11y ago
You should think of all these operations in terms of the state they persist. For example, a database migration doesn't modify the application at all, but only the data directory wherever your database is running (whether mounted in another container or across a network). Whatever process is orchestrating your containers should take responsibility for running setup chores in a separate, ephemeral container from the eventual application container. docker-compose is often not enough, unfortunately.

fpoling 11y ago

I do no think "Don’t create large images" is a good one. I prefer to have a single image that I use to run multiple containers with different services that forms a pod. This greatly simplifies distribution as I need just single image file. It also greatly simplifies updates. But with such image it is useful to add few tools so docker exec -ti container /bin/bash creates a shell that one can use for debugging or verification.

grantlmiller 11y ago

we open-sourced www.fromlatest.io as a way to share docker best practices in a really easy to use way