27 ms·

Simple Dockerfile examples are often broken by default

461 points by itamarst 7y ago

jo-wol 7y ago
The final example in the article is broken. Python interpreter as PID 1 can't handle linux signals.
- itamarst 7y ago
  This is why I have a caveat at the top of the article as well as right after the last example. This particular issue is fixable with `docker run --init`, so not strictly necessary to fix in images.
- deathanatos 7y ago
  It can if you install the appropriate signal handlers, which, if you don't fork, is quite easy. While for a real-world app you're probably right, the article's example is not fleshed out sufficiently to tell one way or another.
eyeareque 7y ago
Not locking it to a specific version is better for security updates. Do you want it to run stable with vulnerabilities or to run secure and broken?
- bdcravens 7y ago
  > Not locking it to a specific version is better for security updates. The idea is that you should take responsibility for your containers and verify fixes and test your application. > Do you want it to run stable with vulnerabilities or to run secure and broken? If these are your two choices, you have a staffing or a workflow problem.
Nullabillity 7y ago
Good points, but it's amusing that his solution to #1 didn't lock down the patch version, nor the distro around it. I think that also makes a decent point for Nix[0], which solves #1-#3 by default (since choosing a particular version of Nixpkgs locks down the whole environment, and considers the build as a DAG of dependencies rather than a linear history). It also supports exporting Docker images, while preserving Nix's richer build caching.[1] [0]: https://nixos.org/nix/ https://nixos.org/nix/ [1]: https://grahamc.com/blog/nix-and-layered-docker-images https://grahamc.com/blog/nix-and-layered-docker-images
- itamarst 7y ago
  Good point, will go fix that. My soon-to-be-ready attempt at a production-ready template (https://pythonspeed.com/products/pythoncontainer/ https://pythonspeed.com/products/pythoncontainer/) covers the tradeoff between point releases vs. not-point-releases, and it does pin the OS. And yes, Nix fixes some of the problems of building a production-ready image, but only a subset.
  - j88439h84 7y ago
    Could you elaborate on the remaining problems with Nix for building Python images?
    - itamarst 7y ago
      Not an expert on Nix, but it's not so much that Nix has problems (though I'm sure it does, my initial research suggested it's not quite there yet for Python packages) but that there other things you need to get right. For example: 1. Signal handling (only one bit of https://hynek.me/articles/docker-signals/ https://hynek.me/articles/docker-signals/ is Dockerfile specific, the rest still applies.) 2. Configuring servers to run correctly in Docker environments (e.g. Gunicorn is broken by default, and some of these issues go beyond Gunicorn: https://pythonspeed.com/articles/gunicorn-in-docker/ https://pythonspeed.com/articles/gunicorn-in-docker/). 3. Not running as root, and dropping capabilities. 4. Building pinned dependencies for Python that you can feed to Nix. 5. Having processes (human and automated) in place to ensure security updates happen. 6. Knowing how to write shell scripts that aren't completely broken (either by not writing them at all and using better language, or by using bash strict mode: http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/) etc.
      - tathougies 7y ago
        > though I'm sure it does, my initial research suggested it's not quite there yet for Python packages) Can you expand on what's missing? I've successfully used nix to cross-compile a pretty substantial python application (+ native extensions, hence the cross compilation), for embedded purposes, and it pretty much worked out of the box. Adding extra dependencies was straightforwards. I think you can use pypi2nix for pinned dependencies, and you can run it periodically for security updates.
jand 7y ago
I do not intend to play down the importance of using docker carefully. But the reproducible build aspect of the critic seems unnecessary to me: Isn't that more a concern of the packaging system? (no python scripter) If your packaging systems supports version selection/locking, then use your packaging system right. If your packaging system cannot pin a version, how should docker solve this?
- itamarst 7y ago
  This issue isn't Docker, Docker itself obviously can't deal with this. The issue is the examples and tutorial people provide on Docker packaging often don't talk about the requirements (pun intended) for reproducible builds at all. Or they don't talk about need to run as not-root. Or they suggest base images that are often broken in subtle ways (Alpine Linux). Or they talk about multi-stage builds for small images, and neglect to explain that you've just destroyed your caching (this is fixable, but you need to know to expect it and how to fix it.) Etc.
  - batbomb 7y ago
    The caching part is (mostly) generally fixable by making sure to always add your source code in two stages - dependency files first, then resolve dependencies, and the rest of the source code later. Rarely is this done though. And definitely alpine+musl doesn't always do what you might expect, and it's often language dependent as to whether or not you'll encounter something strange (not to mention you forfeiting bash)
- zokier 7y ago
  Docker is a packaging system.
  - jand 7y ago
    Thank you for clarifying this. Did you understand the point i tried to make nonetheless or do i need to detail it?
- j88439h84 7y ago
btilly 7y ago
I have a mixed opinion about his first point. There are two basic approaches to take with dependency management. The first version is to lock down every dependency as tightly as you can to avoid accidentally breaking something. Which inevitably leads down the road to everything being locked to something archaic that can't be upgraded easily, and is incompatible with everything else. But with no idea what will break, or how to upgrade. I currently work at a company that went down that path and is now suffering for it. The second version is upgrade early, upgrade often. This will occasionally lead to problems, but they tend to be temporary and easily fixed. And in the long run, your system will age better. Google is an excellent example of a company that does this. The post assumes that the first version should be your model. But having seen both up close and personal, my sympathies actually lie with the second. This is not to say that I'm against reproducible builds. I'm not. But if you want to lock down version numbers for a specific release, have an automated tool supply the right ones for you. And make it trivial to upgrade early, and upgrade often.
- pdonis 7y ago
  > The post assumes that the first version should be your model. No, it doesn't. It just assumes that you want explicit control over when you upgrade. You can always change your Dockerfile or your requirements.txt and build again when you've tested your software against a new Python version or a new version of a package. You can do that as often as you like, so this is perfectly consistent with "upgrade early, upgrade often". But not specifying exact versions in those files means they can get upgraded automatically when you haven't tested your software with them, which can break something.
  - yongjik 7y ago
    From what I've seen, explicit version control does not really work unless there's an organizational force toward timely upgrade. In every company, everyone's busy, nobody has time to look at a service that's been running fine for six months (and risk getting paged "Why did you change it? It suddenly stopped working for us!"). The path of the least resistance is to not upgrade anything that's running "fine", and then old versions and their dependencies accumulate, and when you actually have to upgrade it becomes much more painful. It might work if there's a dedicated team whose mission is upgrade dependencies for everyone in time, but I haven't seen one in action so I'm not sure how well it might work out. (Well, unless you count Google as one such example. But Google does Google things.)
treis 7y ago
(1) and (2) aren't really broken, IMHO. For most cases always using the most up to date version is better than having 100% reproducible builds. After all, you have the docker image that you can distribute if you really need to. Better to pick up security and performance patches as they become available. If those updates break something then you can make the decision to fix on a known good version.
- erik_seaberg 7y ago
  If you always pin, you have history to tell you which versions were good. If you mostly don't, you have to start disassembling a bunch of old images just to figure out what they were built from.
bytematic 7y ago
Probably going to want to use tagged docker repos so that updating certain packages, no matter the language, don't suddenly break your images
kkapelon 7y ago
I am actually preparing my own article titled "Docker antipatterns" that will include many more points like this.
42n4 7y ago
Very good clues!
j88439h84 7y ago
For reproducible builds, `python:3.7` isn't specific enough. python:3.7.3-alpine3.9 is more specific, for example. There aren't supposed to be breaking changes in the bugfix releases, but they'll happen anyway.
- kam 7y ago
  And `python:3.7@sha256:35ff9f44818f8850f1d318aa69c2e7ba61d85e3b93283078c10e56e7d864c183` is even better.
- mistrial9 7y ago
  overfitting here?
- meowface 7y ago
  Ran into this recently. Docker container was running into issues until I changed "python:3.7-alpine" to "python:3.7.3-alpine3.9". It was because a package I was relying on from "apk add" changed between Alpine versions. I could probably safely make it "python:3.7-alpine3.9" (instead of pinning to Python 3.7.3), since the issue was the Alpine version, but at this point I'm starting to really buy into the whole reproducible build thing.
jessemillar 7y ago
> A broken Docker image can lead to production outages, and building best-practices images is a lot harder than it seems. So don’t just copy the first example you find on the web: do your research, and spend some time reading about best practices. While I may not agree with absolutely everything in the article, this final point is paramount. Please don't blindly use technology because you managed to find a copypasta config that runs. Running != good.
- deleted 7y ago
  [deleted]
- zrobotics 7y ago
  Definitely very true. I write more C++ than anything else, and the sheer number of online examples that start with using namespace std; is just staggering. Sure, it works in a toy example posted to stackoverflow, but it will cause problems in larger projects. I think globally there needs to be better emphasis on using best-practices in tutorials and examples; I remember this particular pet-peeve of mine also being present in college textbooks. Especially for content aimed at newbies, it should be frowned upon to show the wrong way to do things, since then it gets harder to show how to do it the right way. I've had people who were surprised to find out that they could type: using std::chrono::duration; using std::cout; instead of pulling in the entire std namespace; simply because they'd only ever seen examples that did it the lazy way. edit: lack of semicolons strikes again!
  - ajross 7y ago
    That's... true in the specific example, becuase C++'s standard library is a huge, promiscuous mess of obvious-looking symbol names that are just asking for a collision with a user name. But in general the notion that we want to isolate "everything" into a namespace is a net loss. Clear and simple abstractions have real value, and short undeclared names are an important part of being clear and simple. The modern convention of separately importing every symbol you use gets really out of hand, when most of the time it really is appropriate that you just declare "my code is using this API" and expect things to work without having to link your program by hand with a giant shipping manifest of symbols at the top of your source files.
batbomb 7y ago
In general, in Go, Java, and Python I've resorted to copying in the Gopkg files, pom.xml, and requirements.txt, and then running the requisite dependency installer for the language (dep, pip, mvn, etc...) and then just copying in the rest of the repo, relying on the .dockeringore with a default-ignore for everything and specifying the individual files/directories you may want to add, and in some cases a rootfs folder when necessary. This seems to be the happy medium for me. I don't have very strong opinions on requirements.txt always being the pinned output from a pip freeze, and it seems like pipenv may actually die in a few years, and poetry will evolve to take the mantle, but I do lots of things with conda anyway.
linuxftw 7y ago
The problems described here are called 'release engineering.' Dockerfiles don't solve release engineering, they provide an abstraction for building a release candidate, putting it through a pipeline, and then tagging a successful build as your release. In other words, the end-container is the immutable object that should be deployed, not the Dockerfile. If you are building the container in each stage of your CI/CD pipeline, you are doing it wrong.
mychael 7y ago
This is an advertisement disguised as a technical post on container security.
- bdcravens 7y ago
  It's a technical post on a company's blog. (which contains links to the company's products, but I didn't feel advertised to) Like thousands that have been posted on HN over the years.
- nirvdrum 7y ago
  Then I wish all advertising were like this. It's very informative and provides a solution rather than just pointing out the problem. I hope this page ends up ranking highly in search results because there are a lot of incomplete Dockerfiles employing questionable practices that sit at the top of search results and proliferate due to cargo culting.
- obalado 7y ago
  its an ad: https://pythonspeed.com/products/pythoncontainer/ https://pythonspeed.com/products/pythoncontainer/ Production-Ready Python Containers is getting closer to release, but it isn’t quite ready. So sign up below to get notified when the template is available for purchase.
neves 7y ago
Isn't it ironic that he isn't pining down the docker version?
mistrial9 7y ago
not everyone is on a upgrade-daily churn, and should not have to be ! if you are externally exposed, sure, because security .. but really, isn't there some room here for different life cycles ?
tuco86 7y ago
I have spent a ridiculous time building this so I'll take the opportunity and share. It builds python wheel packages in a build container and installs them in an app container. Works great for cpython and pypy. Also allows to build for alpine and works for most other languages. We started to build basically everything that way. https://gist.github.com/tuco86/67d84dfb27268b1faf05d2dbb1acb667 https://gist.github.com/tuco86/67d84dfb27268b1faf05d2dbb1acb... Ok, I kind of cheated and added the user just now. Sue me. Also posted this in the other Docker related news. Sue me again.
- Perceptes 7y ago
  Looks like the last line needs to be updated to have the server listen on 8080 instead of 80. (I'm guessing this is left over from before you added the non-root user.)
- rzimmerman 7y ago
  Building the wheels in a build container and copying in the wheelhouse is a great idea. Thanks for posting this.
diminoten 7y ago
"Broken" means "does not work". These examples do work. I'm annoyed by this incongruity. "Not sustainable"/"Not Forward Compatible", etc. would have been preferable.
ru999gol 7y ago
why is running as root in the docker a problem? Isn't the whole point of containers to isolate the container? So what is the difference in a container running root or a user? If there is, wouldn't that be more of a docker bug?
- CameronNemo 7y ago
  If you are in a UID namespace, it likely is not a big deal. But if you were ever to have something escape from a container, it would be a much bigger problem if it was uid 0 in the root namespace.
  - IshKebab 7y ago
    So in other words it isn't really an issue.
    - aflag 7y ago
      If the server is comprimised and paths from the host are mounted inside the container, the attacker could potentially make more damage.
- miduil 7y ago
  The page the article is linking to http://canihaznonprivilegedcontainers.info http://canihaznonprivilegedcontainers.info is mixing up running as root with running docker with --privileged. Latter one renders Docker security to zero, but is barely required. I’m not saying “go run your all your Docker images as root”, but this is clearly FUD. Non-privileged containers are still having "root", just with way fewer capabilities (See Docker [0] docs). I’m not an expert, but I guess depending what you are doing the most problematic capability might be AUDIT_WRITE, because it is not namespaced and could be abused for DOSing syslog. But you might require it for things like sshd, sudo, adduser, passwd, … Depending on how you are holding it the NET_BIND_SERVICE and NET_RAW can be an issue (depends on how your docker network looks like), but the others appear not to be a security issue per-se. This page [1] gives a good overview on default capabilities, though they are also confusing to the reader with "better disable this". I've created an issue, not sure if I have resources to fix their page though. [2] [0]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities https://docs.docker.com/engine/reference/run/#runtime-privil... [1]: https://www.redhat.com/en/blog/secure-your-containers-one-weird-trick https://www.redhat.com/en/blog/secure-your-containers-one-we... [2]: https://github.com/mhausenblas/canihaznonprivilegedcontainers.info/issues/8 https://github.com/mhausenblas/canihaznonprivilegedcontainer...
geggam 7y ago
It seems like the folks making docker files could stand to learn package management with a mature package system before making docker files. I wonder how many use docker after they learn ?
adrianmonk 7y ago
I feel like programmers often fail to grasp the distinction between example code and production-ready code. On one project, we made some example code available, and people would copy and paste it into their project, change a few lines, and launch it into production. Then they were surprised it didn't handle this or that situation or deal with this or that detail. Yeah, no shit it doesn't handle those things, it's example code! You're supposed to read this code along with the documentation so you can get a gist of what the API is like. It's a learning aid, not a software deliverable. Your real code is going to be more complicated. This simplified code exists to get you past the "how the hell does all this fit together at a high level?" hurdle faster. Once you're over that hurdle, you can start on the real implementation.
- thrower123 7y ago
  Hopefully there is documentation; often there is not. It's borderline criminal that nearly all examples omit error-handling code, at least for the broad-strokes classes of errors that you, as the library author, know can and will occur. I hated Java's checked exceptions, but it did let you have some idea what kind of errors were expected to be possible. Most of the time I'm excavating deep layers of source code, monitoring and logging to see what the errors might be, and tossing in catch-all handlers to keep the stack from unwinding too far if at all in doubt.
  - adrianmonk 7y ago
    Yeah, I agree with that. That is usually hard-won knowledge, too, so if it doesn't get transferred, that creates a lot of pain. Unfortunately, a lot of people hate writing documentation and just want to get it over with as quickly as possible. (Ability to create working software doesn't necessarily correlate with ability or desire to share knowledge.) Also, if you look at it in terms of incentives, it's tricky to reward people for making sure the documentation of gotchas is reasonably complete. By definition, these are things that people would probably not see until they're pointed out. So it's hard to tell if someone has really done it. (And when people are busy, things that aren't rewarded are unlikely to happen.)
- dkarl 7y ago
vorticalbox 7y ago
I would move the requirements.txt and pip install to after the user creation seeing as you'll invalidate that cache if your requirements change. Best part is that was brought up as an issue in the article only to do the same thing in an example
est 7y ago
One more broken part is CMD [ "python", "./yourscript.py" ] This breaks if you want to debug yourscript.py on startup. Better use a sh to wrap it.
- deathanatos 7y ago
  Why/what do you think this breaks / what does wrapping it in a shell do for you? E.g., for me, the following Dockerfile: FROM python:3 RUN pip3 install ipdb COPY test.py /test.py CMD ["python", "test.py"] where test.py is: import ipdb; ipdb.set_trace() print('Hello, World.') run as `docker run -ti --rm $IMAGE_ID` works as expected: » docker run -ti --rm 52e98c118dc3 > /test.py(2)<module>() 1 import ipdb; ipdb.set_trace() ----> 2 print('Hello, World.') ipdb> p globals() {'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <_frozen_importlib_external.SourceFileLoader object at 0x7f809a8c8278>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, '__file__': 'test.py', '__cached__': None, 'ipdb': <module 'ipdb' from '/usr/local/lib/python3.7/site-packages/ipdb/__init__.py'>} ipdb> ^D Exiting Debugger. »
  - est 7y ago
    Can you add extra environ before python executes any code?
    - Faaak 7y ago
      Did you even try ? `docker exec -e foo=bar -it ....`
quickthrower2 7y ago
Nice eBomb. Describes the problem, offers value, does it politely so it is HN (and other places) friendly, then "there are still problems..." and then the paid solution. I've seen docker images that do a git clone from the master head to get the source, so basically if their Github account gets hacked. You're f'd.
myroon5 7y ago
This Dockerfile linter would warn you about multiple of these problems and more: https://github.com/hadolint/hadolint https://github.com/hadolint/hadolint
hayd 7y ago
Running pip with sudo doesn't seem a great idea either...
zingmars 7y ago
I feel like the better solution to #4 is setting up UID namespacing for docker instead of (just) creating random users within the container. Even if you create a user, it's still going to run as whatever UID it has within the container (probably 1000 is most likely your UID if you're the only one using said system)
DoctorPenguin 7y ago
Isn't the example on how to make the referenced file better another contribution to the pool of "broken by default" images? Either that or I don't get the argument.

461 points by itamarst 7y ago

jo-wol 7y ago

The final example in the article is broken. Python interpreter as PID 1 can't handle linux signals.

itamarst 7y ago
This is why I have a caveat at the top of the article as well as right after the last example. This particular issue is fixable with `docker run --init`, so not strictly necessary to fix in images.
deathanatos 7y ago
It can if you install the appropriate signal handlers, which, if you don't fork, is quite easy. While for a real-world app you're probably right, the article's example is not fleshed out sufficiently to tell one way or another.

eyeareque 7y ago

Not locking it to a specific version is better for security updates. Do you want it to run stable with vulnerabilities or to run secure and broken?

bdcravens 7y ago
> Not locking it to a specific version is better for security updates. The idea is that you should take responsibility for your containers and verify fixes and test your application. > Do you want it to run stable with vulnerabilities or to run secure and broken? If these are your two choices, you have a staffing or a workflow problem.

Nullabillity 7y ago

Good points, but it's amusing that his solution to #1 didn't lock down the patch version, nor the distro around it. I think that also makes a decent point for Nix[0], which solves #1-#3 by default (since choosing a particular version of Nixpkgs locks down the whole environment, and considers the build as a DAG of dependencies rather than a linear history). It also supports exporting Docker images, while preserving Nix's richer build caching.[1] [0]: https://nixos.org/nix/ https://nixos.org/nix/ [1]: https://grahamc.com/blog/nix-and-layered-docker-images https://grahamc.com/blog/nix-and-layered-docker-images

itamarst 7y ago
Good point, will go fix that. My soon-to-be-ready attempt at a production-ready template (https://pythonspeed.com/products/pythoncontainer/ https://pythonspeed.com/products/pythoncontainer/) covers the tradeoff between point releases vs. not-point-releases, and it does pin the OS. And yes, Nix fixes some of the problems of building a production-ready image, but only a subset.
- j88439h84 7y ago
  Could you elaborate on the remaining problems with Nix for building Python images?
  - itamarst 7y ago
    Not an expert on Nix, but it's not so much that Nix has problems (though I'm sure it does, my initial research suggested it's not quite there yet for Python packages) but that there other things you need to get right. For example: 1. Signal handling (only one bit of https://hynek.me/articles/docker-signals/ https://hynek.me/articles/docker-signals/ is Dockerfile specific, the rest still applies.) 2. Configuring servers to run correctly in Docker environments (e.g. Gunicorn is broken by default, and some of these issues go beyond Gunicorn: https://pythonspeed.com/articles/gunicorn-in-docker/ https://pythonspeed.com/articles/gunicorn-in-docker/). 3. Not running as root, and dropping capabilities. 4. Building pinned dependencies for Python that you can feed to Nix. 5. Having processes (human and automated) in place to ensure security updates happen. 6. Knowing how to write shell scripts that aren't completely broken (either by not writing them at all and using better language, or by using bash strict mode: http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/) etc.
    - tathougies 7y ago
      > though I'm sure it does, my initial research suggested it's not quite there yet for Python packages) Can you expand on what's missing? I've successfully used nix to cross-compile a pretty substantial python application (+ native extensions, hence the cross compilation), for embedded purposes, and it pretty much worked out of the box. Adding extra dependencies was straightforwards. I think you can use pypi2nix for pinned dependencies, and you can run it periodically for security updates.

jand 7y ago

I do not intend to play down the importance of using docker carefully. But the reproducible build aspect of the critic seems unnecessary to me: Isn't that more a concern of the packaging system? (no python scripter) If your packaging systems supports version selection/locking, then use your packaging system right. If your packaging system cannot pin a version, how should docker solve this?

itamarst 7y ago
This issue isn't Docker, Docker itself obviously can't deal with this. The issue is the examples and tutorial people provide on Docker packaging often don't talk about the requirements (pun intended) for reproducible builds at all. Or they don't talk about need to run as not-root. Or they suggest base images that are often broken in subtle ways (Alpine Linux). Or they talk about multi-stage builds for small images, and neglect to explain that you've just destroyed your caching (this is fixable, but you need to know to expect it and how to fix it.) Etc.
- batbomb 7y ago
  The caching part is (mostly) generally fixable by making sure to always add your source code in two stages - dependency files first, then resolve dependencies, and the rest of the source code later. Rarely is this done though. And definitely alpine+musl doesn't always do what you might expect, and it's often language dependent as to whether or not you'll encounter something strange (not to mention you forfeiting bash)
zokier 7y ago
Docker is a packaging system.
- jand 7y ago
  Thank you for clarifying this. Did you understand the point i tried to make nonetheless or do i need to detail it?
j88439h84 7y ago

btilly 7y ago

I have a mixed opinion about his first point. There are two basic approaches to take with dependency management. The first version is to lock down every dependency as tightly as you can to avoid accidentally breaking something. Which inevitably leads down the road to everything being locked to something archaic that can't be upgraded easily, and is incompatible with everything else. But with no idea what will break, or how to upgrade. I currently work at a company that went down that path and is now suffering for it. The second version is upgrade early, upgrade often. This will occasionally lead to problems, but they tend to be temporary and easily fixed. And in the long run, your system will age better. Google is an excellent example of a company that does this. The post assumes that the first version should be your model. But having seen both up close and personal, my sympathies actually lie with the second. This is not to say that I'm against reproducible builds. I'm not. But if you want to lock down version numbers for a specific release, have an automated tool supply the right ones for you. And make it trivial to upgrade early, and upgrade often.

pdonis 7y ago
> The post assumes that the first version should be your model. No, it doesn't. It just assumes that you want explicit control over when you upgrade. You can always change your Dockerfile or your requirements.txt and build again when you've tested your software against a new Python version or a new version of a package. You can do that as often as you like, so this is perfectly consistent with "upgrade early, upgrade often". But not specifying exact versions in those files means they can get upgraded automatically when you haven't tested your software with them, which can break something.
- yongjik 7y ago
  From what I've seen, explicit version control does not really work unless there's an organizational force toward timely upgrade. In every company, everyone's busy, nobody has time to look at a service that's been running fine for six months (and risk getting paged "Why did you change it? It suddenly stopped working for us!"). The path of the least resistance is to not upgrade anything that's running "fine", and then old versions and their dependencies accumulate, and when you actually have to upgrade it becomes much more painful. It might work if there's a dedicated team whose mission is upgrade dependencies for everyone in time, but I haven't seen one in action so I'm not sure how well it might work out. (Well, unless you count Google as one such example. But Google does Google things.)

treis 7y ago

(1) and (2) aren't really broken, IMHO. For most cases always using the most up to date version is better than having 100% reproducible builds. After all, you have the docker image that you can distribute if you really need to. Better to pick up security and performance patches as they become available. If those updates break something then you can make the decision to fix on a known good version.

erik_seaberg 7y ago
If you always pin, you have history to tell you which versions were good. If you mostly don't, you have to start disassembling a bunch of old images just to figure out what they were built from.

bytematic 7y ago

Probably going to want to use tagged docker repos so that updating certain packages, no matter the language, don't suddenly break your images

kkapelon 7y ago

I am actually preparing my own article titled "Docker antipatterns" that will include many more points like this.

42n4 7y ago

Very good clues!

j88439h84 7y ago

For reproducible builds, `python:3.7` isn't specific enough. python:3.7.3-alpine3.9 is more specific, for example. There aren't supposed to be breaking changes in the bugfix releases, but they'll happen anyway.

kam 7y ago
And `python:3.7@sha256:35ff9f44818f8850f1d318aa69c2e7ba61d85e3b93283078c10e56e7d864c183` is even better.
mistrial9 7y ago
overfitting here?
meowface 7y ago
Ran into this recently. Docker container was running into issues until I changed "python:3.7-alpine" to "python:3.7.3-alpine3.9". It was because a package I was relying on from "apk add" changed between Alpine versions. I could probably safely make it "python:3.7-alpine3.9" (instead of pinning to Python 3.7.3), since the issue was the Alpine version, but at this point I'm starting to really buy into the whole reproducible build thing.

jessemillar 7y ago

> A broken Docker image can lead to production outages, and building best-practices images is a lot harder than it seems. So don’t just copy the first example you find on the web: do your research, and spend some time reading about best practices. While I may not agree with absolutely everything in the article, this final point is paramount. Please don't blindly use technology because you managed to find a copypasta config that runs. Running != good.

deleted 7y ago
[deleted]
zrobotics 7y ago
Definitely very true. I write more C++ than anything else, and the sheer number of online examples that start with using namespace std; is just staggering. Sure, it works in a toy example posted to stackoverflow, but it will cause problems in larger projects. I think globally there needs to be better emphasis on using best-practices in tutorials and examples; I remember this particular pet-peeve of mine also being present in college textbooks. Especially for content aimed at newbies, it should be frowned upon to show the wrong way to do things, since then it gets harder to show how to do it the right way. I've had people who were surprised to find out that they could type: using std::chrono::duration; using std::cout; instead of pulling in the entire std namespace; simply because they'd only ever seen examples that did it the lazy way. edit: lack of semicolons strikes again!
- ajross 7y ago
  That's... true in the specific example, becuase C++'s standard library is a huge, promiscuous mess of obvious-looking symbol names that are just asking for a collision with a user name. But in general the notion that we want to isolate "everything" into a namespace is a net loss. Clear and simple abstractions have real value, and short undeclared names are an important part of being clear and simple. The modern convention of separately importing every symbol you use gets really out of hand, when most of the time it really is appropriate that you just declare "my code is using this API" and expect things to work without having to link your program by hand with a giant shipping manifest of symbols at the top of your source files.

batbomb 7y ago

In general, in Go, Java, and Python I've resorted to copying in the Gopkg files, pom.xml, and requirements.txt, and then running the requisite dependency installer for the language (dep, pip, mvn, etc...) and then just copying in the rest of the repo, relying on the .dockeringore with a default-ignore for everything and specifying the individual files/directories you may want to add, and in some cases a rootfs folder when necessary. This seems to be the happy medium for me. I don't have very strong opinions on requirements.txt always being the pinned output from a pip freeze, and it seems like pipenv may actually die in a few years, and poetry will evolve to take the mantle, but I do lots of things with conda anyway.

linuxftw 7y ago

The problems described here are called 'release engineering.' Dockerfiles don't solve release engineering, they provide an abstraction for building a release candidate, putting it through a pipeline, and then tagging a successful build as your release. In other words, the end-container is the immutable object that should be deployed, not the Dockerfile. If you are building the container in each stage of your CI/CD pipeline, you are doing it wrong.

mychael 7y ago

This is an advertisement disguised as a technical post on container security.

bdcravens 7y ago
It's a technical post on a company's blog. (which contains links to the company's products, but I didn't feel advertised to) Like thousands that have been posted on HN over the years.
nirvdrum 7y ago
Then I wish all advertising were like this. It's very informative and provides a solution rather than just pointing out the problem. I hope this page ends up ranking highly in search results because there are a lot of incomplete Dockerfiles employing questionable practices that sit at the top of search results and proliferate due to cargo culting.
obalado 7y ago
its an ad: https://pythonspeed.com/products/pythoncontainer/ https://pythonspeed.com/products/pythoncontainer/ Production-Ready Python Containers is getting closer to release, but it isn’t quite ready. So sign up below to get notified when the template is available for purchase.

neves 7y ago

Isn't it ironic that he isn't pining down the docker version?

mistrial9 7y ago

not everyone is on a upgrade-daily churn, and should not have to be ! if you are externally exposed, sure, because security .. but really, isn't there some room here for different life cycles ?

tuco86 7y ago

I have spent a ridiculous time building this so I'll take the opportunity and share. It builds python wheel packages in a build container and installs them in an app container. Works great for cpython and pypy. Also allows to build for alpine and works for most other languages. We started to build basically everything that way. https://gist.github.com/tuco86/67d84dfb27268b1faf05d2dbb1acb667 https://gist.github.com/tuco86/67d84dfb27268b1faf05d2dbb1acb... Ok, I kind of cheated and added the user just now. Sue me. Also posted this in the other Docker related news. Sue me again.

Perceptes 7y ago
Looks like the last line needs to be updated to have the server listen on 8080 instead of 80. (I'm guessing this is left over from before you added the non-root user.)
rzimmerman 7y ago
Building the wheels in a build container and copying in the wheelhouse is a great idea. Thanks for posting this.

diminoten 7y ago

"Broken" means "does not work". These examples do work. I'm annoyed by this incongruity. "Not sustainable"/"Not Forward Compatible", etc. would have been preferable.

ru999gol 7y ago

why is running as root in the docker a problem? Isn't the whole point of containers to isolate the container? So what is the difference in a container running root or a user? If there is, wouldn't that be more of a docker bug?

CameronNemo 7y ago
If you are in a UID namespace, it likely is not a big deal. But if you were ever to have something escape from a container, it would be a much bigger problem if it was uid 0 in the root namespace.
- IshKebab 7y ago
  So in other words it isn't really an issue.
  - aflag 7y ago
    If the server is comprimised and paths from the host are mounted inside the container, the attacker could potentially make more damage.
miduil 7y ago
The page the article is linking to http://canihaznonprivilegedcontainers.info http://canihaznonprivilegedcontainers.info is mixing up running as root with running docker with --privileged. Latter one renders Docker security to zero, but is barely required. I’m not saying “go run your all your Docker images as root”, but this is clearly FUD. Non-privileged containers are still having "root", just with way fewer capabilities (See Docker [0] docs). I’m not an expert, but I guess depending what you are doing the most problematic capability might be AUDIT_WRITE, because it is not namespaced and could be abused for DOSing syslog. But you might require it for things like sshd, sudo, adduser, passwd, … Depending on how you are holding it the NET_BIND_SERVICE and NET_RAW can be an issue (depends on how your docker network looks like), but the others appear not to be a security issue per-se. This page [1] gives a good overview on default capabilities, though they are also confusing to the reader with "better disable this". I've created an issue, not sure if I have resources to fix their page though. [2] [0]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities https://docs.docker.com/engine/reference/run/#runtime-privil... [1]: https://www.redhat.com/en/blog/secure-your-containers-one-weird-trick https://www.redhat.com/en/blog/secure-your-containers-one-we... [2]: https://github.com/mhausenblas/canihaznonprivilegedcontainers.info/issues/8 https://github.com/mhausenblas/canihaznonprivilegedcontainer...

geggam 7y ago

It seems like the folks making docker files could stand to learn package management with a mature package system before making docker files. I wonder how many use docker after they learn ?

adrianmonk 7y ago

I feel like programmers often fail to grasp the distinction between example code and production-ready code. On one project, we made some example code available, and people would copy and paste it into their project, change a few lines, and launch it into production. Then they were surprised it didn't handle this or that situation or deal with this or that detail. Yeah, no shit it doesn't handle those things, it's example code! You're supposed to read this code along with the documentation so you can get a gist of what the API is like. It's a learning aid, not a software deliverable. Your real code is going to be more complicated. This simplified code exists to get you past the "how the hell does all this fit together at a high level?" hurdle faster. Once you're over that hurdle, you can start on the real implementation.

thrower123 7y ago
Hopefully there is documentation; often there is not. It's borderline criminal that nearly all examples omit error-handling code, at least for the broad-strokes classes of errors that you, as the library author, know can and will occur. I hated Java's checked exceptions, but it did let you have some idea what kind of errors were expected to be possible. Most of the time I'm excavating deep layers of source code, monitoring and logging to see what the errors might be, and tossing in catch-all handlers to keep the stack from unwinding too far if at all in doubt.
- adrianmonk 7y ago
  Yeah, I agree with that. That is usually hard-won knowledge, too, so if it doesn't get transferred, that creates a lot of pain. Unfortunately, a lot of people hate writing documentation and just want to get it over with as quickly as possible. (Ability to create working software doesn't necessarily correlate with ability or desire to share knowledge.) Also, if you look at it in terms of incentives, it's tricky to reward people for making sure the documentation of gotchas is reasonably complete. By definition, these are things that people would probably not see until they're pointed out. So it's hard to tell if someone has really done it. (And when people are busy, things that aren't rewarded are unlikely to happen.)
dkarl 7y ago

vorticalbox 7y ago

I would move the requirements.txt and pip install to after the user creation seeing as you'll invalidate that cache if your requirements change. Best part is that was brought up as an issue in the article only to do the same thing in an example

est 7y ago

One more broken part is CMD [ "python", "./yourscript.py" ] This breaks if you want to debug yourscript.py on startup. Better use a sh to wrap it.

deathanatos 7y ago
Why/what do you think this breaks / what does wrapping it in a shell do for you? E.g., for me, the following Dockerfile: FROM python:3 RUN pip3 install ipdb COPY test.py /test.py CMD ["python", "test.py"] where test.py is: import ipdb; ipdb.set_trace() print('Hello, World.') run as `docker run -ti --rm $IMAGE_ID` works as expected: » docker run -ti --rm 52e98c118dc3 > /test.py(2)<module>() 1 import ipdb; ipdb.set_trace() ----> 2 print('Hello, World.') ipdb> p globals() {'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <_frozen_importlib_external.SourceFileLoader object at 0x7f809a8c8278>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, '__file__': 'test.py', '__cached__': None, 'ipdb': <module 'ipdb' from '/usr/local/lib/python3.7/site-packages/ipdb/__init__.py'>} ipdb> ^D Exiting Debugger. »
- est 7y ago
  Can you add extra environ before python executes any code?
  - Faaak 7y ago
    Did you even try ? `docker exec -e foo=bar -it ....`

quickthrower2 7y ago

Nice eBomb. Describes the problem, offers value, does it politely so it is HN (and other places) friendly, then "there are still problems..." and then the paid solution. I've seen docker images that do a git clone from the master head to get the source, so basically if their Github account gets hacked. You're f'd.

myroon5 7y ago

This Dockerfile linter would warn you about multiple of these problems and more: https://github.com/hadolint/hadolint https://github.com/hadolint/hadolint

hayd 7y ago

Running pip with sudo doesn't seem a great idea either...

zingmars 7y ago

I feel like the better solution to #4 is setting up UID namespacing for docker instead of (just) creating random users within the container. Even if you create a user, it's still going to run as whatever UID it has within the container (probably 1000 is most likely your UID if you're the only one using said system)

DoctorPenguin 7y ago

Isn't the example on how to make the referenced file better another contribution to the pool of "broken by default" images? Either that or I don't get the argument.