7 ms·
Docker Image Insecurity
- snoble 12y agoIt confuses me why they wouldn't just verify the images since they have the signature in the manifest. Is this because they don't want to wait for a complete image before the start streaming through the pipeline? Is this actually a significant time saver?
- Khao 12y agoIf this is the case, it would seem pretty insulting to me as a developer and user of Docker that "time-saving" is more important that security and validating images. I'd rather use a slower tool that is secure.
- thu 12y agoI think this is because they regard the tar'd layers as a transport mechanism, not as the signed payload itself.
- justincormack 12y agoYes that makes sense, as tar is not fully deterministic, so untar and retar might give a different checksum on the same files (eg ordering). However it is generally better to keep the same bits people signed regardless.
- thu 12y agoMaybe you could use the Git packfile format; this is a self-contained compressed Merkle-tree. If you ever need to reconstruct deterministically the tar from that, you can use something like pristine-tar[0]. [0]: https://joeyh.name/code/pristine-tar/ https://joeyh.name/code/pristine-tar/
- thaJeztah 12y agoSounds interesting. Perhaps you should create a proposal for that on the docker issue tracker, so that it can be discussed as a possible alternative?
- rcoder 12y agoThe entire model looks to me like it never had even the most superficial security analysis done. It's like a smorgasbord of insecure decisions: * the false sense of security from putting signatures in the manifests then ignoring them * loading signing certs via the network with no provision for pinning * happily loading untrusted/unsigned images by default (npm, rubygems, installtools, etc. also do this but why repeat their awful design mistake?) * running basically everything as root (because why deal with all those messy permissions?) My sysadmin Spidey-sense has been tingling at the rate of change in the Docker ecosystem since it went from "interesting POC" to "we think it's production ready" in a shockingly short period of time. Things like this sadly confirm that initial pessimistic view.
- efuquen 12y agoNot at all related to docker, but this sort of thing is what makes me happy about communities like Rust. They are taking an incredibly long time to get to 1.0, but they've been progressing methodically and consistently and are trying to get something good out the door instead of bowing to any pressure to release early. Of course, Mozilla is a different beast compared to Docker, Inc., there is less of a profit motive more so than a need to create a revenue stream to stay maintainable and keep creating good new tech. Things like this are really putting everything that is happening with Rocket and the drama around it in perspective.
- lclarkmichalek 12y agoI don't understand why the image distribution is so tightly tied into the main docker codebase. This is why rocket is a thing, because docker is the systemd of the container world. Please stop trying to do everything.
- the_real_bto 12y agoThere are a lot of problems to solve, and docker by itself only gets you part of the way. Orchestrating docker containers is the interesting problem, not so much Docker in itself (for me anyway). The handwringing that docker should do less (as a company), because each company should only solve a small problem is just stupid. If the argument is that the tools should be small and composable, then I agree 100%. Maybe image distribution should be a separate tool from the tool used to run a container. That is actually a pretty good point.
- 23david 12y agoI think the "handwringing that docker should do less" has a lot to do with the fact that just getting Docker containers right is a huge effort in itself, and they have a long way to go even there. It's not a solved problem by a longshot.
- the_real_bto 12y agoI agree, there are problems all around. I was responsible for getting docker adopted on a fairly small (but a number of moving parts) software project. It has been somewhat painful.
- angelbob 12y agoIt's less "handwringing that Docker should do less" and more "get one thing right first."
- the_real_bto 12y agoAgreed. I was referring to the rocket/systemd remarks. There were some pretty big docker hate going on when rocket's announcement was posted here on HN. Most of that hate was because docker was getting into the orchestration business.
- geku 12y agoI would really prefer when Docker, Inc would spend their time and effort in securing their core product rather than extending it all the time by adding more and more features like Machine, Swarm, etc.
- kofalt 12y agoAccording to Red Hat, the current best way to secure your Docker usage is to `127.0.0.1 index.docker.io` and use an alternate transport. The core "translate flags into running container options" works fine IMO, it's the centralized transport causing the issue. Which isn't the end of the world, as distributing tarballs is not exactly a demanding task. As an example / plug, I helped write a (prototype) tool that lets you import a docker image from the registry, then transport / version it separately: https://github.com/polydawn/hroot https://github.com/polydawn/hroot Thus, integrating via `docker load` + `docker export` is possible & reasonable. Linked from the article: https://securityblog.redhat.com/2014/12/18/before-you-initiate-a-docker-pull https://securityblog.redhat.com/2014/12/18/before-you-initia...
- 23david 12y agoThe inevitable CVE's coming from this report will definitely get their attention. Hopefully the adults in the room will help make sure that the Docker team addresses what up until now has been a really lax approach towards security. Who is the architect in charge of this, and do they have any security chops? If not, it's just a matter of $$$ to get a 3rd-party security review before every major release. I've done it before, and it's really not a big deal.
- ef4 12y agoI keep hearing people ask "why won't they just get the core tech right instead of adding all these tangentially related features?". If Docker was just an open source project, it could focus on getting the core tech right. But Docker is also a startup, and the startup can't stay differentiated unless they keep adding bells & whistles, all of which stay tightly integrated. See also "Why there is no Rails Inc" (http://david.heinemeierhansson.com/posts/6-why-theres-no-rails-inc http://david.heinemeierhansson.com/posts/6-why-theres-no-rai...)
- 23david 12y agoThe business model seemed clearer before with the commercialized dockerhub. What is the business model with all these new features? Seems to me that they'll either license tech to enable enterprises to build their own docker platforms, or offer their own hosted platform... dotcloud 2.0
- ABS 12y agopretty clear in Docker Inc CEO words here: https://gigaom.com/2014/12/20/on-docker-coreos-open-source-and-virtualization/ https://gigaom.com/2014/12/20/on-docker-coreos-open-source-a... “The closest analogy I guess I can give you is, for people who think of Docker and containers as a new form of virtualization, so [with] open source we gave away ESX and what we are selling is something akin to vCenter or vSphere.”
- tuffle 12y agoWhich, if you put the pieces together, clarifies that the purpose of new Docker features is to lasso in fragmentation and make sure you're going to buy their vCenter/vSphere rather than any alternative. This also clarifies why things like Rocket are the biggest existential threat facing Docker Inc., and why the mud is being slung.
- ABS 12y agowell, those are two different things, aren't they? the direction is pretty clear. On everything else I'm on the fence but so far the 'batteries included, but swappable' motto has been respected: everything new they have done/announced lately is not in Docker core but rather an external project (think: Swarm, Machine, Compose) that uses the public APIs
- panhandlr 12y agoThis dude is just a worry wart. Plus, docker only has to keep your nodejs bitcoin app running in the cloud secure enough until the VC check clears...
- disjointrevelry 12y agoReminds me of debian and ubuntu's requirement that apt-get is run under root. There is simple ways to get apt-get to run on non-root, but it require giving permission to non-root account to modify important package signature files. But, they're not as bad as docker. It's becoming norm for these US/Silicon companies to give very bad integrity on data.
- rcoder 12y agoThe difference here is that apt-get and its ilk need to modify critical system state basically every time they run, and that state isn't controlled by a persistent daemon. I actually consider this a great tradeoff: yes I have to use sudo to run that one command, but I don't have a long-lived process sitting around pulling data down off the Internet and doing stuff with it while humming along as uid 0. It's also literally one line of code in most UNIX-based languages (syscall.Setuid(<uid that isn't root>) in Go, FWIW) to drop root privileges before doing something unsafe. Even if the main Docker daemon absolutely has to run as root most of the time, it can and should fork and drop that access for anything dealing with moving data between unstrusted (e.g. the Internet, user input, etc.) and trusted (verified, read-only local state) security domains.
- mjquinn 12y agoAs a quick word of caution (which doesn't invalidate anything you've said), Go has a long-standing bug[0] whereby syscall.Setuid doesn't always apply to all threads (on Linux at least) so extra care does have to be taken. [0] https://github.com/golang/go/issues/1435 https://github.com/golang/go/issues/1435
- rcoder 12y agoWow. That's a fun one. It's also a perfect example of why even really amazing teams reinventing a language/tooling ecosystem from scratch stumble over problems that were solved years (or even decades ago) in preceding platforms. I leave it as an exercise for the reader to decide if the "reinvent from scratch" critique is more deserved by Docker, Go, or Linux. That being said, I'm pretty sure even the broken Setuid behavior described there would be good enough to sandbox a thread or child proc that was just handling buffered I/O into and out of the xz binary.
- shykes 12y agoI wish the author had not omitted this crucial paragraph in the announcement he quotes: Note that this feature is still work in progress: for now, if an official image is corrupted or tampered with, Docker will issue a warning but will not prevent it from running. And non-official images are not verified either. This will change in future versions as we harden the code and iron out the inevitable usability quirks. Until then, please don’t rely on this feature for serious security, just yet. So, we've made it pretty clear from the start that we're working on ways to make image distribution more secure, but are not claiming that it's more secure yet.
- Alupis 12y ago> So, we've made it pretty clear from the start that we're working on ways to make image distribution more secure, but are not claiming that it's more secure yet That very clearly makes Docker not secure from it's inception through today currently. That was/is the point of the article. > but are not claiming that it's more secure yet. What does "more secure" mean... it's either not secure or is secure. > This pipeline is performant but completely insecure. Untrusted input should not be processed before verifying its signature. Unfortunately Docker processes images three times before checksum verification is supposed to occur. > However, despite Docker’s claims, image checksums are never actually checked. Things like this are by design and at the core of how Docker does what it does. Docker clearly was not built with security considerations in mind. > Docker exacerbates this situation by running xz as root. This means that if there is a single vulnerability in xz, a call to docker pull could result in the complete compromise of your entire system. The security implications go on and on.
- mattmcknight 12y agoIt doesn't make Docker "not secure", it means that you need to verify image files in the way you verify other files you download. These changes are irrelevant to the security of Docker itself, for the way I use it with my own images. "it's either not secure or is secure." Security is always a matter of degree.
- 12y ago
- oscargrouch 12y agoAs a non-security aware(not a security specialist) developer, this was one of the most instructional and concise little gem about security flaws i've read. You can learn very useful tricks just by reading this. Thank you
- deleted 12y ago[deleted]
- wayoverthere 12y agoParticularly interesting given that some of these problems were pointed out to Docker folks ~4 months ago in the development of the feature. https://github.com/docker/docker/issues/8093#issuecomment-57138688 https://github.com/docker/docker/issues/8093#issuecomment-57...
- efuquen 12y agoRead through that thread, funny thing is 19 days ago someone is pointing out how the docs are misleading after he is informed that docker doesn't verify signatures [1]. @shykes makes the claim on one of the comments here that nobody has every brought up that particular issue before, and while he wasn't the one who responded to this guy someone from the docker core team did. Clearly nobody changed anything in the docs to clarify. Like most things security related people don't care until there is a major exploit or somebody with enough clout complains enough, or more likely than not both. [1]: https://github.com/docker/docker/issues/8093#issuecomment-65612835 https://github.com/docker/docker/issues/8093#issuecomment-65...
- thaJeztah 12y agoNot disputing that the "verified" message is misleading, but the comment you referred to was about "not yet being able to create a signed image". I don't think there's anything yet in the docs with regard to signed image, apart for the release notes[1] mentioning it being a "sneak peak" of a coming feature, that is under development [1]: https://docs.docker.com/v1.3/release-notes/#new-features https://docs.docker.com/v1.3/release-notes/#new-features
- deleted 12y ago[deleted]
- ewindisch 12y agoHello, I'm the lead security engineer at Docker, Inc. There is nothing particularly new in Jonathan's post and I thank him for facilitating a conversation. Image security is of the upmost importance to us. For these reasons, we've concentrated efforts here in both auditing and engineering effort. Engineers here at Docker, our auditors, and community contributors alike have been evaluating this code to many of the same conclusions. Last month, we released Docker 1.3.2 which included limited privilege separation and extending this paradigm has been discussed. I have explicitly called out the need for containerization of the 'xz' process, and to run it in an unprivileged context. I thank Jonathan for reminding us of the need for this work and validating much of what is already in progress. As the recently published CVEs describe, we are expending resources in discovering and fixing security issues in Docker. Yet, I agree the v1 registry has a flawed design and we're aware of it. In September, I requested to become a maintainer of the tarsum code and have also made proposals and pushed PRs toward improving the v1 registry integration. This is not to replace the v2 effort, but to offer improved security for the design we have today. We have a draft for a v2 registry and image format. This and the supporting libtrust library are in the process of being audited by a 3rd-party. This is something we had previously promised the community and are making good on. What code exists today is a technical preview. Unlike the v1 registry and image format, the libtrust and v2 image format code has been designed for a decentralized model. However, as the libtrust and v2 image work, and subsequently, registry protocols are still in draft and security review, it is difficult for us to recommend that users yet attempt deploying these. This is why the developers of that code have not published clear instructions for its use, nor made such recommendations. As this work comes out of review and a specification is finalized, we should expect to see a much better experience and more secure image transport, along with stronger support for on-premises and 3rd-party registries.
- Titanous 12y ago> There is nothing particularly new in Jonathan's post and I thank him for facilitating a conversation. Maybe not new to you or Docker Inc staff, but I don't see any warnings that a pull could result in complete compromise. On top of this, the inaccurate "verified" image message is still in the current release. > I have explicitly called out the need for containerization of the 'xz' process, and to run it in an unprivileged context. Do you have a link to this? > As this work comes out of review and a specification is finalized, we should expect to see a much better experience and more secure image transport Is there a draft specification available for libtrust? > However, as the libtrust and v2 image work, and subsequently, registry protocols are still in draft and security review, it is difficult for us to recommend that users yet attempt deploying these. And yet you deployed it to production in 1.3.
- rab_oof 12y agoEarly on, I asked that images be signed similar to Debian packages, but was met with skepticism and resistance. To me, none of the Docker core devs had a handle on security implications of allowing anyone and everyone to share random bits without being able to prove end-to-end integrity and nonrepudiation. I hope this has changed, Docker is a great app. But if not, Perhaps someone would like to teach them a security lesson? It seems the only way most people actually learn, sadly. :(