11 ms·
Unprivileged Docker Builds – A Proof of Concept
- _muff1nman_ 9y agoThis is hopefully something that buildah can achieve: https://github.com/projectatomic/buildah/issues/171 https://github.com/projectatomic/buildah/issues/171
- freedomben 9y agoThis is interesting, but I'm not sure if I'd stay at a company that refused me root access to my own machine. A culture where such a lack of trust exists is not one that sounds attractive to me.
- jcims 9y agoFair enough, but there are cases where allowing that while also meeting the web of regulatory requirements that beset some industries is difficult if not practically impossible.
- user5994461 9y agoIn layman terms, expect banks, defense, government and healthcare to have a dedicated operation team with root access. Developers do not have it.
- vbezhenar 9y agoCan you run VirtualBox and use it as your working machine instead?
- user5994461 9y agoYes, but I think you are missing the point. Developers can access dev and production machines with non root users, root is never needed to run software. If you are part of the sysadmins that really need root, to manage iptables or system updates for instance, you would have root.
- phs318u 9y agoIf the VM is on the corporate network then it’s the same as connecting an unmanaged device - defeats the purpose of locking down machines. Developer VMs should be on their own VLAN.
- crypt1d 9y agoI've seen that in a couple of IT outsourcing companies, too. Its hilarious because it complicates working with these companies so much that sometimes u just might reconsider hiring the company and do the work yourself. I dont think they realize how much they are shooting themselves in the foot with dumb restrictions like that.
- deleted 9y ago[deleted]
- danenania 9y agoThis is something I've had to consider a lot in the design of EnvKey[1] (a config/secrets manager). I agree that refusing root access on your own machine seems like it's going too far in most cases, but where's the line? For example, EnvKey makes it easy to only give a developer access to development/staging config so that if someone only needs to deal with code and not servers, they'll never see production-level config. Could that get in the way sometimes? Sure. If for whatever reason someone who's normally a pure dev needs to step into ops for a bit, they'll have to ask for upgraded permissions to do so, which could certainly be seen as annoying, and could make someone feel less trusted than he or she would like to be. On the other hand, giving production secrets to every dev undeniably increases the surface area for all kinds of attacks, and I think that even small startups would be well-served by moving on from this as soon as they can. I think the key distinction to make is between real security and security theater. As a developer, I'm willing to give up a little trust and a little efficiency if the argument for why I'm doing it seems valid, but if I'm being asked to jump through extra hoops without any clear benefit attached, I'll probably resent it. So for me, the most relevant question to ask the OP (or a company that wants to implement this) is what's the threat model? What exact attack scenarios is this protecting against? Are those realistic enough to justify the extra hoops? 1 - https://www.envkey.com https://www.envkey.com
- cyphar 9y agoRootless containers are quite useful for things like accessing computing clusters which are shared and thus nobody gets root access on them (such as in universities). Rootless building is quite useful for other reasons (mainly because it allows you to build without ever hitting a privileged codepath which is arguably a huge security improvement), but to be honest it's more of a cool feature you can implement once you have the far more useful tools implemented (such as rootless storage management and rootless containers).
- theothertom 9y agoI've locked down developer workstations before, to prevent things like "connect a USB stick and take a copy of data/code". We did allow running your own VM, where you could be root if you wanted. In general, I've found it's pretty workable to run without root/sudo for development work - there's not _that_ much stuff that you can't just install to ~/bin and run from there.
- nebulous1 9y agodevs can't get root on their own machine? Is this common?
- ironjunkie 9y agoA lot of big companies have that policy. Even for devs.
- Macha 9y agoI've heard of it happening in other companies here in Ireland, and our security/infra team at a large multinational keep making appreciative noises towards companies and concepts about locking down end dev user machines and having dev needed stuff happen on VMs. Luckily they've not got their way yet.
- spc476 9y agoI may depend upon the company. I work at a company what writes software for smart phones [1] and we don't have such restrictions. Yet our parent company (we were bought out several years ago) does deal with financial transactions and has all sorts of regulatory restrictions what can and can't be done with computers. Unfortunately, for some obscure reason, it's only now that we might have to conform to the regulatory compliance issues (no one has been able to answer me "Why now? Why not when we first got bought years ago?") We're trying to fight it but the fix seems to be in. [1] Enhanced caller ID. Part of our software lives on the cell phone. Part lives on the phone network.
- johnnyfaehell 9y agoLooks like the guy who wrote this works for a UK bank, which kinda makes sense since we all know UK banks suck at IT. RBS, TSB, Natwest, etc all have serious technical problems and banks have been known for years of just plain sucking at building web apps.
- akhilcacharya 9y agoIt was like this at one of the companies I worked at. All dev desktops were heavily locked down and you had to file tickets to get software installed - sometimes the IT folks would log into your machine remotely and do it from there. It was kinda neat in a way.
- maltalex 9y agoAccording to the author's Linkedin [0] he's "Lead OpenShift Architect at Barclays" I get the security risks and the fear of malicious insiders at a company such as this. But having expensive, fairly high level employees work around not having root access on their own machines strikes me as odd. The guy does docker for a living and can't run docker on his own machine. Also, recall that he has physical access to said machine (I hope) so if he really was a malicious insider, he could already pretty much own it. Then again, maybe he doesn't. [0]: https://www.linkedin.com/in/ian-miell-694496/ https://www.linkedin.com/in/ian-miell-694496/
- ajackfox 9y agoUnfortunately I think these types of restrictions are fairly common in large organizations, even non-financial. Speaking as someone in a Fortune 500, sometimes the bureaucratic hoops one must jump through to develop in a VM or with docker on ones own machine isn't even worth it.
- eat_veggies 9y agoAt a company I interned at, even getting Linux on a laptop was a multi-week process. I ended up running a Linux VM on virtualbox on Windows, used cmder to ssh into it, and used a shared folder to edit code natively while running it on Linux.
- user5994461 9y agoTo be fair, it makes no sense for the company to allow any desktop on Linux. The desktop wouldn't have the centralized authentication, the system updates, office or the drivers for the printer that are well integrated only on their windows machines.
- mbrumlow 9y agoDid you mean Linux on a desktop? Umm, you can do all of those things with Linux... I can't remember the last time I had any issues with printing on Linux at all.
- arbie 9y agoAccess to large compute farms in most enterprises requires that users do not have root access to the nodes. Being able to run Docker containers on the farm would be fantastic. This is a great article that I will investigate further. Thanks!
- CaliforniaKarl 9y agoNot just on compute farms in enterprises; in academia as well.
- cyphar 9y agoI was actually part of a university research group when I discovered the need for this, and started working on rootless containers as a result -- I gave a talk about this last year[1] (though quite a few things have changed since then). [1]: https://www.youtube.com/watch?v=r6EcUyamu94 https://www.youtube.com/watch?v=r6EcUyamu94
- cyphar 9y agoI would recommend taking a look at [1]. This is the exact usecase I started working on rootless containers for. I talk a bit more about it in a talk I gave last year[2]. [1]: https://rootlesscontaine.rs/ https://rootlesscontaine.rs/ [2]: https://www.youtube.com/watch?v=r6EcUyamu94 https://www.youtube.com/watch?v=r6EcUyamu94
- verst 9y agoRelated to this, so pardon my pitch: We [Azure] literally just launched Azure Container Registry Build. [1] With one command you can build your container in the cloud and have it stored in your registry. Either push the source code from local or have it use a remote git repository. > $ az acr build --registry $ACR_NAME --image $IMAGE_NAME:$TAG_NAME --context . [1] https://docs.microsoft.com/azure/container-registry/container-registry-tutorial-quick-build https://docs.microsoft.com/azure/container-registry/containe...
- dayglo 9y agoYes! now I can ditch my full-fat-vm vsts build agents! Thanks to your team.
- jacques_chester 9y agoI think the key distinction here is unprivileged builds outside a container. Kaniko was aimed at an actually-quite-large problem, which is performing unprivileged building inside a container in a shared cluster. This matters a lot, because otherwise CI/CD is a serious weak point in your overall security model. Especially if you will run PRs that folks send you -- ie, run sight-unseen 3rd-party code in a privileged container. In terms of the OpenShift vs local development experience, we heard something very similar about Cloud Foundry too, so now there's `cf local`[0]. Essentially, Stephen Levine extracted buildpack staging into standalone container images (which now use Kaniko) along with a little CLI tool to help you make the local development experience more like the `cf push` one. As it happens there is a lot of cooperation happening in this space right now -- folks from Google, Pivotal and Red Hat have been comparing notes about container building in the past few months. It's something we share a common interest in improving. At Pivotal we have also begun closer cooperation with Heroku on buildpacks, which is really exciting. The buildpacks model lends itself to some really cool optimisations around image rebasing[1]. Folks heading to Kubecon this week who are interested in container builds should make sure to see the joint presentations by Ben Parees (Red Hat), Steve Speicher (Red Hat) and Matt Moore (Google). Disclosure: I work for Pivotal. [0] https://github.com/cloudfoundry-incubator/cflocal https://github.com/cloudfoundry-incubator/cflocal [1] https://www.youtube.com/watch?v=jsNY4OP3IrE https://www.youtube.com/watch?v=jsNY4OP3IrE
- jacques_chester 9y agoFor folks coming to this later, a correction. The buildpacks team are using the "crane" tool in go-containerregistry[0], rather than Kaniko, as the basis for their daemonless container-building containers. There is so much container-related work coming out of Google right now that I am struggling to keep up. [0] https://github.com/google/go-containerregistry https://github.com/google/go-containerregistry
- dvlsg 9y agoHeh. I wonder if anyone has ever opened a bunch of PRs with bitcoin miners in them, just to see if they could get them to run on someone else's CI.
- ggm 9y agoMaybe its me, but I don't see what the restrictions would be to make an image which had a root FS inside it, without privileges. The "bits" which make it be a root FS don't need you to "mount" it as a root process and run chflags, you can dd into the image, change things, do math, change checksums. Sure, its a lot harder but the principle of what an image is, is: its a stream of bits. If you can modify the stream of bits, you don't need root to mark it as a region which has magic root properties be it a chflags FS state, or a setuid bit of whatever. Also, again it may just be me, but if you are running a hypervisor limited VM image on a stream of bits, and you can modify those bits outside that state, restricting this VM not to have root runtime is slightly odd. This reads like a proscriptive "no root" rule has been metasploited into "we will wake you at 3am to check if your dream, you are running as root" type extremism.
- jacques_chester 9y agoIt's less about "could I write a program to modify bits directly?" and more about "what can I get that I won't have to support myself for the rest of time?". Nothing stops anyone from writing code to interpret Dockerfiles or to directly fiddle with image layers. But taking the cost:value ratio proportional to everything else you need to be doing, it's probably a poor investment of time. Google has economies of scale around this exact problem, which is why they've been pumping out work in this area -- Kaniko, Skaffold, image-rebase, FTL, rules_docker, Jib etc.
- ggm 9y agoSo his story reduces down to the real problem: what tooling can I find, which runs without root, but makes images which include root outcomes, for the set of things I need in an image which can't run un-privileged, and those tools need to work without running setuid() or seteuid() to root. Thats a good story btw. I have people working near me who probably want the same thing from a lower driver, but nonetheless interest in non-root required builds.
- cyphar 9y ago> Maybe its me, but I don't see what the restrictions would be to make an image which had a root FS inside it, without privileges. There are many reasons why those restrictions exist, it's mainly related to what types of files you can create and how you could trivially exploit if the host if things like mknod(2) were allowed as unprivileged users. There's also some more subtle things like distributions having certain directories be "chmod 000" (which root can access because of CAP_DAC_OVERRIDE but ordinary users cannot, and you need to emulate CAP_DAC_OVERRIDE to make it work). In short, yes you would think it's trivial (I definitely did when I implemented umoci's rootless support) but it's actually quite difficult in some places. Also unprivileged FUSE is still not available in the upstream kernel, so you couldn't just write your own filesystem that generates the archives (and even if FUSE was unprivileged it would still be suboptimal over just being more clever about how you create the image).
- scarface74 9y agoNot getting root on your own machine as a developer? That's ridiculous. I left a company where I was the lead developer because I got tired of waiting two weeks for the outside networking consultants to get resources provisioned on AWS. Our own internal network people had to wait on them. Now I work for a much smaller company and they gave me full admin access to the AWS console from day one and I am "just" a senior developer.
- curiousgal 9y agoMost developers working on HPC systems don't have root access.
- chris_st 9y agoExactly our situation. I'd love to be able to build and deploy docker images, but the "docker is root" thing makes this impossible in our shop.
- curiousgal 9y agoHave you looked into Singularity[0]? It supports Docker containers. 0.https://singularity.lbl.gov/index.html https://singularity.lbl.gov/index.html
- marcosnils 9y agohttps://github.com/genuinetools/img https://github.com/genuinetools/img
- peterwwillis 9y ago"Building Container Images Securely on Kubernetes:" "Standalone, daemon-less, unprivileged Dockerfile and OCI compatible container image builder." "TLDR; will work unprivileged on your host not in a container til we work out some kinks." https://blog.jessfraz.com/post/building-container-images-securely-on-kubernetes/ https://blog.jessfraz.com/post/building-container-images-sec...
- tannhaeuser 9y agoThe zoo of build tools and scripts on top of Docker and "orchestration" tools sure reaks of incidental complexity. Could someone explain to me what material problem Docker is solving that couldn't be solved using a statically linked binary as demon?
- empath75 9y agoGreat now deploy that demon and updates across an entire fleet of servers a few times a day. And make sure you have metrics for it, and that you’re efficiently using cpu and memory resources across your cloud footprint.
- tannhaeuser 9y agoDocker doesn't do these things neither for you.
- hanikesn 9y agoMost applications can't be easily condensed done to a single static binary.
- fulafel 9y agoUnpaid advert: LXD supports unprivileged containers quite well.
- cyphar 9y agoYup. I implemented the rootless containers support in runc and I love working with the LXC folks when it comes to topics related to low-level container implementation. It should be noted though that by default LXC does require some privileged (suid) helpers (for networking and cgroups) -- though you can disable them as well. runc by default doesn't, though that's just a matter of defaults and what use-cases we were targeting.
- humbleMouse 9y agoWhat does this approach have that openshift doesn't provide?
- cyphar 9y agoDoesn't require running anything as root. Almost all of OpenShift runs as root.
- zwischenzug 9y agoAuthor here. I worked at a place that had full root for everyone everywhere and introduced docker back in 2014. It was fun. Then I moved to finance where regulatory requirements mean that root is unacceptable. Whatever we think of that (and I now build openshift clusters at home to test, which I'll blog on soonish) it is a reality for a lot of people. Don't get me started on trying to use git on Windows either... Also, I have no admin access to uat or production envs or codebase. It's a challenge.
- zwischenzug 9y agoPS I work in infra now, no longer as a dev. It's the same for us. The sec team, on the other hand...
- alexellisuk 9y agoA PoC was shown of how to do this with buildkit several weeks ago in [0], but in your words - it's also not for the faint of heart (involving patching the Kernel). This is the way of the future - Docker image builds should not need to be privileged (they often are for mounting filesystems) https://github.com/genuinetools/img https://github.com/genuinetools/img [0] Similar: https://github.com/AkihiroSuda/runrootless https://github.com/AkihiroSuda/runrootless
- deleted 9y ago[deleted]
- cyphar 9y ago> A PoC was shown of how to do this with buildkit several weeks ago in [0], but in your words - it's also not for the faint of heart (involving patching the Kernel). Rootless builds work without kernel patches (the "rawproc" stuff mentioned in issues is not going to be merged into the kernel and there are other ways of fixing the issue -- like mounting an empty /proc into the container). I can do builds right now on my machine with orca-build. The main reason it's for the faint of heart is that we don't really have nice wrappers around all of these technologies (runc works perfectly fine, as does umoci, as does orca-build, as does proot, as does ...). Jess's project is quite nice because it takes advantage of the cache stuff BuildKit has, though I personally prefer umoci's way of doing storage and unprivileged building (though I am quite biased ofc). I'm going to port orca-build to Go and probably will take some inspiration from "img" as to how to use Docker's internal Dockerfile parsing library. For more information you can take a look at https://rootlesscontaine.rs/ https://rootlesscontaine.rs/.
- tome 9y agoThe big unanswered question in my mind is "why on earth does being in the docker group give root access?". The relevant section from the Docker manual is > First of all, only trusted users should be allowed to control your Docker daemon. This is a direct consequence of some powerful Docker features. Specifically, Docker allows you to share a directory between the Docker host and a guest container; and it allows you to do so without limiting the access rights of the container. This means that you can start a container where the /host directory is the / directory on your host; and the container can alter your host filesystem without any restriction. https://docs.docker.com/engine/security/security/#control-groups https://docs.docker.com/engine/security/security/#control-gr... This does not seem like a necessary consequence of setting up a daemon for building disk images. What am I missing here? Is this an engineering oversight on the part of Docker or is there some technical reason that forces it to be like this?
- deleted 9y ago[deleted]
- e12e 9y agoAfaik as I've been able to figure out, it should be possible to implement the plan9 filserver in userspace, and combine user-mode file-sharing with convenience and performance (esp. for a local host). Unfortunately, I've never been able to find a combination of utilities that actually work and allow this. Which I guess leaves user-mode nfs - or more exotic things like ipfs.
- zenlikethat 9y agoBasically docker group exists because it's a lot easier to get people to just add themselves to that group than to type 'sudo' over and over again. Sad but true. It's not mandatory to use the docker group though. That's completely optional and you could definitely just 'sudo' whenever you do a 'docker' command. The docker _daemon_ needs to run as root because it needs to be able to do all kinds of privileged system calls to actually set up the containers. But if everyone who interacts with the docker daemon is a sudoer, that's not a problem.
- 9y ago