7 ms·
Rootless Docker in a multi-user environment
- curt15 2y agoHow well does rootless docker work compared to rootless podman?
- sarusso 2y agoIf it works as I understood, in this setup I can see an advantage at an architectural level: in Podman containers images are stored on a per-user basis, while in this setup they would be shared between users, thus using much less disk space (if using the same base images). Besides this, I actually have the same question.
- cpuguy83 2y agoNo both use the exact same techniques to achieve so called "rootless" which is generally considered to be insecure. Rootless docker stores images for the user executing it and that is all, same as podman.
- bjoli 2y agoHow are they considered insecure?
- itsphilos 2y agoI think OP is referring to the "unprivileged user namespaces" [1] feature of Linux, which caused numerous security incidents in the past. AFAIK, this is mainly because with this feature enabled, unprivilged users can create environments/namespaces which allow them to exploit kernel bugs much more easily. Most of them revolve around broken permission checks (read: root inside container but not outside, yet feature X falsely checks for the permissions _inside_). [2] has a nice list of CVEs caused by unprivileged user namespaces. Given that rootful docker e.g. is also prone to causing security issues, it's ultimately an attacker model / pick-your-poison situation though. [1] https://www.man7.org/linux/man-pages/man7/user_namespaces.7.html https://www.man7.org/linux/man-pages/man7/user_namespaces.7.... [2] https://security.stackexchange.com/a/209533 https://security.stackexchange.com/a/209533
- curt15 2y agoDoesn't the Chromium sandbox, the gold standard for browser sandboxes, use user namespaces? https://chromium.googlesource.com/chromium/src/+/HEAD/docs/linux/sandboxing.md https://chromium.googlesource.com/chromium/src/+/HEAD/docs/l...
- sarusso 2y agoOk, but here the OP is doing something a bit different than just rootless Docker, which is to use a "centralised" rootless Docker running as a single, non-privileged user... or am I missing something?
- grudg3 2y agoI wonder why people aren't using Singularity containers instead which seem to be rootless by design.
- yjftsjthsd-h 2y agoAt the risk of showcasing my bubble: Lack of exposure; how many people even know what singularity is or how to use it? I know it's used in scientific HPC, but I don't see evidence of wider adoption.
- firesteelrain 2y agoNever heard of it. Podman or bust
- grudg3 2y agoDefinitely used in HPC environments where multi user is prevalent, I know of it because I used to admin a small HPC
- ratscylla 2y agoI think it is a few things together. The rootless and daemonless design leads to UX differences with Docker. Because of the differences, it isn't just a drop in replacement; porting applications can be a pain if they do anything weird (no network isolation, --containall isn't default and still is a bit different when on, etc). And Docker has a ton of momentum and usage.
- sarusso 2y agoOh man, Singularity… I once wrote: “In the shipping container analogy, you can think about Singularty containers as if they have no walls.” [1] [1] https://sarusso.github.io/blog/container-engines-runtimes-orchestrators.html https://sarusso.github.io/blog/container-engines-runtimes-or...
- smitty1110 2y agoWell, that blog post is getting a bookmark. Next time a junior dev asks me how this BS works, I'm going to show them your diagram.
- lmeyerov 2y agoRelated: If you are into this kind of thing and the extra fun that is GPUs + pydata, we have a 1mo or 2mo project around adding rootless to our GPU graph AI containers & packer flow. Ping build@graphistry.com . Niche but a project we have been wanting for awhile. Base containers get OSS'd etc. This stuff is twisty!
- fenollp 2y agoWhat do you mean by "adding rootless"?
- from-nibly 2y agoI still don't get why multi user. I haven't seen a multi user scenario that makes any sense for any Linux deploy in ages. Is this a shared prod server? Is it a shared Dev server? Why would those be multi user? For prod, why isnt it cattle where you don't ever SSH onto the server? For Dev if you can't just run it all on your local machine why not do something like shipyard.build Somebody tell me what im missing.
- hamandcheese 2y agoNot docker specific, but there are still lots and lots of servers out there running multiple services. Especially in my home-lab! Each service runs under a service-specific user, sometimes with extra hardening applied by systemd. It's a tried-and-true method for gaining some semblance of security boundaries on a shared server without the additional administration overhead of kubernetes or similar. A more relevant use case to industry might be a CI machine that you want to get better utilization out of. Easy, just start of multiple CI runners under different users. "Just use Kubernetes" I hear someone screaming? Well, sometimes you need CI on macOS.
- racingmars 2y agoI run a multi-user Linux server. It's IBM POWER9 hardware and several of my friends in the open-source community who care about testing on something other than x86, or who are interested in different architectures or playing with its unique capabilities such as quad-precision floating point implemented in the hardware appreciate having shell access to it.
- 3np 2y agoLet's say you have some centralized monitoring of various host metrics. The ingestion process needs some amount of privileged access. But you don't want to give it full root. Meanwhile, your actual service probably needs no privileged access, save for some secrets which should also be inaccessible for the monitoring agent process. You may want to run both of these as containers. I guess you may be using SELinux but even so, users and groups are natural parts of expressing and enforcing such constraints.
- 2y ago
- rbut 2y agoHere's our experience with rootless docker. We've used it in a single-user (a docker user) and multi-user (user for each dev) environment. Most, if not all, containers work fine, there are some, like mailcow which don't work well with it. If you have multiple IPs on the one machine, there is a longstanding bug that means you can't bind the same port on different IPs. Eg IP1:80 and IP2:80. The workaround for this is separate rootless docker users + runtime for each container that shares ports, nasty. In a multi-user environment we simply setup rootless docker under each devs user, so they have their own runtime and their own containers isolated from other devs. This works really well. Overall it works well for us.
- Helmut10001 2y agoSite is down? I am getting a connection refused.. Anyway, surprised that so many comments wonder about the usefulness of docker rootless in a shared environment. It is my main approach for separation of concerns in my homelab. I always use docker rootless to share resources with many isolated apps. I wrote a blog post about how to host Mastodon with docker rootless [1]. [1]: https://du.nkel.dev/blog/2023-12-12_mastodon-docker-rootless/ https://du.nkel.dev/blog/2023-12-12_mastodon-docker-rootless...
- rducksherlock 2y agoPossibly, the web server was down due to a large number of requests. I have a lot of stuff on the server running in containers, but limited my httpd instance to 1 CPU core and 1 GB of RAM. Better upgrade it, I guess.
- Helmut10001 2y agoI am still getting ERR_CONNECTION_REFUSED.
- rducksherlock 2y agoHmm, that's weird now. I suspect, the problem may be with DNS, since my domain used to point to another IP less than a week ago. Try to flush the DNS cache, or to connect from another device. Also, try this link: cmtops.dev/posts/rootless-docker-in-multiuser-environment/
- Helmut10001 2y agocmtops.dev/posts/rootless-docker-in-multiuser-environment/ also gives me ERR_CONNECTION_REFUSED Btw. this is from Germany. A geoblock?
- rducksherlock 2y agoNo, I never geoblock people because that's stupid and annoying. I will look into it this week. Busy at work, sorry. You can write me an email to timofey.chuchkanov@cmtops.dev, so we can continue this troubleshoot later without flooding here.
- abound 2y agoThis is a lot of work for something that works out of the box with Podman. Of course, using Podman introduces its own idiosyncrasies, and as someone else noted, the benefit of the approach in the article is that all users share an image cache. Source: I use Podman on a workstation where I SSH in as a bunch of different non-root users, and I've never had to think about it working.
- jeppester 2y agoI've used rootless podman for development for several months now. I had a few issues in the beginning, but in the end the solutions were rather trivial. I had to: - Delete config files from previous podman versions (pre 4) - Enable the docker socket (for my user) - Use docker compose 2 rather than "podman compose" or an older docker compose (shipped with the distro) We mostly use docker-compose files for our dev setups, so I can't say if I'd run into issues with more elaborate setups. But I must say that it works extremely well for me.
- Yasuraka 2y agoI've switched from using podman-compose to using the podman native way of composing services, which uses Kubernetes style manifests https://docs.podman.io/en/stable/markdown/podman-kube-generate.1.html#examples https://docs.podman.io/en/stable/markdown/podman-kube-genera... It also lets you generate those manifests from existing containers or pods. No need to learn the compose spec, less friction dev and prod without stop-gap measures like Kompose
- rducksherlock 2y agoYou are absolutely right. However, if your company has already highly invested into using Docker, the fact of something working OOTB in any other piece of software doesn't convince to make a decision to switch. On top of that, our DevOps team is pretty happy with Docker :)
- nubinetwork 2y agoWhat's the difference between this, and adding users to the docker group? As long as the docker daemon is running, you should be able to spin up containers without needing root.
- IAmLiterallyAB 2y agoThe docker daemon is running as root in that scenario. So anyone in the Docker group can trivially become root by starting a privileged container.