7 ms·

PRoot: User-space implementation of chroot, mount –bind, and binfmt_misc

132 points by nateb2022 2y ago

nine_k 2y ago
Ah! Finally I see a way to try nix (the package manager) on my existing installation without it requiring access to the root directory and other such stuff!
- yjftsjthsd-h 2y ago
  https://nixos.wiki/wiki/Nix_Installation_Guide https://nixos.wiki/wiki/Nix_Installation_Guide discusses that a bit more (also other solutions)
  - lostmsu 2y ago
    Unrelated, but does anyone know what's the deal between wiki.nixos.org and nixos.wiki? The former appears to be actually functional, but the latter ranks much higher in search engines and seems to be incorrectly treated by them as the primary source.
    - Reventlov 2y ago
      Because nixos.wiki was the primary source, until very recently. Then, wiki.nixos.org was introduced, and mass edits were made (without the owner of nixos.wiki being ok with that) trying to redirect people from nixos.wiki to wiki.nixos.org, and now both co-exist. Reminds me a bit the archlinux.fr vs wiki.archlinux.org fr wiki situation from 10 years ago.
    - Voklen 2y ago
      nixos.wiki was the primary wiki before NixOS really had an official one. Now there is the official wiki.nixos.org and the NixOS foundation asked the nixos.wiki maintainer to cooperate on the new wiki but they refused. Now there are multiple problems[1] with nixos.wiki and the maintainer in unresponsive so although unfortunately it's ranked higher in search results, in my experience you'll get better information on the official wiki. [1] https://wiki.nixos.org/wiki/FAQ#Why_is_there_a_new_wiki?_What_is_with_nixos.wiki https://wiki.nixos.org/wiki/FAQ#Why_is_there_a_new_wiki?_Wha...?
      - lostmsu 2y ago
        Ah, too bad the maintainer is unresponsive. If they don't plan to work on it, they'd do a huge service to community by adding permanent redirects to wiki.nixos.org and maybe canonical links.
      - Y_Y 2y ago
        I went to see what the problems were for myself. While I wouldn't run my own sure that way, I was expecting something more serious to justify a migration. I think having the primary wiki within the project is a good idea generally. It does feel like the problems below are just minor annoyances though. --- Why is there a new wiki? What is with nixos.wiki? The old wiki at nixos.wiki has several problems: Many components (mediawiki, php, icu) are severely outdated. MediaWiki 1.29 (EOL 2018), now 1.35 (EOL 2023-12) PHP 7.3.33 (EOL 2021-12) ICU 64.2 Cloudflare DDOS protection makes wiki edits fail sometimes. There is no WYSIWYG editor. The wiki infrastructure, which was supposed to be made public after launch, never ended-up being made public
        yjftsjthsd-h 2y ago
        Many components (mediawiki, php, icu) are severely outdated. The wiki infrastructure, which was supposed to be made public after launch, never ended-up being made public I suspect both of these are a bigger deal to the Nix community than perhaps to the average group.
- kokada 2y ago
  I don't recommend using Nix inside PRoot unless this is your only option (e.g.: Android, since there are no user namespaces). The reason for that is that Nix is syscall heavy and using it in PRoot will be slow, really slow. As someone said, the wiki [1] has some interesting options. From the options I used nix-user-chroot [2] with great success, and while the tool is unmaintained it should still work. It uses Linux's user namespaces instead of intercepting syscalls, and this makes the performance pretty much identical to native. If you want more control like PRoot offers, I recommend bubblewrap instead [2]. Another option undocumented is to just grab a recent nix binary somewhere and run. It will automatically create the Nix store inside `$HOME/.local/share/nix` (if I am not mistaken) and use user namespaces to mount to it. However the last time I tried this didn't work well for a few things (e.g.: Home-Manager), so in general I still think nix-user-chroot is a better choice. [1]: https://wiki.nixos.org/wiki/Nix_Installation_Guide https://wiki.nixos.org/wiki/Nix_Installation_Guide, linking to the official one [2]: https://github.com/containers/bubblewrap https://github.com/containers/bubblewrap [3]: https://wiki.nixos.org/wiki/Nix_Installation_Guide#nix-user-chroot https://wiki.nixos.org/wiki/Nix_Installation_Guide#nix-user-...
- soraminazuki 2y ago
  Nix can do that natively with the --store option. But I'd recommend installing it normally if you do have access to / on your system though. Nix itself doesn't litter files all over the filesystem and can be uninstalled cleanly.
ggm 2y ago
Unless I am mistaken this cannot elide over the fundamental protections Android enforce which make it impossible to do cross-device symlinks to external SD cards in a non-rooted Android: the outer kernel blocks this, even though you can cd into the paths. If (for example) you can overlay/union mount and have a synthetic upper layer FS which manages things, I could use this.
- hks0 2y ago
  proot works by hijacking and altering syscalls. So maybe you could emulate the symlink yourself, by altering the file paths at ptrace level? Thinking more, maybe it works out of the box? Just mount the overlays (there's a cli flag IIRC) and proot takes care of syscalls by definition.
CaliforniaKarl 2y ago
Here's an example of how we've used this. RStudio Server[0] 1.3 and older hard-coded a number of paths, such as the path for storing temporary files: Instead of looking for the TMPDIR environment variable (as specified by POSIX[1]), R Studio Server would always use /tmp. That is extremely annoying, because we set TMPDIR to a path on fast local storage (SATA or NVMe SSDs) that the job scheduler cleans up at the end of the compute job. We do have a last-resort mechanism using pam_namespace[2], such that a user going to `/tmp` actually takes them to `/namespace/tmp/${username}`, but that is per-user, not per-job. If a user has two R Studio jobs, and those two jobs landed on the same host, there would be trouble. So, we used PRoot to wrap R Studio, with /tmp bind-mounted to a directory under TMPDIR. [0]: https://www.rstudio.com/products/rstudio/download-server/ https://www.rstudio.com/products/rstudio/download-server/ [1]: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1... [2]: https://linux.die.net/man/8/pam_namespace https://linux.die.net/man/8/pam_namespace
- godelski 2y ago
  For those that don’t know, you shouldn’t blindly let programs have access to tmp. They can get access to sockets and stuff. If you’re running with systemd there’s a private tmp option for this reason. It’s always best to sandbox programs when you can. Linux has been making this much easier but it’s still non trivial https://salvatoresecurity.com/the-many-perils-of-tmp/ https://salvatoresecurity.com/the-many-perils-of-tmp/
  - westurner 2y ago
    From the systemd.exec man page: https://www.freedesktop.org/software/systemd/man/systemd.exec.html#PrivateTmp= https://www.freedesktop.org/software/systemd/man/systemd.exe... : PrivateTmp=true #JoinsNamespaceOf= `unshare -m` and then bind-mounting a private /tmp at /tmp/systemd-private-/ does the same thing; `systemd-tmpfiles --help`: https://serverfault.com/questions/1010339/how-exactly-to-use-systemd-private-temporary-directory-feature/1142519#1142519 https://serverfault.com/questions/1010339/how-exactly-to-use...
hks0 2y ago
Termux has now a nice wrapper (manager?) for proot [1]. I once wanted to submit a wrapper but got an angry response as the first comment to my PR and it was closed. Guess enough people did that until they decided maybe it's not a bad idea after all. [1]: https://github.com/termux/proot-distro https://github.com/termux/proot-distro
- yjftsjthsd-h 2y ago
  > I once wanted to submit a wrapper but got an angry response as the first comment to my PR and it was closed. I don't follow; you made a PR to add this functionality to proot itself?
  - hks0 2y ago
    Ah no, sorry I was so lost in my thoughts and explained badly. I was taking in the context of Termux app for Android, nothing to do with proot itself. Termux uses apt and deb repos, and provides a patched proot. I wanted to add a package similar to one I linked.
- jillyboel 2y ago
  > I once wanted to submit a wrapper but got an angry response link?
albertzeyer 2y ago
A similar tool for this user-space bind-mount is https://github.com/fritzw/ld-preload-open https://github.com/fritzw/ld-preload-open, which relies on LD_PRELOAD to overwrite common libc functions. Thus this is less reliable as the presented tool which uses ptrace, but it still works reasonably well (I run e.g. PyCharm with it).
- jeduardo 2y ago
  Thanks for sharing this! I had to do exactly the same thing some 10 years ago to get an Oracle instance up and running again. Oracle insisted on using the /tmp location, despite being installed on a different drive, and the disk was full. As I had access to the Oracle system user, but not to the DBA user to change any configuration, I built a similar shared lib and preloaded it to the script. Worked like a charm! Happy to know that there is something _slightly more streamlined_ to do that now.
- AshamedCaptain 2y ago
  There is also Scratchbox, which was used (and probably is still used) to cross-build embedded Linux distributions. https://github.com/sailfishos/scratchbox2 https://github.com/sailfishos/scratchbox2 It also offers CPU transparency and was able to run almost arbitrary desktop software, but specializes in build toolchains.
aloisdg 2y ago
Great name in french
- QuiDortDine 2y ago
  Means "fart" for the curious.
Voklen 2y ago
Can anyone explain why chroot requires root privileges in the first place? Because from my understanding it seems like it should only restrict what you can do rather than grant any new abilities.
- badmintonbaseba 2y ago
  Nowadays you have unprivileged options with mount namespaces. And from an unprivileged user namespace you can probably invoke chroot just fine as "root". `chroot`, the function, has probably some baggage that makes it impossible to enable straight from an unprivileged program? Anyway, a chroot-like program seems to be implementable on top of user namespaces. Rootless docker is mostly that, with more namespace and cgroup isolation.
- Teongot 2y ago
  If you can get a suid root binary into the chroot, then you can control its configuration files to bypass security restrictions. $ ln /usr/bin/sudo ./my-chroot $ echo "$USER ALL=(ALL) NOPASSWD: ALL" > ./my-chroot/etc/sudoers.d/01-oops $ chroot ./my-chroot $ sudo bash modern Linux distributions prevent creation of hard links to suid binaries, but the restrictions on chroot came years before that.
  - cedws 2y ago
    Can’t SUID binaries in chroots be ignored, like the nosuid mount flag?
    - blueflow 2y ago
      Linux namespaces can do that, with a UID namespace, you drop the ability to do SUID, and then you can open a mount namespace and bind-mount and chroot as you like.
- vifon 2y ago
  chroot doesn't nest, there is only one chroot active for a given process at a given moment. If you're inside a chrooted environment and call chroot on a subdirectory without entering it, you regain the access to the parent directories.
rekado 2y ago
`guix pack` can create bundles that use a static proot to make them relocatable: https://hpc.guix.info/blog/2017/10/using-guix-without-being-root/ https://hpc.guix.info/blog/2017/10/using-guix-without-being-... It also supports other more performant ways, but in some situations proot is the best choice.
- Y_Y 2y ago
  But can it pack the whole system? I've been trying to run Guix on my Android for a while without resorting to a full VM. Nix has a custom termux, and lots of distros run under proot under termux[0], but seemingly not Guix. [0] https://wiki.termux.com/wiki/PRoot https://wiki.termux.com/wiki/PRoot
randomtoast 2y ago
If I shell script something together that executes a process in a PRoot environment, would that be similar to Docker?
badmintonbaseba 2y ago
So this works by intercepting file-related syscalls. Does it intercept io_uring?
bieganski 2y ago
related url, for syscall intercepting made easy: https://github.com/bieganski/asstrace/ https://github.com/bieganski/asstrace/ see `pathsubst` example. unfortunately set of use cases for `ptrace`-based solution is limited, due to high performance overhead.
animesh10k 2y ago
this is epic stuff
molticrystal 2y ago
Does this support using gdb to debug a program, because I vaguely remember years ago using a simulated root to install a linux distro on android to compile and test certain ideas while traveling and due to the nature of ptrace , they couldn't be loaded into a debugger.
gbraad 2y ago
have been using proot on Android to start my container image[1] as an alternative OS on top of termux. it is not perfect, but at least you can have a Fedora userspace on an Android tablet. when you add termux:x11 or vnc you have a minimal desktop, and with vscode a dev environment. 1: https://github.com/gbraad-devenv/ https://github.com/gbraad-devenv/

132 points by nateb2022 2y ago

Ah! Finally I see a way to try nix (the package manager) on my existing installation without it requiring access to the root directory and other such stuff!

yjftsjthsd-h 2y ago
https://nixos.wiki/wiki/Nix_Installation_Guide https://nixos.wiki/wiki/Nix_Installation_Guide discusses that a bit more (also other solutions)
- lostmsu 2y ago
  Unrelated, but does anyone know what's the deal between wiki.nixos.org and nixos.wiki? The former appears to be actually functional, but the latter ranks much higher in search engines and seems to be incorrectly treated by them as the primary source.
  - Reventlov 2y ago
    Because nixos.wiki was the primary source, until very recently. Then, wiki.nixos.org was introduced, and mass edits were made (without the owner of nixos.wiki being ok with that) trying to redirect people from nixos.wiki to wiki.nixos.org, and now both co-exist. Reminds me a bit the archlinux.fr vs wiki.archlinux.org fr wiki situation from 10 years ago.
  - Voklen 2y ago
    nixos.wiki was the primary wiki before NixOS really had an official one. Now there is the official wiki.nixos.org and the NixOS foundation asked the nixos.wiki maintainer to cooperate on the new wiki but they refused. Now there are multiple problems[1] with nixos.wiki and the maintainer in unresponsive so although unfortunately it's ranked higher in search results, in my experience you'll get better information on the official wiki. [1] https://wiki.nixos.org/wiki/FAQ#Why_is_there_a_new_wiki?_What_is_with_nixos.wiki https://wiki.nixos.org/wiki/FAQ#Why_is_there_a_new_wiki?_Wha...?
    - lostmsu 2y ago
      Ah, too bad the maintainer is unresponsive. If they don't plan to work on it, they'd do a huge service to community by adding permanent redirects to wiki.nixos.org and maybe canonical links.
    - Y_Y 2y ago
      I went to see what the problems were for myself. While I wouldn't run my own sure that way, I was expecting something more serious to justify a migration. I think having the primary wiki within the project is a good idea generally. It does feel like the problems below are just minor annoyances though. --- Why is there a new wiki? What is with nixos.wiki? The old wiki at nixos.wiki has several problems: Many components (mediawiki, php, icu) are severely outdated. MediaWiki 1.29 (EOL 2018), now 1.35 (EOL 2023-12) PHP 7.3.33 (EOL 2021-12) ICU 64.2 Cloudflare DDOS protection makes wiki edits fail sometimes. There is no WYSIWYG editor. The wiki infrastructure, which was supposed to be made public after launch, never ended-up being made public
      - yjftsjthsd-h 2y ago
        Many components (mediawiki, php, icu) are severely outdated. The wiki infrastructure, which was supposed to be made public after launch, never ended-up being made public I suspect both of these are a bigger deal to the Nix community than perhaps to the average group.
kokada 2y ago
I don't recommend using Nix inside PRoot unless this is your only option (e.g.: Android, since there are no user namespaces). The reason for that is that Nix is syscall heavy and using it in PRoot will be slow, really slow. As someone said, the wiki [1] has some interesting options. From the options I used nix-user-chroot [2] with great success, and while the tool is unmaintained it should still work. It uses Linux's user namespaces instead of intercepting syscalls, and this makes the performance pretty much identical to native. If you want more control like PRoot offers, I recommend bubblewrap instead [2]. Another option undocumented is to just grab a recent nix binary somewhere and run. It will automatically create the Nix store inside `$HOME/.local/share/nix` (if I am not mistaken) and use user namespaces to mount to it. However the last time I tried this didn't work well for a few things (e.g.: Home-Manager), so in general I still think nix-user-chroot is a better choice. [1]: https://wiki.nixos.org/wiki/Nix_Installation_Guide https://wiki.nixos.org/wiki/Nix_Installation_Guide, linking to the official one [2]: https://github.com/containers/bubblewrap https://github.com/containers/bubblewrap [3]: https://wiki.nixos.org/wiki/Nix_Installation_Guide#nix-user-chroot https://wiki.nixos.org/wiki/Nix_Installation_Guide#nix-user-...
soraminazuki 2y ago
Nix can do that natively with the --store option. But I'd recommend installing it normally if you do have access to / on your system though. Nix itself doesn't litter files all over the filesystem and can be uninstalled cleanly.

ggm 2y ago

Unless I am mistaken this cannot elide over the fundamental protections Android enforce which make it impossible to do cross-device symlinks to external SD cards in a non-rooted Android: the outer kernel blocks this, even though you can cd into the paths. If (for example) you can overlay/union mount and have a synthetic upper layer FS which manages things, I could use this.

hks0 2y ago
proot works by hijacking and altering syscalls. So maybe you could emulate the symlink yourself, by altering the file paths at ptrace level? Thinking more, maybe it works out of the box? Just mount the overlays (there's a cli flag IIRC) and proot takes care of syscalls by definition.

CaliforniaKarl 2y ago

Here's an example of how we've used this. RStudio Server[0] 1.3 and older hard-coded a number of paths, such as the path for storing temporary files: Instead of looking for the TMPDIR environment variable (as specified by POSIX[1]), R Studio Server would always use /tmp. That is extremely annoying, because we set TMPDIR to a path on fast local storage (SATA or NVMe SSDs) that the job scheduler cleans up at the end of the compute job. We do have a last-resort mechanism using pam_namespace[2], such that a user going to `/tmp` actually takes them to `/namespace/tmp/${username}`, but that is per-user, not per-job. If a user has two R Studio jobs, and those two jobs landed on the same host, there would be trouble. So, we used PRoot to wrap R Studio, with /tmp bind-mounted to a directory under TMPDIR. [0]: https://www.rstudio.com/products/rstudio/download-server/ https://www.rstudio.com/products/rstudio/download-server/ [1]: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1... [2]: https://linux.die.net/man/8/pam_namespace https://linux.die.net/man/8/pam_namespace

godelski 2y ago
For those that don’t know, you shouldn’t blindly let programs have access to tmp. They can get access to sockets and stuff. If you’re running with systemd there’s a private tmp option for this reason. It’s always best to sandbox programs when you can. Linux has been making this much easier but it’s still non trivial https://salvatoresecurity.com/the-many-perils-of-tmp/ https://salvatoresecurity.com/the-many-perils-of-tmp/
- westurner 2y ago
  From the systemd.exec man page: https://www.freedesktop.org/software/systemd/man/systemd.exec.html#PrivateTmp= https://www.freedesktop.org/software/systemd/man/systemd.exe... : PrivateTmp=true #JoinsNamespaceOf= `unshare -m` and then bind-mounting a private /tmp at /tmp/systemd-private-/ does the same thing; `systemd-tmpfiles --help`: https://serverfault.com/questions/1010339/how-exactly-to-use-systemd-private-temporary-directory-feature/1142519#1142519 https://serverfault.com/questions/1010339/how-exactly-to-use...

hks0 2y ago

Termux has now a nice wrapper (manager?) for proot [1]. I once wanted to submit a wrapper but got an angry response as the first comment to my PR and it was closed. Guess enough people did that until they decided maybe it's not a bad idea after all. [1]: https://github.com/termux/proot-distro https://github.com/termux/proot-distro

yjftsjthsd-h 2y ago
> I once wanted to submit a wrapper but got an angry response as the first comment to my PR and it was closed. I don't follow; you made a PR to add this functionality to proot itself?
- hks0 2y ago
  Ah no, sorry I was so lost in my thoughts and explained badly. I was taking in the context of Termux app for Android, nothing to do with proot itself. Termux uses apt and deb repos, and provides a patched proot. I wanted to add a package similar to one I linked.
jillyboel 2y ago
> I once wanted to submit a wrapper but got an angry response link?

albertzeyer 2y ago

A similar tool for this user-space bind-mount is https://github.com/fritzw/ld-preload-open https://github.com/fritzw/ld-preload-open, which relies on LD_PRELOAD to overwrite common libc functions. Thus this is less reliable as the presented tool which uses ptrace, but it still works reasonably well (I run e.g. PyCharm with it).

jeduardo 2y ago
Thanks for sharing this! I had to do exactly the same thing some 10 years ago to get an Oracle instance up and running again. Oracle insisted on using the /tmp location, despite being installed on a different drive, and the disk was full. As I had access to the Oracle system user, but not to the DBA user to change any configuration, I built a similar shared lib and preloaded it to the script. Worked like a charm! Happy to know that there is something _slightly more streamlined_ to do that now.
AshamedCaptain 2y ago
There is also Scratchbox, which was used (and probably is still used) to cross-build embedded Linux distributions. https://github.com/sailfishos/scratchbox2 https://github.com/sailfishos/scratchbox2 It also offers CPU transparency and was able to run almost arbitrary desktop software, but specializes in build toolchains.

aloisdg 2y ago

Great name in french

QuiDortDine 2y ago
Means "fart" for the curious.

Voklen 2y ago

Can anyone explain why chroot requires root privileges in the first place? Because from my understanding it seems like it should only restrict what you can do rather than grant any new abilities.

badmintonbaseba 2y ago
Nowadays you have unprivileged options with mount namespaces. And from an unprivileged user namespace you can probably invoke chroot just fine as "root". `chroot`, the function, has probably some baggage that makes it impossible to enable straight from an unprivileged program? Anyway, a chroot-like program seems to be implementable on top of user namespaces. Rootless docker is mostly that, with more namespace and cgroup isolation.
Teongot 2y ago
If you can get a suid root binary into the chroot, then you can control its configuration files to bypass security restrictions. $ ln /usr/bin/sudo ./my-chroot $ echo "$USER ALL=(ALL) NOPASSWD: ALL" > ./my-chroot/etc/sudoers.d/01-oops $ chroot ./my-chroot $ sudo bash modern Linux distributions prevent creation of hard links to suid binaries, but the restrictions on chroot came years before that.
- cedws 2y ago
  Can’t SUID binaries in chroots be ignored, like the nosuid mount flag?
  - blueflow 2y ago
    Linux namespaces can do that, with a UID namespace, you drop the ability to do SUID, and then you can open a mount namespace and bind-mount and chroot as you like.
vifon 2y ago
chroot doesn't nest, there is only one chroot active for a given process at a given moment. If you're inside a chrooted environment and call chroot on a subdirectory without entering it, you regain the access to the parent directories.

rekado 2y ago

`guix pack` can create bundles that use a static proot to make them relocatable: https://hpc.guix.info/blog/2017/10/using-guix-without-being-root/ https://hpc.guix.info/blog/2017/10/using-guix-without-being-... It also supports other more performant ways, but in some situations proot is the best choice.

Y_Y 2y ago
But can it pack the whole system? I've been trying to run Guix on my Android for a while without resorting to a full VM. Nix has a custom termux, and lots of distros run under proot under termux[0], but seemingly not Guix. [0] https://wiki.termux.com/wiki/PRoot https://wiki.termux.com/wiki/PRoot

randomtoast 2y ago

If I shell script something together that executes a process in a PRoot environment, would that be similar to Docker?

badmintonbaseba 2y ago

So this works by intercepting file-related syscalls. Does it intercept io_uring?

bieganski 2y ago

related url, for syscall intercepting made easy: https://github.com/bieganski/asstrace/ https://github.com/bieganski/asstrace/ see `pathsubst` example. unfortunately set of use cases for `ptrace`-based solution is limited, due to high performance overhead.

animesh10k 2y ago

this is epic stuff

molticrystal 2y ago

Does this support using gdb to debug a program, because I vaguely remember years ago using a simulated root to install a linux distro on android to compile and test certain ideas while traveling and due to the nature of ptrace , they couldn't be loaded into a debugger.

gbraad 2y ago

have been using proot on Android to start my container image[1] as an alternative OS on top of termux. it is not perfect, but at least you can have a Fedora userspace on an Android tablet. when you add termux:x11 or vnc you have a minimal desktop, and with vscode a dev environment. 1: https://github.com/gbraad-devenv/ https://github.com/gbraad-devenv/