10 ms·
Rpath, or why lld doesn’t work on NixOS
- elikoga 5y agoWow, great investigation article
- setheron 5y agoI actually wrote somethijg recently related to RPATH in Nix you might find interesting https://fzakaria.com/2022/03/14/shrinkwrap-taming-dynamic-shared-objects.html https://fzakaria.com/2022/03/14/shrinkwrap-taming-dynamic-sh...
- fanf2 5y agoBecause I am stubborn, when I am linking with a library installed in a nonstandard place, I usually try to get the configure script to do the right thing, even though it is not always easy. But, just in case I lose the battle, I keep in mind the existence of chrpath, a little utility for changing the rpath in an ELF binary. Because you use it on the final build artefact, there is no way for autoconf or libtool to screw it up.
- kaba0 5y agoAs mentioned by another commenter, the NixOS project does have a similar little program called patchelf, which may be a bit better known?
- slabity 5y agoHow does an article on NixOS talk about the `rpath` issue without also mentioning the `patchelf` utility that NixOS developers created to solve this issue? It's a small tool that lets you modify ELF executables and binaries. It's also the recommended way for NixOS users to modify binaries to work properly. https://github.com/NixOS/patchelf https://github.com/NixOS/patchelf
- kazinator 5y agoI'm pretty sure NixOS isn't the only one doing this hack. The Yocto build system does something similar. It contains its own build-time binary glibc, and patches its tools to point to its own internal library installation. Or something like that. In effect, Yocto has its own build-time distro, which has to run from any filesystem location.
- PaulDavisThe1st 5y agomacOS has also been doing this for years, decades even, as part of the job of install_name_tool
- astrange 5y agoThough you can override paths to libraries with environment variables, as long as the overriding one has the same full install name. That’s better since editing a binary will break its codesigning.
- kazinator 5y agoScripts and environment variables are an ugly hack. Environment variables are dynamically scoped and will cause problems, if you don't care to suppress them from being inherited by child processes. Suppose there are two installations of app. One is the system one, and one is locally installed by the user. The user overrides LD_LIBRARY_PATH when invoking the local app. Suppose that that app is used in such a way that it invokes the system-installed app; that could then find the wrong libraries due to the LD_LIBRARY_PATH being inherited. A program must simply know where its exact pieces are, all by itself, without any external tricks that could influence more than just that program. Search paths (all of them, including PATH) should be left to the user, for arranging the system; the user should be able to manipulate paths in arbitrary ways, yet the application shouldn't break as far as being able to locate and load its own pieces.
- lloeki 5y ago> I'm pretty sure NixOS isn't the only one doing this hack When developing ArchMac I had to do godawful hacks to bog-standard libs because whatever build system decided to hardcode a lib path (or forcefully strip one when it should be hardcoded, I've had to handle both) that I had to manipulate through various means including install_name_tool which is not that different from patchelf†. This kind of issue was not macOS specific, it just turns out the various ways things were built happened to gracefully "work" on most Linux distros by sheer luck but they could have been equally broken. † Not really a surprise when thinking about it, the concept of Nix derivations is not that different from the concept of Darwin bundles/frameworks (in terms of being a self-contained dependency package) so it's only natural similar issues, and thus approaches and tools to tackle them, emerged.
- pvtmert 5y agointerestingly macOS has a really nice solution called @rpath and install_name_tool basically you build your binary and set a rpath, let's say it is /usr/local/lib in your machine mach-o binary stores: @rpath = /usr/local/lib libxyz = @rpath/libxyz.dylib.1 when I want to install those to /opt/something only thing I need to do is install_name_tool -add_rpath /opt/something This will add search directories to binary itself. There are some DYLD_* environment variables too but I'm not sure about them... (Some are SIP protected by the way) PS: It may invalidate signed binaries. Again, not tested such use cases.
- kazinator 5y agoThat's a pretty poor solution compared to just finding libraries relative to the root of the program's installation.
- whateveracct 5y agoYep - Windows of all things handles this the best. You can try to do this with rpath of $ORIGIN. But then you'll probably still run into libc issues -.- Not hard to see why a lot of games focus on Windows and let wine/proton handle Linux.
- deleted 5y ago[deleted]
- kazinator 5y agoAnd the irony? It was someone on Unix who came up with the idea that when foo.c contains #include "bar.h", the same directory where foo.c resides will be searched for that header first, by default, before other places. The idea of doing that for the DLL search in Windows could have been inspired by C.
- ncmncm 5y agoSo you are saying this is why Windows users universally complain about "DLL Hell", and Linux users don't? Or is this how MS finally fixed Windows DLL Hell? (Presuming Windows DLL Hell is, indeed, fixed; I wouldn't know.)
- kazinator 5y agorpath is braindamaged; no program should be using that, and a distro build should almost never be inserting such a thing into executables. I suspect that NixOS is playing with this in order to have a relocatable install: so that is to say, so that user can install NixOS in some subdirectory of a system running some existing distro. Any subdirectory, yet so that programs can find their libraries. If I were in this predicament, rather than perpetrating hacks to patch the rpaths in binaries, I'd fix the dynamic linker to have a better way of locating shared libraries. The linker would determine the path from which the executable is being run, calculate the sysroot location dynamically, then look for libraries in that tree. E.g. /path/to/usr/bin/program would look under /path/to/usr/lib and related places. A possibly nice hack would be to extend the meaning of the rpath variable. Give it a syntax, like say that if it starts with @, then the rest of it denotes a relative sysroot path fragment. E.g. the program that gets installed as /path/to/usr/bin/program would be built with an rpath of "@/usr/bin". So then the dynamic linker sees the @ and does a sysroot calculation. First it strips off the basename to get just the directory part "/path/to/usr/bin". Then it sees, hey, the suffix of "/path/to/usr/bin" matches the "/usr/bin" in the rpath. The suffix is stripped to produce "/path/to" and that path is then used as the root for the library searching. Instead of searching literally in /lib or /usr/lib or whatnot, the "/path/to" part is prefixed to every search place to look in /path/to/lib and /path/to/usr/lib. Patching binaries is very poor; it changes their cryptographic hash like SHA-256. You want your distro to be installing bit-exact stuff from the packages, and treating it as immutable.
- eptcyka 5y agonixOS is built around hashing the outputs of its builds, so you can verify if a build produces the expected output given the same inputs. The reason nixOS patches binaries is so that they can actually find the shared objects they expect when they are not stored in /usr/lib. Since every build output gets its own unique path, this allows for a binary to link against two slightly different versions of the same library, which in practice is never an issue one needs to resolve. However, a more practical issue this solves is that you can have a single sysroot with two binaries that need two slightly different versions of the same library.
- 5y ago
- stabbles 5y agoLinux usually (by convention) provides 1 file and 2 symlinks per lib: liba.so -> liba.so.x -> liba.so.j.k.l. The first one is to make the linker (ld) happy: -la will look for liba.so. The linker puts the SONAME (liba.so.x) in DT_NEEDED. The second symlink's filename corresponds to the SONAME, so that the runtime linker (ld.so) can locate the library by SONAME in rpaths. The third one is the actual library, which can be updated while keeping the same soname & same abi. Now, it would be great if the linker had an option to not only copy the SONAME into DT_NEEDED, but also register the path in which the library was located as an rpath. Cause the situation on Linux is absurd! You pass some flags -L and -l to the compiler/linker, the linker links something and nobody knows what. Then when you run your executable it has to locate this something again, and you can only pray that your libc and binutils/llvm agree on search paths & order. In most cases this does not work, and you must manually pass -Wl,-rpath,/some/path to add a search path. Nobody guarantees that what the linker links is what the runtime linker uses. Of course there are many edge cases: - linking during make without relinking during make install will make your executables register rpaths to build directories instead of install dirs - sometimes you link to a stub lib that should not be used at runtime But still, some guarantee that what you build with is what you run with would be a major user experience improvement for linux.
- jcranmer 5y ago> Cause the situation on Linux is absurd! You pass some flags -L and -l to the compiler/linker, the linker links something and nobody knows what. Then when you run your executable it has to locate this something again, and you can only pray that your libc and binutils/llvm agree on search paths & order. In most cases this does not work, and you must manually pass -Wl,-rpath,/some/path to add a search path. Nobody guarantees that what the linker links is what the runtime linker uses. The issue you're missing is that the build directory is usually not the location of the final binary objects. The actual absolute path of libfoo.so may well be in /builds/runner/foo-package-4df78af0/build/prefix/lib/libfoo.so, which is unlikely to exist on anyone other than the CI's machine (and even on the CI machine itself for too much longer). The actual location will usually be /usr/lib64/libfoo.so, but the library that is linked against may well not be there at the time of linking (particularly in the case where a package is building both a library and an executable that depends on said library in the same package). What you really want is for the relative path to the library to stored in the executable. Unless what you want is to actually use the globally-installed library and not one you're building at the same time. There's no single solution that fits every use case!
- higherhalf 5y agoSpeaking of significantly faster linkers, I personally use mold[1], including when writing in Rust. [1]: https://github.com/rui314/mold https://github.com/rui314/mold
- FL33TW00D 5y agoReally looking forward to MacOS support on Mold!
- saghm 5y agoI'm super interested in trying out mold, but I was a bit taken about seeing that it's AGPL licensed; I have no idea what the implications of that license for a linker would be. Would using it to link into a final binary require sharing the source of the binary when its distributed? What about if its used to link statically into a library, and then that library is linked into another binary?
- moonchild 5y agoNo more than would using a GPL-licensed linker, such as gnu ld or gold.
- rkangel 5y agoYou're running the tool not embedding or linking against its source code. An equivalent example might be if Libreoffice was under the GPL licence[1] - that licence wouldn't have any implications for a spreadsheet you created using it. [1] It's not - it's MPL
- cryptonector 5y agoYes, you need to set the RPATH correctly when building ELF executables and shared objects. You would only not know this if you were only ever building things to install into /usr with dependencies on things in /usr. > Curious observation: dynamic linking on NixOS is not entirely dynamic. Because executables expect to find shared libraries in specific locations marked with hashes of the libraries themselves, it’s not possible to just upgrade .so on disk for all the binaries to pick it up. Dynamic linking is not just about being able to upgrade without re-linking. Dynamic linking is not even primarily about that, not anymore, if it ever was. Dynamic linking is more than anything about semantics that no one has bothered to add to static linking! Static linking for C is stuck in the 1970s. Dynamic linking for C makes C more like C+ -- a different language. Specifically: - with static linking symbol conflicts are a serious problem - with dynamic linking symbol conflicts need not be a problem because with direct binding (Illumos) or versioned symbols (GNU), you get to resolve the bindings correctly at build-time and have them resolve correctly at run-time - at build time you get to list just direct dependencies, and the linker does the rest -- compare to static linking, where you have to list all dependencies only in the final link-edit and then you must flatten the dependency list into some order, and then if there are conflicts, you lose. For all those who keep harping on how static linking is better than dynamic linking, what I would suggest is that what must be done to make static linking not suck is to enrich .a files with the kinds of metadata that ELF adds to shared objects so we can get the same "list only direct dependencies" semantics when static linking as when dynamic linking. And I would note that libtool does this, just... very poorly. What I would do to fix static linking: - have `ld` add a .o to every .a that includes the `-L`/`-R`/-l` arguments given when constructing the .a (normally one does not do this when linking statically!) - have `ld` look in every .a found when doing a final link-edit to recursively find its dependencies, and, most importantly, - provide the same direct binding / versioned symbol semantics as in dynamic linking so that external symbols in dependents are resolved to the correct dependencies in the same way as in dynamic linking. Notionally this is quite simple. But adding this to the various ld implementations would probably be rather a lot of work. Still, if people insist on static linking, this work should be done.
- jcelerier 5y ago> For all those who keep harping on how static linking is better than dynamic linking, what I would suggest is that what must be done to make static linking not suck is to enrich .a files with the kinds of metadata that ELF adds to shared objects so we can get the same "list only direct dependencies" semantics when static linking as when dynamic linking. Or just use cmake which does that automatically
- wyldfire 5y ago> That something is rpath (also known as RUNPATH) Pretty sure that runpath and rpath are distinct and have slightly distinct behavior. Can't fault you much for making the mistake, though. The two are not given names to be easily distinguished.
- nemetroid 5y agoRPATH and RUNPATH are indeed different. However, I believe the -rpath flag is used to set either (--enable/disable-new-dtags determines which one is used), so the confusion is understandable.
- skainswoo 5y agoWhat is the distinction between them? I always thought they were the same thing.
- wyldfire 5y agoThe difference IIRC is the order they're used by the dynamic loader. One is prior to searching LD_LIBRARY_PATH, the other after.
- nemetroid 5y agoThere's also a difference for transitive dependencies. From the Linux manpage for ld.so, about RUNPATH: > Such directories are searched only to find those objects required by DT_NEEDED (direct dependencies) entries and do not apply to those objects' children, which must themselves have their own DT_RUNPATH entries. This is unlike DT_RPATH, which is applied to searches for all children in the dependency tree.
- stabbles 5y agoDepends on your libc. Search order: glibc: rpath > LD_LIBRARY_PATH > runpath > ld.so.cache > default paths. musl: LD_LIBRARY_PATH > rpath=runpath > default paths. Search path inheritance: glibc: rpaths are inherited: When exe depends on libx depends on liby, then liby first considers its own rpaths, then libx's rpaths, then exe's rpaths. HOWEVER if liby specifies runpath, it will not consider rpaths from parents. musl: rpaths and runpaths are the same and always inherited. I verified the glibc/musl sources when writing https://github.com/haampie/libtree https://github.com/haampie/libtree
- mangix 5y agoOr avoid all of this with static libraries :).
- Rucadi 5y agoThis is the only sane way... And for people saying "Buy you can't get security updates". I would rather have a dynamically-linked binary that includes all the dependences in it, at which you can upgrade the dependencies with a tool over the binary, than the madness of shared libraries in system paths. (Well, you kinda get that with appimage and similar).
- benreesman 5y agoOne of my favorite things about NixOS is how easy it makes it to not use shared libraries, AKA the most brain-damaged, over-used, needlessly complex, crime against computing in common use. There are good reasons to use LXC and namespaces and shit, but mostly Docker is a workaround for how fucking stupid dynamic linking is as a default. There are even use cases for .so, but as a default? Someone should be flogged. It was stupid when Sun pushed it with X in the 90s, and it’s stupider now. Also, gold/mold are worth trying in addition to lld. Depends on your software.
- ncmncm 5y agoHere I thought symbolic links were "the most brain-damaged, over-used, needlessly complex, crime against computing in common use." I am not making this up, ask Jeremy Allison, who is in a position to know. Using Nix without relying on symbolic links would be ... challenging.
- colordrops 5y agoWhat's the alternative?
- benreesman 5y agoNix/NixOS is an alternative, but as I've mentioned elsewhere in the thread it's got major, major downsides. Excluding that, you can go the Alpine/Docker route, which isn't terrible but isn't great. Or you can rebuild your box via Ansible or whatever every time you need a new AMI. This is actually a pretty reasonable solution.
- colordrops 5y agoWait, I thought you were criticizing Nix because it uses symbolic links. I originally used Absible but switched to Nix. I found ansible to be too idiosyncratic and brittle to maintain. It also isn't inherently idempotent though it's supposed to be used that way.
- colordrops 5y agoI haven't laughed out loud to a technical post in a long while, and this comment got me: "As this is NixOS, we are not going to barbarically install it globally"
- gigatexal 5y agoI've been admiring NixOS from afar but given the bending users have to do to use it in this particular use-case wouldn't it seem that NixOS should present in a transparent and seamless way libraries in these shell-environments the author creates for the build to surface the necessary libraries in such a way that "standard" tools like ld/ldd can find them? Shouldn't it be on the shoulders of NixOS to make this such that users of NixOS need not require patching tools or hacks etc?
- rkangel 5y agoNixOS changes some standard conventions of Linux filesystem layout and where tools can expect to find things. These are for good reasons and are due to the core of what NixOS is trying to achieve. For building most things those changes are relatively abstracted away (see ld wrapper script for details) and you don't have to know about them. Fiddling around with something low level like a linker is I think a forgivable situation where the abstraction leaks - it is a process that is inextricably linked to where the system puts things. My issue is the last sentence: So… .. turns out there’s more than one lld on NixOS. There’s pkgs.lld, the thing I have been using in the post. And then there’s pkgs.llvmPackages.bintools package, which also contains lld. And that version is actually wrapped into an rpath-setting shell script, the same way ld is. That means that there isn't a problem. NixOS has fixed this, the system works. Except that you have to magically know which package you should be using. This is the sort of problem that I run into with Nix - it's hard to know the correct incantation.
- gigatexal 5y agoThat's fair. Makes my point/post obsolete.