5 ms·

I was confused on what $ORIGIN means, so, macroexpanding the article a bit https://fzakaria.com/2026/06/21/nix-needs-relocatable-binaries https://fzakaria.com/

I was confused on what $ORIGIN means, so, macroexpanding the article a bit https://fzakaria.com/2026/06/21/nix-needs-relocatable-binaries https://fzakaria.com/2026/06/21/nix-needs-relocatable-binari... > The loader in Linux however natively supports the variable $ORIGIN which translates to “the directory containing the executable.” https://man7.org/linux/man-pages/man8/ld.so.8.html https://man7.org/linux/man-pages/man8/ld.so.8.html But, if ld.so supports $ORIGIN already, why does the kernel needs to support it also? Or rather, why can't the kernel leverage ld.so and do this entirely on userspace?

jz391 3mo ago
One point mentioned in the first link you included is support of $ORIGIN in #! scripts - that would need kernel support.
- setheron 3mo ago
  This also can be handled by binfmt_misc; I include a demo of it in a gist.
dgrunwald 3mo ago
$ORIGIN was only supported when the loader was looking for other dependencies. The loader itself (field PT_INTERP) is loaded by the kernel. So prior to this change, every program must hardcode the absolute path to ld.so. With support for an $ORIGIN-relative loader, each program could use its own copy of ld.so.
- charcircuit 3mo ago
  Each program having its own loader is an anti pattern. Such a requirement is overkill. You can have a single loader that supports everything on the system.
  - roblabla 3mo ago
    Not with NixOS. ld.so is tied to a version of glibc, in ways that can be subtly incompatible. And nixos can have multiple glibc version installed on a single machine. Besides, it allows for upgrades/downgrades to be done in a way that's much less error-prone.
    - charcircuit 3mo ago
      >in ways that can be subtly incompatible There is much more value to be had in making glibc properly backward compatible so you can have a single one that can be used with everything than trying to make it so that you can swap everything around, creating extra complexity and compatibility risks.
      - roblabla 3mo ago
        How is that more valuable? It comes with two big pitfalls: - It would still not allow downgrades to work properly - It would cause glibc/ld.so to have a harder time adding new features, as they now need to worry about incompatible versions being used together Meanwhile, having different ld.so has many good use-cases, like simplifying development of ld.so itself, allowing them to swapped during updates in ways that are safer, etc... And the eBPF binfmt support is a rather simple, generic mechanism that is likely to have many other use-cases beyond ld.so. So it's overall a pretty good resolution to the issue?
        charcircuit 3mo ago
        >It would still not allow downgrades to work properly If you really need this, then there are options like swapping the linker with an older version or making a hardcoded linker that now points to an older one. >It would cause glibc/ld.so to have a harder time adding new features, as they now need to worry about incompatible versions being used together Every language runtime has to care about not breaking compatibility with apps that have been released already. This is not a new or unique problem.
      - dspillett 3mo ago
        > […] in making glibc properly backward compatible […] You could only do that going forward though, and would be stuck, at least for a time, with the historic versions that still need extra handling. In an ideal world that wouldn't be needed, but in an ideal world you'd not need multiple versions of glibc at all so we aren't there. Support in the kernel means that it will work even if applied to old packages that for some other reason need a held-back version of glibc. As mentioned elsewhere it also gives the feature to #! directives in scripts too.
  - HexDecOctBin 3mo ago
    Unfortunately, Glibc and loader are linked together intrinsically in the Linux ecosystem. So, if you want to be able to launch a program reliably, shipping your own loader and libc might actually be the only way.
    - VorpalWay 3mo ago
      Glibc is backward compatible though, they even have symbol versioning to provide multiple versions of the same symbol. So as long as you have the same or a newer version of glibc (than what was built against) you should be good to go. And I don't remember hearing about breakages for this. Other libraries on the system is far more hit and miss, but glibc is quite compatible. The other way around is harder though, you can't take a program built against a newer glibc and run it on an older version. So you generally need to spin up a container with some LTS distro and build your binary in it if you want it to be maximally compatible. (However, zig apparently is able to deal with this by shipping a mapping between glibc versions and symbol versions and doing the linking themselves. You can even use zig to link rust code using cargo-zigbuild and get that benefit.) But if you want to be maximally portable: static linking against musl. Though beware that many things are slower in musl, such as the allocator.
      - Joker_vD 3mo ago
        No, glibc isn't backwards compatible; I've had instances when the loader would refuse to load the executable because the installed glibc is too new for it.
        bonzini 3mo ago
        That's not supposed to happen. I would like to have more info.
      - HexDecOctBin 3mo ago
        Glibc is the opposite of backwards compatible. This thread I was a part of explains some issues faced in the past: https://news.ycombinator.com/item?id=47029789 https://news.ycombinator.com/item?id=47029789
  - kazinator 3mo ago
    If I had to choose between 16 dockers, or 16 processes with different ld.so, guess what I'm choosing ...
reinitctxoffset 3mo ago
You have the right idea but the wrong specifics. The boundary you're alluding to isn't the kernel/userspace boundary, it's the `libc` boundary, which is admittedly privileged by convention if not by Ring 0. On most Linux systems this is regrettably `glibc` (in a container where you get to choose everyone chooses the superior option of `musl`), on all Darwin systems this is `libstandard`. This thread is more concerned with Linux, so you are probably dealing with `/lib64/ld-linux-x86-64.so.2`, which operates in userspace but is by convention and opacity quite clearly part of "the system". The kernel modification is a much bigger, much weirder side effect of a weird Nix loyalty test around shebang lines in shell scripts, on which is has opted to be intentionally and violently incompatible with everything for no benefit other than incompatibility.
- seanhunter 3mo ago
  That’s a weird rant which is manifestly false. On Nix the benefit you get from the incompatibility is the system can roll forward and back while it’s running so you can do big changes (eg updating or rolling back) the entire system without affecting running processes. You can also run groups of processes in isolation with different sets (or versions) of packages from the rest of the system. It’s fair to argue about whether those things are really important or whether nix does this in the best way, but to claim that the only benefit is incompatibility is just obvious nonsense.
  - reinitctxoffset 3mo ago
    The privileged path is already there. `/usr/bin/env` is no different from `/bin/bash`, `/bin/sh`, or any other ELF artifact at a known place. The argument is made, the argument is spurious. It is made in the other direction regarding the driver run path, just as religious, opposite ruling from purity court. No one even knows why, the trail goes cold in a mysterious 2012 commit about Mesa, it's literally a performance to bring the plane cargo back. Receipts for claim: #141803. Disagreeing with you doesn't make something a rant. If it's not clear how deep my Nix expertise is I will demonstrate it to any level you like: I'm just as entitled to an opinion as you are and I would appreciate it if the nixpkgs community was a little less rude to anyone who disagrees with some dogma no matter their knowledge. The nixpkgs community is not regarded as healthy or friendly after an internecine faction war that split three ways twice, producing four implementations none of which work, and if that reputation is ever going to mend, it will be because less than every Nix person has precisely zero chill the nanosecond anyone disagrees with them about anything.
matheusmoreira 3mo ago
> why can't the kernel leverage ld.so and do this entirely on userspace? Because Linux is completely independent from its user space. There is no "the loader in Linux", it just happens that GNU wrote one and it's widely used.

jz391 3mo ago

One point mentioned in the first link you included is support of $ORIGIN in #! scripts - that would need kernel support.

setheron 3mo ago
This also can be handled by binfmt_misc; I include a demo of it in a gist.

dgrunwald 3mo ago

$ORIGIN was only supported when the loader was looking for other dependencies. The loader itself (field PT_INTERP) is loaded by the kernel. So prior to this change, every program must hardcode the absolute path to ld.so. With support for an $ORIGIN-relative loader, each program could use its own copy of ld.so.

charcircuit 3mo ago
Each program having its own loader is an anti pattern. Such a requirement is overkill. You can have a single loader that supports everything on the system.
- roblabla 3mo ago
  Not with NixOS. ld.so is tied to a version of glibc, in ways that can be subtly incompatible. And nixos can have multiple glibc version installed on a single machine. Besides, it allows for upgrades/downgrades to be done in a way that's much less error-prone.
  - charcircuit 3mo ago
    >in ways that can be subtly incompatible There is much more value to be had in making glibc properly backward compatible so you can have a single one that can be used with everything than trying to make it so that you can swap everything around, creating extra complexity and compatibility risks.
    - roblabla 3mo ago
      How is that more valuable? It comes with two big pitfalls: - It would still not allow downgrades to work properly - It would cause glibc/ld.so to have a harder time adding new features, as they now need to worry about incompatible versions being used together Meanwhile, having different ld.so has many good use-cases, like simplifying development of ld.so itself, allowing them to swapped during updates in ways that are safer, etc... And the eBPF binfmt support is a rather simple, generic mechanism that is likely to have many other use-cases beyond ld.so. So it's overall a pretty good resolution to the issue?
      - charcircuit 3mo ago
        >It would still not allow downgrades to work properly If you really need this, then there are options like swapping the linker with an older version or making a hardcoded linker that now points to an older one. >It would cause glibc/ld.so to have a harder time adding new features, as they now need to worry about incompatible versions being used together Every language runtime has to care about not breaking compatibility with apps that have been released already. This is not a new or unique problem.
    - dspillett 3mo ago
      > […] in making glibc properly backward compatible […] You could only do that going forward though, and would be stuck, at least for a time, with the historic versions that still need extra handling. In an ideal world that wouldn't be needed, but in an ideal world you'd not need multiple versions of glibc at all so we aren't there. Support in the kernel means that it will work even if applied to old packages that for some other reason need a held-back version of glibc. As mentioned elsewhere it also gives the feature to #! directives in scripts too.
- HexDecOctBin 3mo ago
  Unfortunately, Glibc and loader are linked together intrinsically in the Linux ecosystem. So, if you want to be able to launch a program reliably, shipping your own loader and libc might actually be the only way.
  - VorpalWay 3mo ago
    Glibc is backward compatible though, they even have symbol versioning to provide multiple versions of the same symbol. So as long as you have the same or a newer version of glibc (than what was built against) you should be good to go. And I don't remember hearing about breakages for this. Other libraries on the system is far more hit and miss, but glibc is quite compatible. The other way around is harder though, you can't take a program built against a newer glibc and run it on an older version. So you generally need to spin up a container with some LTS distro and build your binary in it if you want it to be maximally compatible. (However, zig apparently is able to deal with this by shipping a mapping between glibc versions and symbol versions and doing the linking themselves. You can even use zig to link rust code using cargo-zigbuild and get that benefit.) But if you want to be maximally portable: static linking against musl. Though beware that many things are slower in musl, such as the allocator.
    - Joker_vD 3mo ago
      No, glibc isn't backwards compatible; I've had instances when the loader would refuse to load the executable because the installed glibc is too new for it.
      - bonzini 3mo ago
        That's not supposed to happen. I would like to have more info.
    - HexDecOctBin 3mo ago
      Glibc is the opposite of backwards compatible. This thread I was a part of explains some issues faced in the past: https://news.ycombinator.com/item?id=47029789 https://news.ycombinator.com/item?id=47029789
- kazinator 3mo ago
  If I had to choose between 16 dockers, or 16 processes with different ld.so, guess what I'm choosing ...

reinitctxoffset 3mo ago

You have the right idea but the wrong specifics. The boundary you're alluding to isn't the kernel/userspace boundary, it's the `libc` boundary, which is admittedly privileged by convention if not by Ring 0. On most Linux systems this is regrettably `glibc` (in a container where you get to choose everyone chooses the superior option of `musl`), on all Darwin systems this is `libstandard`. This thread is more concerned with Linux, so you are probably dealing with `/lib64/ld-linux-x86-64.so.2`, which operates in userspace but is by convention and opacity quite clearly part of "the system". The kernel modification is a much bigger, much weirder side effect of a weird Nix loyalty test around shebang lines in shell scripts, on which is has opted to be intentionally and violently incompatible with everything for no benefit other than incompatibility.

seanhunter 3mo ago
That’s a weird rant which is manifestly false. On Nix the benefit you get from the incompatibility is the system can roll forward and back while it’s running so you can do big changes (eg updating or rolling back) the entire system without affecting running processes. You can also run groups of processes in isolation with different sets (or versions) of packages from the rest of the system. It’s fair to argue about whether those things are really important or whether nix does this in the best way, but to claim that the only benefit is incompatibility is just obvious nonsense.
- reinitctxoffset 3mo ago
  The privileged path is already there. `/usr/bin/env` is no different from `/bin/bash`, `/bin/sh`, or any other ELF artifact at a known place. The argument is made, the argument is spurious. It is made in the other direction regarding the driver run path, just as religious, opposite ruling from purity court. No one even knows why, the trail goes cold in a mysterious 2012 commit about Mesa, it's literally a performance to bring the plane cargo back. Receipts for claim: #141803. Disagreeing with you doesn't make something a rant. If it's not clear how deep my Nix expertise is I will demonstrate it to any level you like: I'm just as entitled to an opinion as you are and I would appreciate it if the nixpkgs community was a little less rude to anyone who disagrees with some dogma no matter their knowledge. The nixpkgs community is not regarded as healthy or friendly after an internecine faction war that split three ways twice, producing four implementations none of which work, and if that reputation is ever going to mend, it will be because less than every Nix person has precisely zero chill the nanosecond anyone disagrees with them about anything.

matheusmoreira 3mo ago

> why can't the kernel leverage ld.so and do this entirely on userspace? Because Linux is completely independent from its user space. There is no "the loader in Linux", it just happens that GNU wrote one and it's widely used.