9 ms·

Unfortunately this is exactly the type of stuff that makes supporting commercial apps on linux a nightmare. Weird crashes due to weird linking of system librari

Unfortunately this is exactly the type of stuff that makes supporting commercial apps on linux a nightmare. Weird crashes due to weird linking of system libraries. Common distros are very adamant about dynamic linking everything in order to support the use case of "core library has vulnerability, upgrade it in place without rebuilding consuming apps." Along with a desire to avoid "dll hell" and force a single canonical version of every library systemwide. This leads to these sorts of issues. Windows gets around it by letting applications put the DLLs they care about beside the executable, and having it check there first by default.

gabcoh 3y ago
Can Linux not trivially do the same thing as windows with LD_PRELOAD? If so why is this more of an issue on Linux than Windows? Is it really less a technical challenge and more just a matter of Linux getting less support from upstream developers?
- bravetraveler 3y ago
  I was thinking/wondering this myself. Not to reinvent the wheel - more toss an idea around, but a 'venv for LD_PRELOAD' sounds like it'd deal with this pretty handily Not... in a way I'd use as a distribution/release maintainer. Probably as an administrator [of my LAN]
  - gabcoh 3y ago
    Such things already exist. Eg. Appimage or even docker.
    - bravetraveler 3y ago
      Right, but I don't really want to get into a distribution model - the hack suits me fine :) More an exercise in curiosity than anything Flatpak (or Snap, ew) probably deals with it fine today, Steam's there
    - lnxg33k1 3y ago
      and even that has been managed to be split between snap appimage and flatpak :D (sorry not meant to offend, long time linux day-to-day user here, but it was just ironic for me to point out fragmentation of fragmentation ^^)
  - 0x457 3y ago
    That's Nix with extra steps.
    - bravetraveler 3y ago
      I specifically said I'm not really trying to solution this, lol. More toying with the LD_PRELOAD aspect than anything Nix is neat, and I don't think I've used it enough to be too critical - but in some ways it feels like 'extra steps' I wanted to make a 'reproducible' installation (ala kickstart, not strictly binary)... but it felt very much like distribution work; declaring dependencies and the like
      - 0x457 3y ago
        Oh, nix is an extra mile. A lot could be improved, but that's what I'm using to deal with dependencies.
        bravetraveler 3y ago
        Gotcha, I don't feel so floundering now! I plan to spend more time with it, I see a lot of merit The amount of control is great, but the docs could use some work. For my simple goals (install Sway, Ansible, some other things) it was a broadsword when I need a butter knife
        0x457 3y ago
        What sold me on nix is home-manager and flakes: I can easily bootstrap my environment anywhere nix is available.
  - never_inline 3y ago
    There are tools which overwrite linked libraries, eg: chrpath.
- stabbles 3y ago
  LD_PRELOAD is too global to be useful, it's hard to scope it to one process (and not child processes). macOS is better in the sense that it clears DYLD_* variables when the dynamic linker has done its work and the process starts. (Although that can also be painful when you want to run a shell script and set DYLD_* outside)
  - nly 3y ago
    You can compile binaries with additional relative library paths in to them that will take priority over /usr/lib64
    - josephg 3y ago
      How? Maybe this should be better documented & recommended. I suppose at some point you're just statically linking with more steps - though for a problem like this it might be worth it.
      - xioxox 3y ago
        See the ELF rpath, which can be set by the linker. This can be modified after using patchelf.
      - rkeene2 3y ago
        $ORIGIN has been pretty well-known and documented for a very long time
      - yrro 3y ago
        https://sourceware.org/binutils/docs-2.40/ld/Options.html#index-runtime-library-search-path https://sourceware.org/binutils/docs-2.40/ld/Options.html#in...
      - ywei3410 3y ago
        The other comments have already covered the how, but I'd like to add that the mechanism used extensively in Nix [1]. [1] https://nixos.org/ https://nixos.org/
  - 3y ago
- aidenn0 3y ago
  This sounds like it's an interaction with the GPU driver though, which could also happen on windows...
- LispSporks22 3y ago
  It can be done by setting rpath to origin, even post compilation using the patchelf tool. Works great with C shared libraries. Perhaps ABI issues with C++ shared libs introduces other problems.
  - JonChesterfield 3y ago
    With the warning that rpath!=runpath, both are called rpath, and which you get depends on your linker and whether you also pass -Wl,--disable-new-dtags Runpath is the default, and also the one that is non-transitive and overridden by environment variables.
- admax88qqq 3y ago
  Yes linux _can_, the machinery is there, but culturally the common distros do not. And the defaults do not. On windows I can literally drop a DLL next to an executable and it will pick it up. On linux I have to do a wrapper script to set LD_PRELOAD, or mutate the binary's rpath to get it to load. It's not really a question of capability, but a question of culture and defaults that makes linux hard to support. Debian for example goes through great pains (or used to at least) to unbundle shared libraries such as openssl from projects like chromium.
doublepg23 3y ago
The funny thing is in on Fedora in 2023 I don't feel like I'm missing out on most software.
ho_schi 3y ago
Aehm. That is what a lot of closed-source applications do on Linux. And Valve does that, too. The open-source ones are maintained in the packing system and kept lean.
eikenberry 3y ago
Isn't this exactly the use case for which flatpaks are designed? Isn't Redhat/Fedora in the process of adopting them as the primary way to support third party/proprietary graphical apps like Steam? Doesn't the current Steam flatpak avoid this issue? TLDR; isn't this already addressed?
- MichaelZuo 3y ago
  It would appear so, I don't understand why this blog post is so popular.
nsajko 3y ago
Completely off base. If you want to distribute your application to users yourself (instead of letting the distro take care of that), then distribute all dependencies together with it.
- q3k 3y ago
  Which libGL.so should I be distributing alongside my application?
- marcthe12 3y ago
  There are a few dependencies that can not be easily vendored (At least not recommended). Mesa is probably the biggest example (An this case was caused by a mesa dependency). You can vendor them technically or even static link them but then you might end up with limited hardware support. The only alternative is to setup a mini opengl distro.
mjg59 3y ago
This is largely solved with approaches such as Flatpak or Snap, but graphics drivers are still an issue - they're expected to be supplied by the distribution, and components of them end up in-process in the application even if the rest of the application's runtime is shipped with the application. If there's an incompatibility between the application runtime and assumptions made by that driver code (as there appears to be in this case - TF2 ships its own malloc() implementation, but the graphics driver code ends up using it inconsistently and so blows up) then you're going to have problems. I don't think there's anything about Windows that would fundamentally change things here. Windows apps aren't shipping their own graphics drivers, even if they're bundling everything else.
- cesarb 3y ago
  > TF2 ships its own malloc() implementation, but the graphics driver code ends up using it inconsistently and so blows up) then you're going to have problems. I don't think there's anything about Windows that would fundamentally change things here. Yes, there is: on Windows, due to the way DLL linking works there, the graphics driver wouldn't use the malloc() implementation from TF2. The flat linking namespace in which you can globally replace the memory allocator for every dynamic library does not exist on Windows; if the graphics driver is linked to the memory allocator from the C library, it will get the memory allocator from that C library, not from some other DLL in the same process. That's not to say Windows is free of dynamic linking problems. While on Linux it's mostly only NSS and the graphics driver (and only when explicitly requested), on Windows it's common for unrelated third party software to inject DLLs and threads all over every process on the system. And it's not uncommon for these injected DLLs to do things like hooking into system DLLs (by overwriting the entry point of exported functions, or even internal functions), leading to hard-to-diagnose crashes when things are not like they expected.
  - eklitzke 3y ago
    Linux does not have a flat linking namespace (for example, see RTLD_NEXT in the dlsym man page). If it did, this wouldn't be a problem because everything would use the libc malloc or everything would use the tcmalloc implementation. It's just that glibc exports pretty much everything as a weak symbol, most programs and libraries that link against libc bind to whatever ld.so picks as the first in the search object search order, and messing with linker settings is the last thing most developers want to think about. I just read TFA so I haven't dived into all the details but my wild speculation is that the graphics driver is doing an aligned new and an unaligned delete. The alignment parameters in the new and delete give them different symbol names but you're free to implement them by dispatching to the unaligned version, so this wouldn't necessarily cause a problem (even in C++17 mode) if the malloc implementation doesn't actually handle them differently. However if tcmalloc was built for C++14 then it would only have the unaligned operators, and therefore the aligned new would resolve to the glibc implementation and the unaligned delete would resolve to the tcmalloc implementation, or something like that.
sebazzz 3y ago
> Unfortunately this is exactly the type of stuff that makes supporting commercial apps on linux a nightmare. Weird crashes due to weird linking of system libraries. That is the true reason containers were born, isn't it? The kernel is perfect, the public interface of the kernel is perfect. Userspace is a mess. Let's fix it by adding a layer between and have a userspace per application.

gabcoh 3y ago

Can Linux not trivially do the same thing as windows with LD_PRELOAD? If so why is this more of an issue on Linux than Windows? Is it really less a technical challenge and more just a matter of Linux getting less support from upstream developers?

bravetraveler 3y ago
I was thinking/wondering this myself. Not to reinvent the wheel - more toss an idea around, but a 'venv for LD_PRELOAD' sounds like it'd deal with this pretty handily Not... in a way I'd use as a distribution/release maintainer. Probably as an administrator [of my LAN]
- gabcoh 3y ago
  Such things already exist. Eg. Appimage or even docker.
  - bravetraveler 3y ago
    Right, but I don't really want to get into a distribution model - the hack suits me fine :) More an exercise in curiosity than anything Flatpak (or Snap, ew) probably deals with it fine today, Steam's there
  - lnxg33k1 3y ago
    and even that has been managed to be split between snap appimage and flatpak :D (sorry not meant to offend, long time linux day-to-day user here, but it was just ironic for me to point out fragmentation of fragmentation ^^)
- 0x457 3y ago
  That's Nix with extra steps.
  - bravetraveler 3y ago
    I specifically said I'm not really trying to solution this, lol. More toying with the LD_PRELOAD aspect than anything Nix is neat, and I don't think I've used it enough to be too critical - but in some ways it feels like 'extra steps' I wanted to make a 'reproducible' installation (ala kickstart, not strictly binary)... but it felt very much like distribution work; declaring dependencies and the like
    - 0x457 3y ago
      Oh, nix is an extra mile. A lot could be improved, but that's what I'm using to deal with dependencies.
      - bravetraveler 3y ago
        Gotcha, I don't feel so floundering now! I plan to spend more time with it, I see a lot of merit The amount of control is great, but the docs could use some work. For my simple goals (install Sway, Ansible, some other things) it was a broadsword when I need a butter knife
        0x457 3y ago
        What sold me on nix is home-manager and flakes: I can easily bootstrap my environment anywhere nix is available.
- never_inline 3y ago
  There are tools which overwrite linked libraries, eg: chrpath.
stabbles 3y ago
LD_PRELOAD is too global to be useful, it's hard to scope it to one process (and not child processes). macOS is better in the sense that it clears DYLD_* variables when the dynamic linker has done its work and the process starts. (Although that can also be painful when you want to run a shell script and set DYLD_* outside)
- nly 3y ago
  You can compile binaries with additional relative library paths in to them that will take priority over /usr/lib64
  - josephg 3y ago
    How? Maybe this should be better documented & recommended. I suppose at some point you're just statically linking with more steps - though for a problem like this it might be worth it.
    - xioxox 3y ago
      See the ELF rpath, which can be set by the linker. This can be modified after using patchelf.
    - rkeene2 3y ago
      $ORIGIN has been pretty well-known and documented for a very long time
    - yrro 3y ago
      https://sourceware.org/binutils/docs-2.40/ld/Options.html#index-runtime-library-search-path https://sourceware.org/binutils/docs-2.40/ld/Options.html#in...
    - ywei3410 3y ago
      The other comments have already covered the how, but I'd like to add that the mechanism used extensively in Nix [1]. [1] https://nixos.org/ https://nixos.org/
- 3y ago
aidenn0 3y ago
This sounds like it's an interaction with the GPU driver though, which could also happen on windows...
LispSporks22 3y ago
It can be done by setting rpath to origin, even post compilation using the patchelf tool. Works great with C shared libraries. Perhaps ABI issues with C++ shared libs introduces other problems.
- JonChesterfield 3y ago
  With the warning that rpath!=runpath, both are called rpath, and which you get depends on your linker and whether you also pass -Wl,--disable-new-dtags Runpath is the default, and also the one that is non-transitive and overridden by environment variables.
admax88qqq 3y ago
Yes linux _can_, the machinery is there, but culturally the common distros do not. And the defaults do not. On windows I can literally drop a DLL next to an executable and it will pick it up. On linux I have to do a wrapper script to set LD_PRELOAD, or mutate the binary's rpath to get it to load. It's not really a question of capability, but a question of culture and defaults that makes linux hard to support. Debian for example goes through great pains (or used to at least) to unbundle shared libraries such as openssl from projects like chromium.

doublepg23 3y ago

The funny thing is in on Fedora in 2023 I don't feel like I'm missing out on most software.

ho_schi 3y ago

Aehm. That is what a lot of closed-source applications do on Linux. And Valve does that, too. The open-source ones are maintained in the packing system and kept lean.

eikenberry 3y ago

Isn't this exactly the use case for which flatpaks are designed? Isn't Redhat/Fedora in the process of adopting them as the primary way to support third party/proprietary graphical apps like Steam? Doesn't the current Steam flatpak avoid this issue? TLDR; isn't this already addressed?

MichaelZuo 3y ago
It would appear so, I don't understand why this blog post is so popular.

nsajko 3y ago

Completely off base. If you want to distribute your application to users yourself (instead of letting the distro take care of that), then distribute all dependencies together with it.

q3k 3y ago
Which libGL.so should I be distributing alongside my application?
marcthe12 3y ago
There are a few dependencies that can not be easily vendored (At least not recommended). Mesa is probably the biggest example (An this case was caused by a mesa dependency). You can vendor them technically or even static link them but then you might end up with limited hardware support. The only alternative is to setup a mini opengl distro.

mjg59 3y ago

This is largely solved with approaches such as Flatpak or Snap, but graphics drivers are still an issue - they're expected to be supplied by the distribution, and components of them end up in-process in the application even if the rest of the application's runtime is shipped with the application. If there's an incompatibility between the application runtime and assumptions made by that driver code (as there appears to be in this case - TF2 ships its own malloc() implementation, but the graphics driver code ends up using it inconsistently and so blows up) then you're going to have problems. I don't think there's anything about Windows that would fundamentally change things here. Windows apps aren't shipping their own graphics drivers, even if they're bundling everything else.

cesarb 3y ago
> TF2 ships its own malloc() implementation, but the graphics driver code ends up using it inconsistently and so blows up) then you're going to have problems. I don't think there's anything about Windows that would fundamentally change things here. Yes, there is: on Windows, due to the way DLL linking works there, the graphics driver wouldn't use the malloc() implementation from TF2. The flat linking namespace in which you can globally replace the memory allocator for every dynamic library does not exist on Windows; if the graphics driver is linked to the memory allocator from the C library, it will get the memory allocator from that C library, not from some other DLL in the same process. That's not to say Windows is free of dynamic linking problems. While on Linux it's mostly only NSS and the graphics driver (and only when explicitly requested), on Windows it's common for unrelated third party software to inject DLLs and threads all over every process on the system. And it's not uncommon for these injected DLLs to do things like hooking into system DLLs (by overwriting the entry point of exported functions, or even internal functions), leading to hard-to-diagnose crashes when things are not like they expected.
- eklitzke 3y ago
  Linux does not have a flat linking namespace (for example, see RTLD_NEXT in the dlsym man page). If it did, this wouldn't be a problem because everything would use the libc malloc or everything would use the tcmalloc implementation. It's just that glibc exports pretty much everything as a weak symbol, most programs and libraries that link against libc bind to whatever ld.so picks as the first in the search object search order, and messing with linker settings is the last thing most developers want to think about. I just read TFA so I haven't dived into all the details but my wild speculation is that the graphics driver is doing an aligned new and an unaligned delete. The alignment parameters in the new and delete give them different symbol names but you're free to implement them by dispatching to the unaligned version, so this wouldn't necessarily cause a problem (even in C++17 mode) if the malloc implementation doesn't actually handle them differently. However if tcmalloc was built for C++14 then it would only have the unaligned operators, and therefore the aligned new would resolve to the glibc implementation and the unaligned delete would resolve to the tcmalloc implementation, or something like that.

sebazzz 3y ago

> Unfortunately this is exactly the type of stuff that makes supporting commercial apps on linux a nightmare. Weird crashes due to weird linking of system libraries. That is the true reason containers were born, isn't it? The kernel is perfect, the public interface of the kernel is perfect. Userspace is a mess. Let's fix it by adding a layer between and have a userspace per application.