4 ms·

The xz debacle made me keenly aware of what a weak point the linker is, and now I kinda want a system that doesn't have a dynamic linker at all.

by kibwen 3y ago

klysm 3y ago
I’ve been convinced static linking is the way to go for a number of years. The benefits of dynamic linking don’t make sense anymore imo
- intelVISA 3y ago
  Awoken
- tssge 3y ago
  On single user systems, sure. However multi user systems can benefit greatly from dynamic linking; some highly multi user systems become impossible to create at all with only static linking (RAM demand becomes too great). Dynamic linking also leads to better CPU cache utilization and thus higher performance; provided if multiple applications are actually utilizing the dynamically linked library. While RAM is somewhat abundant these days, L1/L2/L3 cache on your CPU most definitely isn't. Application startup times are also of higher performance when part of the code is already in memory. EDIT: Let me elaborate a bit on what I mean by "benefiting greatly" with a real world example. I have a system with around 300 tenants with a minimum of 500 PHP processes online at any given time. A single PHP process is able to load 70MB worth of dynamically linked libraries (PHP extensions) if so chosen by the tenant. I'll just skip libc and other such libraries to give static linking a chance. With dynamic linking, the maximum memory requirement is 70MB for said extensions. With static linking, the maximum memory requirement is 35GB for said extensions. Dynamic linking thus leads to 99,8% cost saving on memory usage. Bear in mind this is at minimum load: static linking requirements will scale linearly with load. Not to even get started on the performance benefits this brings...
  - gpm 3y ago
    Are these extensions standard across your users, or are they user supplied? If the former is the case, in a world where dynamic linking wasn't "the way things were done", you'd just just statically link them all into the PHP interpreters binary, and then only run the code to actually start them for the users that want them. Linux shares memory between the different instances of the same executable the same way it does between different instances of a shared library, there should be no difference in cost. Today, I doubt php supports this, particularly the "only run the code to actually start them" part, but in principle it's a lot simpler than dynamic linking to do that. If the latter is the case, how are you getting any savings today?
    - tssge 3y ago
      >Are these extensions standard across your users, or are they user supplied? These extensions are standard PHP extensions, so they are the same for each process and user. Extensions like MySQL driver, Postgres driver, ImageMagick bindings and such. You are correct that no savings would be possible if these were different across each user.
      - a1369209993 3y ago
        > These extensions are standard PHP extensions, so they are the same for each process and user. Then with static linking, the maximum memory requirement is 70MB for said extensions (statically linked into the PHP executable).
        tssge 3y ago
        Are you basically saying there is no difference in memory usage when comparing static linking and dynamic linking? Any links to more info on the matter? I tested this on the machine with a static executable I compiled it looks to me that memory is not shared when statically linked. I thought the point of dynamic linking anyways was to save memory by sharing it (and disk space, though these days we have more than enough for executable code). Though it could be just that I misunderstand the output of smaps_rollup.
- josephg 3y ago
  I go back and forth on this. On MacOS, I do like that the UI libraries for all applications are shipped with the OS. When the OS updates, the UI for all installed programs updates to include the new look and feel. Apple can also update them to change the way that applications render to the screen. These libraries are quite large, and they probably shouldn't be shipped with every application. If they were, it would make it a lot harder to compile an app to work with multiple versions of macos. Apps would probably break a lot more when the OS updates. For shell scripts and such, I agree and I'd be happier if everything was statically linked. My biggest criticism of dynamic linking is how ridiculous the situation is on debian and similar OSes. Imagine if I make some software with rust (or javascript or python or something). My software might pull in 20-100 dependencies from cargo, npm, pip, etc. If I want my software to appear in apt, the debian maintainers insist on adding a mirror of all of those dependencies into apt. So apt sort of maintains a crappy, out of date mirror of a bunch of other language-specific package managers. And as far as I know, they do this work by hand - adding misery to misery. Nobody wants that. Least of all the software developers - who will get bug reports for their own software compiled with mysteriously out-of-date dependencies that they aren't testing themselves. Its horrible.
  - AnonymousPlanet 3y ago
    I don't understand what you think pip, npm etc. have to do with dynamic linking. Those libraries get loaded in the respective interpreter, the linker doesn't come into action at all. And security wise npm, pip and the likes are nightmares waiting to be exploited xz style. No static linking will save you from it.
    - josephg 3y ago
      It maps. If you throw a nodejs or python package in apt, you can either include all of its js / python dependencies in the package itself or make separate apt packages for each of the projects dependencies and have the apt package depend on those packages. The difference is pretty similar to static / dynamic linking. Including the dependencies gives the package author explicit control over the versions of all your dependencies. But depending on external apt packages means you can install security updates to shared libraries system wide. (Think log4j or xz). And as I understand it, that is what Debian prefers. In JavaScript projects it’s even become common practice to use a bundler to “compile” server side JavaScript (with all your dependencies) into a single large unreadable .js file. Doing that can reduce memory usage and dramatically improve startup time. The bundled code usually has dead code elimination. It’s more or less identical to static linking - except the resulting artifact is another JavaScript file.
singron 3y ago
The linker is a convenient way to get your code to run in the right place, but the linker runs in userspace, and your code can do anything the linker does. E.g. you can scan function tables, remap pages r/w, and interpose functions. The OpenBSD folks have some good ideas against these issues, although they are targeted at things like ROP exploits. E.g. pledge, which disables certain syscalls, and immutable memory maps (https://marc.info/?l=openbsd-tech&m=166203784715942 https://marc.info/?l=openbsd-tech&m=166203784715942). A really scary version of the xz backdoor is if it had an entire implementation of openssh in its own ctor function. Then it wouldn't even need to interpose anything. It's way harder to get rid of ctors.
- marcosdumay 3y ago
  > E.g. you can scan function tables, remap pages r/w, and interpose functions. Your code doesn't start doing those things without notice due to external interference. But a linker will add them, on your behalf, without raising any flags.
jiveturkey 3y ago
Could you expand on how xz took advantage of the dynamic loader (I think that's why you most likely meant)? If you built your binary statically against the compromised version of xz, you'd still suffer wouldn't you? Or did it depend on dynamic linking in some way? I'm not really that familiar with the mechanism it used, so sorry if that's a basic question. So, eg an entire distro built statically would still have been fully compromised. And without dynamic libs, you wouldn't be able to just update the single lib, you'd have to update everything. So to the extent your own software is statically linked, your own software may be fine (because you never picked up the new update) but you are still vulnerable if the statically linked system had the bad lib linked in.
- jcranmer 3y ago
  The xz hack used an ifunc resolver to load its payload, where the ifunc resolver is a function that the dynamic loader calls to figure out what the address of a symbol should be (i.e., to make something dependent on the current hardware capabilities).
cryptonector 3y ago
You might want static linking, but static link semantics are still stuck in 1978. We really need to teach the static linker a bunch of tricks that the dynamic linker knows. Also, as u/singron notes, a debugger can do to a statically linked executable everything that TFA's unicorns.so can do. Static linking only makes that a wee bit harder, not that much harder, and recall that once a unicorns executable is published then the fact that static linking made it harder up to that time becomes completely irrelevant.
- Dwedit 3y ago
  Meanwhile in Windows land, "static linking" is very much a thing, but it means you dynamically link only to the basic Windows API libraries (Kernel32, User32, Gdi32, Ole32, Advapi32, etc...). You can't have a windows program that doesn't have some way to import API functions. System call numbers are shuffled with every version change of Windows.
- marcosdumay 3y ago
  > You might want static linking Personally, I really want to go the other way around, with microkernel-like userspace services. That way everybody can drop the pretense that the software author did any verification on his dependencies, and we can add real access controls to the ABI. But yeah, the current situation is close to the worst possible one. It would improve if we change things on any direction at all.
  - cryptonector 3y ago
    > Personally, I really want to go the other way around, with microkernel-like userspace services. Yes! This requires very small protocols. Indeed, systemd has such a tiny protocol, and sshd did not need to link to libsystemd to use that protocol, but linking with libsystemd was what Debian had chosen to do.
    - marcosdumay 3y ago
      Except that when only systemd does it, everybody still keeps the pretense and nobody works on security. And the systemd protocol is made in a way completely unfit for using everywhere. (Not by systemd's fault.)
theamk 3y ago
The general idea works fine without static linker though... Just replace LD_PRELOAD manipulation with PATH manipulation, and create wrapper binaries for common apps. Slightly more detectable, but not by much.
rurban 3y ago
You are confusing the loader with the linker. The loader is the problem, always was. Just think of ldd exploits
thenonameguy 3y ago
NixOS/GNU Guix is uniquely positioned in this area, as it tracks all dependencies explicitly, with exact versions. If there is a paradigm where this can be achieved on the OS level, they are the closest to it today. See this related talk from NixOS 2022: https://www.youtube.com/watch?v=HZKFe4mCkr4 https://www.youtube.com/watch?v=HZKFe4mCkr4
nottorp 3y ago
Two different "philosophical" answers to this: 1. Patching known holes after the fact is just an arm's race that will never end. Better to do secure foundations. But that leads to 2: 2. We can probably build a software environment that is a lot more secure than what we have now, but no one could afford to pay for building it, and possibly no one could afford the operating costs to use it.

The xz debacle made me keenly aware of what a weak point the linker is, and now I kinda want a system that doesn't have a dynamic linker at all.

by kibwen 3y ago

The xz debacle made me keenly aware of what a weak point the linker is, and now I kinda want a system that doesn't have a dynamic linker at all.

klysm 3y ago

I’ve been convinced static linking is the way to go for a number of years. The benefits of dynamic linking don’t make sense anymore imo

intelVISA 3y ago
Awoken
tssge 3y ago
On single user systems, sure. However multi user systems can benefit greatly from dynamic linking; some highly multi user systems become impossible to create at all with only static linking (RAM demand becomes too great). Dynamic linking also leads to better CPU cache utilization and thus higher performance; provided if multiple applications are actually utilizing the dynamically linked library. While RAM is somewhat abundant these days, L1/L2/L3 cache on your CPU most definitely isn't. Application startup times are also of higher performance when part of the code is already in memory. EDIT: Let me elaborate a bit on what I mean by "benefiting greatly" with a real world example. I have a system with around 300 tenants with a minimum of 500 PHP processes online at any given time. A single PHP process is able to load 70MB worth of dynamically linked libraries (PHP extensions) if so chosen by the tenant. I'll just skip libc and other such libraries to give static linking a chance. With dynamic linking, the maximum memory requirement is 70MB for said extensions. With static linking, the maximum memory requirement is 35GB for said extensions. Dynamic linking thus leads to 99,8% cost saving on memory usage. Bear in mind this is at minimum load: static linking requirements will scale linearly with load. Not to even get started on the performance benefits this brings...
- gpm 3y ago
  Are these extensions standard across your users, or are they user supplied? If the former is the case, in a world where dynamic linking wasn't "the way things were done", you'd just just statically link them all into the PHP interpreters binary, and then only run the code to actually start them for the users that want them. Linux shares memory between the different instances of the same executable the same way it does between different instances of a shared library, there should be no difference in cost. Today, I doubt php supports this, particularly the "only run the code to actually start them" part, but in principle it's a lot simpler than dynamic linking to do that. If the latter is the case, how are you getting any savings today?
  - tssge 3y ago
    >Are these extensions standard across your users, or are they user supplied? These extensions are standard PHP extensions, so they are the same for each process and user. Extensions like MySQL driver, Postgres driver, ImageMagick bindings and such. You are correct that no savings would be possible if these were different across each user.
    - a1369209993 3y ago
      > These extensions are standard PHP extensions, so they are the same for each process and user. Then with static linking, the maximum memory requirement is 70MB for said extensions (statically linked into the PHP executable).
      - tssge 3y ago
        Are you basically saying there is no difference in memory usage when comparing static linking and dynamic linking? Any links to more info on the matter? I tested this on the machine with a static executable I compiled it looks to me that memory is not shared when statically linked. I thought the point of dynamic linking anyways was to save memory by sharing it (and disk space, though these days we have more than enough for executable code). Though it could be just that I misunderstand the output of smaps_rollup.
josephg 3y ago
I go back and forth on this. On MacOS, I do like that the UI libraries for all applications are shipped with the OS. When the OS updates, the UI for all installed programs updates to include the new look and feel. Apple can also update them to change the way that applications render to the screen. These libraries are quite large, and they probably shouldn't be shipped with every application. If they were, it would make it a lot harder to compile an app to work with multiple versions of macos. Apps would probably break a lot more when the OS updates. For shell scripts and such, I agree and I'd be happier if everything was statically linked. My biggest criticism of dynamic linking is how ridiculous the situation is on debian and similar OSes. Imagine if I make some software with rust (or javascript or python or something). My software might pull in 20-100 dependencies from cargo, npm, pip, etc. If I want my software to appear in apt, the debian maintainers insist on adding a mirror of all of those dependencies into apt. So apt sort of maintains a crappy, out of date mirror of a bunch of other language-specific package managers. And as far as I know, they do this work by hand - adding misery to misery. Nobody wants that. Least of all the software developers - who will get bug reports for their own software compiled with mysteriously out-of-date dependencies that they aren't testing themselves. Its horrible.
- AnonymousPlanet 3y ago
  I don't understand what you think pip, npm etc. have to do with dynamic linking. Those libraries get loaded in the respective interpreter, the linker doesn't come into action at all. And security wise npm, pip and the likes are nightmares waiting to be exploited xz style. No static linking will save you from it.
  - josephg 3y ago
    It maps. If you throw a nodejs or python package in apt, you can either include all of its js / python dependencies in the package itself or make separate apt packages for each of the projects dependencies and have the apt package depend on those packages. The difference is pretty similar to static / dynamic linking. Including the dependencies gives the package author explicit control over the versions of all your dependencies. But depending on external apt packages means you can install security updates to shared libraries system wide. (Think log4j or xz). And as I understand it, that is what Debian prefers. In JavaScript projects it’s even become common practice to use a bundler to “compile” server side JavaScript (with all your dependencies) into a single large unreadable .js file. Doing that can reduce memory usage and dramatically improve startup time. The bundled code usually has dead code elimination. It’s more or less identical to static linking - except the resulting artifact is another JavaScript file.

singron 3y ago

The linker is a convenient way to get your code to run in the right place, but the linker runs in userspace, and your code can do anything the linker does. E.g. you can scan function tables, remap pages r/w, and interpose functions. The OpenBSD folks have some good ideas against these issues, although they are targeted at things like ROP exploits. E.g. pledge, which disables certain syscalls, and immutable memory maps (https://marc.info/?l=openbsd-tech&m=166203784715942 https://marc.info/?l=openbsd-tech&m=166203784715942). A really scary version of the xz backdoor is if it had an entire implementation of openssh in its own ctor function. Then it wouldn't even need to interpose anything. It's way harder to get rid of ctors.

marcosdumay 3y ago
> E.g. you can scan function tables, remap pages r/w, and interpose functions. Your code doesn't start doing those things without notice due to external interference. But a linker will add them, on your behalf, without raising any flags.

jiveturkey 3y ago

Could you expand on how xz took advantage of the dynamic loader (I think that's why you most likely meant)? If you built your binary statically against the compromised version of xz, you'd still suffer wouldn't you? Or did it depend on dynamic linking in some way? I'm not really that familiar with the mechanism it used, so sorry if that's a basic question. So, eg an entire distro built statically would still have been fully compromised. And without dynamic libs, you wouldn't be able to just update the single lib, you'd have to update everything. So to the extent your own software is statically linked, your own software may be fine (because you never picked up the new update) but you are still vulnerable if the statically linked system had the bad lib linked in.

jcranmer 3y ago
The xz hack used an ifunc resolver to load its payload, where the ifunc resolver is a function that the dynamic loader calls to figure out what the address of a symbol should be (i.e., to make something dependent on the current hardware capabilities).

cryptonector 3y ago

You might want static linking, but static link semantics are still stuck in 1978. We really need to teach the static linker a bunch of tricks that the dynamic linker knows. Also, as u/singron notes, a debugger can do to a statically linked executable everything that TFA's unicorns.so can do. Static linking only makes that a wee bit harder, not that much harder, and recall that once a unicorns executable is published then the fact that static linking made it harder up to that time becomes completely irrelevant.

Dwedit 3y ago
Meanwhile in Windows land, "static linking" is very much a thing, but it means you dynamically link only to the basic Windows API libraries (Kernel32, User32, Gdi32, Ole32, Advapi32, etc...). You can't have a windows program that doesn't have some way to import API functions. System call numbers are shuffled with every version change of Windows.
marcosdumay 3y ago
> You might want static linking Personally, I really want to go the other way around, with microkernel-like userspace services. That way everybody can drop the pretense that the software author did any verification on his dependencies, and we can add real access controls to the ABI. But yeah, the current situation is close to the worst possible one. It would improve if we change things on any direction at all.
- cryptonector 3y ago
  > Personally, I really want to go the other way around, with microkernel-like userspace services. Yes! This requires very small protocols. Indeed, systemd has such a tiny protocol, and sshd did not need to link to libsystemd to use that protocol, but linking with libsystemd was what Debian had chosen to do.
  - marcosdumay 3y ago
    Except that when only systemd does it, everybody still keeps the pretense and nobody works on security. And the systemd protocol is made in a way completely unfit for using everywhere. (Not by systemd's fault.)

theamk 3y ago

The general idea works fine without static linker though... Just replace LD_PRELOAD manipulation with PATH manipulation, and create wrapper binaries for common apps. Slightly more detectable, but not by much.

rurban 3y ago

You are confusing the loader with the linker. The loader is the problem, always was. Just think of ldd exploits

thenonameguy 3y ago

NixOS/GNU Guix is uniquely positioned in this area, as it tracks all dependencies explicitly, with exact versions. If there is a paradigm where this can be achieved on the OS level, they are the closest to it today. See this related talk from NixOS 2022: https://www.youtube.com/watch?v=HZKFe4mCkr4 https://www.youtube.com/watch?v=HZKFe4mCkr4

nottorp 3y ago

Two different "philosophical" answers to this: 1. Patching known holes after the fact is just an arm's race that will never end. Better to do secure foundations. But that leads to 2: 2. We can probably build a software environment that is a lot more secure than what we have now, but no one could afford to pay for building it, and possibly no one could afford the operating costs to use it.