5 ms·

dynamic linking to me seems like solving a problem that filesystems should. deduplicating data.

ahepp 3y ago
Deduplication, in the sense of physical space reduction, seems like the least strong argument one could make for dynamic linking these days. The strongest argument I can think of, is enabling system managers (distro maintainers, even end users) to update dependencies. This might be to apply a security patch, enable some kind of tracing for profiling an application, and so on.
pjmlp 3y ago
Plugins have nothing to do with filesystems.
watermelon0 3y ago
In practice, dynamic linking might not be achiving that much in the sense of deduplication: https://drewdevault.com/dynlib.html https://drewdevault.com/dynlib.html
bayindirh 3y ago
As a bonus, shared libraries are deduplicated in RAM, too. Plus you don't need to recompile the universe because of a patch.
brabel 3y ago
That's an interesting way to look at it. If all dynamic libs lived on a read-only location, then the file system could actually only store the libs in one place and the other "copies" would be just symlinks to that... and when the OS loaded such lib, it would automatically know that despite being in different locations, the libs were the same (they're all symlinks to the same place). Is this something that has been attempted before?
- nurple 3y ago
  This is exactly how nix works, except instead of symlinks it actually modifies the binaries at buld-time to point its dynamic library paths to absolute paths in the store which includes a source-derived hash in place of a version.
- fl7305 3y ago
  > Is this something that has been attempted before? Yes, some file systems implement "deduplication".
- surajrmal 3y ago
  You can store the files as if you name the actual location of the file as a hash of its contents and symlink the file to that location, you naturally get deduplication. Fuchsia does this [1]. You still end up wanting to try and coordinate your packages to share as many deps as possible for resource optimization reasons, but you no longer depend on it. [1]: https://fuchsia.dev/fuchsia-src/concepts/filesystems/blobfs https://fuchsia.dev/fuchsia-src/concepts/filesystems/blobfs
- chrisweekly 3y ago
  Slight tangent, but in the nodejs / npm ecosystem, that's one of the things that makes pnpm unique (and IMHO far superior to npm or yarn) -- its node_modules are deduped using symlinks.
- pjc50 3y ago
  Traditionally this is what /usr/lib is for.
- forgotmyinfo 3y ago
  This is basically dynamic linking: you have foo.so sitting somewhere on disk, and multiple processes can just load it whenever. It can't be read-only, though; you need to add new libraries to that directory eventually.
sltkr 3y ago
They are complementary. If you want to reuse the shared parts of binaries, then you need a way to separate the binary image into parts that are specific to the application and libraries that can be reused across binary images, plus some metadata how to reconstruct the image from its parts. That's what exactly what dynamic libraries provide. In general, it's much easier to link a binary with its libraries than go in to the opposite direction (extracting common code from static linked binaries), because once you statically link a binary the library code will vary slightly due to differences in memory addresses, compiler optimizations, unused code that has been omitted, different input library versions, etc. Even if you were, in theory, able to write a complex filesystem driver that is able to extract the common parts of statically linked libraries so they can be deduplicated, then to reconstruct the original binary in memory, you'd have to perform something similar to dynamic linking, except now in the kernel, which really isn't an improvement.
- adamgordonbell 3y ago
  > If you want to reuse the shared parts of binaries But aren't we, when we use OCI images as a packaging mechanism, using containers to essentially throw away that sharing and arrive at a complicated version of static linking, where everything dynamically linked is shipped with the program? Same goes for arguments about ease of patching things. When the software's package is actually an image, you are patching each image individually that is running on the system.
  - forgotmyinfo 3y ago
    Someone did once joke that Docker is just static linking for millennials. I chuckle because there's a kernel of truth in there. ;)
    - wredcoll 3y ago
      Yeah, maybe, but the cute thing here is that docker also "statically links" the config files, history db, image directory, etc etc.
palata 3y ago
They do more than that. For instance, they can be swapped easily. Think for instance a security library being updated by the distro security team. It also makes it easier to depend on an LGPL library, nicely allowing the users to modify the LGPL library without having to recompile your program.
- zer00eyz 3y ago
  >>> Think for instance a security library being updated by the distro security team. And when your distro doesn't update? When the distro is slow to change? Flatpak, Appimage, Snaps (why?)... upstream devs are bypassing distro maintainers in a lot of cases.
  - palata 3y ago
    I am not saying that they solve all problems in the world. I am just saying that they have merits. In your particular example, if you are not happy with your distro, you are free to find another one. Some distros are extremely fast to change. > upstream devs are bypassing distro maintainers in a lot of cases IMO, for open source projects, upstream devs should not distribute their library. They should let package maintainers do it for their distro.
  - forgotmyinfo 3y ago
    Then there's probably a reason, like the changes haven't been tested yet or verified to make sure they don't FUBAR your machine. It's okay to go a little slower to ensure reliability. Plus, do you really want the latest bugs in HEAD anyway?
otabdeveloper4 3y ago
No, mostly it solves the problem of runtime library loading and swapping. E.g., Linux VDSO can only work as a shared library.
forgotmyinfo 3y ago
Filesystems solve linking insofar as the libraries themselves are on disk; otherwise, this is squarely the responsibility of the loader.

Deduplication, in the sense of physical space reduction, seems like the least strong argument one could make for dynamic linking these days. The strongest argument I can think of, is enabling system managers (distro maintainers, even end users) to update dependencies. This might be to apply a security patch, enable some kind of tracing for profiling an application, and so on.

pjmlp 3y ago

Plugins have nothing to do with filesystems.

watermelon0 3y ago

In practice, dynamic linking might not be achiving that much in the sense of deduplication: https://drewdevault.com/dynlib.html https://drewdevault.com/dynlib.html

bayindirh 3y ago

As a bonus, shared libraries are deduplicated in RAM, too. Plus you don't need to recompile the universe because of a patch.

brabel 3y ago

That's an interesting way to look at it. If all dynamic libs lived on a read-only location, then the file system could actually only store the libs in one place and the other "copies" would be just symlinks to that... and when the OS loaded such lib, it would automatically know that despite being in different locations, the libs were the same (they're all symlinks to the same place). Is this something that has been attempted before?

nurple 3y ago
This is exactly how nix works, except instead of symlinks it actually modifies the binaries at buld-time to point its dynamic library paths to absolute paths in the store which includes a source-derived hash in place of a version.
fl7305 3y ago
> Is this something that has been attempted before? Yes, some file systems implement "deduplication".
surajrmal 3y ago
You can store the files as if you name the actual location of the file as a hash of its contents and symlink the file to that location, you naturally get deduplication. Fuchsia does this [1]. You still end up wanting to try and coordinate your packages to share as many deps as possible for resource optimization reasons, but you no longer depend on it. [1]: https://fuchsia.dev/fuchsia-src/concepts/filesystems/blobfs https://fuchsia.dev/fuchsia-src/concepts/filesystems/blobfs
chrisweekly 3y ago
Slight tangent, but in the nodejs / npm ecosystem, that's one of the things that makes pnpm unique (and IMHO far superior to npm or yarn) -- its node_modules are deduped using symlinks.
pjc50 3y ago
Traditionally this is what /usr/lib is for.
forgotmyinfo 3y ago
This is basically dynamic linking: you have foo.so sitting somewhere on disk, and multiple processes can just load it whenever. It can't be read-only, though; you need to add new libraries to that directory eventually.

sltkr 3y ago

They are complementary. If you want to reuse the shared parts of binaries, then you need a way to separate the binary image into parts that are specific to the application and libraries that can be reused across binary images, plus some metadata how to reconstruct the image from its parts. That's what exactly what dynamic libraries provide. In general, it's much easier to link a binary with its libraries than go in to the opposite direction (extracting common code from static linked binaries), because once you statically link a binary the library code will vary slightly due to differences in memory addresses, compiler optimizations, unused code that has been omitted, different input library versions, etc. Even if you were, in theory, able to write a complex filesystem driver that is able to extract the common parts of statically linked libraries so they can be deduplicated, then to reconstruct the original binary in memory, you'd have to perform something similar to dynamic linking, except now in the kernel, which really isn't an improvement.

adamgordonbell 3y ago
> If you want to reuse the shared parts of binaries But aren't we, when we use OCI images as a packaging mechanism, using containers to essentially throw away that sharing and arrive at a complicated version of static linking, where everything dynamically linked is shipped with the program? Same goes for arguments about ease of patching things. When the software's package is actually an image, you are patching each image individually that is running on the system.
- forgotmyinfo 3y ago
  Someone did once joke that Docker is just static linking for millennials. I chuckle because there's a kernel of truth in there. ;)
  - wredcoll 3y ago
    Yeah, maybe, but the cute thing here is that docker also "statically links" the config files, history db, image directory, etc etc.

palata 3y ago

They do more than that. For instance, they can be swapped easily. Think for instance a security library being updated by the distro security team. It also makes it easier to depend on an LGPL library, nicely allowing the users to modify the LGPL library without having to recompile your program.

zer00eyz 3y ago
>>> Think for instance a security library being updated by the distro security team. And when your distro doesn't update? When the distro is slow to change? Flatpak, Appimage, Snaps (why?)... upstream devs are bypassing distro maintainers in a lot of cases.
- palata 3y ago
  I am not saying that they solve all problems in the world. I am just saying that they have merits. In your particular example, if you are not happy with your distro, you are free to find another one. Some distros are extremely fast to change. > upstream devs are bypassing distro maintainers in a lot of cases IMO, for open source projects, upstream devs should not distribute their library. They should let package maintainers do it for their distro.
- forgotmyinfo 3y ago
  Then there's probably a reason, like the changes haven't been tested yet or verified to make sure they don't FUBAR your machine. It's okay to go a little slower to ensure reliability. Plus, do you really want the latest bugs in HEAD anyway?

otabdeveloper4 3y ago

No, mostly it solves the problem of runtime library loading and swapping. E.g., Linux VDSO can only work as a shared library.

forgotmyinfo 3y ago

Filesystems solve linking insofar as the libraries themselves are on disk; otherwise, this is squarely the responsibility of the loader.