4 ms·

Call me crazy, but it would be nice if binaries were bundled with all their dependencies but then the OS maps identical files to the same memory region to save

Call me crazy, but it would be nice if binaries were bundled with all their dependencies but then the OS maps identical files to the same memory region to save space+perf. You get the best of both worlds?

dahfizz 8y ago
This is exactly the approach of newer application formats in the Linux world such as flatpak and snaps. The problem, though, is that all the libraries are duplicated on disk. Flatpak and snaps get around this by recognizing when there is a shared dependency and only downloading it once. But if you "bake in" all your dependecies into a single executable file, that would cause pretty significant inflation for program sizes. This may be less of an issue now as disk space gets cheaper and cheaper, but it still feels like a step in the wrong direction.
- wtetzner 8y ago
  It would be cool to have a filesystem that unique'd storage of identical objects under the hood. E.g. even though the user sees a particular filename, it's actual stored using a SHA to detect duplicates. If you edit a file, it could do a copy-on-write, to ensure you're not inadvertently changing a different file.
  - db48x 8y ago
    Deduplicating filesystems do exist; ZFS is one of the better examples. Of course, maintaining the indexes turns out to become a larger percentage of your cpu time, and those indexes have to live somewhere so they occupy a lot of memory. (You could reduce that impact by doing file-level rather than block -level deduplication, but then you also lose many of the benefits) If you want something that's higher level, there are tools like git annex that do this.
    - ComputerGuru 8y ago
      ZFS (and others) feature block-based deduplication, not rule-level deduplication which is cheap and fast (though less “effective”).
      - ComputerGuru 8y ago
        *file-level
  - rdc12 8y ago
    There was Venti in Plan9 that had this style of approach. [1] https://www.usenix.org/legacy/events/fast02/quinlan/quinlan_html/ https://www.usenix.org/legacy/events/fast02/quinlan/quinlan_...
- zem 8y ago
  the other issue is that when a bug or vulnerability is patched in the shared code, you no longer get to fix every dependency by updating a single synamic library.
  - amenghra 8y ago
    If it’s easy to inspect the packages, then you just recursively fix everywhere. It’s better than having static libraries that you cannot fix without recompiling from source.
- zamalek 8y ago
  This is true of Win 3.1 and all versions subsequent (unless a JIT is involved). The problem with this is DLL hell. You need a central repository of images for mapping, but that leads to conflicts. Possibly a good opportunity for CAS?
gpm 8y ago
Honestly the best part of shared libraries in my opinion isn't that they save space and memory. It's that when there is a security bug in libc I don't have to update everything.
- pjmlp 8y ago
  And make it relatively painless to create plugins with more performance vs using IPC. On the other hand if they crash or contain memory corruption bugs, the host process suffers as well.
- foota 8y ago
  The corollary being that you don't need to worry about breaking things when upgrading libc
  - gpm 8y ago
    Well sure, but at least for networked services (which is unfortunately a substantial fraction of the binaries on my computer) I'd happily take the security over functionality tradeoff. If things don't work, I notice and fix it. If things aren't secure, I don't notice, get hacked, and my files get ransomed back to me.
    - zaarn 8y ago
      You could have both, in theory. Ie, use static linking unless a compatible version of the library is present on disk with a higher version number at runtime.
      - AnIdiotOnTheNet 8y ago
        That won't prevent the higher version from breaking your application.
        zaarn 8y ago
        In theory it could, simply remove the newer version and the app uses it's internal library again. Or alternatively, like LD_PRELOAD, just have an env variable; LD_DONT_PRELOAD. Though that would make it more complicated.
Animats 8y ago
But then address space randomization won't work.
amw-zero 8y ago
That’s exactly what a shared library is right? It’s not crazy, it’s existed for a very long time.

dahfizz 8y ago

This is exactly the approach of newer application formats in the Linux world such as flatpak and snaps. The problem, though, is that all the libraries are duplicated on disk. Flatpak and snaps get around this by recognizing when there is a shared dependency and only downloading it once. But if you "bake in" all your dependecies into a single executable file, that would cause pretty significant inflation for program sizes. This may be less of an issue now as disk space gets cheaper and cheaper, but it still feels like a step in the wrong direction.

wtetzner 8y ago
It would be cool to have a filesystem that unique'd storage of identical objects under the hood. E.g. even though the user sees a particular filename, it's actual stored using a SHA to detect duplicates. If you edit a file, it could do a copy-on-write, to ensure you're not inadvertently changing a different file.
- db48x 8y ago
  Deduplicating filesystems do exist; ZFS is one of the better examples. Of course, maintaining the indexes turns out to become a larger percentage of your cpu time, and those indexes have to live somewhere so they occupy a lot of memory. (You could reduce that impact by doing file-level rather than block -level deduplication, but then you also lose many of the benefits) If you want something that's higher level, there are tools like git annex that do this.
  - ComputerGuru 8y ago
    ZFS (and others) feature block-based deduplication, not rule-level deduplication which is cheap and fast (though less “effective”).
    - ComputerGuru 8y ago
      *file-level
- rdc12 8y ago
  There was Venti in Plan9 that had this style of approach. [1] https://www.usenix.org/legacy/events/fast02/quinlan/quinlan_html/ https://www.usenix.org/legacy/events/fast02/quinlan/quinlan_...
zem 8y ago
the other issue is that when a bug or vulnerability is patched in the shared code, you no longer get to fix every dependency by updating a single synamic library.
- amenghra 8y ago
  If it’s easy to inspect the packages, then you just recursively fix everywhere. It’s better than having static libraries that you cannot fix without recompiling from source.
zamalek 8y ago
This is true of Win 3.1 and all versions subsequent (unless a JIT is involved). The problem with this is DLL hell. You need a central repository of images for mapping, but that leads to conflicts. Possibly a good opportunity for CAS?

gpm 8y ago

Honestly the best part of shared libraries in my opinion isn't that they save space and memory. It's that when there is a security bug in libc I don't have to update everything.

pjmlp 8y ago
And make it relatively painless to create plugins with more performance vs using IPC. On the other hand if they crash or contain memory corruption bugs, the host process suffers as well.
foota 8y ago
The corollary being that you don't need to worry about breaking things when upgrading libc
- gpm 8y ago
  Well sure, but at least for networked services (which is unfortunately a substantial fraction of the binaries on my computer) I'd happily take the security over functionality tradeoff. If things don't work, I notice and fix it. If things aren't secure, I don't notice, get hacked, and my files get ransomed back to me.
  - zaarn 8y ago
    You could have both, in theory. Ie, use static linking unless a compatible version of the library is present on disk with a higher version number at runtime.
    - AnIdiotOnTheNet 8y ago
      That won't prevent the higher version from breaking your application.
      - zaarn 8y ago
        In theory it could, simply remove the newer version and the app uses it's internal library again. Or alternatively, like LD_PRELOAD, just have an env variable; LD_DONT_PRELOAD. Though that would make it more complicated.

Animats 8y ago

But then address space randomization won't work.

amw-zero 8y ago

That’s exactly what a shared library is right? It’s not crazy, it’s existed for a very long time.