4 ms·

If you add 9MB to each binary on the system by statically linking them, and your system runs 200 programs (including system services) on average, your system no

by cyphar 6y ago

If you add 9MB to each binary on the system by statically linking them, and your system runs 200 programs (including system services) on average, your system now uses about 2GB more memory (to be fair, probably not all the time but it does increase memory pressure needlessly). Shared libraries aren't just about storage space. They also provide page cache sharing (memory for a shared library is only mapped once and the mapping is shared by different programs). A slight aside but they also provide the ability to apply security updates sanely to all programs using that library on the system (just update the library and restart the programs, as opposed to having to install a rebuilt version of every program that uses the library). Is this a game-changing feature? These days probably not, but it is (again) just needless waste. And I say this as someone who is currently developing a shared library in Rust that will probably be included in a lot of distributions because I expect quite a fair amount of container runtimes will end up using it. (But I do also work for a Linux distribution.)

SigmundA 6y ago
I used to think that way, but have come to the conclusion disk and memory issues should be handled by page / block level deduplication. Security updates are a plus, but introducing bugs through a shared library is a minus. I would prefer we focus on robust application sandboxing instead. At the server datacenter level its pretty much how its handled anyway, you have isolated vm / containers sitting on deduped storage.
- Hizonner 6y ago
  Sandboxing doesn't actually solve the biggest security problems, because most systems are there to run applications. That means that the interesting things on those systems are inside of some application's sandbox. It doesn't matter if I can't get root if I can get all the actually important assets on the system by cracking the application that actually processes those data. ... and in my experience, most application developers, and especially the developers who are the most truculently hostile to working with the system versions of libraries and the most likely to demand to bundle in their own copies for "stability" purposes, are really horrible about paying attention to security issues in their third-party dependencies. When I'm being a user and a sysadmin rather than a developer or a packager, and I have the choice of trusting a developer versus trusting a packager, then, all else being equal, I will tend to trust the packager. Not that I trust either one that much.
- tln 6y ago
  But the linker doesn't try to keep code in a dedupe friendly format. I don't think this would help on a pile of Go/rust binaries. Maybe there would be better sharing if I extracted the ELF sections first. I just ran an experiment on the 20 largest items in my Mac's /usr/local/bin, many of which are Go binaries for file in `find /usr/local/bin -perm +u+x | xargs ls -SL | head -20`; do split -a4 -b4k $file `basename $file`; done ls | xargs md5 > md5.txt cut -d' ' -f4 md5.txt | sort | uniq -c | cut -c1-4 | sort | uniq -c This gives: (number of 4kb blocks, frequency of sharing) 128021 1 12105 2 13 3 1 4 1 1137 so about 10% of blocks are shared... once. And 1 blocks is shared 1137 (all zeroes). To your second point, we still want fast security updates and more secure code even with sandboxing! EG, sandboxing would not help with heartbleed -- critical data leaked.
  - SigmundA 6y ago
    System wide shared libs is the opposite were we are trying to optimize space at the file level rather than block, and the linker instead optimizes for that along with all the other baggage of shared libs. So if block level dedupe where the norm linkers would optimize for that instead. Coming from the .Net / Node world you don't statically link but you typically don't have shared libs, all the libs are in the same folder as the executable and dedupe works just fine there because you have just a bunch of duplicated files in different directories. Same with containers and Vm's all duplicating a bunch of the same system files. You get isolation and reuse because its done at a lower abstraction. As far as security, again my point was a trade off, yes you could potentially get security updates faster along with breaking changes faster. If the sandbox apps had a good way to stay up to date, then they can update upstream after being integrated and tested with the app. If you have an app store of some sort it can watch for and notify to the authors of security issues in libs these use and prompt them to upgrade or be taken down. I know the app store argument probably isn't popular around here but from my experience it is a far more secure solution especially for the non technical user. If something is really so widely used it should be a api provided by the OS, otherwise it should be bundled with the app individually and code sharing is done in source repos / build systems, at least thats my opinion after all these years.
    - 6y ago
- lrem 6y ago
  You can't deduplicate if your 200 applications are using 20 minor versions times 50 mixes of compile flags. You're lucky if any two share the same output.
  - SigmundA 6y ago
    If your apps are using 20 different minor revisions and 50 different compile flags they prob can't share those libraries anyway? At least if some parts are the same they will dedupe at the block level even if files are different.
    - lrem 6y ago
      Indeed, you can't share in that case. The work of the distributor, as described in the article, is to make them share by unifying all those and requiring libraries to have stable ABI.
- ptx 6y ago
  > memory issues should be handled by page / block level deduplication I seem to recall ZFS deduplication needing tons of RAM. Wouldn't the memory deduplication you suggest similarly require even more memory?
  - SigmundA 6y ago
    Both Mac and Windows already do memory compression. I don't know how ZFS works but the dedup I have seen runs in the background consuming minimal resources. I would think if your mem-mapping a file that has been depdupped then its could just memap the already deduped block and cache in memory and not consume more memory if another file uses the same block and is memmapped but not sure if any OS does that.
- avasthe 6y ago
  > Security updates are a plus, but introducing bugs through a shared library is a minus. The minus you mention is a consequence of "Move break fast things" webshit philosophy that's so pervasive these days. Depending on undocumented behaviour and not properly documenting behaviour. Edit: and forgot to mention ignorance towards security updates caused by modern PL package version locking etc.. It is a culture problem and technology can only do so much to fix that.
MCOfficer 6y ago
9MB is a lot. For scale: I have a rust application; assets (font + music), GUI via GPU, networking, unzipping, a bit of cryptography, it's all in there. 400 dependencies in total (yes, i don't like it). By these metrics it's by far the largest rust application I've seen thus far. When fully optimized, it's 11MB in size.
- cyphar 6y ago
  I just used the metric from the article: > librsvg-2.so (version 2.40.21, C only) - 1408840 bytes > librsvg-2.so (version 2.49.3, Rust only) - 9899120 bytes For your application, I imagine most of your application's code size is in your dependencies -- so if each crate was a separate shared library (for instance) then you'd end up reducing the amount of duplicated code if you had 50 applications that all needed to use RustCrypto.

by cyphar 6y ago

SigmundA 6y ago

I used to think that way, but have come to the conclusion disk and memory issues should be handled by page / block level deduplication. Security updates are a plus, but introducing bugs through a shared library is a minus. I would prefer we focus on robust application sandboxing instead. At the server datacenter level its pretty much how its handled anyway, you have isolated vm / containers sitting on deduped storage.

Hizonner 6y ago
Sandboxing doesn't actually solve the biggest security problems, because most systems are there to run applications. That means that the interesting things on those systems are inside of some application's sandbox. It doesn't matter if I can't get root if I can get all the actually important assets on the system by cracking the application that actually processes those data. ... and in my experience, most application developers, and especially the developers who are the most truculently hostile to working with the system versions of libraries and the most likely to demand to bundle in their own copies for "stability" purposes, are really horrible about paying attention to security issues in their third-party dependencies. When I'm being a user and a sysadmin rather than a developer or a packager, and I have the choice of trusting a developer versus trusting a packager, then, all else being equal, I will tend to trust the packager. Not that I trust either one that much.
tln 6y ago
But the linker doesn't try to keep code in a dedupe friendly format. I don't think this would help on a pile of Go/rust binaries. Maybe there would be better sharing if I extracted the ELF sections first. I just ran an experiment on the 20 largest items in my Mac's /usr/local/bin, many of which are Go binaries for file in `find /usr/local/bin -perm +u+x | xargs ls -SL | head -20`; do split -a4 -b4k $file `basename $file`; done ls | xargs md5 > md5.txt cut -d' ' -f4 md5.txt | sort | uniq -c | cut -c1-4 | sort | uniq -c This gives: (number of 4kb blocks, frequency of sharing) 128021 1 12105 2 13 3 1 4 1 1137 so about 10% of blocks are shared... once. And 1 blocks is shared 1137 (all zeroes). To your second point, we still want fast security updates and more secure code even with sandboxing! EG, sandboxing would not help with heartbleed -- critical data leaked.
- SigmundA 6y ago
  System wide shared libs is the opposite were we are trying to optimize space at the file level rather than block, and the linker instead optimizes for that along with all the other baggage of shared libs. So if block level dedupe where the norm linkers would optimize for that instead. Coming from the .Net / Node world you don't statically link but you typically don't have shared libs, all the libs are in the same folder as the executable and dedupe works just fine there because you have just a bunch of duplicated files in different directories. Same with containers and Vm's all duplicating a bunch of the same system files. You get isolation and reuse because its done at a lower abstraction. As far as security, again my point was a trade off, yes you could potentially get security updates faster along with breaking changes faster. If the sandbox apps had a good way to stay up to date, then they can update upstream after being integrated and tested with the app. If you have an app store of some sort it can watch for and notify to the authors of security issues in libs these use and prompt them to upgrade or be taken down. I know the app store argument probably isn't popular around here but from my experience it is a far more secure solution especially for the non technical user. If something is really so widely used it should be a api provided by the OS, otherwise it should be bundled with the app individually and code sharing is done in source repos / build systems, at least thats my opinion after all these years.
  - 6y ago
lrem 6y ago
You can't deduplicate if your 200 applications are using 20 minor versions times 50 mixes of compile flags. You're lucky if any two share the same output.
- SigmundA 6y ago
  If your apps are using 20 different minor revisions and 50 different compile flags they prob can't share those libraries anyway? At least if some parts are the same they will dedupe at the block level even if files are different.
  - lrem 6y ago
    Indeed, you can't share in that case. The work of the distributor, as described in the article, is to make them share by unifying all those and requiring libraries to have stable ABI.
ptx 6y ago
> memory issues should be handled by page / block level deduplication I seem to recall ZFS deduplication needing tons of RAM. Wouldn't the memory deduplication you suggest similarly require even more memory?
- SigmundA 6y ago
  Both Mac and Windows already do memory compression. I don't know how ZFS works but the dedup I have seen runs in the background consuming minimal resources. I would think if your mem-mapping a file that has been depdupped then its could just memap the already deduped block and cache in memory and not consume more memory if another file uses the same block and is memmapped but not sure if any OS does that.
avasthe 6y ago
> Security updates are a plus, but introducing bugs through a shared library is a minus. The minus you mention is a consequence of "Move break fast things" webshit philosophy that's so pervasive these days. Depending on undocumented behaviour and not properly documenting behaviour. Edit: and forgot to mention ignorance towards security updates caused by modern PL package version locking etc.. It is a culture problem and technology can only do so much to fix that.

MCOfficer 6y ago

9MB is a lot. For scale: I have a rust application; assets (font + music), GUI via GPU, networking, unzipping, a bit of cryptography, it's all in there. 400 dependencies in total (yes, i don't like it). By these metrics it's by far the largest rust application I've seen thus far. When fully optimized, it's 11MB in size.

cyphar 6y ago
I just used the metric from the article: > librsvg-2.so (version 2.40.21, C only) - 1408840 bytes > librsvg-2.so (version 2.49.3, Rust only) - 9899120 bytes For your application, I imagine most of your application's code size is in your dependencies -- so if each crate was a separate shared library (for instance) then you'd end up reducing the amount of duplicated code if you had 50 applications that all needed to use RustCrypto.