5 ms·

Something worth noting with shared dependencies is that yes, they save on disk space, but they also save on memory. A 200MB non-shared dependency will take up 6

Something worth noting with shared dependencies is that yes, they save on disk space, but they also save on memory. A 200MB non-shared dependency will take up 600MB across three apps, but a 200MB shared dependency can be loaded on it's own and save 400 megabytes. (Most operating systems manage this at the physical page level, by mapping multiple instances of a shared library to the same physical memory pages.) 400 megabytes of memory usage is probably worth more than 400 megabytes of storage. It may not be a make-or break thing on it's own, but it's one of the reasons Linux can run on lower-end devices.

whatshisface 2y ago
Isn't 200MB a little large for a dependency? The Linux kernel is ~30MB.
- odo1242 2y ago
  It's around the size of OpenCV, to be specific. I do see your argument though.
packetlost 2y ago
When you statically compile an application, you only store text (code) of functions you actually use, generally, so unless you're using all 400Mb of code you're not going to have a 400Mb+ binary. I don't think I've ever seen a dependency that was 400Mb of compiled code if you stripped debug information and weren't embedding graphical assets, so I'm not sure how relevant this is in the first place. 400Mb of opcodes is... a lot.
- vvanders 2y ago
  Yes, this is lost in most discussions when it comes to DSOs. Not only do you have the complexity of versioning and vending but also you can't optimize with LTO and other techniques(which can make a significant difference in final binary size). If you've got 10-15+ consumers of a shared library or want to do plugins/hot-code reloading and have a solid versioning story by all means vend a DSO. If you don't however I would strongly recommend trying to keep all dependencies static and letting LTO/LTCG do its thing.
  - intelVISA 2y ago
    Indeed, I'm biased but often the arguments in favor of shared objs have fundamental misconceptions that betray a lack of lower-level tinkering that makes it hard to take seriously.
    - saagarjha 2y ago
      The same can be said for any position in this debate.
- AshamedCaptain 2y ago
  I have seen a binary that is approximately 1.1GB of text. That is without debug symbols. With debug symbols it would hit GDB address space overflow bugs all the time. You have not seen what 30 year old engineering software houses can produce. And this is not the largest software by any chance. Also, sibling comments argue that kernel samepage merging can help avoid the bloat of static linking. But here what you argue will make every copy of the shared libraries oh-so-slightly-different and therefore prevent KSM from working at all. Really, no one is thinking this through very well. Even distributions that do static linking in all but name (such as NixOS) do still technically use dynamic linking for the disk space and memory savings.
  - packetlost 2y ago
    Your poor I$. The fact of the matter is outside of GPU drivers and poorly designed GUI applications, that's an extreme statistical outlier.
    - lostmsu 2y ago
      You mean GPU libraries, right? Just watch how these become used in every other app over the next few years. And yes, the fact that you won't use those apps does not make it easier for the rest of us.
    - AshamedCaptain 2y ago
      Sum the GPU driver and the mandatory LLVM requirement that comes with it, and you already have almost half a gigabyte of text per exec. No wonder that your logic forces you to literally dismiss all GUI applications as "extreme statistical outliers" out of hand. For the record, the 1.1GB executable I'm thinking about is a popular _terminal-only_ simulation package. No GUI.
      - nightowl_games 2y ago
        Can't you just tell us what it is?
      - packetlost 2y ago
pradn 2y ago
It's possible for the OS to recognize that several pages have the same content (ie: doing a hash) and then de-duplicate them. This can happen across multiple applications. It's easiest for read-only pages, but you can swing it for mutable pages as well. You just have to copy the page on the first write (ie: copy-on-write). I don't know which OSs do this, but I know hypervisors certainly do this across multiple VMs.
- deleted 2y ago
  [deleted]
- ChocolateGod 2y ago
  Correct me if I'm wrong Linux only supports KSM (memory-deduping) between processes when doing it between VMs, as QEMU provides information to the kernel to perform it.
  - yjftsjthsd-h 2y ago
    https://www.kernel.org/doc/html/latest/admin-guide/mm/ksm.html https://www.kernel.org/doc/html/latest/admin-guide/mm/ksm.ht... > KSM was originally developed for use with KVM (where it was known as Kernel Shared Memory), to fit more virtual machines into physical memory, by sharing the data common between them. But it can be useful to any application which generates many instances of the same data Although... > KSM only operates on those areas of address space which an application has advised to be likely candidates for merging, by using the madvise(2) system call
    - pradn 2y ago
      I wonder if you could just madvise the entire address space. My hunch is that it's for performance reasons only - fewer pages to scan, hash, and de-duplicate.
      - MrDrMcCoy 2y ago
        You can. There are custom kernel builds that do this, as well as shim loaders that do that madvise call. However, those are considerably less practical than the recent systemd feature that allows you to do it per unit. If applied to the system scope, it effectively covers the whole system. However, there are caveats: 1. You need a much better newer systemd than your distro likely packages. 2. KSM dedupe scans aren't free. Your system can spend its time doing that scan or it can spend its doing work with duplicate pages. Only relatively idle or highly homogenous systems would be free of the penalty. 3. For applications, especially statically linked ones, the duplicated code is not super likely to fall along page boundaries, thus the actual detectable duplication will be relatively low. That said, it's still great for densely deploying a high traffic microservice that's more memory than CPU bound.
bbatha 2y ago
Only if you don’t have LTO on. If you have LTO on you’re likely to use a fraction of the shared dependency size even across multiple apps.
- odo1242 2y ago
  That is a good point.
indigodaddy 2y ago
I remember many years ago VMWare developed a technology to take advantage of these shared library savings across VMs as well. https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.resmgmt.doc/GUID-FEAC3A43-C57E-49A2-8303-B06DBC9054C5.html https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsp...
- abhinavk 2y ago
  Linux’s KVM has it too. It’s called KSM.
  - indigodaddy 2y ago
    ah right, forgot about that as well
Lerc 2y ago
In practice I don't think this results in memory savings. By having a shared library and shared memory use, you also have distributed the blame for the size of the application. It would be true that this save memory if applications did not increase their memory requirements over time, but the fast is that they do, and the rate at which they increase their memory use seems to be dictated not by how much memory they intrinsically need but how much is available to them. There are notable exceptions. AI models, Image manipulation programs etc. do actually require enough memory to store the relevant data. On the other hand I have used a machine where the volume control sitting in the system tray used almost 2% of the system RAM. Static linking enables the cause of memory use to be more clearly identified, That enables people to see who is wasting resources. When people can see who is wasting resources, there is a higher incentive to not waste them.
- pessimizer 2y ago
  This is a law of averages argument. There is no rational argument for bloat in order to protect software from bloat. This is like saying that it doesn't matter that we waste money, because we're going to spend the entire budget anyway.
  - Lerc 2y ago
    I think it's more like, If we pay money to someone to audit our books, we are more likely to achieve more within our budget.
anotherhue 2y ago
Assuming it is worth it with modern memory sizes (I think not), could this present a negative on NUMA systems? Forcing contention when a simple copy would have been sufficient? I assume there's something optimising that away but I'm not well versed.
tdtd 2y ago
This is certainly true on Windows, where loaded DLLs share the base addresses across processes even with ASLR enabled, but is it the case on Linux, where ASLR forces randomization of .so base addresses per process, so relocations will make the data in their pages distinct? Or is it the case that on modern architectures with IP-relative addressing (like x64) that relocations are so uncommon that most library pages contain none?
- saagarjha 2y ago
  Code pages are intentionally designed to stay clean; relocations are applied elsewhere in pages that are different for each process.
839302 2y ago
Hola. No
intelVISA 2y ago
Please don't soapbox under false pretenses - in no world does this "200mb non-shared dependency" comparison exist: that would imply static linkage is just appending a shared object into the executable at compile-time.

whatshisface 2y ago

Isn't 200MB a little large for a dependency? The Linux kernel is ~30MB.

odo1242 2y ago
It's around the size of OpenCV, to be specific. I do see your argument though.

packetlost 2y ago

When you statically compile an application, you only store text (code) of functions you actually use, generally, so unless you're using all 400Mb of code you're not going to have a 400Mb+ binary. I don't think I've ever seen a dependency that was 400Mb of compiled code if you stripped debug information and weren't embedding graphical assets, so I'm not sure how relevant this is in the first place. 400Mb of opcodes is... a lot.

vvanders 2y ago
Yes, this is lost in most discussions when it comes to DSOs. Not only do you have the complexity of versioning and vending but also you can't optimize with LTO and other techniques(which can make a significant difference in final binary size). If you've got 10-15+ consumers of a shared library or want to do plugins/hot-code reloading and have a solid versioning story by all means vend a DSO. If you don't however I would strongly recommend trying to keep all dependencies static and letting LTO/LTCG do its thing.
- intelVISA 2y ago
  Indeed, I'm biased but often the arguments in favor of shared objs have fundamental misconceptions that betray a lack of lower-level tinkering that makes it hard to take seriously.
  - saagarjha 2y ago
    The same can be said for any position in this debate.
AshamedCaptain 2y ago
I have seen a binary that is approximately 1.1GB of text. That is without debug symbols. With debug symbols it would hit GDB address space overflow bugs all the time. You have not seen what 30 year old engineering software houses can produce. And this is not the largest software by any chance. Also, sibling comments argue that kernel samepage merging can help avoid the bloat of static linking. But here what you argue will make every copy of the shared libraries oh-so-slightly-different and therefore prevent KSM from working at all. Really, no one is thinking this through very well. Even distributions that do static linking in all but name (such as NixOS) do still technically use dynamic linking for the disk space and memory savings.
- packetlost 2y ago
  Your poor I$. The fact of the matter is outside of GPU drivers and poorly designed GUI applications, that's an extreme statistical outlier.
  - lostmsu 2y ago
    You mean GPU libraries, right? Just watch how these become used in every other app over the next few years. And yes, the fact that you won't use those apps does not make it easier for the rest of us.
  - AshamedCaptain 2y ago
    Sum the GPU driver and the mandatory LLVM requirement that comes with it, and you already have almost half a gigabyte of text per exec. No wonder that your logic forces you to literally dismiss all GUI applications as "extreme statistical outliers" out of hand. For the record, the 1.1GB executable I'm thinking about is a popular _terminal-only_ simulation package. No GUI.
    - nightowl_games 2y ago
      Can't you just tell us what it is?
    - packetlost 2y ago

pradn 2y ago

It's possible for the OS to recognize that several pages have the same content (ie: doing a hash) and then de-duplicate them. This can happen across multiple applications. It's easiest for read-only pages, but you can swing it for mutable pages as well. You just have to copy the page on the first write (ie: copy-on-write). I don't know which OSs do this, but I know hypervisors certainly do this across multiple VMs.

deleted 2y ago
[deleted]
ChocolateGod 2y ago
Correct me if I'm wrong Linux only supports KSM (memory-deduping) between processes when doing it between VMs, as QEMU provides information to the kernel to perform it.
- yjftsjthsd-h 2y ago
  https://www.kernel.org/doc/html/latest/admin-guide/mm/ksm.html https://www.kernel.org/doc/html/latest/admin-guide/mm/ksm.ht... > KSM was originally developed for use with KVM (where it was known as Kernel Shared Memory), to fit more virtual machines into physical memory, by sharing the data common between them. But it can be useful to any application which generates many instances of the same data Although... > KSM only operates on those areas of address space which an application has advised to be likely candidates for merging, by using the madvise(2) system call
  - pradn 2y ago
    I wonder if you could just madvise the entire address space. My hunch is that it's for performance reasons only - fewer pages to scan, hash, and de-duplicate.
    - MrDrMcCoy 2y ago
      You can. There are custom kernel builds that do this, as well as shim loaders that do that madvise call. However, those are considerably less practical than the recent systemd feature that allows you to do it per unit. If applied to the system scope, it effectively covers the whole system. However, there are caveats: 1. You need a much better newer systemd than your distro likely packages. 2. KSM dedupe scans aren't free. Your system can spend its time doing that scan or it can spend its doing work with duplicate pages. Only relatively idle or highly homogenous systems would be free of the penalty. 3. For applications, especially statically linked ones, the duplicated code is not super likely to fall along page boundaries, thus the actual detectable duplication will be relatively low. That said, it's still great for densely deploying a high traffic microservice that's more memory than CPU bound.

bbatha 2y ago

Only if you don’t have LTO on. If you have LTO on you’re likely to use a fraction of the shared dependency size even across multiple apps.

odo1242 2y ago
That is a good point.

indigodaddy 2y ago

I remember many years ago VMWare developed a technology to take advantage of these shared library savings across VMs as well. https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.resmgmt.doc/GUID-FEAC3A43-C57E-49A2-8303-B06DBC9054C5.html https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsp...

abhinavk 2y ago
Linux’s KVM has it too. It’s called KSM.
- indigodaddy 2y ago
  ah right, forgot about that as well

Lerc 2y ago

In practice I don't think this results in memory savings. By having a shared library and shared memory use, you also have distributed the blame for the size of the application. It would be true that this save memory if applications did not increase their memory requirements over time, but the fast is that they do, and the rate at which they increase their memory use seems to be dictated not by how much memory they intrinsically need but how much is available to them. There are notable exceptions. AI models, Image manipulation programs etc. do actually require enough memory to store the relevant data. On the other hand I have used a machine where the volume control sitting in the system tray used almost 2% of the system RAM. Static linking enables the cause of memory use to be more clearly identified, That enables people to see who is wasting resources. When people can see who is wasting resources, there is a higher incentive to not waste them.

pessimizer 2y ago
This is a law of averages argument. There is no rational argument for bloat in order to protect software from bloat. This is like saying that it doesn't matter that we waste money, because we're going to spend the entire budget anyway.
- Lerc 2y ago
  I think it's more like, If we pay money to someone to audit our books, we are more likely to achieve more within our budget.

anotherhue 2y ago

Assuming it is worth it with modern memory sizes (I think not), could this present a negative on NUMA systems? Forcing contention when a simple copy would have been sufficient? I assume there's something optimising that away but I'm not well versed.

tdtd 2y ago

This is certainly true on Windows, where loaded DLLs share the base addresses across processes even with ASLR enabled, but is it the case on Linux, where ASLR forces randomization of .so base addresses per process, so relocations will make the data in their pages distinct? Or is it the case that on modern architectures with IP-relative addressing (like x64) that relocations are so uncommon that most library pages contain none?

saagarjha 2y ago
Code pages are intentionally designed to stay clean; relocations are applied elsewhere in pages that are different for each process.

839302 2y ago

Hola. No

intelVISA 2y ago

Please don't soapbox under false pretenses - in no world does this "200mb non-shared dependency" comparison exist: that would imply static linkage is just appending a shared object into the executable at compile-time.