5 ms·
> The first reason was disk space: object files could simply be smaller. Nowadays, for all intents and purposes, disk space is free and unlimited. It's not for
by jcranmer 7y ago
> The first reason was disk space: object files could simply be smaller. Nowadays, for all intents and purposes, disk space is free and unlimited.
It's not for disk space, it's runtime RAM occupation usage. The code for a .so is going to be slightly larger than the equivalent .a. At the very least, you'll need to burn extra space for the PLT and GOT. And PIC relocations are going to tend to burn more space than static linking relocations.
The advantage of a .so you're talking about is that you can load a library shared by multiple processes at a fixed virtual address location that's the same for all processes, and thus you only burn the library once in RAM. For static libraries, the embedded addresses are going to be slightly different depending on their corresponding libraries, so even magic deduplication isn't going to be able to keep you from having the library in multiple places in physical memory.
- ThemalSpan 7y agoTo your first point: iOS takes this idea to its logical conclusion and it seems to be an important aspect of iOS’s performance. https://iphonedevwiki.net/index.php/Dyld_shared_cache https://iphonedevwiki.net/index.php/Dyld_shared_cache
- comex 7y agomacOS as well.
- loeg 7y agoicache, TLB entries, and even L1 are much more limited than main memory, especially on embedded devices. By linking everything into a single object, iOS can save on wasted padding and use superpages to map the shared library into each process in a TLB-efficient way.
- CamperBob2 7y agoRAM is free, too, at least as far as library code size is concerned.
- mikeash 7y agoA typical system has dozens or hundreds of processes running even when nothing is going on. If they all had their own copy of every system library in RAM, it wouldn’t look so free. Especially on mobile devices.
- zingermc 7y agoWhat if the OS could dedupe pages?
- kelnos 7y agoDo any OSes actually do this, though? If not, this is academic until one actually does. Linux does have something called "KSM" that de-dupes memory pages, but it appears that it's only so a hypervisor can de-dupe guest OS pages? Odd that it's not more general than that. Though I guess there is a cost: the KSM code needs to scan through memory to find duplicate pages, which I can't imagine is fast.
- loeg 7y agoOn the fly memory dedupe is incredibly expensive. Especially compared to just not duplicating in the first place. Also, it's quite likely that static linked libraries wouldn't be located at the exact same page offsets every time they're linked in to a binary, making page-level dedupe not useful.
- adrianmonk 7y agoYou'd need to arrange for the linking process to result in identical (and identically-aligned!) code pages. If you're going to all the trouble to ensure that, why not just write it all to a file and call it a shared library?
- bayareanative 7y agoExactly. Emperors' new clothes from naive convention abandonment of solved problems.
- 7y ago
- kazinator 7y agoModern shared libs load at different addresses in different processes. For that to work, they have to be compiled with relocatable code generation, and mapped read-only. Ancient Linux distributions once had something called "a.out shared libs". These were mapped to fixed addresses. The distribution had to work out non-conflicting addresses for all available libraries. The build procedure for a.out shared libs involved intercepting the assembly output of GCC and doing some text processing on it.
- deleted 7y ago[deleted]
- dexen 7y agoA modest proposal: let's use only static libraries (plugins aside). Let's help KSM efficiently merge the R/X, and the R/O memory pages of the static libraries of running processes. To maintain the current performance, let's give KSM a hint: the hint would be a string of standardized, canonical library pathname, including name and version number. During the (static) linking of libraries, an extra metadata item would be stored in the ELF file: the canonical pathnames of all the static libraries used, along with base addresses of R/X segments (code) and R/O segments (r/o data). Upon forking a process, the information would be provided to kernel, and passed over to the KSM, to use as basis of same-page lookup and possible merging of the pages. This effectively inverts the current mechanism. Currently when libraries are dynamically loaded, the dynloader uses actual file pathname of the shlib as the key for operation opposite to "merging" - i.e., re-uses the already loaded R/X and R/O pages, by adding proper memory mapping. The proposed change is three-fold: 1) extend the linker to add the metadata to ELF, 2) extend in-kernel ELF interpreter to extract the info upon exec() and friends, 3a) extend the KSM with a limited mode, where it would look up & merge only the hinted memory regions, in linear fashion, right after an exec() & friends. An alternative to 3a), to avoid fussing about with KSM: 3b) modify the VM subsystem to extend current swap/SHM so it provides an unique address range for each static lib canonical pathname. Requesting pages from this address range would map from the shared pages, if any process already loaded one such. To handle adversarial processes on single machine, a further extension where crypto signatures are checked is possible.