3 ms·
Dynamic linking doesn't even really save disk space https://drewdevault.com/dynlib https://drewdevault.com/dynlib The vast majority of libraries on your system
by Gigachad 4y ago
Dynamic linking doesn't even really save disk space https://drewdevault.com/dynlib https://drewdevault.com/dynlib
The vast majority of libraries on your system are used by only one program. I'd imagine dynamic linking also freezes progress and improvements on libraries because it's extremely difficult to roll out changes without breaking packages that haven't been tested and updated for them.
- rektide 4y agoWe run hundreds of processes on big nodes at work & we absolutely would save hella hella memory & have way better cache utilization if we could share the massive massive massive reams of code each process duplicates for itself. Static linking is basically like running Electron. You take a wonderful multi-machine capable environment then use it to run a single thing, exacerbating the size of what should be minor overhead. As a Linux user, there's like 30 dependencies used by basically half of what I run. If you click in to the article & look at the first chart, it shows this clearly: it tapers down quickly yes but so what? What actually matters? There are a good solid chunk of core libraries used en volume. You dont save much disk space, yes. But many libraries are used by many-hundreds of consumers. Thats where the pain is, that's the loss, that's where you are burning memory space again and again and again with every process you load, and worse, is the cache you are contesting for no reason. You could share that. You should. Giving it up is only because someone has decided it's too hard & too difficult to try,.. and sometimes they are right. But most Linux distros have been working through & managing it & doing ok for years.
- charcircuit 4y ago>We run hundreds of processes on big nodes at work & we absolutely would save hella hella memory & have way better cache utilization if we could share the massive massive massive reams of code each process duplicates for itself. Assuming those hundreds are mostly duplicates why can't they all share their entire code section?
- rektide 4y agoThere's ~60 different things we run. It would be tempting to try to make some kind of a group scheduler. But on the other hand, we get some nice load levelling/load-averaging from having mixed processes. Ultimately there's not really any reason for us to have static linking, but it's just "how it's done" now and it's unquestioned and there aren't as many well established alternatives as there ought to be.
- giantrobot 4y ago> The vast majority of libraries on your system are used by only one program. I find this is a Linux-ism rather than the common case on Windows or macOS. With Windows and macOS a given version of the OS is a fixed set of libraries and services available for third party software. Even third party libraries tend to call into the system libraries for some things. On Linux the only fixed facility of the OS is going to be kernel syscalls since even the libc can differ between distros.
- josephg 4y agoI think dynamic libraries which are only used by a single program are even more common on windows and macos. Look at how many DLLs ship with applications, and get installed inside the Program Files/MyApp directories! (Seriously - do a search for .dll files there. They're everywhere). Practically all of those DLLs are only be used by a single program. In each case, the price is being paid for dynamic linking (in startup time and the memory cost from not having dead code elimination). But with none of the benefits. MacOS and iOS apps have the same problem - but I think the modern apple security policies make it even worse. As far as I know there's no way for a mac application bundle to install a shared library on macos or ios which is made available to other applications. The "Drag the .app to the Applications folder" requires that every application ships with its own set of "shared libraries". Outside of the shared libraries provided by the operating system, why would applications ever use shared libraries when they won't ever be shared anyway? (If its not clear, I'm using the terms "shared library" and "dynamic library" interchangeably here because they mean the same thing.)
- eschaton 4y agoOn Apple’s platforms the primary motivator for sharing code within an application is that an application isn’t just one component. You may have an application that has any number of app extensions in it that provide access to its features in various settings; if you build that functionality into a framework and use it from the app itself and any app extensions, it’ll still only be loaded into memory once even though there might be several processes mapping it. Creating frameworks and libraries also helps the developer break down software into more testable components, which doesn’t necessarily require them to be dynamic but if you do go that route you remove an axis of variance that can result in different behavior. For example, a static library on UNIX is just an indexed archive of object files, so it doesn’t have a single overall set of imports and exports or a single set of initializers to run at load, whereas a dynamic library is a fully-linked module with its own set of dependencies, imports, exports, and initializers. If you assume equivalence between them you’re eventually going to be wrong, often in hard-to-diagnose ways.
- comex 4y agoThat post keeps being linked in these discussions but it's highly misleading. > Do your installed programs share dynamic libraries? > Findings: not really The posted findings actually show that common libraries ranging from libc to libX11 are used by high-hundreds to thousands of binaries. The fact that there are also a lot of not-widely-shared dynamic libraries doesn't seem particularly significant. > Wouldn't statically linked executables be huge? > Findings: not really > On average, dynamically linked executables use only 4.6% of the symbols on offer from their dependencies. A good linker will remove unused symbols. This analysis is completely wrong because it ignores the effect of transitive dependencies. If all exported symbols are independent and don't depend on anything else, then using 4.6% of the symbols means the linker can strip the library down to 4.6% of the size. Simple libraries like libc or libm are pretty close to that ideal. But for more complex libraries, the exported symbols are just the entry points for a deep nest of interlinked functions, and often a single symbol will transitively depend on a large fraction of the library.
- josephg 4y ago> This analysis is completely wrong because it ignores the effect of transitive dependencies. I hear your criticism, and I'd love to see the corrected results for this. My instinct is that most fully statically linked binaries wouldn't be that much bigger than their dynamically linked counterparts even with transitive inclusion. But I don't know for sure. Honestly I'm surprised its not easier to run these experiments. If dyld can traverse the call graph through dynamic libraries, why can't it just output the result as a single statically linked binary? I don't know how useful this would be - and it would miss out on inlining & optimization opportunities. But it'd be great for these sort of questions.
- atq2119 4y agoAt least with ELF, shared libraries are organized into segments and you really only have a handful of them. The information about where individual function boundaries are has been thrown away and in principle this can matter, e.g. if cold code was moved away from hot code. Therefore, the linker can't just split the segments apart. As for your instinct, it'd be wildly wrong depending on the use case. Maybe if you only link with libc, sure. But now consider linking against something like a GUI framework...