6 ms·
Hidden dependencies in Linux binaries
- surajrmal 3y agoThe tool is interesting, but doesn't account for the fact that some shared libraries opened via dlopen are done so lazily. So it might miss those if you haven't executed a code path that triggers them to load. The other side of not accidentally loading more into your process than you thought is breaking down shared libraries into increasingly smaller sizes. In its limit I imagine it would be akin to a function per shared library, which probably defeats the point a bit.
- rwmj 3y agoFrom the article it seems like that might be the point, to visualise what the binary actually uses, eg. if you use different preferences.
- Retr0id 3y agoIf the code path that loads the library is never hit, does it really count as a dependency?
- sooperserieous 3y agoIt does if someone is intentionally sending it down that code path for exactly that reason.
- marcosdumay 3y agoIf your tests don't exercise a bug, is it still there?
- eqvinox 3y agoLazy binding for dlopen is disabled when the LD_BIND_NOW environment variable is set to a nonempty value, cf. https://man7.org/linux/man-pages/man3/dlopen.3.html https://man7.org/linux/man-pages/man3/dlopen.3.html
- tedunangst 3y agoThat doesn't mean the program is going execute all the dlopen() calls at start up.
- eqvinox 3y agoThat is true, but the root comment was specifically referring to dependencies of dlopen'ed libs not getting loaded. That one is 'fixable'. (Btw, I'm pretty sure dlopen itself can't be lazy, due to needing to run constructors; the root comment is a bit vaguely worded... but ofc that only matters after dlopen is called.)
- ris 3y ago> Meanwhile, when I use CUDA instead of Vulkan, I get serenity back. CUDA FTW! Just because the complexity is hidden from you doesn't mean it's not there. You have no idea what is statically bundled into the CUDA libs.
- dosshell 3y agoI agree with you, hidden is worse. But we do know what it can not static link to, any GPL library, which many indirect dependencies are.
- qwertox 3y agoIf static and dynamic libraries use the same interface, shouldn't they be detectable in both cases? Or is it removed at compile time?
- dosshell 3y agoFirst IANACC (I'm not a compiler programmer), but this is my understanding: What do you mean by interface? A dynamic library is handled very different from a static one. A dynamic library is loaded into the process virtual memory address space. There will be a tree trace there of loaded libraries. (I would guess this program walks this tree. But there may be better ways i do not know of that this program utilize) In the world of gnu/linux a static library is more or less a collection of object files. The linker, to my best knowledge, will not treat the content of the static libraries different than from your own code. LTO can take place. In the final elf the static library will be indistinguishable from your own code. My experience of the symbole table in elf files is limited and I do not know if they could help to unwrap static library dependencies. (A debug symbol table would of course help).
- heftig 3y agoI think you mean the LGPL? It allows you to "convey a combined work under terms of your choice" as long as the LGPL-covered part can be modified, which can be achieved either via dynamic linking or by providing the proprietary code as bare object files to relink statically. The GPL doesn't have this exception.
- rwmj 3y agoI was debugging a crash in vlc today - actually in the Intel VDPAU driver - and debuginfod (which dynamically downloads the debuginfo for everything in a coredump) took a good 15 minutes to run. If you look at the 'ldd /usr/bin/vlc' output it's only about 10 libraries, but it loads dozens and dozens more dynamically using dlopen, and I think probably those libraries dlopen even more. This tool could be pretty useful to visualise that.
- quotemstr 3y agoWe're in this situation because we're using a model of dynamic linking that's decades out of date. Why aren't we using process-isolated sandboxed components talking over io_uring-based low-latency IPC to express most software dependencies? The vast majority of these dependencies absolutely do not need to be co-located with their users. Consider liblzma: would liblzma-as-a-service really be that bad, especially if the service client and service could share memory pages for zero-copy data transfer, just as we already do for, e.g. video decode? Or consider React Native: RN works by having an application thread send a GUI scene to a renderer thread, which then adjusts a native widget tree to match what the GUI thread wants. Why do these threads have to be in the same process? You're doing a thread switch anyway to jump from the GUI thread to the renderer thread: is switching address spaces at the same time going to kill you? Especially if the two threads live on different cores and nothing has to "switch"? Both dynamic linking and static linking should be rare in modern software ecosystems. We need to instead reinvigorate the idea of agent-based component systems with strongly isolated components.
- rwmj 3y agoEssentially path dependency. It would be almost impossible to change how it works now.
- vlovich123 3y agoI don't think this stuff is as hard as you make it out to be. Consider that companies like Apple regularly change how things are done successfully. It just requires a good plan, time, & budget. Wayland is one example of what that looks like & it's not a good story. Pulse audio followed by pipewire is another example of migrations happening. I suspect this would probably be slightly worse than Wayland unless some kind of transparent shim could be written for each boundary so that it can be slotted in transparently.
- Analemma_ 3y agoApple can regularly change how things are done because they have absolute control over their platform and use an "our way or the highway" approach to breaking changes, where developers have to go along or lose access to a lucrative market. This approach really really would not work on Linux: consider that the rollout of systemd was one-tenth as dictatorial as is SOP for changes from Apple, and it caused legions of Linux users to scream for Poettering's head on a stick.
- JonChesterfield 3y agoThe glibc separation into multiple shared libraries is such a weird thing. Anyone happen to know how that happened? See musl for an example where they put it all in one lib and thus avoid a whole pile of failure modes.
- eqvinox 3y agoPOSIX requires it, cf. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/c99.html#tag_20_11_13_01 https://pubs.opengroup.org/onlinepubs/9699919799/utilities/c... However, that "requirement" doesn't prevent you from shipping an empty libm (or other libs listed there.) (The actual reason is probably that glibc is old enough to have lived in a time where you cared about saving time and space by not linking the math functions when you didn't need them...)
- eqvinox 3y agoActually, now that I'm looking at the list… I certainly don't have a "libxnet" on my system ;D
- o11c 3y agoA lot of them are stubs nowadays actually.
- ajross 3y agoReally the big finding here is that Xlib et. al. get pulled in to GPU compute tools, because access to GPU contexts has traditionally been mediated by the desktop subsystem, because the GPU was traditionally "owned" by the rendering layers of the device abstraction. The bug here is much more a changing hardware paradigm than it is an issue with shared library dependencies that recapitulate it. Things moved and the software layers kludged along instead of reworking from scratch. Obviously what's needed is a layer somewhere in the device stack that "owns" the GPU resources and doles them out to desktop rendering and compute clients as needed, without the two needing to know about each other. But that's a ton more work than just untangling some symbol dependencies!
- kingspact 3y ago[flagged]
- RajT88 3y agoOn windows, this is Dependency Walker versus ProcExp. Similar eye-goggling results. https://www.dependencywalker.com/ https://www.dependencywalker.com/ https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer https://learn.microsoft.com/en-us/sysinternals/downloads/pro...
- pjmlp 3y agoDependency walker used to be part of the Windows SDK, never got the point why the removal, given its utility.
- qwertox 3y agoWhen I google for "libvulkan_virtio" I get zero results. What does it do?
- epilys 3y agovirtio-gpu with vulkan command passthrough to host, https://docs.mesa3d.org/drivers/venus.html https://docs.mesa3d.org/drivers/venus.html
- qwertox 3y agoThank you. So is this a Vulkan emulator which does not send the commands into a software renderer but rather to the host's GPU? What reserves the resources on the GPU, also this driver? Can one reserve resources explicitly through the API or does this happen dynamically, as-needed? Because if explicitly, then I'd wonder if this is also part of the library, of the Vulkan spec, or if it is some Mesa offering.
- epilys 3y agoIt's basically allowing you to use the vulkan API on the virtualized guest, by writing vulkan API commands in a ring buffer in memory that is visible both by guest and host. These memory regions are only alive and accessible as long as the allocation lives, which is controlled by virtio control commands (in specific, create, map, unmap, destroy BLOB where a blob is the shared memory allocation). This allows textures shaders and generally large amounts of data to skip being copied to and from the virtqueues, which is the usual method of virtio communication. So to answer your question, if you use the Vulkan API on a guest to for example query the available Vulkan devices, if the correct mesa library is installed and virtio-gpu Venus is available, you will be able to use resources on the host with the Vulkan API.
- YarickR2 3y agoJust wait until Lennart pushes his idea of doing linking entirely via dlopen() in systemd (see the story from a few days ago). Last bits of sane and efficient means to track dependencies will be gone forever after that. Good luck creating any lean Docker/k8s images without pulling in systemd-based stack after that.
- nerdponx 3y ago> see the story from a few days ago I missed it, can you add a link or at least the post title I can search for?
- bheadmaster 3y ago> Just wait until Lennart pushes his idea of doing linking entirely via dlopen() in systemd (see the story from a few days ago) Could you paste a link to the story? I haven't been able to find it through search engines, and I'd love to read the rationale of such idea...
- dsissitka 3y agoLooks like https://news.ycombinator.com/item?id=40014724 https://news.ycombinator.com/item?id=40014724.
- deleted 3y ago[deleted]
- tedunangst 3y agoWait until /lib is split into /lib.d with /lib.d/default and /lib.d/available and....
- dmwilcox 3y agoI'm impatiently waiting for the systemd shell and editor /jk (and to be clear I hope this is a joke but I worry sometimes)
- nerdponx 3y agoI am a perpetual newbie when it comes to things like this. What's the advantage of dlopen()-ing instead of dynamic linking?
- tux3 3y agoFaster program start, and potentially making it a soft-dependency instead of a hard dep (or letting you fallback on something else if it's not there)
- d3m0t3p 3y agoYou can load libs at runtime with dl-open, for exemple if you need a feature you can load the corresponding lib. While with dynamic linking you'd load everything when the process is launched slowing the launch
- sgtnoodle 3y agoA more obscure use would be for loading multiple instances of a singleton library. This is especially helpful in something like a unit test suite, where you want each test case to start in a cleanly initialized state. If the code under test has a bunch of globally initialized variables, reloading the library at runtime is one of only a few possible ways of doing it.
- deleted 3y ago[deleted]