3 ms·

Dynamic linking and dynamic loading aren't necessarily the same thing, are they? I imagine one could statically link in some C code to a rust program that uses

Dynamic linking and dynamic loading aren't necessarily the same thing, are they? I imagine one could statically link in some C code to a rust program that uses dlopen, for example, or even just do the syscall directly in rust (assuming that's possible).

pjmlp 6y ago
Yes they are, exactly the same OS infrastructure. It doesn't matter if it is the kernel or your application that calls dlopen(). How do imagine that something like ld.so gets implemented?
- sgtnoodle 6y ago
  The difference, I think, is that dynamic linking necessarily binds local symbols to the dynamically loaded foreign code, while dynamic loading simply loads the foreign code into the same address space and provides an API for looking up symbols' addresses by name. There's no inherent need for the ABI to match, as long as you don't directly try to call into a function; you can write adapters to explicitly handle the dynamically loaded code's ABI, like python does with ctypes.
  - pjmlp 6y ago
    Nope, that is the same infrastructure, just the details change across how they get exposed across OSes, specially non-POSIX ones.
    - sgtnoodle 6y ago
      At the end of the day it's all machine code running on silicon. What point are you trying to make? I was responding to the claim that a lack of good support for dynamic linking in a language prevents it from using shared objects for plugins and modules. My point was that dynamic loading is an operating system feature implemented through system calls, and can therefore be done in any language that can do a system call, whether or not the language makes it easy for you. I also claim that dynamic linking is not equivalent to loading, but I agree with you that they are related to each other and linking is indeed implemented on top of the system calls that provide dynamic loading. Dynamic linking specifically implies (for me) that non-static application symbols are resolved one way or another before the program counter jumps to main().
      - coliveira 6y ago
        In abstract you could separate these things, but in modern OSs there is little to no difference between these concepts. Dynamic loading is implemented using dynamic linking and if you don't want to use dynamic linking then you have to reinvent the wheel down to the OS level. You will have a very hard time to make dynamic loading available in any architecture following this route.
      - jgilias 6y ago
        Just dropping by to say that you're right. You can absolutely _load_ dynamic libraries from Rust. I've done this. But of course, the executable is not dynamically _linked_ to the requested libraries. E.g., running `ldd` on my executable won't show which shared libraries it is or should be linked against. Rather, it will just fail with an error if the requested library is not there. There are crates for loading dynamic libraries that use `dlopen` under the hood.
      - pjmlp 6y ago
        First of all, depending on the OS there are no syscalls to call to, Linux is the exception here. Everyone else has to go through dlopen, LoadLibrary, or whatever alternative the OS offers. Which is exactly the same code path that at some level the OS dynamic linker relies on. If you are statically linking everything, then there are no libraries to give input to dlopen anyway, it doesn't load .a files, and .so is not what we want to support anyway. But lets keep the assumption that in the world of static linking advocacy, explicitly loading dynamic libraries is acceptable. SO now, not only has the application to do the job of mapping dynamic library symbols to function pointers, they are constrained to C ABI function calls, which is not Rust anyway. As for you Linux based assumption, there are OSes, where dynamic linking doesn't take place before the PC jumps to main, you can do deferred dynamic linking. The call sites are marked for triggering a page fault on call, which only when it happens, will the OS look around for code to load. This is how managed languages like Java and C# work, which many keep forgetting there are also AOT compilers, just in case the VM arguments comes into play. But since we are talking about systems languages, you can find this feature on Windows and Aix, just to give two examples how the OS world isn't everywhere a Linux clone. https://docs.microsoft.com/en-us/cpp/build/reference/linker-support-for-delay-loaded-dlls?view=msvc-160 https://docs.microsoft.com/en-us/cpp/build/reference/linker-... https://www.ibm.com/support/knowledgecenter/ssw_aix_71/generalprogramming/lazyload.html https://www.ibm.com/support/knowledgecenter/ssw_aix_71/gener...
- a1369209993 6y ago
  > It doesn't matter if it is the kernel or your application that calls dlopen(). My application calls mmap and mprotect(PROT_EXEC), not dlopen.
  - sgtnoodle 6y ago
    Neat, memory mapping is a nifty concept. Are you loading a raw binary blob and directly jumping into it then? That's hardcore. I'm not familiar with the inner workings of dlopen, but I wouldn't be surprised if dlopen itself is mostly userspace code that knows how to interpret the ELF format, and internally it uses mmap as the actual system call. Maybe it has its own system call, though, to let the kernel do most of the work? I've seen some really cool uses of mmap. It's nice for reducing the number of memory copies when you're churning through a bunch of data on disk, and often necessary when using shared memory for low overhead inter-process communication. I worked on a project that needed publish-subscribe message passing semantics with fine-grained synchronization across dozens of processes, for example, and a lock-free shared memory "disrupter" queue structure was able to handle millions of messages per second, while sockets could only handle a few hundred thousand or so. I've seen some unfortunate uses as well. Sometimes folk like the idea of manipulating files on disk as arrays in memory, and use mmap even though there's no practical performance need for it. The code often times ends up harder to read than it would have had it used file descriptors. Also, the performance ends up worse because the implemented algorithm's access patterns unknowingly cause lots of page faults and disk seeks because the data is an "array". Often times there's an alternative way to do the same work sequentially with an additional in-memory data structure, and using file descriptors naturally biases one towards doing that.
  - pjmlp 6y ago
    Try that on an OS that isn't a Linux distribution.
    - a1369209993 6y ago
      Works fine on BSD (though it's been long enough that I forget which BSD) as long as you ignore the parts of the documentation written by facist morons. Haven't had occasion to beat proper linking behaviour into a Windows machine, but what I've heard suggests it'd be a pain the ass, yes. (Then again, Windows isn't a OS, it's a shitty GUI for DOS that bloated until it could read email. Kinda like OSX, except there isn't a vaguely decent OS buried under the manure.)

pjmlp 6y ago

Yes they are, exactly the same OS infrastructure. It doesn't matter if it is the kernel or your application that calls dlopen(). How do imagine that something like ld.so gets implemented?

sgtnoodle 6y ago
The difference, I think, is that dynamic linking necessarily binds local symbols to the dynamically loaded foreign code, while dynamic loading simply loads the foreign code into the same address space and provides an API for looking up symbols' addresses by name. There's no inherent need for the ABI to match, as long as you don't directly try to call into a function; you can write adapters to explicitly handle the dynamically loaded code's ABI, like python does with ctypes.
- pjmlp 6y ago
  Nope, that is the same infrastructure, just the details change across how they get exposed across OSes, specially non-POSIX ones.
  - sgtnoodle 6y ago
    At the end of the day it's all machine code running on silicon. What point are you trying to make? I was responding to the claim that a lack of good support for dynamic linking in a language prevents it from using shared objects for plugins and modules. My point was that dynamic loading is an operating system feature implemented through system calls, and can therefore be done in any language that can do a system call, whether or not the language makes it easy for you. I also claim that dynamic linking is not equivalent to loading, but I agree with you that they are related to each other and linking is indeed implemented on top of the system calls that provide dynamic loading. Dynamic linking specifically implies (for me) that non-static application symbols are resolved one way or another before the program counter jumps to main().
    - coliveira 6y ago
      In abstract you could separate these things, but in modern OSs there is little to no difference between these concepts. Dynamic loading is implemented using dynamic linking and if you don't want to use dynamic linking then you have to reinvent the wheel down to the OS level. You will have a very hard time to make dynamic loading available in any architecture following this route.
    - jgilias 6y ago
      Just dropping by to say that you're right. You can absolutely _load_ dynamic libraries from Rust. I've done this. But of course, the executable is not dynamically _linked_ to the requested libraries. E.g., running `ldd` on my executable won't show which shared libraries it is or should be linked against. Rather, it will just fail with an error if the requested library is not there. There are crates for loading dynamic libraries that use `dlopen` under the hood.
    - pjmlp 6y ago
      First of all, depending on the OS there are no syscalls to call to, Linux is the exception here. Everyone else has to go through dlopen, LoadLibrary, or whatever alternative the OS offers. Which is exactly the same code path that at some level the OS dynamic linker relies on. If you are statically linking everything, then there are no libraries to give input to dlopen anyway, it doesn't load .a files, and .so is not what we want to support anyway. But lets keep the assumption that in the world of static linking advocacy, explicitly loading dynamic libraries is acceptable. SO now, not only has the application to do the job of mapping dynamic library symbols to function pointers, they are constrained to C ABI function calls, which is not Rust anyway. As for you Linux based assumption, there are OSes, where dynamic linking doesn't take place before the PC jumps to main, you can do deferred dynamic linking. The call sites are marked for triggering a page fault on call, which only when it happens, will the OS look around for code to load. This is how managed languages like Java and C# work, which many keep forgetting there are also AOT compilers, just in case the VM arguments comes into play. But since we are talking about systems languages, you can find this feature on Windows and Aix, just to give two examples how the OS world isn't everywhere a Linux clone. https://docs.microsoft.com/en-us/cpp/build/reference/linker-support-for-delay-loaded-dlls?view=msvc-160 https://docs.microsoft.com/en-us/cpp/build/reference/linker-... https://www.ibm.com/support/knowledgecenter/ssw_aix_71/generalprogramming/lazyload.html https://www.ibm.com/support/knowledgecenter/ssw_aix_71/gener...
a1369209993 6y ago
> It doesn't matter if it is the kernel or your application that calls dlopen(). My application calls mmap and mprotect(PROT_EXEC), not dlopen.
- sgtnoodle 6y ago
  Neat, memory mapping is a nifty concept. Are you loading a raw binary blob and directly jumping into it then? That's hardcore. I'm not familiar with the inner workings of dlopen, but I wouldn't be surprised if dlopen itself is mostly userspace code that knows how to interpret the ELF format, and internally it uses mmap as the actual system call. Maybe it has its own system call, though, to let the kernel do most of the work? I've seen some really cool uses of mmap. It's nice for reducing the number of memory copies when you're churning through a bunch of data on disk, and often necessary when using shared memory for low overhead inter-process communication. I worked on a project that needed publish-subscribe message passing semantics with fine-grained synchronization across dozens of processes, for example, and a lock-free shared memory "disrupter" queue structure was able to handle millions of messages per second, while sockets could only handle a few hundred thousand or so. I've seen some unfortunate uses as well. Sometimes folk like the idea of manipulating files on disk as arrays in memory, and use mmap even though there's no practical performance need for it. The code often times ends up harder to read than it would have had it used file descriptors. Also, the performance ends up worse because the implemented algorithm's access patterns unknowingly cause lots of page faults and disk seeks because the data is an "array". Often times there's an alternative way to do the same work sequentially with an additional in-memory data structure, and using file descriptors naturally biases one towards doing that.
- pjmlp 6y ago
  Try that on an OS that isn't a Linux distribution.
  - a1369209993 6y ago
    Works fine on BSD (though it's been long enough that I forget which BSD) as long as you ignore the parts of the documentation written by facist morons. Haven't had occasion to beat proper linking behaviour into a Windows machine, but what I've heard suggests it'd be a pain the ass, yes. (Then again, Windows isn't a OS, it's a shitty GUI for DOS that bloated until it could read email. Kinda like OSX, except there isn't a vaguely decent OS buried under the manure.)