4 ms·

You don't link all of Libc

30 points by ingve 2y ago

chasil 2y ago
When I boot OpenBSD, I see: reordering: ld.so libc libcrypto sshd. How does that play into this?
- caspper69 2y ago
  Not related. The blog post is a mini-rant complaining about how other people are complaining about having to link "all of libc" on BSD when their standard libraries are not based on libc (libc is the interface to the kernel on BSD). He's talking about static linking. Honestly, I'm not terribly sure I get it, as almost all "other languages" link to libc in some way on most OSes, and don't call native kernel interfaces directly at all. But maybe it's more common than I think. Famously, Go went this route and it caused them no end of consternation with their FFI, and they've gone back in the other direction of linking with libc directly. In your case, reordering has to do with ASLR (address randomization) in dynamic libraries. OpenBSD takes ASLR a step further and so not only randomizes the libraries themselves, but the archives inside the library and the functions themselves- they call it KARL. That's what it's doing when you see that message.
  - chasil 2y ago
    Right, but does that mean that /bin/ls changes at every boot, because some of what it took from libc is dynamic?
    - caspper69 2y ago
      The article states that /bin/ls is statically linked. That means that the parts it uses from libc have long-since been taken and merged directly into the ls binary. So this randomization has no effect. Even if ls were dynamically linked, the ls binary itself wouldn't change at every boot; instead its import/jump table, generated dynamically by ld.so as part of the OS's executable loading process, would simply point to different (and random) offsets within libc. It would have a different in-memory representation after each boot, but its on-disk representation would remain the same.
  - kazinator 2y ago
    On Linux, your language run-time can call into the kernel directly with itsown syscall assembly language stubs. The kernel interface is something public and stable, independently of the C library (of which there are several available, incidentally: glibc, uClibc, musl, ...).
    - caspper69 2y ago
      Right, I understand that. You can do the same on Windows too. My point was that most languages' runtimes don't do that, not that they can't do that.
      - andrewf 2y ago
        I think NT syscalls can change. https://j00ru.vexillium.org/syscalls/nt/64/ https://j00ru.vexillium.org/syscalls/nt/64/ You're meant to go through ntdll.dll, but even that interface is documented as private/unstable.
        caspper69 2y ago
        You're absolutely right, they can. And they have. But the reality is more like this: https://xkcd.com/1172/ https://xkcd.com/1172/
        immibis 2y ago
        You're meant to go through published interfaces, such as the ones in kernel32.dll. ntdll is an implementation detail and so are syscalls.
      - kazinator 2y ago
        Because then they would face additional portability hurdles that they can avoid by targeting POSIX. Even decent porting to Windows is possible (speaking of which) via POSIX.
- wahern 2y ago
  Not directly related, but note what is mentioned in the post about the purpose of individual object files for each function. What might be considered an anachronism is being leveraged for improved security. What's happening (IIRC) with the reordering is that all those otherwise dynamically linked binaries (sshd is a PIE binary) are being themselves relinked from scratch (OpenBSD ships with the .o files), but shuffling the order of .o files passed to the linker. This effectively enhances ASLR. The way ASLR normally works is that each binary (program or shared library) is loaded into a random base address, but all the functions defined in each binary are fixed relative to each other. In stock installations libc's fopen is at a fixed offset from libc's snprintf, for example, and this will be true for every program loading libc.so. In the above, libc.so is relinked with a randomized ordering to the .o object files. The result is that relative offsets for libc functions on your individual system will be different from every other system, including from your previous boot, hopefully making exploits (e.g. utilizing ROP chains) more difficult. More modern projects like libcrypto and sshd tend to group related functions into the same .o file, so all the functions in each .o will still have a fixed offset from each other, but they still have quite alot of individual .o files to shuffle. Note that OpenBSD does the same thing for the kernel.
rcxdude 2y ago
You can do better than working at the object file level. With --gc-sections and --ffunction-sections, the linker can include only the relevant functions into your binary (whether your libc tries to minimise internal dependencies is another question)
- Neywiny 2y ago
  I think even better than functions it can optimize out data sections too. For example if I declare a huge array and don't use it, it doesn't show up in the binary. Not always true but the are times it is.
kazinator 2y ago
If you use dynamic linking, you will get all of libc mapped into your address space. It won't all be faulted in; just the parts you call. The parts you fault in should point to the same physical frame of memory that other processes have faulted in that have also called the same part of the library. That's why it's a "shared library".
- alsoforgotmypwd 2y ago
  Yeap. I roll my eyes at the people who want to static link everything and think they're somehow better or purer when they're just wasting RAM for nothing.
  - ericye16 2y ago
    Static linker supporter here, not to start a flame war but I like static linking because it means I don't have to worry about version mismatches of shared libraries between what I ship and what the user gets.
    - caspper69 2y ago
      Your complaint is more related to how the OS you're running on handles dynamic libraries, rather than anything inherent to dynamic libraries themselves. It is possible to version libraries and serve up the correct version(s) to different applications simultaneously. Now, consider the case of a Linux distro with a few thousand binaries. Should a defect be found in a common library, the burden of updating, say, 10,000 servers might rapidly become a headache. In this case, a shared library probably makes sense. Also, if you want to create extensible software (like a plugin system), dynamic libraries are one of the most expedient and well supported ways to do so. This is not necessarily the traditional use case for shared libs, but it's not uncommon either. And this use case normally doesn't have those versioning problems either (as it leans more to the dynamic side than the shared side). All that being said- static linking also has its advantages; as you stated, you can be guaranteed that your users are executing the library code you intended and that nothing breaks. It's also faster, because you're not waiting for ld.so to resolve your imports at startup & first use (nit: yes, I know you also can force all resolution to finalize at startup at the expense of slightly longer startup time). Further, you can be sure that someone hasn't hooked your library calls to do something nefarious. The downside, of course, is that you have to re-build and re-deploy your entire application on every library's bugfix/security updates. As with anything related to computing, everything has tradeoffs. Some tradeoffs may be dealbreakers for your use case, but they might not be for someone else's.
kazinator 2y ago
With the GNU linker, you would have to go out of your way to link all of a library, like: ld ... --whole-archive libwhatever.a --no-whole-archive Into the 1990s, it was still a big deal whether a linker could eliminate individual unused functions rather than just remove unused object files. I recall this ability is called function linking. In GCC they wanted function linking for the libgcc.a functions that provide run-time support, like numeric functions (e.g. division) needed by some generated code to make up for what the machine's instruction set does not provide. GCC's build process simulated (maybe still does?) function linking for libgcc.a by using the preprocessor to compile the file with these routines multiple times, separating the output into multiple .o files containing just one function. These days we have a libgcc as a shared library.

30 points by ingve 2y ago

chasil 2y ago

When I boot OpenBSD, I see: reordering: ld.so libc libcrypto sshd. How does that play into this?

caspper69 2y ago
Not related. The blog post is a mini-rant complaining about how other people are complaining about having to link "all of libc" on BSD when their standard libraries are not based on libc (libc is the interface to the kernel on BSD). He's talking about static linking. Honestly, I'm not terribly sure I get it, as almost all "other languages" link to libc in some way on most OSes, and don't call native kernel interfaces directly at all. But maybe it's more common than I think. Famously, Go went this route and it caused them no end of consternation with their FFI, and they've gone back in the other direction of linking with libc directly. In your case, reordering has to do with ASLR (address randomization) in dynamic libraries. OpenBSD takes ASLR a step further and so not only randomizes the libraries themselves, but the archives inside the library and the functions themselves- they call it KARL. That's what it's doing when you see that message.
- chasil 2y ago
  Right, but does that mean that /bin/ls changes at every boot, because some of what it took from libc is dynamic?
  - caspper69 2y ago
    The article states that /bin/ls is statically linked. That means that the parts it uses from libc have long-since been taken and merged directly into the ls binary. So this randomization has no effect. Even if ls were dynamically linked, the ls binary itself wouldn't change at every boot; instead its import/jump table, generated dynamically by ld.so as part of the OS's executable loading process, would simply point to different (and random) offsets within libc. It would have a different in-memory representation after each boot, but its on-disk representation would remain the same.
- kazinator 2y ago
  On Linux, your language run-time can call into the kernel directly with itsown syscall assembly language stubs. The kernel interface is something public and stable, independently of the C library (of which there are several available, incidentally: glibc, uClibc, musl, ...).
  - caspper69 2y ago
    Right, I understand that. You can do the same on Windows too. My point was that most languages' runtimes don't do that, not that they can't do that.
    - andrewf 2y ago
      I think NT syscalls can change. https://j00ru.vexillium.org/syscalls/nt/64/ https://j00ru.vexillium.org/syscalls/nt/64/ You're meant to go through ntdll.dll, but even that interface is documented as private/unstable.
      - caspper69 2y ago
        You're absolutely right, they can. And they have. But the reality is more like this: https://xkcd.com/1172/ https://xkcd.com/1172/
      - immibis 2y ago
        You're meant to go through published interfaces, such as the ones in kernel32.dll. ntdll is an implementation detail and so are syscalls.
    - kazinator 2y ago
      Because then they would face additional portability hurdles that they can avoid by targeting POSIX. Even decent porting to Windows is possible (speaking of which) via POSIX.
wahern 2y ago
Not directly related, but note what is mentioned in the post about the purpose of individual object files for each function. What might be considered an anachronism is being leveraged for improved security. What's happening (IIRC) with the reordering is that all those otherwise dynamically linked binaries (sshd is a PIE binary) are being themselves relinked from scratch (OpenBSD ships with the .o files), but shuffling the order of .o files passed to the linker. This effectively enhances ASLR. The way ASLR normally works is that each binary (program or shared library) is loaded into a random base address, but all the functions defined in each binary are fixed relative to each other. In stock installations libc's fopen is at a fixed offset from libc's snprintf, for example, and this will be true for every program loading libc.so. In the above, libc.so is relinked with a randomized ordering to the .o object files. The result is that relative offsets for libc functions on your individual system will be different from every other system, including from your previous boot, hopefully making exploits (e.g. utilizing ROP chains) more difficult. More modern projects like libcrypto and sshd tend to group related functions into the same .o file, so all the functions in each .o will still have a fixed offset from each other, but they still have quite alot of individual .o files to shuffle. Note that OpenBSD does the same thing for the kernel.

rcxdude 2y ago

You can do better than working at the object file level. With --gc-sections and --ffunction-sections, the linker can include only the relevant functions into your binary (whether your libc tries to minimise internal dependencies is another question)

Neywiny 2y ago
I think even better than functions it can optimize out data sections too. For example if I declare a huge array and don't use it, it doesn't show up in the binary. Not always true but the are times it is.

kazinator 2y ago

If you use dynamic linking, you will get all of libc mapped into your address space. It won't all be faulted in; just the parts you call. The parts you fault in should point to the same physical frame of memory that other processes have faulted in that have also called the same part of the library. That's why it's a "shared library".

alsoforgotmypwd 2y ago
Yeap. I roll my eyes at the people who want to static link everything and think they're somehow better or purer when they're just wasting RAM for nothing.
- ericye16 2y ago
  Static linker supporter here, not to start a flame war but I like static linking because it means I don't have to worry about version mismatches of shared libraries between what I ship and what the user gets.
  - caspper69 2y ago
    Your complaint is more related to how the OS you're running on handles dynamic libraries, rather than anything inherent to dynamic libraries themselves. It is possible to version libraries and serve up the correct version(s) to different applications simultaneously. Now, consider the case of a Linux distro with a few thousand binaries. Should a defect be found in a common library, the burden of updating, say, 10,000 servers might rapidly become a headache. In this case, a shared library probably makes sense. Also, if you want to create extensible software (like a plugin system), dynamic libraries are one of the most expedient and well supported ways to do so. This is not necessarily the traditional use case for shared libs, but it's not uncommon either. And this use case normally doesn't have those versioning problems either (as it leans more to the dynamic side than the shared side). All that being said- static linking also has its advantages; as you stated, you can be guaranteed that your users are executing the library code you intended and that nothing breaks. It's also faster, because you're not waiting for ld.so to resolve your imports at startup & first use (nit: yes, I know you also can force all resolution to finalize at startup at the expense of slightly longer startup time). Further, you can be sure that someone hasn't hooked your library calls to do something nefarious. The downside, of course, is that you have to re-build and re-deploy your entire application on every library's bugfix/security updates. As with anything related to computing, everything has tradeoffs. Some tradeoffs may be dealbreakers for your use case, but they might not be for someone else's.

kazinator 2y ago

With the GNU linker, you would have to go out of your way to link all of a library, like: ld ... --whole-archive libwhatever.a --no-whole-archive Into the 1990s, it was still a big deal whether a linker could eliminate individual unused functions rather than just remove unused object files. I recall this ability is called function linking. In GCC they wanted function linking for the libgcc.a functions that provide run-time support, like numeric functions (e.g. division) needed by some generated code to make up for what the machine's instruction set does not provide. GCC's build process simulated (maybe still does?) function linking for libgcc.a by using the preprocessor to compile the file with these routines multiple times, separating the output into multiple .o files containing just one function. These days we have a libgcc as a shared library.