6 ms·

I really love the Go has gone with static binaries. There are so few use cases where we really need to be distributing libraries separately from apps and link t

by pbiggar 11y ago

I really love the Go has gone with static binaries. There are so few use cases where we really need to be distributing libraries separately from apps and link them dynamically later. It's 2015, static binaries only takes a tiny bit of memory (relative to how much is available and what we use for other things), and both disk space and bandwidth for downloading binaries is pretty abundant. Why are we still worrying about ABIs!! [edit: clarified it isn't about the type of linking, it's about separate distribution of libs from apps that use the libs).

millstone 11y ago
Imagine writing, say, Firefox, without dynamic linking. You would have to provide a separate version of Firefox for every version of the OS, and when the next version of the OS is released, your current Firefox would stop working! It's not about saving memory or disk space (although those are still very significant, especially on mobile). It's about allowing a program's components to evolve separately.
- pbiggar 11y ago
  I don't understand why you would need a separate version of Firefox for every version of the OS?
  - millstone 11y ago
    Consider Copy/Paste. This is a system feature, and all apps must agree on where the clipboard is and how to access it. If the system implementation of the clipboard changes (say, in the next OS), then apps that statically linked against the old implementation will not be able to Copy/Paste with those in the new implementation. You also have the reverse problem: apps compiled against the new implementation will break when run on the prior OS. Dynamic linking is one way to ensure that all apps automatically get the correct clipboard implementation for the system they are running on. Of course this applies to all system features, not just copy/paste.
    - pbiggar 11y ago
      I'm not saying you can completely avoid integrating with the OS (after all, you need to communicate with the OS if you want to run). I'm saying that if you use something like libjpeg, you should statically compile it in. Application libraries should be part of the application (system libraries, then, should be part of the system).
      - millstone 11y ago
        For sure, I agree. There's not much cause for bundling a dynamic library with your app. But you still need dynamic linking for the system. Of course C++ is really terrible at this. Its binary interface is very fragile: you can't add variables or virtual methods without breaking clients, and of course different STLs cannot interoperate. So for the case of C++ specifically, ABI compatibility is lipstick on a pig. Still, the gcc guys deserve props for their abundance of caution, and it does pay dividends for other tools (e.g. gdb) that have to understand the ABI.
  - pcwalton 11y ago
    For one, on some OS's (like Windows), the syscall interface is unstable, so you really want to use kernel32.dll and friends or you're signing up for pain. Even if you're on an OS with stable syscalls, there are plenty of vendor-specific DLLs you have to work with: GPU drivers, for example.
    - cremno 11y ago
      >For one, on some OS's (like Windows), the syscall interface is unstable True but if someone is dealing with syscalls on Windows they're programming something very unusual. >so you really want to use kernel32.dll and friends or you're signing up for pain. Why? There is no pain involved unless a function that isn't implemented in older versions is imported.
      - millstone 11y ago
        You don't have to use syscalls directly. If you statically link a library that makes system calls, then a syscall interface change will break your app. But if you wrap all system calls in a dylib, then that dylib insulates you against syscall interface changes.
        cremno 11y ago
        Syscalls are irrelevant for almost everybody programming Windows applications. I didn't write about Mac OS X.
  - pjc50 11y ago
    On Windows at least, all the common controls and the file dialogs are dynamically loaded, because they can be different across different versions. The file open/save dialogs embed Explorer and all its shell extensions, each of which is a DLL. These can vary from system to system. Something similar is present on Linux with libnss; if you want to do username-userid lookups, you have to load the relevant libraries at runtime.
ajross 11y ago
The graves of the untold millions (billions?) of systems compromised by malware injected through statically linked zlib and libjpeg blobs grow restless... The value of dynamically linking common system libraries isn't performance (well, anymore), it's that your friendly distro maintainers can do a far, far better job than you at maintaining that software for you as bugs are fixed over time. Now, Go isn't subject to the kind of severity of bug that C is, and their package management may be slick enough to make straightforward recompilation into the default deployment mode. But if that's true it's true in spite of the drawbacks of static linkage, not because of it.
- pbiggar 11y ago
  I think I have a (slightly futuristic and not-yet-common) worldview where this is ridiculous. I agree it's real in the present world though. If you assume that: - people stop using a full OS to deploy their apps (in a containerized world, for example), so there's only a tiny attack surface, - that only the application's dependencies exist (no reliance on distro-provided too) and are full specified (eg in a Gemfile.lock or equivalent), - that a dev team is responsible for knowing about security vulns in their dependencies when they happen, - that things like libjpeg are run in separate services which are completely locked down and dont have the ability to compromise another system, Then this isn't an issue. Lots of big ifs there, I'm aware.
  - pcwalton 11y ago
    Proxying all image decoding to a separate process-per-image isn't likely to be practical anytime soon. Browsers already struggle with too many processes even with process-per-tab.
    - pbiggar 11y ago
      I think we live in different worlds. It seems all your examples are downloadable software. I live in a cloudy/SaaS/distributed systems world.
    - ajross 11y ago
madars 11y ago
Because you probably don't want to recompile gazillion binaries if one library has an exploitable security vulnerability.
- wereHamster 11y ago
  CPU cycles are cheap. Every major distribution has a build server. Determining dependencies is easy with proper package manager. Rebuilding all binaries should be as easy as pressing one button, and not take more than a few hours.
  - Alupis 11y ago
    > Rebuilding all binaries should be as easy as pressing one button, and not take more than a few hours If only that were true. There are some packages which take hours to compile by themselves, even on beefy servers.
    - xaa 11y ago
      Out of curiosity, an example? The only programs I've compiled that remotely approach an hour are the Linux kernel, GCC, and ATLAS. The first two are so fundamental that there's no point in recompiling for security reasons (if your current version is compromised, you should assume the resulting binary from a recompilation to be compromised). ATLAS is a specialized package that can be replaced in common circumstances with faster-to-compile packages. IIRC, you can go from zero to a complete desktop Gentoo system in 2-4 hours on an i7 desktop. But even this long is admittedly an annoyance most users would not want to endure on a regular basis for a modest increase in security. The main reason people use Gentoo seems to be configurability, not security.
      - ahomescu1 11y ago
        > Out of curiosity, an example? The only programs I've compiled that remotely approach an hour are the Linux kernel, GCC, and ATLAS. Don't know about GP, but I can think of a few more: KDE, OpenOffice, Firefox (that alone takes about 2 hours on my quad-core Phenom), Chromium.
Sanddancer 11y ago
In addition to the flaws that this brings when there's any sort of severe bug, it also makes third party addons to your software much more painful. Dynamic linking lets a user, at runtime, add new libraries and new functionality, without having to stop their workflow, recompile the binary, then start their workflow again. If the program's source is several hundred megabytes, like many user facing apps tend to be these days, that can lead to a lot of painful waiting. The upsides to being able to dynamically add functionality severely outweigh the problems of dynamic versioning.
ksherlock 11y ago
static linking won't prevent this problem. Consider a build where clang (c++11), gcc5 (c++11), and gcc4 (c++03) are used. gcc5 uses the new abi, gcc4 uses the old abi, clang uses the new abi headers but links to the old abi (since it doesn't yet support the new name mangling scheme).
- pbiggar 11y ago
  No, static _linking_ wont solving this problem. Getting rid of linking as a concept gets rid of this.
  - pcwalton 11y ago
    You want to eliminate separate compilation? Even Golang isn't doing the equivalent of LTO for all builds anymore. (It changed in 1.2 or thereabouts, I think.)
    - pbiggar 11y ago
      I'm being a little imprecise with my language (too much time since I've worked on C++), sorry. I want to eliminate the idea that libraries are distributed separately from the application that uses them.
      - pcwalton 11y ago
        So every game should vendor all the device-specific OpenGL implementations for every GPU they want to run on into their tree?
        pbiggar 11y ago
        They already do on my xbox. But no, of course I'm not suggesting that - and that's a good counter-example, thanks. Really, I believe that the idea that the OS distributes libraries like libjpeg as archaic (distros strike me as an awful idea, tbh). That's part of the application, and it's the application's responsibility to ship a fix.
dottrap 11y ago
This isn't really a dynamic vs static linking problem. This is an ABI problem and affects static libraries too depending how each library was compiled and what its dependencies are. Go avoids this because the language designed in simpler ABI requirements, and also doesn't allow binary libraries at all (everything is expected to be source code). This completely avoids the problem since you only have one compiler producing all the components. But this isn't always a good thing. There are some good reasons for prebuilt libraries. (I had the unfortunate need to recently compile a huge C++ library on a slow ARM based embedded machine without the help of a cross-compiler. It took about 14 hours to compile. And I had to do it twice because the first time the build flags were wrong.) For more interesting perspective on dynamic vs. static libraries, read: iOS Static Libraries Are, Like, Really Bad, And Stuff (Radar 15800975)http://landonf.org/code/ios/Radar_15800975_iOS_Frameworks.20140112.html http://landonf.org/code/ios/Radar_15800975_iOS_Frameworks.20...
- pbiggar 11y ago
  > Go avoids this because the language designed in simpler ABI requirements, and also doesn't allow binary libraries at all. Yes, this is what I'm getting at! This mess of linking and ABIs is a mess, and it shouldn't exist unless it must, and it only must in very few cases. Thanks for the link, I'll take a read.
  - felixgallo 11y ago
    For instance, all those cases where you'd like security issues fixed by updating a shared library, rather than finding and recompiling and redistributing every program on your system that may have used that library.
    - marssaxman 11y ago
      As an end user, I'd like to ensure that it's impossible to change the behavior of multiple unrelated applications by upgrading something on my system. I don't want a program to change until I decide to upgrade that specific program.
GauntletWizard 11y ago
Shared libraries still make sense for unix/linux distributions; There's a lot of little binaries, a lot of little processes, and the size of the full libstdc (or even one pruned to the binaries necessities) is high overhead compared to their actual size.

by pbiggar 11y ago

millstone 11y ago

Imagine writing, say, Firefox, without dynamic linking. You would have to provide a separate version of Firefox for every version of the OS, and when the next version of the OS is released, your current Firefox would stop working! It's not about saving memory or disk space (although those are still very significant, especially on mobile). It's about allowing a program's components to evolve separately.

pbiggar 11y ago
I don't understand why you would need a separate version of Firefox for every version of the OS?
- millstone 11y ago
  Consider Copy/Paste. This is a system feature, and all apps must agree on where the clipboard is and how to access it. If the system implementation of the clipboard changes (say, in the next OS), then apps that statically linked against the old implementation will not be able to Copy/Paste with those in the new implementation. You also have the reverse problem: apps compiled against the new implementation will break when run on the prior OS. Dynamic linking is one way to ensure that all apps automatically get the correct clipboard implementation for the system they are running on. Of course this applies to all system features, not just copy/paste.
  - pbiggar 11y ago
    I'm not saying you can completely avoid integrating with the OS (after all, you need to communicate with the OS if you want to run). I'm saying that if you use something like libjpeg, you should statically compile it in. Application libraries should be part of the application (system libraries, then, should be part of the system).
    - millstone 11y ago
      For sure, I agree. There's not much cause for bundling a dynamic library with your app. But you still need dynamic linking for the system. Of course C++ is really terrible at this. Its binary interface is very fragile: you can't add variables or virtual methods without breaking clients, and of course different STLs cannot interoperate. So for the case of C++ specifically, ABI compatibility is lipstick on a pig. Still, the gcc guys deserve props for their abundance of caution, and it does pay dividends for other tools (e.g. gdb) that have to understand the ABI.
- pcwalton 11y ago
  For one, on some OS's (like Windows), the syscall interface is unstable, so you really want to use kernel32.dll and friends or you're signing up for pain. Even if you're on an OS with stable syscalls, there are plenty of vendor-specific DLLs you have to work with: GPU drivers, for example.
  - cremno 11y ago
    >For one, on some OS's (like Windows), the syscall interface is unstable True but if someone is dealing with syscalls on Windows they're programming something very unusual. >so you really want to use kernel32.dll and friends or you're signing up for pain. Why? There is no pain involved unless a function that isn't implemented in older versions is imported.
    - millstone 11y ago
      You don't have to use syscalls directly. If you statically link a library that makes system calls, then a syscall interface change will break your app. But if you wrap all system calls in a dylib, then that dylib insulates you against syscall interface changes.
      - cremno 11y ago
        Syscalls are irrelevant for almost everybody programming Windows applications. I didn't write about Mac OS X.
- pjc50 11y ago
  On Windows at least, all the common controls and the file dialogs are dynamically loaded, because they can be different across different versions. The file open/save dialogs embed Explorer and all its shell extensions, each of which is a DLL. These can vary from system to system. Something similar is present on Linux with libnss; if you want to do username-userid lookups, you have to load the relevant libraries at runtime.

ajross 11y ago

The graves of the untold millions (billions?) of systems compromised by malware injected through statically linked zlib and libjpeg blobs grow restless... The value of dynamically linking common system libraries isn't performance (well, anymore), it's that your friendly distro maintainers can do a far, far better job than you at maintaining that software for you as bugs are fixed over time. Now, Go isn't subject to the kind of severity of bug that C is, and their package management may be slick enough to make straightforward recompilation into the default deployment mode. But if that's true it's true in spite of the drawbacks of static linkage, not because of it.

pbiggar 11y ago
I think I have a (slightly futuristic and not-yet-common) worldview where this is ridiculous. I agree it's real in the present world though. If you assume that: - people stop using a full OS to deploy their apps (in a containerized world, for example), so there's only a tiny attack surface, - that only the application's dependencies exist (no reliance on distro-provided too) and are full specified (eg in a Gemfile.lock or equivalent), - that a dev team is responsible for knowing about security vulns in their dependencies when they happen, - that things like libjpeg are run in separate services which are completely locked down and dont have the ability to compromise another system, Then this isn't an issue. Lots of big ifs there, I'm aware.
- pcwalton 11y ago
  Proxying all image decoding to a separate process-per-image isn't likely to be practical anytime soon. Browsers already struggle with too many processes even with process-per-tab.
  - pbiggar 11y ago
    I think we live in different worlds. It seems all your examples are downloadable software. I live in a cloudy/SaaS/distributed systems world.
  - ajross 11y ago

madars 11y ago

Because you probably don't want to recompile gazillion binaries if one library has an exploitable security vulnerability.

wereHamster 11y ago
CPU cycles are cheap. Every major distribution has a build server. Determining dependencies is easy with proper package manager. Rebuilding all binaries should be as easy as pressing one button, and not take more than a few hours.
- Alupis 11y ago
  > Rebuilding all binaries should be as easy as pressing one button, and not take more than a few hours If only that were true. There are some packages which take hours to compile by themselves, even on beefy servers.
  - xaa 11y ago
    Out of curiosity, an example? The only programs I've compiled that remotely approach an hour are the Linux kernel, GCC, and ATLAS. The first two are so fundamental that there's no point in recompiling for security reasons (if your current version is compromised, you should assume the resulting binary from a recompilation to be compromised). ATLAS is a specialized package that can be replaced in common circumstances with faster-to-compile packages. IIRC, you can go from zero to a complete desktop Gentoo system in 2-4 hours on an i7 desktop. But even this long is admittedly an annoyance most users would not want to endure on a regular basis for a modest increase in security. The main reason people use Gentoo seems to be configurability, not security.
    - ahomescu1 11y ago
      > Out of curiosity, an example? The only programs I've compiled that remotely approach an hour are the Linux kernel, GCC, and ATLAS. Don't know about GP, but I can think of a few more: KDE, OpenOffice, Firefox (that alone takes about 2 hours on my quad-core Phenom), Chromium.

Sanddancer 11y ago

In addition to the flaws that this brings when there's any sort of severe bug, it also makes third party addons to your software much more painful. Dynamic linking lets a user, at runtime, add new libraries and new functionality, without having to stop their workflow, recompile the binary, then start their workflow again. If the program's source is several hundred megabytes, like many user facing apps tend to be these days, that can lead to a lot of painful waiting. The upsides to being able to dynamically add functionality severely outweigh the problems of dynamic versioning.

ksherlock 11y ago

static linking won't prevent this problem. Consider a build where clang (c++11), gcc5 (c++11), and gcc4 (c++03) are used. gcc5 uses the new abi, gcc4 uses the old abi, clang uses the new abi headers but links to the old abi (since it doesn't yet support the new name mangling scheme).

pbiggar 11y ago
No, static _linking_ wont solving this problem. Getting rid of linking as a concept gets rid of this.
- pcwalton 11y ago
  You want to eliminate separate compilation? Even Golang isn't doing the equivalent of LTO for all builds anymore. (It changed in 1.2 or thereabouts, I think.)
  - pbiggar 11y ago
    I'm being a little imprecise with my language (too much time since I've worked on C++), sorry. I want to eliminate the idea that libraries are distributed separately from the application that uses them.
    - pcwalton 11y ago
      So every game should vendor all the device-specific OpenGL implementations for every GPU they want to run on into their tree?
      - pbiggar 11y ago
        They already do on my xbox. But no, of course I'm not suggesting that - and that's a good counter-example, thanks. Really, I believe that the idea that the OS distributes libraries like libjpeg as archaic (distros strike me as an awful idea, tbh). That's part of the application, and it's the application's responsibility to ship a fix.

dottrap 11y ago

This isn't really a dynamic vs static linking problem. This is an ABI problem and affects static libraries too depending how each library was compiled and what its dependencies are. Go avoids this because the language designed in simpler ABI requirements, and also doesn't allow binary libraries at all (everything is expected to be source code). This completely avoids the problem since you only have one compiler producing all the components. But this isn't always a good thing. There are some good reasons for prebuilt libraries. (I had the unfortunate need to recently compile a huge C++ library on a slow ARM based embedded machine without the help of a cross-compiler. It took about 14 hours to compile. And I had to do it twice because the first time the build flags were wrong.) For more interesting perspective on dynamic vs. static libraries, read: iOS Static Libraries Are, Like, Really Bad, And Stuff (Radar 15800975)http://landonf.org/code/ios/Radar_15800975_iOS_Frameworks.20140112.html http://landonf.org/code/ios/Radar_15800975_iOS_Frameworks.20...

pbiggar 11y ago
> Go avoids this because the language designed in simpler ABI requirements, and also doesn't allow binary libraries at all. Yes, this is what I'm getting at! This mess of linking and ABIs is a mess, and it shouldn't exist unless it must, and it only must in very few cases. Thanks for the link, I'll take a read.
- felixgallo 11y ago
  For instance, all those cases where you'd like security issues fixed by updating a shared library, rather than finding and recompiling and redistributing every program on your system that may have used that library.
  - marssaxman 11y ago
    As an end user, I'd like to ensure that it's impossible to change the behavior of multiple unrelated applications by upgrading something on my system. I don't want a program to change until I decide to upgrade that specific program.

GauntletWizard 11y ago

Shared libraries still make sense for unix/linux distributions; There's a lot of little binaries, a lot of little processes, and the size of the full libstdc (or even one pruned to the binaries necessities) is high overhead compared to their actual size.