5 ms·

It's really weird to me that they're talking about balancing static vs. dynamic in terms of the dev-end build time speed vs running the software load times. To

It's really weird to me that they're talking about balancing static vs. dynamic in terms of the dev-end build time speed vs running the software load times. To me the dynamic vs static balancing act is about compatibility vs security where dynamic linked applications easily get lib security updates but the binary may or may not work with your system libs and statically compiling is more compatible but doesn't automatically get system lib security updates.

liuliu 3y ago
They are talking about app-bundled dylibs, which cannot be updated separately due to the whole-bundle signing.
- notacoward 3y ago
  In other words it's a fix for a somewhat Apple-specific (and in this case Apple-created) problem?
  - astrange 3y ago
    iOS definitely isn't the only platform where apps ship their own versions of DLLs.
    - notacoward 3y ago
      No, not the only one, but the converse isn't rare either. Whether it's "bundles" or containers, a lot of people act as though bloat and slow build times because of static linking are inevitable when in fact the pain is self inflicted.
      - deleted 3y ago
        [deleted]
  - conradev 3y ago
    The problem is not Apple-specific and their solution could be useful elsewhere. The specific optimization this achieves is during build time only: these new files are static libraries that are quicker to link. It is a small shift of some of the linking pipeline into the (parallel) builds of dependent libraries, rather than heaping it all onto the linker at the end of the build. The linker has to essentially re-link from scratch for every small change. Parallelization has long been known as the best way to speed up linking. This Apple change comes in addition to rewriting their linker to be massively parallel. Mold did it first, though: https://github.com/rui314/mold https://github.com/rui314/mold edit: lld did it first, mold improved upon it A faster development cycle is one of the coolest improvements that Zig brings – lightning fast recompiles, bringing the native development cycle closer to web speed. More languages should get this capability, and Apple needs it for Swift to get instantly updating SwiftUI previews. Static linking is required to get the best performance: no linker runtime cost, cross-language LTO and dead code elimination If this optimization is generally applicable and developers find it worthwhile, I could imagine this making its way to GCC-land. At the very least, gold needs replacing/rewriting to be competitive now. edited: for clarity
    - ndesaulniers 3y ago
      > Mold did it first, though: https://github.com/rui314/mold https://github.com/rui314/mold Before LLD?
      - conradev 3y ago
        I had forgotten about LLD, you are right, but also, they’re from the same author! :P
      - covector 3y ago
        lld has parallelism here and there, as well as ld64. I do believe ld64 is technically the first linker that started parsing object files in parallel. Parallel linking was conjectured impossible for a while without breaking static archive semantics (AB-BA problem). Michael has a good explanation in https://www.youtube.com/watch?v=ONGVraXbJOo https://www.youtube.com/watch?v=ONGVraXbJOo (minute 3 and beyond).
- layer8 3y ago
  Whole-bundle signing wouldn’t need to prevent security updates of individual libraries to be loaded when provided on the side, similar to cryptexes.
RcouF1uZ4gsC 3y ago
I think the whole security thing of relying on dynamic linking is a red herring. It is neither necessary nor sufficient. It will miss private dynamic libraries, Docker containers, potentially virtual envs. And it has a huge cost in terms of maintenance and backwards compatibility. In a world where we have regular builds from source, this model really doesn't have a place.
- notacoward 3y ago
  > It will miss private dynamic libraries, Docker containers, potentially virtual envs. Yes, it will. Does that make dynamic libraries a red herring, or does it mean these approaches all make the same mistake? Please think before you answer. Just because something's common doesn't mean it's wise or necessary. > it has a huge cost in terms of maintenance and backwards compatibility How much are those "huge costs" unique to that approach? Maintenance and backwards compatibility are still problems even with static linking or embedding into containers or any other alternative. It's disingenuous to count common problems against only one approach. The real tradeoff is between problems that are unique to each, and it's possible to disagree about which choice is best without stacking the deck for one side.
  - nicoburns 3y ago
    Ultimately if the app is unmaintained, then it's likely to have security issues even if it's using dynamic libraries. If it is maintained then it shouldn't be a big deal to update it.
    - deleted 3y ago
      [deleted]
    - layer8 3y ago
      Security is not black and white. Better to be able to fix some security issues of an unmaintained app than none. This is a bit of a tangent, but I don’t think it is sustainable in the long run to require the ever-growing volume of software in use to all be actively maintained. We should find a model of software development where we can achieve sufficient security without having to maintain and rebuild the world all the time.
hexomancer 3y ago
Frankly, I think dynamic libraries are less secure (at least when talking about a desktop environment, not servers) because sometimes package managers update dynamic library dependencies without actually checking if binary compatibility was preserved in the new version, causing various crashes and bugs (I experienced this first hand with an archlinux package of a software I was developing).
- lgg 3y ago
  That may true on Linux, but it is not the case on Apple platforms. On Darwin the base system is immutable and the dylibs embedded in an application bundle cannot be changed without invalidating the codesignature. In order to have this sort of issue occur you need to opt out of multiple security settings that are enabled by default (such as the hardened runtime and library validation) AND then be sloppy with your use of relative paths or dlopen() calls.
  - hexomancer 3y ago
    Yeah this is the right way of doing it. The archlinux (and most other distributions') way is profoundly stupid.
  - asveikau 3y ago
    I think you totally missed what the issue is here. In that scenario, Apple could still patch a system lib and it can break your application. It's not a question of the library updates being untrustworthy and code signing by the vendors fixes it. It's the library updates themselves breaking shit, not intentionally. Static linking prevents that, at the cost of disk space and memory and missing out on updates that might not (usually won't) break your app. Otoh, if you told me apple is more careful about breaking ABIs with updates to shared libraries, that is believable.
    - lgg 3y ago
      The GP specifically talked about an inadvertent dylib hijacking, which is prevented by the mechanisms I described. You are talking about a platform ABI break, which while unfortunate does occasionally happen due to significant technical issues (or sometimes by accident). Apple does spend a significant amount of effort to avoid breaking supported ABIs. There have definitely been issues though, and especially early in Mac OS X while learning how to deal with upstream open source projects that don't care about ABI. In this specific case the result was Apple funding the development of libc++ and factoring it into libc++ and libc++abi specifically do prevent this sort of breakage in the future. Another example would be about a decade ago when Apple removed the ssl headers for the SDKs and told developers to either use SecureTransport or include their own SSL libraries, since depending on openssl's ABI was not feasible.

liuliu 3y ago

They are talking about app-bundled dylibs, which cannot be updated separately due to the whole-bundle signing.

notacoward 3y ago
In other words it's a fix for a somewhat Apple-specific (and in this case Apple-created) problem?
- astrange 3y ago
  iOS definitely isn't the only platform where apps ship their own versions of DLLs.
  - notacoward 3y ago
    No, not the only one, but the converse isn't rare either. Whether it's "bundles" or containers, a lot of people act as though bloat and slow build times because of static linking are inevitable when in fact the pain is self inflicted.
    - deleted 3y ago
      [deleted]
- conradev 3y ago
  The problem is not Apple-specific and their solution could be useful elsewhere. The specific optimization this achieves is during build time only: these new files are static libraries that are quicker to link. It is a small shift of some of the linking pipeline into the (parallel) builds of dependent libraries, rather than heaping it all onto the linker at the end of the build. The linker has to essentially re-link from scratch for every small change. Parallelization has long been known as the best way to speed up linking. This Apple change comes in addition to rewriting their linker to be massively parallel. Mold did it first, though: https://github.com/rui314/mold https://github.com/rui314/mold edit: lld did it first, mold improved upon it A faster development cycle is one of the coolest improvements that Zig brings – lightning fast recompiles, bringing the native development cycle closer to web speed. More languages should get this capability, and Apple needs it for Swift to get instantly updating SwiftUI previews. Static linking is required to get the best performance: no linker runtime cost, cross-language LTO and dead code elimination If this optimization is generally applicable and developers find it worthwhile, I could imagine this making its way to GCC-land. At the very least, gold needs replacing/rewriting to be competitive now. edited: for clarity
  - ndesaulniers 3y ago
    > Mold did it first, though: https://github.com/rui314/mold https://github.com/rui314/mold Before LLD?
    - conradev 3y ago
      I had forgotten about LLD, you are right, but also, they’re from the same author! :P
    - covector 3y ago
      lld has parallelism here and there, as well as ld64. I do believe ld64 is technically the first linker that started parsing object files in parallel. Parallel linking was conjectured impossible for a while without breaking static archive semantics (AB-BA problem). Michael has a good explanation in https://www.youtube.com/watch?v=ONGVraXbJOo https://www.youtube.com/watch?v=ONGVraXbJOo (minute 3 and beyond).
layer8 3y ago
Whole-bundle signing wouldn’t need to prevent security updates of individual libraries to be loaded when provided on the side, similar to cryptexes.

RcouF1uZ4gsC 3y ago

I think the whole security thing of relying on dynamic linking is a red herring. It is neither necessary nor sufficient. It will miss private dynamic libraries, Docker containers, potentially virtual envs. And it has a huge cost in terms of maintenance and backwards compatibility. In a world where we have regular builds from source, this model really doesn't have a place.

notacoward 3y ago
> It will miss private dynamic libraries, Docker containers, potentially virtual envs. Yes, it will. Does that make dynamic libraries a red herring, or does it mean these approaches all make the same mistake? Please think before you answer. Just because something's common doesn't mean it's wise or necessary. > it has a huge cost in terms of maintenance and backwards compatibility How much are those "huge costs" unique to that approach? Maintenance and backwards compatibility are still problems even with static linking or embedding into containers or any other alternative. It's disingenuous to count common problems against only one approach. The real tradeoff is between problems that are unique to each, and it's possible to disagree about which choice is best without stacking the deck for one side.
- nicoburns 3y ago
  Ultimately if the app is unmaintained, then it's likely to have security issues even if it's using dynamic libraries. If it is maintained then it shouldn't be a big deal to update it.
  - deleted 3y ago
    [deleted]
  - layer8 3y ago
    Security is not black and white. Better to be able to fix some security issues of an unmaintained app than none. This is a bit of a tangent, but I don’t think it is sustainable in the long run to require the ever-growing volume of software in use to all be actively maintained. We should find a model of software development where we can achieve sufficient security without having to maintain and rebuild the world all the time.

hexomancer 3y ago

Frankly, I think dynamic libraries are less secure (at least when talking about a desktop environment, not servers) because sometimes package managers update dynamic library dependencies without actually checking if binary compatibility was preserved in the new version, causing various crashes and bugs (I experienced this first hand with an archlinux package of a software I was developing).

lgg 3y ago
That may true on Linux, but it is not the case on Apple platforms. On Darwin the base system is immutable and the dylibs embedded in an application bundle cannot be changed without invalidating the codesignature. In order to have this sort of issue occur you need to opt out of multiple security settings that are enabled by default (such as the hardened runtime and library validation) AND then be sloppy with your use of relative paths or dlopen() calls.
- hexomancer 3y ago
  Yeah this is the right way of doing it. The archlinux (and most other distributions') way is profoundly stupid.
- asveikau 3y ago
  I think you totally missed what the issue is here. In that scenario, Apple could still patch a system lib and it can break your application. It's not a question of the library updates being untrustworthy and code signing by the vendors fixes it. It's the library updates themselves breaking shit, not intentionally. Static linking prevents that, at the cost of disk space and memory and missing out on updates that might not (usually won't) break your app. Otoh, if you told me apple is more careful about breaking ABIs with updates to shared libraries, that is believable.
  - lgg 3y ago
    The GP specifically talked about an inadvertent dylib hijacking, which is prevented by the mechanisms I described. You are talking about a platform ABI break, which while unfortunate does occasionally happen due to significant technical issues (or sometimes by accident). Apple does spend a significant amount of effort to avoid breaking supported ABIs. There have definitely been issues though, and especially early in Mac OS X while learning how to deal with upstream open source projects that don't care about ABI. In this specific case the result was Apple funding the development of libc++ and factoring it into libc++ and libc++abi specifically do prevent this sort of breakage in the future. Another example would be about a decade ago when Apple removed the ssl headers for the SDKs and told developers to either use SecureTransport or include their own SSL libraries, since depending on openssl's ABI was not feasible.