5 ms·

This looks really hype-y. AFAICT a "merged" binary is just a statically linked one. The only problem it solves is a self-inflicted one - failure of the old stat

This looks really hype-y. AFAICT a "merged" binary is just a statically linked one. The only problem it solves is a self-inflicted one - failure of the old static linker to prune or de-duplicate stuff in linked libraries (particularly ObjC-specific stuff). It notably does not solve the main problem with static linking - old insecure code which would have been fixed with an updated dynamic library but is instead "stranded" by having already been copied into executables (or bundles and no reasonable person would quibble about that) never to change again. Also: provenance, supply chain, etc. Yes, I'm aware that superkuh beat me to this last point, and also that updating dynamic libraries can also cause breakage. But I still think it's important to note that this isn't really advancing the state of the art like Apple would like you to believe. It's just a new middle-of-the-road approach with its own possibly-positive tradeoffs and pitfalls. Nothing here that wasn't already considered at least thirty years ago, and whether they were right or wrong to choose another path is less relevant than the fact that it's not new.

ridiculous_fish 3y ago
The problem being tackled here is link time in debug builds. This affects all platforms.
- roqi 3y ago
  > The problem being tackled here is link time in debug builds. This affects all platforms. I've worked on many projects, big and small, and the link time of debug builds was never a problem that was worth fixing. In fact, even today I was discussing with a coworker the inefficiencies of the project's build process, we literally commented that having to only link a huge subproject is a major blessing.
  - compiler-guy 3y ago
    For Google at least, link times are sufficiently important that the company has rewritten the linker from scratch twice--both open source. The Gnu-Gold linker which ships as part of gnu binutils and subsequently the ELF version of llvm's lld. So although you might not encounter issues with link times (debug or otherwise), it is a multi-million dollar problem for big companies like Google and Apple. Both in resources and engineer time.
    - roqi 3y ago
      > So although you might not encounter issues with link times (debug or otherwise), it is a multi-million dollar problem for big companies like Google and Apple. Both in resources and engineer time. I appreciate your appeal to authority, but I worked at a FANG on a project that had over 20 GB worth of static and dynamic/shared libraries. Linking was never an issue.
      - compiler-guy 3y ago
        Err, "Google has rewritten the linker twice. Both times with the stated goal to make link times much faster." isn't an appeal to authority. It's evidence that the company has found speeding up linking to be worth millions of dollars. Otherwise it wouldn't have done it. They surely weren't doing it for fun.
        roqi 3y ago
        > Err, "Google has rewritten the linker twice. Both times with the stated goal to make link times much faster." isn't an appeal to authority. It is, and a very weak one considering Google has a history of getting people to work on promotion-oriented projects. https://news.ycombinator.com/item?id=31261488 https://news.ycombinator.com/item?id=31261488 > It's evidence that the company has found speeding up linking to be worth millions of dollars. It really isn't. You're buying into the fallacy that a FANG can never do wrong and all their developers are infallible and walk on water. All you're able to say is google did this and google did that, and you're telling that to a guy who has first-hand experience on how this and that is actually made. You never mentioned any technical aspect or more importantly performance differences. You only name-dropped Google, and to a guy who already worked at a FANG. Linking was never an issue.
        deleted 3y ago
        [deleted]
        KerrAvon 3y ago
        There are many FAANG customers who care about link time; some of them are also FAANGs, but certainly not all. You're falling into the libertarian trap of thinking that because it didn't happen in your experience, it could not possibly happen to anyone.
roqi 3y ago
> But I still think it's important to note that this isn't really advancing the state of the art like Apple would like you to believe. It's just a new middle-of-the-road approach with its own possibly-positive tradeoffs and pitfalls. It also seems that this new library format barely solves any problem and in the process bumps up the number of library types that developers need to onboard and possibly support.
- Reason077 3y ago
  > "It also seems that this new library format ... bumps up the number of library types that developers need to onboard and possibly support." No, a mergeable library is just a special type of dynamic library, which adds some additional features and metadata. I can't imagine why you'd distribute both mergeable and non-mergeable dylibs? In fact, rather than having to support more library types, developers may actually have to support less - because it may eliminate the need to ship both a static lib and a .dylib with your framework?
lgg 3y ago
There is a bit more to it than that. Yes, it was always possible to use a mess of build rules and shell scripts to make your debug and release builds swap between fully static and dynamic libraries, but it was a lot of work, and was difficult to maintain. The novelty of mergeable dylibs is that they now make it trivial to switch between the two without all of that work. In particular it solves two large problems people tended to run into: 1. Static archives and dynamic libraries have different semantics with respect to symbol lookup. In particular, due to two level namespaces multiple dylibs can exports the same symbol without it being a runtime collision since the binary importing them stores the library a symbol came from in addition to the name. This is different from static archives where you have sets of symbols brought in by .o file. That means it is often non-trivial to switch between dynamic libraries and static archives. Mergeable libraries solve this by allowing you to use the semantics of dynamic libraries and two level namespaces for content that will be statically linked. 2. Most people use frameworks, not raw dylibs. They do that for a lot of reasons, but the biggest one is to allow them to distribute non-code resources that are associated with the library. This is a common problem that has been solved in various ways (Windows embeds the resources in the DLL files, classic Mac OS depended on resource forks, etc). Mergeable dylibs are completely supported by the runtime in such a way that enough of the dylib's identity is preserved so that things like NSBundle continue to work as a way to find the bundles resources despite the code itself being merged.
compiler-guy 3y ago
I'm not seeing any hype. They have shipped a nice new feature for their build system. I don't see any claims of it advancing the state of the art, or life-changing, just that it improves certain work flows in an easy way, and provide a convenient way to use it for their developers.

ridiculous_fish 3y ago

The problem being tackled here is link time in debug builds. This affects all platforms.

roqi 3y ago
> The problem being tackled here is link time in debug builds. This affects all platforms. I've worked on many projects, big and small, and the link time of debug builds was never a problem that was worth fixing. In fact, even today I was discussing with a coworker the inefficiencies of the project's build process, we literally commented that having to only link a huge subproject is a major blessing.
- compiler-guy 3y ago
  For Google at least, link times are sufficiently important that the company has rewritten the linker from scratch twice--both open source. The Gnu-Gold linker which ships as part of gnu binutils and subsequently the ELF version of llvm's lld. So although you might not encounter issues with link times (debug or otherwise), it is a multi-million dollar problem for big companies like Google and Apple. Both in resources and engineer time.
  - roqi 3y ago
    > So although you might not encounter issues with link times (debug or otherwise), it is a multi-million dollar problem for big companies like Google and Apple. Both in resources and engineer time. I appreciate your appeal to authority, but I worked at a FANG on a project that had over 20 GB worth of static and dynamic/shared libraries. Linking was never an issue.
    - compiler-guy 3y ago
      Err, "Google has rewritten the linker twice. Both times with the stated goal to make link times much faster." isn't an appeal to authority. It's evidence that the company has found speeding up linking to be worth millions of dollars. Otherwise it wouldn't have done it. They surely weren't doing it for fun.
      - roqi 3y ago
        > Err, "Google has rewritten the linker twice. Both times with the stated goal to make link times much faster." isn't an appeal to authority. It is, and a very weak one considering Google has a history of getting people to work on promotion-oriented projects. https://news.ycombinator.com/item?id=31261488 https://news.ycombinator.com/item?id=31261488 > It's evidence that the company has found speeding up linking to be worth millions of dollars. It really isn't. You're buying into the fallacy that a FANG can never do wrong and all their developers are infallible and walk on water. All you're able to say is google did this and google did that, and you're telling that to a guy who has first-hand experience on how this and that is actually made. You never mentioned any technical aspect or more importantly performance differences. You only name-dropped Google, and to a guy who already worked at a FANG. Linking was never an issue.
        deleted 3y ago
        [deleted]
        KerrAvon 3y ago
        There are many FAANG customers who care about link time; some of them are also FAANGs, but certainly not all. You're falling into the libertarian trap of thinking that because it didn't happen in your experience, it could not possibly happen to anyone.

roqi 3y ago

> But I still think it's important to note that this isn't really advancing the state of the art like Apple would like you to believe. It's just a new middle-of-the-road approach with its own possibly-positive tradeoffs and pitfalls. It also seems that this new library format barely solves any problem and in the process bumps up the number of library types that developers need to onboard and possibly support.

Reason077 3y ago
> "It also seems that this new library format ... bumps up the number of library types that developers need to onboard and possibly support." No, a mergeable library is just a special type of dynamic library, which adds some additional features and metadata. I can't imagine why you'd distribute both mergeable and non-mergeable dylibs? In fact, rather than having to support more library types, developers may actually have to support less - because it may eliminate the need to ship both a static lib and a .dylib with your framework?

lgg 3y ago

There is a bit more to it than that. Yes, it was always possible to use a mess of build rules and shell scripts to make your debug and release builds swap between fully static and dynamic libraries, but it was a lot of work, and was difficult to maintain. The novelty of mergeable dylibs is that they now make it trivial to switch between the two without all of that work. In particular it solves two large problems people tended to run into: 1. Static archives and dynamic libraries have different semantics with respect to symbol lookup. In particular, due to two level namespaces multiple dylibs can exports the same symbol without it being a runtime collision since the binary importing them stores the library a symbol came from in addition to the name. This is different from static archives where you have sets of symbols brought in by .o file. That means it is often non-trivial to switch between dynamic libraries and static archives. Mergeable libraries solve this by allowing you to use the semantics of dynamic libraries and two level namespaces for content that will be statically linked. 2. Most people use frameworks, not raw dylibs. They do that for a lot of reasons, but the biggest one is to allow them to distribute non-code resources that are associated with the library. This is a common problem that has been solved in various ways (Windows embeds the resources in the DLL files, classic Mac OS depended on resource forks, etc). Mergeable dylibs are completely supported by the runtime in such a way that enough of the dylib's identity is preserved so that things like NSBundle continue to work as a way to find the bundles resources despite the code itself being merged.

compiler-guy 3y ago

I'm not seeing any hype. They have shipped a nice new feature for their build system. I don't see any claims of it advancing the state of the art, or life-changing, just that it improves certain work flows in an easy way, and provide a convenient way to use it for their developers.