10 ms·
Mergeable libraries [video]
- covector 3y agoDocumentation at https://developer.apple.com/documentation/xcode/configuring-your-project-to-use-mergeable-libraries https://developer.apple.com/documentation/xcode/configuring-...
- enriquto 3y agogive me static αpε or give me death!
- superkuh 3y agoIt's really weird to me that they're talking about balancing static vs. dynamic in terms of the dev-end build time speed vs running the software load times. To me the dynamic vs static balancing act is about compatibility vs security where dynamic linked applications easily get lib security updates but the binary may or may not work with your system libs and statically compiling is more compatible but doesn't automatically get system lib security updates.
- liuliu 3y agoThey are talking about app-bundled dylibs, which cannot be updated separately due to the whole-bundle signing.
- notacoward 3y agoIn other words it's a fix for a somewhat Apple-specific (and in this case Apple-created) problem?
- astrange 3y agoiOS definitely isn't the only platform where apps ship their own versions of DLLs.
- notacoward 3y agoNo, not the only one, but the converse isn't rare either. Whether it's "bundles" or containers, a lot of people act as though bloat and slow build times because of static linking are inevitable when in fact the pain is self inflicted.
- deleted 3y ago[deleted]
- conradev 3y agoThe problem is not Apple-specific and their solution could be useful elsewhere. The specific optimization this achieves is during build time only: these new files are static libraries that are quicker to link. It is a small shift of some of the linking pipeline into the (parallel) builds of dependent libraries, rather than heaping it all onto the linker at the end of the build. The linker has to essentially re-link from scratch for every small change. Parallelization has long been known as the best way to speed up linking. This Apple change comes in addition to rewriting their linker to be massively parallel. Mold did it first, though: https://github.com/rui314/mold https://github.com/rui314/mold edit: lld did it first, mold improved upon it A faster development cycle is one of the coolest improvements that Zig brings – lightning fast recompiles, bringing the native development cycle closer to web speed. More languages should get this capability, and Apple needs it for Swift to get instantly updating SwiftUI previews. Static linking is required to get the best performance: no linker runtime cost, cross-language LTO and dead code elimination If this optimization is generally applicable and developers find it worthwhile, I could imagine this making its way to GCC-land. At the very least, gold needs replacing/rewriting to be competitive now. edited: for clarity
- ndesaulniers 3y ago> Mold did it first, though: https://github.com/rui314/mold https://github.com/rui314/mold Before LLD?
- conradev 3y agoI had forgotten about LLD, you are right, but also, they’re from the same author! :P
- covector 3y agolld has parallelism here and there, as well as ld64. I do believe ld64 is technically the first linker that started parsing object files in parallel. Parallel linking was conjectured impossible for a while without breaking static archive semantics (AB-BA problem). Michael has a good explanation in https://www.youtube.com/watch?v=ONGVraXbJOo https://www.youtube.com/watch?v=ONGVraXbJOo (minute 3 and beyond).
- layer8 3y agoWhole-bundle signing wouldn’t need to prevent security updates of individual libraries to be loaded when provided on the side, similar to cryptexes.
- RcouF1uZ4gsC 3y agoI think the whole security thing of relying on dynamic linking is a red herring. It is neither necessary nor sufficient. It will miss private dynamic libraries, Docker containers, potentially virtual envs. And it has a huge cost in terms of maintenance and backwards compatibility. In a world where we have regular builds from source, this model really doesn't have a place.
- notacoward 3y ago> It will miss private dynamic libraries, Docker containers, potentially virtual envs. Yes, it will. Does that make dynamic libraries a red herring, or does it mean these approaches all make the same mistake? Please think before you answer. Just because something's common doesn't mean it's wise or necessary. > it has a huge cost in terms of maintenance and backwards compatibility How much are those "huge costs" unique to that approach? Maintenance and backwards compatibility are still problems even with static linking or embedding into containers or any other alternative. It's disingenuous to count common problems against only one approach. The real tradeoff is between problems that are unique to each, and it's possible to disagree about which choice is best without stacking the deck for one side.
- nicoburns 3y agoUltimately if the app is unmaintained, then it's likely to have security issues even if it's using dynamic libraries. If it is maintained then it shouldn't be a big deal to update it.
- deleted 3y ago[deleted]
- layer8 3y agoSecurity is not black and white. Better to be able to fix some security issues of an unmaintained app than none. This is a bit of a tangent, but I don’t think it is sustainable in the long run to require the ever-growing volume of software in use to all be actively maintained. We should find a model of software development where we can achieve sufficient security without having to maintain and rebuild the world all the time.
- hexomancer 3y agoFrankly, I think dynamic libraries are less secure (at least when talking about a desktop environment, not servers) because sometimes package managers update dynamic library dependencies without actually checking if binary compatibility was preserved in the new version, causing various crashes and bugs (I experienced this first hand with an archlinux package of a software I was developing).
- lgg 3y agoThat may true on Linux, but it is not the case on Apple platforms. On Darwin the base system is immutable and the dylibs embedded in an application bundle cannot be changed without invalidating the codesignature. In order to have this sort of issue occur you need to opt out of multiple security settings that are enabled by default (such as the hardened runtime and library validation) AND then be sloppy with your use of relative paths or dlopen() calls.
- hexomancer 3y agoYeah this is the right way of doing it. The archlinux (and most other distributions') way is profoundly stupid.
- asveikau 3y agoI think you totally missed what the issue is here. In that scenario, Apple could still patch a system lib and it can break your application. It's not a question of the library updates being untrustworthy and code signing by the vendors fixes it. It's the library updates themselves breaking shit, not intentionally. Static linking prevents that, at the cost of disk space and memory and missing out on updates that might not (usually won't) break your app. Otoh, if you told me apple is more careful about breaking ABIs with updates to shared libraries, that is believable.
- lgg 3y agoThe GP specifically talked about an inadvertent dylib hijacking, which is prevented by the mechanisms I described. You are talking about a platform ABI break, which while unfortunate does occasionally happen due to significant technical issues (or sometimes by accident). Apple does spend a significant amount of effort to avoid breaking supported ABIs. There have definitely been issues though, and especially early in Mac OS X while learning how to deal with upstream open source projects that don't care about ABI. In this specific case the result was Apple funding the development of libc++ and factoring it into libc++ and libc++abi specifically do prevent this sort of breakage in the future. Another example would be about a decade ago when Apple removed the ssl headers for the SDKs and told developers to either use SecureTransport or include their own SSL libraries, since depending on openssl's ABI was not feasible.
- liuliu 3y agoWeird. Yes, link time is a big issue at development time, but with the new development of ld-prime and sold, seems trading that with dylib is not a big of a deal. On the other hand, the other big use of dylib I saw for apps to embed (other than providing 3rd-party libraries for other people to use), is to share code between main app binary and extensions, which mergeable library doesn't seem to support. I guess the biggest use is indeed to deliver 3rd-party close-source libraries?
- compiler-guy 3y agoMany, perhaps most?, dylibs that a Mac app ships with are all within the app bundle itself, and signed as such, so dylib sharing is not a common use case.
- liuliu 3y agoYeah, and the only reason they don't static link is because these dylibs are some 3rd-party closed-source libraries to avoid a lot of version clashes comparing to ship static lib naively. Otherwise you put some code in dylib so your app extension can share some code with the main app, but that is tricky due to different RAM usage restrictions between app extension v.s. main app.
- remram 3y agoRAM usage restrictions?
- liuliu 3y agoFor example: https://tailscale.com/blog/go-linker/ https://tailscale.com/blog/go-linker/ talks about how they slim down the Go runtime to workaround the 15MiB memory limit for network extension.
- remram 3y agoBut what does that have to do with dylibs? If you have RAM usage restrictions, aren't they for the whole process, so the main binary and all its dylibs?
- notacoward 3y agoThis looks really hype-y. AFAICT a "merged" binary is just a statically linked one. The only problem it solves is a self-inflicted one - failure of the old static linker to prune or de-duplicate stuff in linked libraries (particularly ObjC-specific stuff). It notably does not solve the main problem with static linking - old insecure code which would have been fixed with an updated dynamic library but is instead "stranded" by having already been copied into executables (or bundles and no reasonable person would quibble about that) never to change again. Also: provenance, supply chain, etc. Yes, I'm aware that superkuh beat me to this last point, and also that updating dynamic libraries can also cause breakage. But I still think it's important to note that this isn't really advancing the state of the art like Apple would like you to believe. It's just a new middle-of-the-road approach with its own possibly-positive tradeoffs and pitfalls. Nothing here that wasn't already considered at least thirty years ago, and whether they were right or wrong to choose another path is less relevant than the fact that it's not new.
- ridiculous_fish 3y agoThe problem being tackled here is link time in debug builds. This affects all platforms.
- roqi 3y ago> The problem being tackled here is link time in debug builds. This affects all platforms. I've worked on many projects, big and small, and the link time of debug builds was never a problem that was worth fixing. In fact, even today I was discussing with a coworker the inefficiencies of the project's build process, we literally commented that having to only link a huge subproject is a major blessing.
- compiler-guy 3y agoFor Google at least, link times are sufficiently important that the company has rewritten the linker from scratch twice--both open source. The Gnu-Gold linker which ships as part of gnu binutils and subsequently the ELF version of llvm's lld. So although you might not encounter issues with link times (debug or otherwise), it is a multi-million dollar problem for big companies like Google and Apple. Both in resources and engineer time.
- jevinskie 3y agoThey better keep the new linker open source like they did for ld64 (despite delays), or it is going to become a private ABI nightmare.
- baybal2 3y agoThis sounds very much like... prelink?
- viraptor 3y agoNot quite, but it's closer to statifier https://statifier.sourceforge.net/ https://statifier.sourceforge.net/ or ermine https://www.magicermine.com/ https://www.magicermine.com/ You can definitely do most of the same process in hacky ways on Linux already. I don't think you can use it for making the merged shared lib to use for compilation later... But it's only a few hacks away.
- planede 3y agoFrom the video: > Mergeable metadata roughly doubles the size of the dylib. I think they just bundle the static lib and dynamic lib together. I think there is a wasted opportunity to do something much smarter here, but maybe that will follow. We already compile static libraries with PIC, and use the same object files to compile dynamic libraries. In theory the two could be bundled in a way that shares most of the data, as the machine code is literally the same. They are both linked from the same object files.
- viraptor 3y agoReminds me of some recent work in nixpkgs where the idea is to get a mostly-static builds that only depend on libsystem on MacOS. https://github.com/NixOS/nixpkgs/issues/214611 https://github.com/NixOS/nixpkgs/issues/214611 At some point the difference does seem to become mostly "what do you do during linking".
- ChrisMarshallNY 3y agoIt won't make much difference to me, until it's supported in SPM (which will probably be soon).
- plorkyeran 3y ago> That does mean if you rely on bundle lookup support, you should update your minimum deployment version to iOS 12 or later to use mergeable libraries. But if you don't rely on these APIs, you can disable this support with the new linker option -no_merged_libraries_hook. Then you won't need to update your app's deployment version. I guess no one told the linker team that Xcode 15 was going to bump the minimum deployment target to iOS 12? Sort of awkward that no one caught that this bit of the talk isn't actually applicable to anyone.
- deleted 3y ago[deleted]
- JimDabell 3y agoHardly anybody supports iOS 11 these days anyway. Every device capable of running iOS 11 can be upgraded to iOS 12, which was released almost five years ago.
- mkoubaa 3y agoReally happy to see this and the child in me hopes to see something like this get "up streamed" into the ELF spec.
- graderjs 3y agoWill this make it possible to link in standalone libraries as like a single bundle easily and distribute that without having to worry about local installations? E.g., I tried to bundle ffmpeg as a library with my voice memo transcription MacOS app^0, but it was too difficult, so I just went with building a standalone ffmpeg binary (~40MB), and instrumenting it with a bash script that I called from the Objective-C code, ha ha ha! :) 0: https://apps.apple.com/us/app/wisprnote/id1671480366?mt=12 https://apps.apple.com/us/app/wisprnote/id1671480366?mt=12
- harrisi 3y agoJust a side note, what you're doing sounds like a violation of the FFmpeg license. https://www.ffmpeg.org/legal.html https://www.ffmpeg.org/legal.html
- pertymcpert 3y agoWhat exactly are they violating? Isn't running a binary with a bash script ok?
- harrisi 3y agoJust based on what the comment says, they're distributing a compiled FFmpeg, presumably not with source or attribution. I can't check to see if there's information in the app but there's no mention of it on the store page or anywhere else I can find either. They could be fine, but going through the checklist FFmpeg provides for legal considerations (not legal advice), they seem to be doing the opposite of all of them.
- pertymcpert 3y agoI haven't checked their specific app ($119.99) but it's common for packages to have OSS attribution and copyright notices as a dialog or something that the user can click on to see. Since they're presumably not modifying ffmpeg, there's also no source that needs to be provided. For example, my Chrysler car infotainment has an option in the system settings to see all of the copyright notices and OSS info that goes into the system.