5 ms·

Apple Releases New Static Linker

167 points by hasheddan 3y ago

WhereIsTheTruth 3y ago
Hopefully this will support more architectures in the future, that's pretty impressive
- scottlamb 3y ago
  > Hopefully this will support more architectures in the future, that's pretty impressive What architectures would it support? I assume it handles aarch64. It may already handle x86-64. macOS and iOS don't run on any others. I doubt Apple would be interested in making it support other OSs that run on more architectures. (It's probably a faux pax of me to say this, but I wonder if the parent comment was written by a LLM...)
  - photonbeam 3y ago
    They would probably be internally supporting riscv, given job postings
    - scottlamb 3y ago
      Interesting point. Maybe. On the other hand, afaik they're just doing risc-v on tiny things like security enclaves and they might have a right different executable format and toolchain for those.
  - WhereIsTheTruth 3y ago
    > I wonder if the parent comment was written by a LLM. That's pretty disrespectful, I expect an apology from you > I doubt Apple would be interested in making it support other OSs that run on more architectures I don't think Apple is only using ARM and their chip for their whole stack, in fact, their servers run something else, and they support software that runs on other OSes wich run plenty of different architectures, including X86_64, and they are exploring a plethora of architectures [1] > I assume it handles > It may already handle "Faux pas" indeed, let's stop assuming and let's stay positive, nice and respectful to each other [1] - https://www.techpowerup.com/298936/report-apple-to-move-a-part-of-its-embedded-cores-to-risc-v-stepping-away-from-arm-isa https://www.techpowerup.com/298936/report-apple-to-move-a-pa...
    - scottlamb 3y ago
      I'll keep my suspicions to myself next time. Folks are increasingly going to wonder this when there's a short/vague comment that doesn't quite match their understanding of the world. I wouldn't have wondered the same with the additional context you've supplied here, e.g.: > I don't think Apple is only using ARM and their chip for their whole stack, in fact, their servers run something else, and they support software that runs on other OSes wich run plenty of different architectures, including X86_64, and they are exploring a plethora of architectures [1] Now I understand why you suggested it might run on other architectures / OSs. I'm skeptical though. They may develop/run software on Linux/Windows, but I haven't seen any examples of them releasing a development toolchain (e.g. any part of Xcode) for those OSs (with the obvious exception of platform-independent LLVM improvements). Likewise microprocessor-level stuff (afaict everything they do with risc-v).
  - sacnoradhq 3y ago
    Do you also go around accusing people of being witches? How on Earth would you know, much less prove, text originated from LLM? Your comments come across as meaningless if you're on a mission to disrespect commentors frivolously.
EscapeFromNY 3y ago
Here's a benchmark from the author of the mold linker: https://twitter.com/nabbisen/status/1665936938837344256 https://twitter.com/nabbisen/status/1665936938837344256 > I installed Xcode 15 Beta to try out the new Apple's linker. It seems that my mold/sold linker is still a bit faster than theirs, but the gap has now reduced to ~60% when linking the sold linker itself on my Mac Studio. Apple's new linker is in the same league as mine. Great job!
- covector 3y ago
  It looks like the gap between the two is very little (< 100ms) and the Apple linker performs slightly better than mold on large applications: https://twitter.com/SmileyKeith/status/1666168273199845377 https://twitter.com/SmileyKeith/status/1666168273199845377
- armchairhacker 3y ago
  Why didn't Apple just use mold?
  - taspeotis 3y ago
    It's AGPL?
    - canucker2016 3y ago
      ...and the macos/iOS-supported version, named 'sold', is commercial and non-AGPL. see https://github.com/bluewhalesystems/sold https://github.com/bluewhalesystems/sold
      - shagie 3y ago
        At $8/user/month (while I realize that still likely be pocket change to Apple)... how much is that and what is the risk that the company changes the licensing once you're using it? Large companies (Apple, Amazon, and the like) would likely buy the company rather than license their product (see Dark Sky (now WeatherKit), Kiva (now Amazon Robotics - https://www.fromscratchradio.org/show/mick-mountz https://www.fromscratchradio.org/show/mick-mountz 23:22 )) so that they can't lose control over something that they want when management for one of there tools changes. And so why use mold (license problems and they wouldn't want to impose the AGPL on all their developers), or be a reseller of sold ($8/month/user - how many developers are in the Apple developer program) - it comes down to "it is better for us to develop our own and release it so that there is certainty about the license for any developers who use it."
sunday_serif 3y ago
Can anyone recommend a resource on how linkers work other than the "Linkers and Loaders" book? Maybe a linker repo that is easier to read through, or a more modern book or series of articles on the topic? No shame on the book, it is a classic, but it is certainly a bit long in the tooth. I am very interested in linkers, but find them a bit inaccessible to learn about.
- canucker2016 3y ago
  Look at Gold and Mold Gold blog posts: https://www.airs.com/blog/archives/38 https://www.airs.com/blog/archives/38 and https://en.wikipedia.org/wiki/Gold_(linker) https://en.wikipedia.org/wiki/Gold_(linker) Design and implementation of Mold: https://github.com/rui314/mold/blob/main/docs/design.md https://github.com/rui314/mold/blob/main/docs/design.md
- jjtheblunt 3y ago
  you might get good mileage out of web pages for ld and dl_open and related commands and functions.
- zzzred 3y ago
  https://opensource.apple.com/source/ld64/ld64-136/doc/design/linker.html https://opensource.apple.com/source/ld64/ld64-136/doc/design...
dzdt 3y ago
In my dream world 64 bit OS, every program would be statically relocated at install time. It would permanently have assigned its own segment of the global system address space. Every library it depends on would also have a permanently assigned segment of the global address space. Also at install time the application would run all its initialization steps including GUI object allocation, with GUI object handles permanently assigned in their own segment of a global address space. The linked and initialized state of the application would be stored. Then launching the application would be more similar to a resume-from-suspend operation: memmap the pre-initialized state into its already reserved segment of address space and reactive the frozen gui objects. Probably I'm naive: I've never built any of this kind of low-level OS stuff. But I really do think there is room to re-envision some things which were decided in the 1970's based on constraints of 1970's hardware.
- dietr1ch 3y ago
  Sounds cool until you think that you have to maintain this after new things are installed, removed and upgraded. ASLR is around for a reason, having everything statically fixed may bring security concerns. And, how do you run multiple programs out of the same binary?
  - dzdt 3y ago
    Running multiple instances: launch+fork with copy-on-write semantics? Re: ASLR, yeah I think you would lose most of that mitigation. Would it be worth it to make launching even the biggest programs instantaneous? The usability upside could be something amazing; maybe some other security mitigations could take the place of ASLR.
    - dietr1ch 3y ago
      Security mitigations given everything you can run has a fixed address sounds cool, but wouldn't the copy-on-write semantics kind of break that? I guess you might still be able to do something though.
- l33t233372 3y ago
CalChris 3y ago
Apple calls their (now old) linker, ld64, a static linker. On the other hand, dyld is their dynamic linker. This is wrong but Apple then is a $2.8T company. They should call ld64 their system linker or just their linker. They have of late altogether disabled static linking for 3rd part applications. All applications must now start with dyld. You only have dynamically linked applications. This was done for several reasons, but application start time and process security were probably tied for first on the list. ld64 is a fork of the old and slow GCC ld linker. It has accumulated a thousand switches and features and hacks over the years. The move to only dynamic linking meant that most of them can never be used again. So now is a great good time to do a complete rewrite, especially with lld and mold as models; both were architected by the awesome Rui Ueyama. Throw away all the crap, parallelize everything and yeah, it should be fast.
- eschaton 3y ago
  What do you mean by “disabled static linking?” You can make and use static libraries if you want, but on Darwin-based platforms it’s never been supported to make direct system calls; everything has been required to link libSystem.dylib for system call access all along, and dyld has been unavoidable all along.
- covector 3y ago
  > ld64 is a fork of the old and slow GCC ld linker. ld64 was a complete rewrite, circa 2005, courtesy of Nick Kledzik. `opensource.apple.com` has still the old Changelog(s). https://github.com/rotten-apples/ld64/blob/master/ChangeLog https://github.com/rotten-apples/ld64/blob/master/ChangeLog
  - CalChris 3y ago
    Thanks for the correction. I did not know this. He even has some patents on dynamic linking. https://patents.justia.com/inventor/nick-kledzik https://patents.justia.com/inventor/nick-kledzik

167 points by hasheddan 3y ago

WhereIsTheTruth 3y ago

Hopefully this will support more architectures in the future, that's pretty impressive

scottlamb 3y ago
> Hopefully this will support more architectures in the future, that's pretty impressive What architectures would it support? I assume it handles aarch64. It may already handle x86-64. macOS and iOS don't run on any others. I doubt Apple would be interested in making it support other OSs that run on more architectures. (It's probably a faux pax of me to say this, but I wonder if the parent comment was written by a LLM...)
- photonbeam 3y ago
  They would probably be internally supporting riscv, given job postings
  - scottlamb 3y ago
    Interesting point. Maybe. On the other hand, afaik they're just doing risc-v on tiny things like security enclaves and they might have a right different executable format and toolchain for those.
- WhereIsTheTruth 3y ago
  > I wonder if the parent comment was written by a LLM. That's pretty disrespectful, I expect an apology from you > I doubt Apple would be interested in making it support other OSs that run on more architectures I don't think Apple is only using ARM and their chip for their whole stack, in fact, their servers run something else, and they support software that runs on other OSes wich run plenty of different architectures, including X86_64, and they are exploring a plethora of architectures [1] > I assume it handles > It may already handle "Faux pas" indeed, let's stop assuming and let's stay positive, nice and respectful to each other [1] - https://www.techpowerup.com/298936/report-apple-to-move-a-part-of-its-embedded-cores-to-risc-v-stepping-away-from-arm-isa https://www.techpowerup.com/298936/report-apple-to-move-a-pa...
  - scottlamb 3y ago
    I'll keep my suspicions to myself next time. Folks are increasingly going to wonder this when there's a short/vague comment that doesn't quite match their understanding of the world. I wouldn't have wondered the same with the additional context you've supplied here, e.g.: > I don't think Apple is only using ARM and their chip for their whole stack, in fact, their servers run something else, and they support software that runs on other OSes wich run plenty of different architectures, including X86_64, and they are exploring a plethora of architectures [1] Now I understand why you suggested it might run on other architectures / OSs. I'm skeptical though. They may develop/run software on Linux/Windows, but I haven't seen any examples of them releasing a development toolchain (e.g. any part of Xcode) for those OSs (with the obvious exception of platform-independent LLVM improvements). Likewise microprocessor-level stuff (afaict everything they do with risc-v).
- sacnoradhq 3y ago
  Do you also go around accusing people of being witches? How on Earth would you know, much less prove, text originated from LLM? Your comments come across as meaningless if you're on a mission to disrespect commentors frivolously.

EscapeFromNY 3y ago

Here's a benchmark from the author of the mold linker: https://twitter.com/nabbisen/status/1665936938837344256 https://twitter.com/nabbisen/status/1665936938837344256 > I installed Xcode 15 Beta to try out the new Apple's linker. It seems that my mold/sold linker is still a bit faster than theirs, but the gap has now reduced to ~60% when linking the sold linker itself on my Mac Studio. Apple's new linker is in the same league as mine. Great job!

covector 3y ago
It looks like the gap between the two is very little (< 100ms) and the Apple linker performs slightly better than mold on large applications: https://twitter.com/SmileyKeith/status/1666168273199845377 https://twitter.com/SmileyKeith/status/1666168273199845377
armchairhacker 3y ago
Why didn't Apple just use mold?
- taspeotis 3y ago
  It's AGPL?
  - canucker2016 3y ago
    ...and the macos/iOS-supported version, named 'sold', is commercial and non-AGPL. see https://github.com/bluewhalesystems/sold https://github.com/bluewhalesystems/sold
    - shagie 3y ago
      At $8/user/month (while I realize that still likely be pocket change to Apple)... how much is that and what is the risk that the company changes the licensing once you're using it? Large companies (Apple, Amazon, and the like) would likely buy the company rather than license their product (see Dark Sky (now WeatherKit), Kiva (now Amazon Robotics - https://www.fromscratchradio.org/show/mick-mountz https://www.fromscratchradio.org/show/mick-mountz 23:22 )) so that they can't lose control over something that they want when management for one of there tools changes. And so why use mold (license problems and they wouldn't want to impose the AGPL on all their developers), or be a reseller of sold ($8/month/user - how many developers are in the Apple developer program) - it comes down to "it is better for us to develop our own and release it so that there is certainty about the license for any developers who use it."

sunday_serif 3y ago

Can anyone recommend a resource on how linkers work other than the "Linkers and Loaders" book? Maybe a linker repo that is easier to read through, or a more modern book or series of articles on the topic? No shame on the book, it is a classic, but it is certainly a bit long in the tooth. I am very interested in linkers, but find them a bit inaccessible to learn about.

canucker2016 3y ago
Look at Gold and Mold Gold blog posts: https://www.airs.com/blog/archives/38 https://www.airs.com/blog/archives/38 and https://en.wikipedia.org/wiki/Gold_(linker) https://en.wikipedia.org/wiki/Gold_(linker) Design and implementation of Mold: https://github.com/rui314/mold/blob/main/docs/design.md https://github.com/rui314/mold/blob/main/docs/design.md
jjtheblunt 3y ago
you might get good mileage out of web pages for ld and dl_open and related commands and functions.
zzzred 3y ago
https://opensource.apple.com/source/ld64/ld64-136/doc/design/linker.html https://opensource.apple.com/source/ld64/ld64-136/doc/design...

dzdt 3y ago

In my dream world 64 bit OS, every program would be statically relocated at install time. It would permanently have assigned its own segment of the global system address space. Every library it depends on would also have a permanently assigned segment of the global address space. Also at install time the application would run all its initialization steps including GUI object allocation, with GUI object handles permanently assigned in their own segment of a global address space. The linked and initialized state of the application would be stored. Then launching the application would be more similar to a resume-from-suspend operation: memmap the pre-initialized state into its already reserved segment of address space and reactive the frozen gui objects. Probably I'm naive: I've never built any of this kind of low-level OS stuff. But I really do think there is room to re-envision some things which were decided in the 1970's based on constraints of 1970's hardware.

dietr1ch 3y ago
Sounds cool until you think that you have to maintain this after new things are installed, removed and upgraded. ASLR is around for a reason, having everything statically fixed may bring security concerns. And, how do you run multiple programs out of the same binary?
- dzdt 3y ago
  Running multiple instances: launch+fork with copy-on-write semantics? Re: ASLR, yeah I think you would lose most of that mitigation. Would it be worth it to make launching even the biggest programs instantaneous? The usability upside could be something amazing; maybe some other security mitigations could take the place of ASLR.
  - dietr1ch 3y ago
    Security mitigations given everything you can run has a fixed address sounds cool, but wouldn't the copy-on-write semantics kind of break that? I guess you might still be able to do something though.
l33t233372 3y ago

CalChris 3y ago

Apple calls their (now old) linker, ld64, a static linker. On the other hand, dyld is their dynamic linker. This is wrong but Apple then is a $2.8T company. They should call ld64 their system linker or just their linker. They have of late altogether disabled static linking for 3rd part applications. All applications must now start with dyld. You only have dynamically linked applications. This was done for several reasons, but application start time and process security were probably tied for first on the list. ld64 is a fork of the old and slow GCC ld linker. It has accumulated a thousand switches and features and hacks over the years. The move to only dynamic linking meant that most of them can never be used again. So now is a great good time to do a complete rewrite, especially with lld and mold as models; both were architected by the awesome Rui Ueyama. Throw away all the crap, parallelize everything and yeah, it should be fast.

eschaton 3y ago
What do you mean by “disabled static linking?” You can make and use static libraries if you want, but on Darwin-based platforms it’s never been supported to make direct system calls; everything has been required to link libSystem.dylib for system call access all along, and dyld has been unavoidable all along.
covector 3y ago
> ld64 is a fork of the old and slow GCC ld linker. ld64 was a complete rewrite, circa 2005, courtesy of Nick Kledzik. `opensource.apple.com` has still the old Changelog(s). https://github.com/rotten-apples/ld64/blob/master/ChangeLog https://github.com/rotten-apples/ld64/blob/master/ChangeLog
- CalChris 3y ago
  Thanks for the correction. I did not know this. He even has some patents on dynamic linking. https://patents.justia.com/inventor/nick-kledzik https://patents.justia.com/inventor/nick-kledzik