8 ms·

Achieving an open-source implementation of Apple Code Signing and notarization

309 points by indygreg2 4y ago

benced 4y ago
Beyond clownish that apple didn’t just release this themselves. Kudos to the creators for doing the job apple should have done.
xrd 4y ago
This is so fantastic. I wouldn't have believed it was possible had I not seen it here. I maintain multiple old OSX machines solely for this purpose; migrating to Linux for this step makes me feel so much safer for my CI build systems.
Sytten 4y ago
The only thing left to solve is to actually build cross platform. You can do with rust cross but you have to build your own docker images on macos first since apple licensing prohibits redistribution of binaries. It would save us so much money in github CI minutes.
- funhatch 4y ago
  https://pmbaty.com/iosbuildenv/ https://pmbaty.com/iosbuildenv/
- bogwog 4y ago
  You mean building mac/ios apps on Linux? That’s relatively easy to do, with the only questionable/legally gray area being access to the iOS/Mac SDKs. I’m not a lawyer, but it’s my understanding that the Google v Oracle case established that APIs aren’t protected by copyright. So if we can use the headers from e.g. the iOS SDK on Linux wtohout worrying about copyright issues (Apple developer agreements may be an issue though), it should be possible to autogenerate stub libraries for linking purposes. With that, you can compile and link a binary for iOS or Mac on Linux. Next, you’ll need to implement the packaging and signing portion. Packaging (generating an app bundle) is easy, but codesigning requires Apple’s codesign utility, which doesn’t work outside of Mac, even with a project like Darling (last I checked, anyways). But if the OP does implement a working codesign alternative, then that means it should be possible to create an iOS/Mac app that can be distributed on the App Store entirely from Linux with no Xcode, virtual machines, cloud macs, etc. The only gate left then is the Apple tax/signing certificate you need to buy for $99/year. But that might change with all of the antitrust pressure Apple is facing, including the recent EU law that would force Apple to allow competing app stores.
  - TazeTSchnitzel 4y ago
    Zig already does this I think, they committed the Apple headers to their repo.
  - tssva 4y ago
    "I’m not a lawyer, but it’s my understanding that the Google v Oracle case established that APIs aren’t protected by copyright." I'm no lawyer either but I have read the Supreme Court's decision in the case. It doesn't address whether APIs are protected by copyright. The Supreme Court generally makes the most limited ruling needed in order to resolve a case. In this case there were two issues at play. -Are API's copyrightable? -If so, was Google's use of the APIs fair use under copyright law as a jury had previously ruled? In order to make the most limited ruling needed to decide the case the Supreme Court first addressed the fair use question with an assumption that APIs could be copyrighted. The Court decided that Google's use did fall under fair use and so the case could be decided by the fair use question alone. They therefore didn't bother to address the question of whether APIs can be copyrighted. As it stands right now there is a U.S. Court of Appeals ruling in the Oracle case that APIs are copyrightable which until the Supreme Court does address the issue means they are as far as the courts are concerned. Fair use is a case-by-case defense and the Supreme Court's decision doesn't mean that all use of an API is fair use. This is all very much still all up in the air and any use of the headers from the iOS SDK should be concerned about copyright issues.
- roblabla 4y ago
  This is actually a solved problem, using osxcross[0]. The experience is honestly very smooth, and we don't require any apple proprietary binaries. The only thing apple-proprietary is their SDK (containing the header files for compiling, and tbd files for linking), which can be downloaded from apple's website (at least if you have a developer account), or from various GitHub projects archiving them. [0]: https://github.com/tpoechtrager/osxcross https://github.com/tpoechtrager/osxcross
- tedmielczarek 4y ago
  We achieved this for Firefox macOS builds years ago. (Greg was there for that work as well. ) You do need to have an SDK accessible, but storing one internally and using it in CI was deemed acceptable. The impact on CI build times was hard to overstate. We were already building our own clang binaries for various reasons, so it was mostly a matter of making an SDK available and ensuring that the right compiler options were passed in (since running clang on macOS sets a bunch of defaults that you don't get even if you pass -target x86_64-apple-darwin).
xet7 4y ago
How to build GUI apps for iOS and macOS with Rust ? Can this notarize apps made with other programming languages?
- norman784 4y ago
  As far I can tell, you can compile the binary and sign your app. # From a Git checkout $ cargo run --bin rcodesign -- --help $ cargo install --bin rcodesign # Remote install. $ cargo install --git https://github.com/indygreg/PyOxidizer https://github.com/indygreg/PyOxidizer --branch main --bin rcodesign apple-codesign
  - rvz 4y ago
    - steveklabnik 4y ago
      Your parent is clearly answering the OP’s second question but not the first, come on. I’m aware of production iOS apps using Rust that are either: 1. Using rust for core logic but still using Swift/obj c for UI. 2. Pure Rust but don’t use native widgets. Like games or similar custom UI. That said it is in my understanding theoretically possible but I’m not aware of a paved cow path here. It’s also not my area of specialty though.
- TazeTSchnitzel 4y ago
  You could use Qt and cross-compile.
- indygreg2 4y ago
  rcodesign can sign and notarize applications written in any language. From rcodesign's perspective, Rust is an implementation detail of the tool itself.
ridiculous_fish 4y ago
I think Apple's official code signing tool is also open source? https://github.com/Apple-FOSS-Mirror/security_systemkeychain/blob/master/src/codesign.cpp https://github.com/Apple-FOSS-Mirror/security_systemkeychain... xar is another tool for codesigning Mac installer packages, which runs on Linux. I've used it successfully in the past. https://users.wfu.edu/cottrell/productsign/productsign_linux.html https://users.wfu.edu/cottrell/productsign/productsign_linux...
- bri3d 4y ago
  Good pointers indeed - Apple's code signing tool is open source, but heavily dependent on Cocoa/OSX libraries like CoreFoundation. I'm not sure if it works with any of the open-source reimplementations of these libraries, although I kind of want to try now, just for fun - has anyone done this already? Plus, it's not Written in Rust (TM)! `xar` only signs packages, not Mach-O binaries, at least as far as I know?
  - bogwog 4y ago
    > Plus, it's not Written in Rust (TM)! It could be co-authored by Xi Jinping and Mark Zuckerberg in COBOL and hosted on NPM, and that wouldn’t stop me from using an open source codesign alternative that runs on Linux.
    - thewebcount 4y ago
      Can you elaborate? Since I mostly work on macOS stuff, I don't understand the workflow that leads to needing this sort of tool. What's the process of writing software for either macOS or iOS that doesn't involve needing macOS or iOS at some point? Like you probably need to test on those systems if you're supporting software for them, right? Is it because development is done on Linux, and only testing is done on, say, macOS? Sorry if this is a stupid question. I just haven't run into this sort of situation before and am trying to better understand it.
      - bogwog 4y ago
deleted 4y ago
[deleted]
giomasce 4y ago
What does notarization mean here?
- duped 4y ago
  https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution https://developer.apple.com/documentation/security/notarizin...
- salawat 4y ago
  Realistically? It's an additional static analysis hoop to jump through and another brick in Apple's walled garden in terms of trying to lock-out generic programs written by anyone who can't be arsed to fiddle with cryptography.
deknos 4y ago
is there a public format for Code or Keysigning Servers? Like SCEP/EST, but for other stuff?
- nmnvzr 4y ago
  From what I can remember the signature portion is documented in CMS RFC and the file format is only available from Apple's open source libsecurity.
nimbius 4y ago
this feels like a job for lemur and smallstep https://smallstep.com/docs/step-ca https://smallstep.com/docs/step-ca https://github.com/Netflix/lemur https://github.com/Netflix/lemur
infotogivenm 4y ago
I’ve always used gon ( https://github.com/mitchellh/gon https://github.com/mitchellh/gon ) for this, which is open source golang, but I don’t think it supports mach-o embedding. I’ll have to try this tool out.
- indygreg2 4y ago
  Note that gon is a glorified front-end for executing processes like `codesign`, `altool`, and even `ditto` for zip file generation. This Rust implementation, by contrast, has all the functionality implemented in pure Rust: there is no calling out to external processes for anything. You could drop the statically linked `rcodesign` executable into a Linux container with no other files and it would work. That's not to discredit gon or its authors: it is a fantastic tool for streamlining common functionality. But the mechanism is completely different.
  - infotogivenm 4y ago
    Never knew it wrapped shell commands - good to know!
neilk 4y ago
Kudos to these developers. I wish them success. If I can do a small derail: I and some contributors had this mostly working a few iOS versions ago with isign (https://github.com/isignpy/isign https://github.com/isignpy/isign). This is befor notarization. Announced several times to HN but didn’t seem to be interesting to many people. This was a spin-off from our work at a testing company. I’m not an iOS developer. (That may be why I could never see what to do with this.) I tried to make it into something, but I could never figure out who actually wanted this. What potential uses are foreseen for rcodedesign? - Weird custom signing. We did a project for a large financial company to make it compatible with hardware security module signing, but they never implemented it due to the pandemic changing priorities, and then someone discovered there was a little-known MacOS API to do something similar anyway, so our project was shelved. (We got paid though). - “Alternative” app distribution. From time to time I am contacted by people who run alternative app stores, either distributing hacked versions of paid apps, or who run app store in countries under embargo from Western countries. (They have banks, their customers have iPhones, they want a way to distribute iOS apps). The ethical issues and legal risks seemed significant so I never pursued that. If someone wants to go do that, I guess that’s a use case. - Build pipeline. Maybe, but you can’t fully build a iOS app on Linux so it didn’t seem like a win. - Mass app production. I am aware of some app white-labeling concepts that used isign back in the day. There was a guy who had a generic app for local news outlets, and essentially uploaded it to the App Store dozens of times under different names, for new versions. isign was more scriptable (and I wrote a multi-resign option for this use case) but I didn’t see a way to turn this into a business. - Testing on real devices. Even the original use case (re-signing apps to run on local device labs) has been mostly obviated. Hosting and scripting on MacOS has gotten better. - ??? something else? I even reached out to saurik about this at a conference. Cydia worked on jailbroken devices, so it didn’t need signing. To my surprise saurik was angry with me. He said that many, many companies had figured this out internally so he was skeptical we would ever release it. (We did!) So… again kudos, but what can we do with this? I’m not dissing it, I’m genuinely baffled. I also thought this would be important and useful but couldn’t figure it out.
- nicoburns 4y ago
  Regarding use case: It could certainly be useful for macOS CLI tools. Languages like Go and Zig can cross-compile seamlessly from linux (and Rust is nearly but not quite there). With this you could compile, sign and notarize a ready-to-publish binary without ever touching a macOS machine.
candiddevmike 4y ago
Will this upload to the app store?
gilgoomesh 4y ago
> There are probably thousands of companies and individuals who have wanted to release Apple software from non-macOS operating systems. (The existence and popularity of tools like fastlane seems to confirm this.) I'm not sure the existence/popularity of fastlane supports the non-macOS operating systems argument since its primary purpose is as a wrapper around Apple's clunky/undocumented build and upload tools on macOS.
xer0x 4y ago
I came here to whine about Apple Code signing, and how this seems like a terrible thing to reproduce, until I realized this breaks free of Apple, and is hopefully much more sane! Well done!
- denkmoon 4y ago
  It's only good when it's not Apple doing it? You know you can go disable code signing verification on macos with a reboot and a single command right?
  - Diti 4y ago
    The comment you replied to means "reproduce" as in reproducible-builds[.org]. Needing to be on macOS to sign Mach-O binaries makes builds less reproducible than also having a Linux way of signing the builds.
pmontra 4y ago
I have a customer that runs a dynamic analysis service. They add a library to the IPA and create a new one. They need to to sign it. It's automated on a Mac but they won't need a Mac anymore if they could run code signing on a Linux VM like the rest of their infrastructure. No more sending files and messages back and forth between the two environments. A huge win.
pheasantquiff 4y ago
Does this mean I can now develop iOS (iphone/ipad) apps, start to finish, without asking Apple's permission _at any stage_, and then sideload these apps onto my personal iphone/ipad? (assuming as mentioned elsewhere that I can develop such apps on linux, without needing MacOS)

309 points by indygreg2 4y ago

Beyond clownish that apple didn’t just release this themselves. Kudos to the creators for doing the job apple should have done.

xrd 4y ago

This is so fantastic. I wouldn't have believed it was possible had I not seen it here. I maintain multiple old OSX machines solely for this purpose; migrating to Linux for this step makes me feel so much safer for my CI build systems.

Sytten 4y ago

The only thing left to solve is to actually build cross platform. You can do with rust cross but you have to build your own docker images on macos first since apple licensing prohibits redistribution of binaries. It would save us so much money in github CI minutes.

funhatch 4y ago
https://pmbaty.com/iosbuildenv/ https://pmbaty.com/iosbuildenv/
bogwog 4y ago
You mean building mac/ios apps on Linux? That’s relatively easy to do, with the only questionable/legally gray area being access to the iOS/Mac SDKs. I’m not a lawyer, but it’s my understanding that the Google v Oracle case established that APIs aren’t protected by copyright. So if we can use the headers from e.g. the iOS SDK on Linux wtohout worrying about copyright issues (Apple developer agreements may be an issue though), it should be possible to autogenerate stub libraries for linking purposes. With that, you can compile and link a binary for iOS or Mac on Linux. Next, you’ll need to implement the packaging and signing portion. Packaging (generating an app bundle) is easy, but codesigning requires Apple’s codesign utility, which doesn’t work outside of Mac, even with a project like Darling (last I checked, anyways). But if the OP does implement a working codesign alternative, then that means it should be possible to create an iOS/Mac app that can be distributed on the App Store entirely from Linux with no Xcode, virtual machines, cloud macs, etc. The only gate left then is the Apple tax/signing certificate you need to buy for $99/year. But that might change with all of the antitrust pressure Apple is facing, including the recent EU law that would force Apple to allow competing app stores.
- TazeTSchnitzel 4y ago
  Zig already does this I think, they committed the Apple headers to their repo.
- tssva 4y ago
  "I’m not a lawyer, but it’s my understanding that the Google v Oracle case established that APIs aren’t protected by copyright." I'm no lawyer either but I have read the Supreme Court's decision in the case. It doesn't address whether APIs are protected by copyright. The Supreme Court generally makes the most limited ruling needed in order to resolve a case. In this case there were two issues at play. -Are API's copyrightable? -If so, was Google's use of the APIs fair use under copyright law as a jury had previously ruled? In order to make the most limited ruling needed to decide the case the Supreme Court first addressed the fair use question with an assumption that APIs could be copyrighted. The Court decided that Google's use did fall under fair use and so the case could be decided by the fair use question alone. They therefore didn't bother to address the question of whether APIs can be copyrighted. As it stands right now there is a U.S. Court of Appeals ruling in the Oracle case that APIs are copyrightable which until the Supreme Court does address the issue means they are as far as the courts are concerned. Fair use is a case-by-case defense and the Supreme Court's decision doesn't mean that all use of an API is fair use. This is all very much still all up in the air and any use of the headers from the iOS SDK should be concerned about copyright issues.
roblabla 4y ago
This is actually a solved problem, using osxcross[0]. The experience is honestly very smooth, and we don't require any apple proprietary binaries. The only thing apple-proprietary is their SDK (containing the header files for compiling, and tbd files for linking), which can be downloaded from apple's website (at least if you have a developer account), or from various GitHub projects archiving them. [0]: https://github.com/tpoechtrager/osxcross https://github.com/tpoechtrager/osxcross
tedmielczarek 4y ago
We achieved this for Firefox macOS builds years ago. (Greg was there for that work as well. ) You do need to have an SDK accessible, but storing one internally and using it in CI was deemed acceptable. The impact on CI build times was hard to overstate. We were already building our own clang binaries for various reasons, so it was mostly a matter of making an SDK available and ensuring that the right compiler options were passed in (since running clang on macOS sets a bunch of defaults that you don't get even if you pass -target x86_64-apple-darwin).

xet7 4y ago

How to build GUI apps for iOS and macOS with Rust ? Can this notarize apps made with other programming languages?

norman784 4y ago
As far I can tell, you can compile the binary and sign your app. # From a Git checkout $ cargo run --bin rcodesign -- --help $ cargo install --bin rcodesign # Remote install. $ cargo install --git https://github.com/indygreg/PyOxidizer https://github.com/indygreg/PyOxidizer --branch main --bin rcodesign apple-codesign
- rvz 4y ago
  - steveklabnik 4y ago
    Your parent is clearly answering the OP’s second question but not the first, come on. I’m aware of production iOS apps using Rust that are either: 1. Using rust for core logic but still using Swift/obj c for UI. 2. Pure Rust but don’t use native widgets. Like games or similar custom UI. That said it is in my understanding theoretically possible but I’m not aware of a paved cow path here. It’s also not my area of specialty though.
TazeTSchnitzel 4y ago
You could use Qt and cross-compile.
indygreg2 4y ago
rcodesign can sign and notarize applications written in any language. From rcodesign's perspective, Rust is an implementation detail of the tool itself.

ridiculous_fish 4y ago

I think Apple's official code signing tool is also open source? https://github.com/Apple-FOSS-Mirror/security_systemkeychain/blob/master/src/codesign.cpp https://github.com/Apple-FOSS-Mirror/security_systemkeychain... xar is another tool for codesigning Mac installer packages, which runs on Linux. I've used it successfully in the past. https://users.wfu.edu/cottrell/productsign/productsign_linux.html https://users.wfu.edu/cottrell/productsign/productsign_linux...

bri3d 4y ago
Good pointers indeed - Apple's code signing tool is open source, but heavily dependent on Cocoa/OSX libraries like CoreFoundation. I'm not sure if it works with any of the open-source reimplementations of these libraries, although I kind of want to try now, just for fun - has anyone done this already? Plus, it's not Written in Rust (TM)! `xar` only signs packages, not Mach-O binaries, at least as far as I know?
- bogwog 4y ago
  > Plus, it's not Written in Rust (TM)! It could be co-authored by Xi Jinping and Mark Zuckerberg in COBOL and hosted on NPM, and that wouldn’t stop me from using an open source codesign alternative that runs on Linux.
  - thewebcount 4y ago
    Can you elaborate? Since I mostly work on macOS stuff, I don't understand the workflow that leads to needing this sort of tool. What's the process of writing software for either macOS or iOS that doesn't involve needing macOS or iOS at some point? Like you probably need to test on those systems if you're supporting software for them, right? Is it because development is done on Linux, and only testing is done on, say, macOS? Sorry if this is a stupid question. I just haven't run into this sort of situation before and am trying to better understand it.
    - bogwog 4y ago

deleted 4y ago

[deleted]

giomasce 4y ago

What does notarization mean here?

duped 4y ago
https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution https://developer.apple.com/documentation/security/notarizin...
salawat 4y ago
Realistically? It's an additional static analysis hoop to jump through and another brick in Apple's walled garden in terms of trying to lock-out generic programs written by anyone who can't be arsed to fiddle with cryptography.

deknos 4y ago

is there a public format for Code or Keysigning Servers? Like SCEP/EST, but for other stuff?

nmnvzr 4y ago
From what I can remember the signature portion is documented in CMS RFC and the file format is only available from Apple's open source libsecurity.

nimbius 4y ago

this feels like a job for lemur and smallstep https://smallstep.com/docs/step-ca https://smallstep.com/docs/step-ca https://github.com/Netflix/lemur https://github.com/Netflix/lemur

infotogivenm 4y ago

I’ve always used gon ( https://github.com/mitchellh/gon https://github.com/mitchellh/gon ) for this, which is open source golang, but I don’t think it supports mach-o embedding. I’ll have to try this tool out.

indygreg2 4y ago
Note that gon is a glorified front-end for executing processes like `codesign`, `altool`, and even `ditto` for zip file generation. This Rust implementation, by contrast, has all the functionality implemented in pure Rust: there is no calling out to external processes for anything. You could drop the statically linked `rcodesign` executable into a Linux container with no other files and it would work. That's not to discredit gon or its authors: it is a fantastic tool for streamlining common functionality. But the mechanism is completely different.
- infotogivenm 4y ago
  Never knew it wrapped shell commands - good to know!

neilk 4y ago

Kudos to these developers. I wish them success. If I can do a small derail: I and some contributors had this mostly working a few iOS versions ago with isign (https://github.com/isignpy/isign https://github.com/isignpy/isign). This is befor notarization. Announced several times to HN but didn’t seem to be interesting to many people. This was a spin-off from our work at a testing company. I’m not an iOS developer. (That may be why I could never see what to do with this.) I tried to make it into something, but I could never figure out who actually wanted this. What potential uses are foreseen for rcodedesign? - Weird custom signing. We did a project for a large financial company to make it compatible with hardware security module signing, but they never implemented it due to the pandemic changing priorities, and then someone discovered there was a little-known MacOS API to do something similar anyway, so our project was shelved. (We got paid though). - “Alternative” app distribution. From time to time I am contacted by people who run alternative app stores, either distributing hacked versions of paid apps, or who run app store in countries under embargo from Western countries. (They have banks, their customers have iPhones, they want a way to distribute iOS apps). The ethical issues and legal risks seemed significant so I never pursued that. If someone wants to go do that, I guess that’s a use case. - Build pipeline. Maybe, but you can’t fully build a iOS app on Linux so it didn’t seem like a win. - Mass app production. I am aware of some app white-labeling concepts that used isign back in the day. There was a guy who had a generic app for local news outlets, and essentially uploaded it to the App Store dozens of times under different names, for new versions. isign was more scriptable (and I wrote a multi-resign option for this use case) but I didn’t see a way to turn this into a business. - Testing on real devices. Even the original use case (re-signing apps to run on local device labs) has been mostly obviated. Hosting and scripting on MacOS has gotten better. - ??? something else? I even reached out to saurik about this at a conference. Cydia worked on jailbroken devices, so it didn’t need signing. To my surprise saurik was angry with me. He said that many, many companies had figured this out internally so he was skeptical we would ever release it. (We did!) So… again kudos, but what can we do with this? I’m not dissing it, I’m genuinely baffled. I also thought this would be important and useful but couldn’t figure it out.

nicoburns 4y ago
Regarding use case: It could certainly be useful for macOS CLI tools. Languages like Go and Zig can cross-compile seamlessly from linux (and Rust is nearly but not quite there). With this you could compile, sign and notarize a ready-to-publish binary without ever touching a macOS machine.

candiddevmike 4y ago

Will this upload to the app store?

gilgoomesh 4y ago

> There are probably thousands of companies and individuals who have wanted to release Apple software from non-macOS operating systems. (The existence and popularity of tools like fastlane seems to confirm this.) I'm not sure the existence/popularity of fastlane supports the non-macOS operating systems argument since its primary purpose is as a wrapper around Apple's clunky/undocumented build and upload tools on macOS.

xer0x 4y ago

I came here to whine about Apple Code signing, and how this seems like a terrible thing to reproduce, until I realized this breaks free of Apple, and is hopefully much more sane! Well done!

denkmoon 4y ago
It's only good when it's not Apple doing it? You know you can go disable code signing verification on macos with a reboot and a single command right?
- Diti 4y ago
  The comment you replied to means "reproduce" as in reproducible-builds[.org]. Needing to be on macOS to sign Mach-O binaries makes builds less reproducible than also having a Linux way of signing the builds.

pmontra 4y ago

I have a customer that runs a dynamic analysis service. They add a library to the IPA and create a new one. They need to to sign it. It's automated on a Mac but they won't need a Mac anymore if they could run code signing on a Linux VM like the rest of their infrastructure. No more sending files and messages back and forth between the two environments. A huge win.

pheasantquiff 4y ago

Does this mean I can now develop iOS (iphone/ipad) apps, start to finish, without asking Apple's permission _at any stage_, and then sideload these apps onto my personal iphone/ipad? (assuming as mentioned elsewhere that I can develop such apps on linux, without needing MacOS)