11 ms·

A brief history of code signing at Mozilla

246 points by todsacerdoti 2y ago

SilasX 2y ago
Wow. No mention in this history/hagiography of the 2019 global add-on outage where (after banning unsigned add-ons even if you enabled them in about:config) they forgot to update their certs? Some of the HN threads: https://news.ycombinator.com/item?id=20421948 https://news.ycombinator.com/item?id=20421948 https://news.ycombinator.com/item?id=19871989 https://news.ycombinator.com/item?id=19871989 Edit: removed speculation if consequences of privacy loss, but come on … that’s still pretty serious.
- jvehent 2y ago
  Yeah. Sorry about that... We didn't think that the intermediate CA expiring would break the signatures, because code signing generally doesn't care about expiration, but we never tested the code path until the intermediate expired and the signatures broke. That was a hard lesson to learn...
  - extraduder_ire 2y ago
    I've seen the idea floated for combatting non signing related time-based bugs, but I'm a firm believer in having at least one machine run tests with its time set artificially far in the future (e.g. 1 year) to catch these ahead of time where possible.
    - jvehent 2y ago
      that was very much one of the lesson learned
- peterldowns 2y ago
  > The one that probably got people killed Got any more info on this? Seems like an absurd exaggeration.
  - alexsereno 2y ago
    It’s such an insane take that proof would be amazing
  - hansvm 2y ago
    I doubt there's proof (much like how China harvesting organs from certain races and religions didn't have solid proof till recently), but it's in the right ballpark. 50+ journalists are killed (and documented as such) each year, for their journalism, despite their attempts to maintain privacy. Privacy measures being suddenly dropped wouldn't make those numbers better. Toss in another 150+ human rights activist murders per year, dozens of whistleblowers, 60+ transgender people, and any number of other at-risk groups who tend to rely on technical measures to preserve their privacy and safety abroad. Did that info being leaked definitely get them killed? No clue. Did it "probably [get] people killed"? I don't know how those people are typically uncovered, so it's hard to say if a few privacy extensions actually mattered. It doesn't seem like an outlandish claim though (and with a prior that those extensions do matter, it's a likely claim).
scosman 2y ago
Does anyone have a windows code signing system they like? Something on GitHub actions ideally. I’m distributing unsigned binaries not because I care about the $200 for a cert, but I don’t want to spend hours figuring out a signing flow.
- tonyedgecombe 2y ago
  > but I don’t want to spend hours figuring out a signing flow. That part is trivial, the real pain comes from dealing with the certificate providers.
- landr0id 2y ago
  I haven't tried it but have been looking into the same problem. This is probably the best bet (does require an Azure account though): https://github.com/Azure/trusted-signing-action https://github.com/Azure/trusted-signing-action
  - scosman 2y ago
    Azure looked promising, but their trusted signing service won't issue a cert unless your corp is 3 years old. Doesn't look like you can bring your own keys here.
    - paulddraper 2y ago
      > unless your corp is 3 years old ...and you need to have a corporation? I thought corporations were pseudo-humans. Turns out it may be the other way around.
    - knallfrosch 2y ago
      Microsoft Learn: Import HSM-protected keys to Key Vault (BYOK) https://learn.microsoft.com/en-us/azure/key-vault/keys/hsm-protected-keys-byok https://learn.microsoft.com/en-us/azure/key-vault/keys/hsm-p... I guess that allows you to get your own cert (example: from DigiKey) for your own HSM (example: YubiKey) and then upload it. That's what we researched before we abandonded the approach and kept code-signing by manually downloading the build, signing on a specific machine where only one architect had access. What could ever go wrong?
jvehent 2y ago
Hey Ben! (I made and used to maintain Autograph at Mozilla)
- bhearsum_ 2y ago
  Hey Julien, long time! We should catch up!
  - kangs 2y ago
    hey Mozillians (:
    - bhearsum2 2y ago
      Hey buddy!!
  - jvehent 2y ago
    Ping me on signal when you get a chance!
Vinnl 2y ago
One thing I appreciate about being at Mozilla is that there are quite a few old-timers around, and that so much work always happens in public. A lot of institutional knowledge gets preserved that way!
- RunSet 2y ago
  You might ask the old-timers whether any of them knows why this bug is unfixed since 2009. https://bugzilla.mozilla.org/show_bug.cgi?id=505521 https://bugzilla.mozilla.org/show_bug.cgi?id=505521 Surely there is some rich lore to be mined.
jaharios 2y ago
Does everything need to be signed? Use a tool to much and you end up becoming it's slave. For a long time a had problems with some websites not loading videos when opened in background because I had an addon which disabled the page visibility api. So I download the source of the addon ( straight from the addons store right click the install button and save as..) changed the manifest to exclude the said sites, zip it back and install it again? Well your addon needs to be signed to be installed permanent even when you manualy install it. I understand that this is to keep tech-illiterate users safe from malicious addons. So is there a flag/about:config setting for advanced users to enable which skips the check? Nope, there was one but not anymore. So what options do I have? Either use beta version of Firefox as my main browser (no thanks). Or create an account as a developer, upload my addon and wait for Mozzila to check and sign my addon (that I created for myself).
- braiamp 2y ago
  The reason for that is like you expect, someone abused that, and we do not have nice things anymore. That's the same reason why apt-get doesn't allow you to remove essential packages on Debian, without modifying a configuration file.
  - bravetraveler 2y ago
    DNF has a similar mechanism, I appreciate it when poorly-written third party package manifests try to remove/deprecate more than they should
  - megous 2y ago
    I can not run my own extension in Firefox by modifying a config file. It's not possible. Not even if I don't let dishonest actors anywhere near my Firefox install. I can murder some trees and poison the environment for all of us, to do pointless mutli-hour re-builds of Firefox for each release and point release to have it accept my add-ons, though. I've also never seen a reason, why I can't at least place my CA into Firefox /usr/lib/firefox folder or /etc/firefox and have it be respected. Or just place local extensions there and have firefox not require signatures for them, because there's no way these can be installed accidentally from web by clicking some link. And if someone can trick me into modifying /usr/lib, they can just trick me into replacing Firefox completly with their malwared build, so signing will not save me anyway.
deleted 2y ago
[deleted]
mike_hearn 2y ago
Code signing is a trip alright. I make a build tool called Conveyor that simplifies all this way down for developers [1] as part of making shipping desktop apps a single command, and damn has it been a lot of work. First challenge: the tools provided by the OS developers only run on their OS and are of questionable quality, with bad error messages, poor documentation and unintuitive defaults. So I used a mix of open source and custom code to make signing work for every OS on any OS with good errors, docs and sample apps. The open source libs out there are slowly catching up with this, but they don't solve every part of the task and aren't integrated with everything else you need to ship apps (like laying out a working app tree, adding a software update engine etc). It's nice because it lets you ship from cheap Linux CI workers. Second challenge: you get a proliferation of private keys. Keys to sign your Mac and Windows binaries, keys to sign your Mac update feeds, keys to sign apt repositories. So, Conveyor can derive them all from a single root key that you can back up on paper with a pen. Except for Windows where this worked up until Microsoft started requiring all keys to be held in an HSM. Third challenge: said HSMs. They're a pain. As the Mozilla blog alludes, some tools like to require things to be done with a GUI. So, Conveyor can detect PKCS#11 drivers and drive the signing process itself using passphrases supplied via CI secrets or similar. It can also store the root key in the Mac keychain if you want to use that, which is a pretty nice way to keep it secure as macOS will ensure that only Conveyor can read it. Fourth challenge: Windows CAs decided to squeeze extra revenue from devs by exploiting the new HSM requirement's incompatibility with cloud CI. They started offering "cloud HSMs" that can be used without any hardware, just a textual credential again (so removing much of the security an HSM is supposed to create!) and these are popular because everyone wants to ship from CI. Unfortunately these services all charge extremely high prices per signature once you go over a fairly small monthly allowance. If you want to ship builds on a daily basis, for instance, you can easily blow hundreds or even thousands of dollars on signatures alone. So Conveyor has a disk cache that stores signatures for binaries, and separates them from the underlying DLLs/EXEs so the cache doesn't fill up too fast even if you're shipping very large binaries, and a GitHub Action to integrate that cache with GitHub's own. All this saves a lot of signatures and therefore money. Fifth challenge: Windows chooses to identify apps by the hash of their X.500 name as decided by their CA, but CAs constantly change their mind on how to format this name and change it for pointless reasons. Because the Windows ecosystem hates developers this can easily break app reputations/updates that are checking code signatures, and nobody at the CAs or Microsoft cares. So Conveyor has a whole system that recognizes when this has occurred and can drive an automatic reinstallation to repair things. The user doesn't notice except for a short delay on app startup (and seeing a progress bar). Sixth challenge: platforms like the JVM that like to hide native code inside ZIP files, which may also contain native libs for the wrong CPU architecture or platform. Conveyor finds them, deletes the irrelevant binaries, signs the ones that remain and puts them back (or extracts them into the app directory and ensures they get found). Seventh challenge: enterprises that have custom internal signing servers. So you can write a little script that submits binaries to the server and gets them back again. This works with all the above stuff too, when relevant. Eighth challenge: MSIX. Microsoft's only non-deprecated packaging tech has a lot of benefits, and allows non-admin users to install packages, but the signature format is pure madness. By far the craziest scheme I ever saw. Reverse engineered and implemented. Next challenge on the list: Azure Trusted Signing, Microsoft's new CA. It's marginally cheaper than the others, so some users want to use it, but it also doesn't provide a standard driver or use standard protocols. It'll require custom support. Looking to the future, the path forward is to use kernel sandboxing tech to eliminate signing requirements for our users when possible. For apps that can fit inside the sandbox it should be possible for us to sign and distribute their apps for them, with full code signing only required if the app has special needs. That would give a much more web-like experience for non-web apps, where a developer can just create a scaffolded app with one command, ship it with a second and not think about any of the platform specific details or code signing at all. Unfortunately this is a way off yet. [1] https://hydraulic.dev/ https://hydraulic.dev/
jonathantf2 2y ago
Slightly off-topic but my god that font is hard to read.
tingletech 2y ago
Once upon a time circa 1997 I made a javascript based windows shell for kiosk mode public terminal across 15 odd campus libraries. I had to buy a signing certificate from some CA to use the API that let me get rid of the netscape navigator chrome. All you could do on the computers was run netscape navigator, and but there was a menu that popped up on the left that had a menu of different library related web pages, and you could not tell that was really a browser window. I also had over 100 plugins we needed to install for all the licensed databases. The terminals ran Novell Netware on NT. I forget what software we used now, but they booted from a read only partition, and we created a new boot image when we shipped updates and someone would have to walk around with CDs and upgrade each workstation. The department that ran the public computer labs on campus had to sign off on the security before they would let me put them on the campus network. I also had a shrinkwrapped netscape enterprise server with its server side javascript, and bought a shrinkwrapped sybase -- both at the same time I bought a Sun desktop computer. I bought all 3 at the campus bookstore and paid with a G/L code. I learned solaris, web stuff, javascript, and databases on that job. The novell netware was a directory server based on LDAP or x.509.
rednafi 2y ago
Interesting history. Signing software is neat, but I wish there were a single standard like HTTP. Instead, we have a fractal mess of GPG, Sigstore, Cosign, and package-specific solutions (npm, PyPI, Maven, etc.), all doing their own thing. This makes me doubt how often these signatures are actually verified on client machines. For example, I've never done it manually because the process is so tedious. Some tools automatically check signatures, but even that is fragmented.
- mike_hearn 2y ago
  In this context code signing mostly means Windows and macOS signatures, which are definitely checked by various things in the OS and third party tooling.
causality0 2y ago
You're going to write a history of code signing at Mozilla and not even mention the time they fucked their users by negligently letting a certificate expire and forcibly disabling every extension on every Firefox instance worldwide?

246 points by todsacerdoti 2y ago

Wow. No mention in this history/hagiography of the 2019 global add-on outage where (after banning unsigned add-ons even if you enabled them in about:config) they forgot to update their certs? Some of the HN threads: https://news.ycombinator.com/item?id=20421948 https://news.ycombinator.com/item?id=20421948 https://news.ycombinator.com/item?id=19871989 https://news.ycombinator.com/item?id=19871989 Edit: removed speculation if consequences of privacy loss, but come on … that’s still pretty serious.

jvehent 2y ago
Yeah. Sorry about that... We didn't think that the intermediate CA expiring would break the signatures, because code signing generally doesn't care about expiration, but we never tested the code path until the intermediate expired and the signatures broke. That was a hard lesson to learn...
- extraduder_ire 2y ago
  I've seen the idea floated for combatting non signing related time-based bugs, but I'm a firm believer in having at least one machine run tests with its time set artificially far in the future (e.g. 1 year) to catch these ahead of time where possible.
  - jvehent 2y ago
    that was very much one of the lesson learned
peterldowns 2y ago
> The one that probably got people killed Got any more info on this? Seems like an absurd exaggeration.
- alexsereno 2y ago
  It’s such an insane take that proof would be amazing
- hansvm 2y ago
  I doubt there's proof (much like how China harvesting organs from certain races and religions didn't have solid proof till recently), but it's in the right ballpark. 50+ journalists are killed (and documented as such) each year, for their journalism, despite their attempts to maintain privacy. Privacy measures being suddenly dropped wouldn't make those numbers better. Toss in another 150+ human rights activist murders per year, dozens of whistleblowers, 60+ transgender people, and any number of other at-risk groups who tend to rely on technical measures to preserve their privacy and safety abroad. Did that info being leaked definitely get them killed? No clue. Did it "probably [get] people killed"? I don't know how those people are typically uncovered, so it's hard to say if a few privacy extensions actually mattered. It doesn't seem like an outlandish claim though (and with a prior that those extensions do matter, it's a likely claim).

scosman 2y ago

Does anyone have a windows code signing system they like? Something on GitHub actions ideally. I’m distributing unsigned binaries not because I care about the $200 for a cert, but I don’t want to spend hours figuring out a signing flow.

tonyedgecombe 2y ago
> but I don’t want to spend hours figuring out a signing flow. That part is trivial, the real pain comes from dealing with the certificate providers.
landr0id 2y ago
I haven't tried it but have been looking into the same problem. This is probably the best bet (does require an Azure account though): https://github.com/Azure/trusted-signing-action https://github.com/Azure/trusted-signing-action
- scosman 2y ago
  Azure looked promising, but their trusted signing service won't issue a cert unless your corp is 3 years old. Doesn't look like you can bring your own keys here.
  - paulddraper 2y ago
    > unless your corp is 3 years old ...and you need to have a corporation? I thought corporations were pseudo-humans. Turns out it may be the other way around.
  - knallfrosch 2y ago
    Microsoft Learn: Import HSM-protected keys to Key Vault (BYOK) https://learn.microsoft.com/en-us/azure/key-vault/keys/hsm-protected-keys-byok https://learn.microsoft.com/en-us/azure/key-vault/keys/hsm-p... I guess that allows you to get your own cert (example: from DigiKey) for your own HSM (example: YubiKey) and then upload it. That's what we researched before we abandonded the approach and kept code-signing by manually downloading the build, signing on a specific machine where only one architect had access. What could ever go wrong?

jvehent 2y ago

Hey Ben! (I made and used to maintain Autograph at Mozilla)

bhearsum_ 2y ago
Hey Julien, long time! We should catch up!
- kangs 2y ago
  hey Mozillians (:
  - bhearsum2 2y ago
    Hey buddy!!
- jvehent 2y ago
  Ping me on signal when you get a chance!

Vinnl 2y ago

One thing I appreciate about being at Mozilla is that there are quite a few old-timers around, and that so much work always happens in public. A lot of institutional knowledge gets preserved that way!

RunSet 2y ago
You might ask the old-timers whether any of them knows why this bug is unfixed since 2009. https://bugzilla.mozilla.org/show_bug.cgi?id=505521 https://bugzilla.mozilla.org/show_bug.cgi?id=505521 Surely there is some rich lore to be mined.

jaharios 2y ago

Does everything need to be signed? Use a tool to much and you end up becoming it's slave. For a long time a had problems with some websites not loading videos when opened in background because I had an addon which disabled the page visibility api. So I download the source of the addon ( straight from the addons store right click the install button and save as..) changed the manifest to exclude the said sites, zip it back and install it again? Well your addon needs to be signed to be installed permanent even when you manualy install it. I understand that this is to keep tech-illiterate users safe from malicious addons. So is there a flag/about:config setting for advanced users to enable which skips the check? Nope, there was one but not anymore. So what options do I have? Either use beta version of Firefox as my main browser (no thanks). Or create an account as a developer, upload my addon and wait for Mozzila to check and sign my addon (that I created for myself).

braiamp 2y ago
The reason for that is like you expect, someone abused that, and we do not have nice things anymore. That's the same reason why apt-get doesn't allow you to remove essential packages on Debian, without modifying a configuration file.
- bravetraveler 2y ago
  DNF has a similar mechanism, I appreciate it when poorly-written third party package manifests try to remove/deprecate more than they should
- megous 2y ago
  I can not run my own extension in Firefox by modifying a config file. It's not possible. Not even if I don't let dishonest actors anywhere near my Firefox install. I can murder some trees and poison the environment for all of us, to do pointless mutli-hour re-builds of Firefox for each release and point release to have it accept my add-ons, though. I've also never seen a reason, why I can't at least place my CA into Firefox /usr/lib/firefox folder or /etc/firefox and have it be respected. Or just place local extensions there and have firefox not require signatures for them, because there's no way these can be installed accidentally from web by clicking some link. And if someone can trick me into modifying /usr/lib, they can just trick me into replacing Firefox completly with their malwared build, so signing will not save me anyway.

deleted 2y ago

[deleted]

mike_hearn 2y ago

Code signing is a trip alright. I make a build tool called Conveyor that simplifies all this way down for developers [1] as part of making shipping desktop apps a single command, and damn has it been a lot of work. First challenge: the tools provided by the OS developers only run on their OS and are of questionable quality, with bad error messages, poor documentation and unintuitive defaults. So I used a mix of open source and custom code to make signing work for every OS on any OS with good errors, docs and sample apps. The open source libs out there are slowly catching up with this, but they don't solve every part of the task and aren't integrated with everything else you need to ship apps (like laying out a working app tree, adding a software update engine etc). It's nice because it lets you ship from cheap Linux CI workers. Second challenge: you get a proliferation of private keys. Keys to sign your Mac and Windows binaries, keys to sign your Mac update feeds, keys to sign apt repositories. So, Conveyor can derive them all from a single root key that you can back up on paper with a pen. Except for Windows where this worked up until Microsoft started requiring all keys to be held in an HSM. Third challenge: said HSMs. They're a pain. As the Mozilla blog alludes, some tools like to require things to be done with a GUI. So, Conveyor can detect PKCS#11 drivers and drive the signing process itself using passphrases supplied via CI secrets or similar. It can also store the root key in the Mac keychain if you want to use that, which is a pretty nice way to keep it secure as macOS will ensure that only Conveyor can read it. Fourth challenge: Windows CAs decided to squeeze extra revenue from devs by exploiting the new HSM requirement's incompatibility with cloud CI. They started offering "cloud HSMs" that can be used without any hardware, just a textual credential again (so removing much of the security an HSM is supposed to create!) and these are popular because everyone wants to ship from CI. Unfortunately these services all charge extremely high prices per signature once you go over a fairly small monthly allowance. If you want to ship builds on a daily basis, for instance, you can easily blow hundreds or even thousands of dollars on signatures alone. So Conveyor has a disk cache that stores signatures for binaries, and separates them from the underlying DLLs/EXEs so the cache doesn't fill up too fast even if you're shipping very large binaries, and a GitHub Action to integrate that cache with GitHub's own. All this saves a lot of signatures and therefore money. Fifth challenge: Windows chooses to identify apps by the hash of their X.500 name as decided by their CA, but CAs constantly change their mind on how to format this name and change it for pointless reasons. Because the Windows ecosystem hates developers this can easily break app reputations/updates that are checking code signatures, and nobody at the CAs or Microsoft cares. So Conveyor has a whole system that recognizes when this has occurred and can drive an automatic reinstallation to repair things. The user doesn't notice except for a short delay on app startup (and seeing a progress bar). Sixth challenge: platforms like the JVM that like to hide native code inside ZIP files, which may also contain native libs for the wrong CPU architecture or platform. Conveyor finds them, deletes the irrelevant binaries, signs the ones that remain and puts them back (or extracts them into the app directory and ensures they get found). Seventh challenge: enterprises that have custom internal signing servers. So you can write a little script that submits binaries to the server and gets them back again. This works with all the above stuff too, when relevant. Eighth challenge: MSIX. Microsoft's only non-deprecated packaging tech has a lot of benefits, and allows non-admin users to install packages, but the signature format is pure madness. By far the craziest scheme I ever saw. Reverse engineered and implemented. Next challenge on the list: Azure Trusted Signing, Microsoft's new CA. It's marginally cheaper than the others, so some users want to use it, but it also doesn't provide a standard driver or use standard protocols. It'll require custom support. Looking to the future, the path forward is to use kernel sandboxing tech to eliminate signing requirements for our users when possible. For apps that can fit inside the sandbox it should be possible for us to sign and distribute their apps for them, with full code signing only required if the app has special needs. That would give a much more web-like experience for non-web apps, where a developer can just create a scaffolded app with one command, ship it with a second and not think about any of the platform specific details or code signing at all. Unfortunately this is a way off yet. [1] https://hydraulic.dev/ https://hydraulic.dev/

jonathantf2 2y ago

Slightly off-topic but my god that font is hard to read.

tingletech 2y ago

Once upon a time circa 1997 I made a javascript based windows shell for kiosk mode public terminal across 15 odd campus libraries. I had to buy a signing certificate from some CA to use the API that let me get rid of the netscape navigator chrome. All you could do on the computers was run netscape navigator, and but there was a menu that popped up on the left that had a menu of different library related web pages, and you could not tell that was really a browser window. I also had over 100 plugins we needed to install for all the licensed databases. The terminals ran Novell Netware on NT. I forget what software we used now, but they booted from a read only partition, and we created a new boot image when we shipped updates and someone would have to walk around with CDs and upgrade each workstation. The department that ran the public computer labs on campus had to sign off on the security before they would let me put them on the campus network. I also had a shrinkwrapped netscape enterprise server with its server side javascript, and bought a shrinkwrapped sybase -- both at the same time I bought a Sun desktop computer. I bought all 3 at the campus bookstore and paid with a G/L code. I learned solaris, web stuff, javascript, and databases on that job. The novell netware was a directory server based on LDAP or x.509.

rednafi 2y ago

Interesting history. Signing software is neat, but I wish there were a single standard like HTTP. Instead, we have a fractal mess of GPG, Sigstore, Cosign, and package-specific solutions (npm, PyPI, Maven, etc.), all doing their own thing. This makes me doubt how often these signatures are actually verified on client machines. For example, I've never done it manually because the process is so tedious. Some tools automatically check signatures, but even that is fragmented.

mike_hearn 2y ago
In this context code signing mostly means Windows and macOS signatures, which are definitely checked by various things in the OS and third party tooling.

causality0 2y ago

You're going to write a history of code signing at Mozilla and not even mention the time they fucked their users by negligently letting a certificate expire and forcibly disabling every extension on every Firefox instance worldwide?