17 ms·

Removing PGP from PyPI

187 points by dlor 3y ago

westurner 3y ago
Now that you have removed GPG ASC signature upload support, is there any way for publishers to add cryptographic signatures to packages that they upload to pypi? FWIU only "the server signs uploads" part of TUF was ever implemented? Why do we use GPG ASC signatures instead of just a checksum over the same channel?
- woodruffw 3y ago
  > Why do we use GPG ASC signatures instead of just a checksum over the same channel? Could you elaborate on what you mean by this? PyPI computes and supplies a digest for every uploaded distribution, so you can already cross-check integrity for any hosted distribution. GPG was nominally meant to provide authenticity for distributions, but it never really served this purpose. That's why it's being removed.
  - westurner 3y ago
    1) the server signs what's uploaded using one or more TUF keys shared to RAM on every pypi upload server. 2) the client uploads a cryptographic signature (made using their own key) along with the package, and the corresponding public key is trusted to upload for that package name, and the client retrieves said public key and verifies the downloaded package's cryptographic signature before installing. FWIU, 1 (PyPI signs uploads with TUF) was implemented, but 2 (users sign their own packages before uploading the signed package and signature, (and then 1)) was never implemented?
    - woodruffw 3y ago
      Your understanding is a little off: we worked on integrating TUF into PyPI for a while, but ran into some scalability/distributability issues with the reference implementation. It's been a few years, but my recollection was that the reference implementation assumed a lot of local filesystem state, which wasn't compatible with Warehouse's deployment (no local state other than tempfiles, everything in object storage). To the best of my knowledge, the current state of TUF for PyPI is that we performed a trusted setup ceremony for the TUF roots[1], but that no signatures were ever produced from those roots. For the time being, we're looking at solutions that have less operational overhead: Sigstore[2] is the main one, and it uses TUF under the hood to provide the root of trust. [1]: https://www.youtube.com/watch?v=jjAq7S49eow&t=1s https://www.youtube.com/watch?v=jjAq7S49eow&t=1s [2]: https://www.sigstore.dev/ https://www.sigstore.dev/
      - westurner 3y ago
        Where does the user specify the cryptographic key to sign a package before uploading? Serverside TUF keys are implemented FWICS, but clientside digital publisher signatures (like e.g. MS Windows .exe's have had when you view the "Properties" of the file for many years now) are not yet implemented. Hopefully I'm just out of date.
        woodruffw 3y ago
        > Where does the user specify the cryptographic key to sign a package before uploading? With Sigstore, they perform an OIDC flow against an identity provider: that results in a verifiable identity credential, which is then bound to a short-lived (~15m) signing key that produces the signatures. That signing key is simultaneously attested through a traditional X.509 PKI (it gets a certificate, that certificate is uploaded to an append-only transparency log, etc.). So: in the normal flow, the user never directly specifies the cryptographic key -- the scheme ensures that they have a strong, ephemeral one generated for them on the spot (and only on their client device, in RAM). That key gets bound to their long-lived identity, so verifiers don't need to know which key they're verifying; they only need to determine whether they trust the identity itself (which can be an email address, a GitHub repository, etc.).
        westurner 3y ago
        What command(s) do I pass to pip/twine/build_pyproject.toml to build, upload, and install a package with a key/cert that users should trust for e.g. psf/requests?
      - trishankdatadog 3y ago
        python-tuf [1] back then assumed that everything was manipulated locally, yes, but a lot has changed since then: you can now read/write metadata entirely in memory, and integrate with different key management backend systems such as GCP. More importantly, I should point out that while Sigstore's Fulcio will help with key management (think of it as a managed GPG, if you will), it will not help with securely mapping software projects to their respective OIDC identities. Without this, how will verifiers know in a secure yet scalable way which Fulcio keys _should_ be used? Otherwise, we would then be back to the GPG PKI problem with its web of trust. This is where PEP 480 [2] can help: you can use TUF (especially after TAP 18 [3]) to do this secure mapping. Marina Moore has also written a proposal called Transparent TUF [4] for having Sigstore manage such a TUF repository for registries like PyPI. This is not to mention the other benefits that TUF can give you (e.g., protection from freeze, rollback, and mix-and-match attacks). We should definitely continue discussing this sometime. [1] https://github.com/theupdateframework/python-tuf https://github.com/theupdateframework/python-tuf [2] https://peps.python.org/pep-0480/ https://peps.python.org/pep-0480/ [3] https://github.com/theupdateframework/taps/blob/master/tap18.md https://github.com/theupdateframework/taps/blob/master/tap18... [4] https://docs.google.com/document/d/1WPOXLMV1ASQryTRZJbdg3wWRR4ckK558kUnxn7Eixaw/edit?usp=sharing https://docs.google.com/document/d/1WPOXLMV1ASQryTRZJbdg3wWR...
  - westurner 3y ago
    > Why do we use GPG ASC signatures instead of just a checksum over the same channel? You can include an md5sum or a sha512sum string next to the URL that the package is downloaded from (for users to optionally check after downloading a package); but if that checksum string is uploaded over the same channel (HTTPS/TLS w/ a CA cert bundle) as the package, the checksum string could have been MITM'd/tampered with, too. A cryptographically-signed checksum can be verified once the pubkey is retrieved over a different channel (GPG: HKP is HTTPS/TLS with cert pinning IIRC), and a MITM would have to spend a lot of money to forge that digital publisher signature. Twine COULD/SHOULD download uploads to check the PyPI TUF signature, which could/should be shipped as a const in twine? And then Twine should check publisher signatures against which trusted map of package names to trusted keys?
- ilyt 3y ago
  Signature tells you who signed it. Of course, if you haven't put any effort in system to end-to-end verify whether it's right signature it doesn't matter.
  - westurner 3y ago
    pip checks that a given was signed with the pypi key but does not check for a signature from the publisher. And now there's no way to host any type of cryptographic signatures on pypi. There is no e2e: pypi signs what's uploaded. (Noting also that packages don't have to be encrypted in order to have cryptographic signatures; only the signature is encrypted, not the whole package)
    - ilyt 3y ago
      Yeah the whole thing looks like throwing away baby with bathwater; the package should * get a signature for author ("the actual author published it") + some metadata with list of valid signing keys (in case project have more authors or just for key rotation * get a signature for hosting provider that confirms "yes, that actual user logged in and uploaded the package" * (the hardest part) key management on client side so the user have to do least amount of work possible in when downloading/updating valid package. If user doesn't want to go to effort to validate whether the public key of author is valid so be it but at very least system should alert on tampering with the provider (checking the hosting signature) or the author key changing (compromised credentials to the hosting provider). It still doesn't prevent "the attacker steals key off author's machine" but that is by FAR the rarest case and could be pretty reasonably prevented by just using hardware tokens. Hell, fund them for key contributors.
Arnavion 3y ago
>Of those 1069 unique keys, about 30% of them were not discoverable on major public keyservers, making it difficult or impossible to meaningfully verify those signatures. I don't know if it applies to any of those 1069 keys, but note that there is a way of hosting PGP keys that does not depend on key servers: WKD https://datatracker.ietf.org/doc/draft-koch-openpgp-webkey-service/16/ https://datatracker.ietf.org/doc/draft-koch-openpgp-webkey-s... . You host the key at a .well-known URI under the domain of the email address. It's a draft as you can see, but I've seen a few people using it (including myself), and GnuPG supports it.
- woodruffw 3y ago
  This is interesting, but it doesn't really solve the key distribution problem: with well-known hosting you now have a (weak) binding to some DNS path, but you're not given any indication of how to discover that path. It's also not clear that a DNS identity is beneficial to PyPI in particular (since PyPI doesn't associate or namespace packages at all w/r/t DNS). More generally, these kinds of improvements are not a sufficient reason to retain PGP: even with a sensible identity binding and key distribution, PGP is still a mess under the hood. The security of a codesigning scheme is always the weakest signature and/or key, and PGP's flexibility more or less ensures that that weakest link will always be extremely weak.
  - Arnavion 3y ago
    Right. I've never used PyPI, but TFA makes it sound like the existing support for signing is "We allow the uploader to upload a signature, and the downloader can look up the key indicated in the signature to do the verification." Is that correct? If so, then yes there is a key ID involved but no email address, so a generic downloader would have no choice but to look it up from a key server.
    - woodruffw 3y ago
      That's correct! PyPI's support for PGP is very old -- it's hard to get an exact date, but I think it's been around since the very earliest versions of the index (well before it was a storing index like it is now). If I had to guess (speculate wildly), my guess would be that the original implementation was done with a healthy SKS network and strong set in mind -- without those things, PGP's already weak identity primitives are more or less nonexistent with just signatures.
KRAKRISMOTT 3y ago
What are we switching to? Does Pypi support ECDSA?
- woodruffw 3y ago
  Just for disambiguation: ECDSA is a signing algorithm, not a protocol or toolkit like PGP. PGP can produce ECDSA signatures through an extension RFC, but it's not a core part of OpenPGP. There is no immediate replacement, because the overwhelming majority of packages never bothered to sign with PGP (and all evidence points to the overwhelming majority of signatures never being verified). In other words, this is much closer to removing "dead" code than to killing an active feature. Longer term, the plan is to integrate Sigstore[1]-based signatures. [1]: https://www.sigstore.dev/ https://www.sigstore.dev/
- jossclimb 3y ago
  sigstore I hope.
  - aborsy 3y ago
    I’m not sure if I understand this correctly, but, this basically seems to be a CA, with SSO-type of proof of identity, short lived certificates and transparency logs? How an OIDC identity is obtained and secured is not treated. It brings useful organization to PKI, but the problem remains. You have to delegate trust to identity providers: Google, GitHub, etc. Keybase was interesting, but the project seems semi-dead.
    - woodruffw 3y ago
      > How an OIDC identity is obtained and secured is not treated. It brings useful organization to PKI, but the problem remains. You have to delegate trust to identity providers: Google, GitHub, etc. Yes, this is a fundamental (and, IMO, reasonable) assumption in Sigstore. The trust argument for large IdPs is that they (1) have the institutional ability and resources (like incident response) to maintain their service, (2) have strong incentives to maintain and improve the overall security of their providers (billions of accounts on the Internet are bound to SSO via Google, etc.), and (3) that any failures in those providers are already catastrophic, so reducing the number of moving and potentially failing parts is a net win in terms of security.
usr1106 3y ago
Isn't that throwing out the baby with the bathwater? There seem to be non-neglible risks of installing malware from PyPI according to various headlines recently. But instead of improving security measures that don't work well they just remove them?
- donaldstufft 3y ago
  Removing security features that don't work is a separate concern from making security features that do work. Nobody who has done any serious work on PyPI security in the past 15 years thinks that GPG will play a part in the future of PyPI security. It's support was entirely vestigial, served no practical purpose, and never would.
  - 7to2 3y ago
    [flagged]
    - gerikson 3y ago
      Note that the person you're replying to is the PyPi maintainer responsible for removing GPG from PyPi. The rationale is expanded here: https://news.ycombinator.com/item?id=36050190 https://news.ycombinator.com/item?id=36050190
      - 7to2 3y ago
        [flagged]
        dang 3y ago
        Attacking other users will get you banned here. No more of this, please. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
    - dang 3y ago
      Please make your substantive points without swipes, in keeping with the site guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html. If someone else is wrong, it suffices to neutrally explain to them, and the rest of us, how they are wrong. Adding putdowns not only doesn't help, it harms the community and discredits your point, so please don't do that. Edit: please see https://news.ycombinator.com/item?id=36092913 https://news.ycombinator.com/item?id=36092913 also. You broke the site guidelines badly in this thread.
rwmj 3y ago
So they're removing PGP signatures, which certainly have some issues, and replacing them with ... nothing?
- sowbug 3y ago
  The research article cited in the announcement is titled "PGP signatures on PyPI: worse than useless." That's the issue. Pretending there is a security solution in place is worse than being upfront that there is none. If you look down and notice that your seatbelt is actually made out of angel hair pasta, you might drive more carefully. Hopefully you'll also get a better car.
  - rwmj 3y ago
    But they're not "worse than useless", that article was wrong. PGP/GPG are without doubt problematic, they have weak points (like use of SHA-1, some keys that could not be located, and terrible UI) but they are not worse than having no traceability of the package at all between the author and PyPI.
    - bandrami 3y ago
      It's like the larger holy war against self-signed certificates in TLS. They are strictly better than plaintext but there is software that will prefer a plaintext connection to self-signed TLS.
      - Avamander 3y ago
        Unverifiable security is not "strictly better" than plaintext.
        bandrami 3y ago
        It is. In both cases you may be MiTMed, but with plaintext that MiTM session may also be eavesdropped upon. This isn't really debateable. Unverified sig simply is strictly better.
tzs 3y ago
> In the last 3 years, about 50k signatures had been uploaded to PyPI by 1069 unique keys. Of those 1069 unique keys, about 30% of them were not discoverable on major public keyservers, making it difficult or impossible to meaningfully verify those signatures. Of the remaining 71%, nearly half of them were unable to be meaningfully verified at the time of the audit (2023-05-19) Why not include the public key in the package? 99% of the time what I want out of package signing is to know that the new version of the package I'm downloading is from the same people as the old version. I don't actually need to know who those people are...just that they are the same people as before.
- hannob 3y ago
  What happens if the developer looses his key? Or if it expires? pypi could show a warning that the key has changed. Which is not an actionable or helpful warning. And then everyone gets used to seeing these warnings every now and then. And you won nothing. Getting signatures to do something useful is hard.
  - bombolo 3y ago
    > What happens if the developer looses his key? Or if it expires? What happens if a developer loses their google titan key that is required to login into pypi?
    - hannob 3y ago
      They either have their backup codes or there's probably a manual process the pypi team can get them their account back if they can sufficiently show they are the real developers. If you have any form of automated signature verification you basically need a concept how to handle recovery. But if this concept comes down to "trust pypi", then you really can just skip the whole thing and rely on pypi giving you the right packages and https to secure the connection).
- woodruffw 3y ago
  > Why not include the public key in the package? Because PyPI (or an attacker) could always substitute a new key. There's very little value in the signature and key coming from the same source: the key (and its justified identity) always need to come from a source of trust, not the source that's being verified. > 99% of the time what I want out of package signing is to know that the new version of the package I'm downloading is from the same people as the old version. I don't actually need to know who those people are...just that they are the same people as before. This might be a misunderstanding, but I don't think you actually want this: lots of large packages have multiple release managers (and contributors who come and go); you don't want to manually resolve each new human identity that appears for a package distribution. What most people actually want is a strong cryptographic attestation that the package distribution came from the same source as the thing hosting the source code, since both that service and the owner of the repository are presumed trusted. Notably, PGP is incapable of providing either of these: you only get key IDs, which are neither strong human identities nor a strong binding to a service. Key IDs might correspond to keys with email (or other identities) in them, but that's (1) not guaranteed, and (2) not a strong proof of identity (since anybody can claim any identity in a PGP key).
jwilk 3y ago
Two days ago: https://news.ycombinator.com/item?id=36021172 https://news.ycombinator.com/item?id=36021172 ("PGP signatures on PyPI: worse than useless", >150 comments)
reidrac 3y ago
I have been thinking about this in the context of Java libraries (really using Scala, but bear with me). If the repo requires a GPG signature, they could also ask for the public key of the developer making the releases (e.g. when they make the account), and they could sign it with their key at that point. Then make available the package, the signature, and the signed public key. Then I only need to trust the repo's key (in this case PyPi). Does this make any sense?
- woodruffw 3y ago
  > Does this make any sense? It makes sense in terms of trusting the package index, but it's inverted from the original design goal: the point of end-user signatures on package indices is to eliminate unnecessary package index trust, not reinforce it. If you already trust the package index, then mandating HTTPS and strong cryptographic digests is going to be far more effective (and secure) than some kind of PGP key attestation scheme.
  - reidrac 3y ago
    The package index only hosts the packages, but doesn't release them. The dev releasing the package is who signs it. Without an easy way to verify the keys, the signatures are useless. Which is why PiPy is removing the GPG keys all together.
    - woodruffw 3y ago
      > The package index only hosts the packages, but doesn't release them. The dev releasing the package is who signs it. I know that; the GP is describing a countersigning scheme, where the package index (qua trusted entity) countersigns for the signing key, which the dev then uses to sign for their package. > Without an easy way to verify the keys, the signatures are useless. Which is why PiPy is removing the GPG keys all together. Agreed entirely; I'm the one who wrote the analysis in the linked announcement :-)
zokier 3y ago
At least you can't blame pypi for ignoring the report, and tbh I find this response time remarkably quick. It wouldn't have been far fetched to imagine someone in their position just trying to ignore/downplay/dispute this sort of reports.
- deleted 3y ago
  [deleted]
- masklinn 3y ago
  As the author of the post noted above, the pypi maintainers have been wanting to get rid of pgp for awhile. The post gave them excellent additional justification to.
WhyNotHugo 3y ago
When many developers didn't use 2FA they pushed for them to enable 2FA within a deadline. It sounds like the same approach could have been used for PyPI. E.g.: an attempt to make the feature useful before declaring it dead forever.
- woodruffw 3y ago
  This has very little to do with 2FA: PGP signing has been de facto dead for years on PyPI, and this change has no effect on publishing workflows: PyPI will still accept uploads that contain signatures, and just ignores them now. It's also not accurate to say that PyPI failed to make 2FA useful: it was deployed for over two years before the 2FA mandate for critical projects went into effect. That mandate also came with free hardware keys for everyone affected.
- masklinn 3y ago
  No. 2FA is a feature for pypi, and developers. The entire purpose of pgp sigs was external, it was for distributions to use. Distributions don’t use it, therefore it’s worthless, just just overhead and technical debt.
  - LtWorf 3y ago
    Debian checks PGP signatures of releases.
    - adamckay 3y ago
      For Python packages served by PyPI?
      - donaldstufft 3y ago
        Sometimes? There's no global policy of doing it in Debian, it's up to individual package maintainers inside of Debian to enable it (it defaults to off AFAIK) and to hardcode the key that they expect the package to be signed by. In the cases that it is used, AFAIK it is only used by Debian's uscan program, which is sort of like the Debian version of Dependabot, it tells them when there is a new version of something to package. As far as I know, the process of packaging that new version is still manual, and relies on the maintainer downloading the package and packaging it, so they may or may not use the signature in that case. How useful this is, is up for debate. Many years ago when I first started taking over releasing pip, that caused the pip GPG key to change, and the reaction of the Debian maintainer at the time was to just comment out the signature bit and fall back to no signature.
bryanlyon 3y ago
I came here thinking they were removing the PGP package from PyPi, but they're just removing a barely-used signature system? I don't know why they have to remove it though. I doubt it requires much maintenance now that it's already in place. Even if only 37% of keys are verifiable, that's infinitely more than will be verifiable if they remove the PGP support.
- Avamander 3y ago
  > Even if only 37% of keys are verifiable, that's infinitely more than will be verifiable if they remove the PGP support. Discoverable. That does not really verify anything about the key, its identities or the supposed signer. It boils down to almost entirely to just an overcomplicated hashing system.
- tedivm 3y ago
  They address your comment directly in their post- > While it doesn't represent a massive operational burden to continue to support it, it does require any new features that touch the storage of files to be made aware of and capable of handling these PGP signatures, which is a non zero cost on the maintainers and contributors of PyPI.
rvz 3y ago
PGP is a solution in search of a problem. We have given it decades for it to be useful and it turns out that it is an enormous security failure. It needed to go. Sigstore [0] on the other hand makes more sense to use instead of problem. [0] https://www.sigstore.dev https://www.sigstore.dev
- msm_ 3y ago
  This reads like an advertisement. I routinely use GPG, and it is useful for me. It's not perfect (far from perfect, really), but it's a solution for multiple of my problems. I don't know much about the solution you promote, but as usual with many "PGP killers" it replaces one very specific application of PGP and ignores all the others. Which is ok! Doing one thing and doing it well is the Unix philosophy after all. But it's not something I have use for, and it's not a viable replacement for GPG.
  - tptacek 3y ago
    If doing one thing and doing it well is the core of the Unix philosophy, PGP is (cryptographically) the antithesis of that. It's a Swiss Army Knife that does none of its tasks well by modern standards.
- LtWorf 3y ago
  I'll let my boss know we must stop signing our releases and having our software automatically check if the new version is legit then. We will instead switch to use some thing with a fluffy corporate website that tells absolutely nothing.
NotYourLawyer 3y ago
Interesting timeline. The Yossarian article that TFA cites and that I assumed was the impetus here was published two days ago on 5/21. But the audit was two days earlier on 5/19.
- woodruffw 3y ago
  I originally ran the audit on 3/27 (IIRC), and then ran it a few additional times as I fixed data quality issues in my scripts (the ones linked in the post). The last time I ran it was on 5/19-5/20, when I was finalizing the post. You can also see that I did a new release of `pgpkeydump` at around the same time, to add some more extracted datapoints. PyPI's admins have been wanting to remove PGP support for years; all I did was provide the final nudge.
zzzeek 3y ago
> Of those 1069 unique keys, about 30% of them were not discoverable on major public keyservers, making it difficult or impossible to meaningfully verify those signatures. Of the remaining 71%, nearly half of them were unable to be meaningfully verified at the time of the audit (2023-05-19) 2. so...*reject those packages*. if you use a PGP key that isn't properly available or verifiable, reject it. That way every package with a PGP key will have 100% "key is properly discoverable" rate. it's not really reasonable to just drop this feature because most packages don't use it. Packages with tens of millions of downloads (like mine) make up a small percentage of total packages, but this small number of packages makes up a huge proportion of actual downloads, and package signing is most useful for these kinds of packages. if the adoption of "proper PGP keys" were ranked by packages/ downloads rather than "packages" alone, these rates would be much different.
- donaldstufft 3y ago
  I don't believe they would. Looking at the top 20 packages in the last month by download (packages with hundreds of millions of downloads), only 1 of them shipped a GPG signature with their most recent release. I haven't asked the author of that one, but I do know them and I suspect they agree with the idea that it's not a valuable thing and they do it largely because it exists.
  - oefrha 3y ago
    > they do it largely because it exists. That’s me. I used to upload signatures to PyPI only because it’s a thing that exists and it’s not much trouble. I’d be counted among the valid 36%, but I doubt anyone ever verified even one of the hundreds of sigs I uploaded over the years. I eventually stopped due to the pointlessness.
- 7to2 3y ago
  That quote doesn't make any sense even if we stopped at the first part. I PGP-sign my packages and my key is not on any public key server. It's on my website. This reasoning lacks rigor and seems to only serve as an excuse to remove a feature that some pypi devs didn't like without offering an alternative for security guarantees that it provided.
eduction 3y ago
So they examined everything uploaded to PyPi with a signature over three years, including old versions, and classify those packages whose signing key is expired today, possibly years later, as "impossible to meaningfully verify." Never mind that the package may have been verifiable with a valid key for a full year or two before the key expired, and in the meantime may have been superseded by a newer version. They also say they can't "meaningfully verify" packages if the key does not have "binding identify information," by which they presumably mean automatically verifiable binding identity information, which usually means someone verified an email from keys.openpgp.org. This is a really narrow way to establish "binding identity information." For example someone who is a PyPi author and publicly links their PGP key from a (https) website on the same domain as the email on the key would not count. A well known longtime PyPi author with a well known key would not count. The ad hoc, out of band nature of how PGP keys are trusted is not remotely new - PyPi would have known from the very start of adopting PGP that many keys would not be automatically verifiable. It makes little sense to turn around now and act like this is some surprising thing. This has the smell of "we didn't want to bother supporting PGP any more because it's hard so we came up with an excuse." No need for an excuse, though: Just be honest about it and let the chips fall where they may, if you really don't want to support PGP. God knows there are valid reasons for not having the energy to deal with PGP. (FWIW I think it's a good solution for packages, for those who can navigate the tooling, but on the other hand I'm not volunteering my time to run PyPi.) P.S. There is a link in their post saying PGP has "documented issues." The specific issue described in the linked document is "packaging signing is not the holy grail" and a list of known things about PGP, like that verification of keys is ad hoc. It also concludes that there is no known better alternative.
- tptacek 3y ago
  There are much, much better solutions for packaging!
  - c0l0 3y ago
    And which of those are in use/available at the Python Package Index?
jpgvm 3y ago
I don't understand how Java can get this right with Maven Central and co but newer languages can't. Having a slight barrier to entry which is essentially "you must learn why signing is important for users of your library and this is how to do it", a) really isn't that bad and b) doesn't result in less quality packages being uploaded c) if it acts like any sort of filter that seems to be a good thing. Maven Central isn't short of high quality packages and no high quality OSS Java libraries are missing so the filter aspect isn't culling anything important. Java, Apt, RPM, etc all have this and have absolutely gigantic numbers of packages so the argument that it's too hard really just doesn't hold water. Doing so requires reading/understanding these ~3 pages of docs: https://central.sonatype.org/publish/requirements/gpg/ https://central.sonatype.org/publish/requirements/gpg/
- blibble 3y ago
  > I don't understand how Java can get this right with Maven Central and co but newer languages can't. it's the magic combination of pushing their own agenda (vs. that of their users), mixed with ineptitude
- donaldstufft 3y ago
  I don't believe that Maven Central's use of GPG is providing a meaningful security control here, so I would dispute the idea that they're doing it "right".
  - jpgvm 3y ago
    At the very least there are a) more active keys b) those keys are available on keyservers and c) it's being used by the major packages in the ecosystem correctly. i.e Spring, Jackson, Quarkus, Logback, Apache-sphere, Google-sphere, etc. So while it might not be providing meaningful security for lower-tier packages it's definitely doing it's job for top tier packages like these that are relied on by hundreds of thousands of projects.
- B1FF_PSUVM 3y ago
  > newer languages can't. Python (1991) is older than Java (1995) (irrelevant factoid, but still ...)
forgotmypw17 3y ago
What an amazing opportunity for someone to add a new way of integrating PGP authentication by writing two short scripts: One to compile a list of file hashes and PGP-sign them. One to validate these hashes against the provided signatures.
jxy 3y ago
I don't understand the argument. Isn't the whole point of PGP establishing some kind of chain of trust? If pypi.org has it's public key, it could sign a few major distributors's keys, and for smaller/individual packages I could either choose to always trust the same public key or don't use the package. It's not a centralized system to begin with. It's not pypi.org's responsibility to identify and verify all the keys belong to who say they belong. Pypi.org's unable to verify individual identities shouldn't impact the overall usefulness of the PGP for package distribution and verification.
lgxz 3y ago
Replace a 31% effective solution with no solution? very impressive
sacnoradhq 3y ago
So how are Python packages signed? Are they just shipping rando code without any sort of E2E assurance? FWIW, Ruby also did a piss-poor job of handling gem signing by making it both difficult and optional. How fucking hard is it to get to the level of code release assurance as Debian or Fedora? Manage GPG keys, signfest them, and enforce a policy.
7to2 3y ago
Trust on first use is absolutely a valid use of PGP signatures that is being used in many real world systems (ask me how I know). You finding that PGP isn't being used they way you think it should does not justify removing it without providing a replacement. Why on earth wasn't the community asked before you implemented this change? > Given all of this, the continued support of uploading PGP signatures to PyPI is no longer defensible. While it doesn't represent a massive operational burden to continue to support it, it does require any new features that touch the storage of files to be made aware of and capable of handling these PGP signatures, which is a non zero cost on the maintainers and contributors of PyPI. This uninformed reasoning is what's indefensible.

187 points by dlor 3y ago

Now that you have removed GPG ASC signature upload support, is there any way for publishers to add cryptographic signatures to packages that they upload to pypi? FWIU only "the server signs uploads" part of TUF was ever implemented? Why do we use GPG ASC signatures instead of just a checksum over the same channel?

woodruffw 3y ago
> Why do we use GPG ASC signatures instead of just a checksum over the same channel? Could you elaborate on what you mean by this? PyPI computes and supplies a digest for every uploaded distribution, so you can already cross-check integrity for any hosted distribution. GPG was nominally meant to provide authenticity for distributions, but it never really served this purpose. That's why it's being removed.
- westurner 3y ago
  1) the server signs what's uploaded using one or more TUF keys shared to RAM on every pypi upload server. 2) the client uploads a cryptographic signature (made using their own key) along with the package, and the corresponding public key is trusted to upload for that package name, and the client retrieves said public key and verifies the downloaded package's cryptographic signature before installing. FWIU, 1 (PyPI signs uploads with TUF) was implemented, but 2 (users sign their own packages before uploading the signed package and signature, (and then 1)) was never implemented?
  - woodruffw 3y ago
    Your understanding is a little off: we worked on integrating TUF into PyPI for a while, but ran into some scalability/distributability issues with the reference implementation. It's been a few years, but my recollection was that the reference implementation assumed a lot of local filesystem state, which wasn't compatible with Warehouse's deployment (no local state other than tempfiles, everything in object storage). To the best of my knowledge, the current state of TUF for PyPI is that we performed a trusted setup ceremony for the TUF roots[1], but that no signatures were ever produced from those roots. For the time being, we're looking at solutions that have less operational overhead: Sigstore[2] is the main one, and it uses TUF under the hood to provide the root of trust. [1]: https://www.youtube.com/watch?v=jjAq7S49eow&t=1s https://www.youtube.com/watch?v=jjAq7S49eow&t=1s [2]: https://www.sigstore.dev/ https://www.sigstore.dev/
    - westurner 3y ago
      Where does the user specify the cryptographic key to sign a package before uploading? Serverside TUF keys are implemented FWICS, but clientside digital publisher signatures (like e.g. MS Windows .exe's have had when you view the "Properties" of the file for many years now) are not yet implemented. Hopefully I'm just out of date.
      - woodruffw 3y ago
        > Where does the user specify the cryptographic key to sign a package before uploading? With Sigstore, they perform an OIDC flow against an identity provider: that results in a verifiable identity credential, which is then bound to a short-lived (~15m) signing key that produces the signatures. That signing key is simultaneously attested through a traditional X.509 PKI (it gets a certificate, that certificate is uploaded to an append-only transparency log, etc.). So: in the normal flow, the user never directly specifies the cryptographic key -- the scheme ensures that they have a strong, ephemeral one generated for them on the spot (and only on their client device, in RAM). That key gets bound to their long-lived identity, so verifiers don't need to know which key they're verifying; they only need to determine whether they trust the identity itself (which can be an email address, a GitHub repository, etc.).
        westurner 3y ago
        What command(s) do I pass to pip/twine/build_pyproject.toml to build, upload, and install a package with a key/cert that users should trust for e.g. psf/requests?
    - trishankdatadog 3y ago
      python-tuf [1] back then assumed that everything was manipulated locally, yes, but a lot has changed since then: you can now read/write metadata entirely in memory, and integrate with different key management backend systems such as GCP. More importantly, I should point out that while Sigstore's Fulcio will help with key management (think of it as a managed GPG, if you will), it will not help with securely mapping software projects to their respective OIDC identities. Without this, how will verifiers know in a secure yet scalable way which Fulcio keys _should_ be used? Otherwise, we would then be back to the GPG PKI problem with its web of trust. This is where PEP 480 [2] can help: you can use TUF (especially after TAP 18 [3]) to do this secure mapping. Marina Moore has also written a proposal called Transparent TUF [4] for having Sigstore manage such a TUF repository for registries like PyPI. This is not to mention the other benefits that TUF can give you (e.g., protection from freeze, rollback, and mix-and-match attacks). We should definitely continue discussing this sometime. [1] https://github.com/theupdateframework/python-tuf https://github.com/theupdateframework/python-tuf [2] https://peps.python.org/pep-0480/ https://peps.python.org/pep-0480/ [3] https://github.com/theupdateframework/taps/blob/master/tap18.md https://github.com/theupdateframework/taps/blob/master/tap18... [4] https://docs.google.com/document/d/1WPOXLMV1ASQryTRZJbdg3wWRR4ckK558kUnxn7Eixaw/edit?usp=sharing https://docs.google.com/document/d/1WPOXLMV1ASQryTRZJbdg3wWR...
- westurner 3y ago
  > Why do we use GPG ASC signatures instead of just a checksum over the same channel? You can include an md5sum or a sha512sum string next to the URL that the package is downloaded from (for users to optionally check after downloading a package); but if that checksum string is uploaded over the same channel (HTTPS/TLS w/ a CA cert bundle) as the package, the checksum string could have been MITM'd/tampered with, too. A cryptographically-signed checksum can be verified once the pubkey is retrieved over a different channel (GPG: HKP is HTTPS/TLS with cert pinning IIRC), and a MITM would have to spend a lot of money to forge that digital publisher signature. Twine COULD/SHOULD download uploads to check the PyPI TUF signature, which could/should be shipped as a const in twine? And then Twine should check publisher signatures against which trusted map of package names to trusted keys?
ilyt 3y ago
Signature tells you who signed it. Of course, if you haven't put any effort in system to end-to-end verify whether it's right signature it doesn't matter.
- westurner 3y ago
  pip checks that a given was signed with the pypi key but does not check for a signature from the publisher. And now there's no way to host any type of cryptographic signatures on pypi. There is no e2e: pypi signs what's uploaded. (Noting also that packages don't have to be encrypted in order to have cryptographic signatures; only the signature is encrypted, not the whole package)
  - ilyt 3y ago
    Yeah the whole thing looks like throwing away baby with bathwater; the package should * get a signature for author ("the actual author published it") + some metadata with list of valid signing keys (in case project have more authors or just for key rotation * get a signature for hosting provider that confirms "yes, that actual user logged in and uploaded the package" * (the hardest part) key management on client side so the user have to do least amount of work possible in when downloading/updating valid package. If user doesn't want to go to effort to validate whether the public key of author is valid so be it but at very least system should alert on tampering with the provider (checking the hosting signature) or the author key changing (compromised credentials to the hosting provider). It still doesn't prevent "the attacker steals key off author's machine" but that is by FAR the rarest case and could be pretty reasonably prevented by just using hardware tokens. Hell, fund them for key contributors.

Arnavion 3y ago

>Of those 1069 unique keys, about 30% of them were not discoverable on major public keyservers, making it difficult or impossible to meaningfully verify those signatures. I don't know if it applies to any of those 1069 keys, but note that there is a way of hosting PGP keys that does not depend on key servers: WKD https://datatracker.ietf.org/doc/draft-koch-openpgp-webkey-service/16/ https://datatracker.ietf.org/doc/draft-koch-openpgp-webkey-s... . You host the key at a .well-known URI under the domain of the email address. It's a draft as you can see, but I've seen a few people using it (including myself), and GnuPG supports it.

woodruffw 3y ago
This is interesting, but it doesn't really solve the key distribution problem: with well-known hosting you now have a (weak) binding to some DNS path, but you're not given any indication of how to discover that path. It's also not clear that a DNS identity is beneficial to PyPI in particular (since PyPI doesn't associate or namespace packages at all w/r/t DNS). More generally, these kinds of improvements are not a sufficient reason to retain PGP: even with a sensible identity binding and key distribution, PGP is still a mess under the hood. The security of a codesigning scheme is always the weakest signature and/or key, and PGP's flexibility more or less ensures that that weakest link will always be extremely weak.
- Arnavion 3y ago
  Right. I've never used PyPI, but TFA makes it sound like the existing support for signing is "We allow the uploader to upload a signature, and the downloader can look up the key indicated in the signature to do the verification." Is that correct? If so, then yes there is a key ID involved but no email address, so a generic downloader would have no choice but to look it up from a key server.
  - woodruffw 3y ago
    That's correct! PyPI's support for PGP is very old -- it's hard to get an exact date, but I think it's been around since the very earliest versions of the index (well before it was a storing index like it is now). If I had to guess (speculate wildly), my guess would be that the original implementation was done with a healthy SKS network and strong set in mind -- without those things, PGP's already weak identity primitives are more or less nonexistent with just signatures.

KRAKRISMOTT 3y ago

What are we switching to? Does Pypi support ECDSA?

woodruffw 3y ago
Just for disambiguation: ECDSA is a signing algorithm, not a protocol or toolkit like PGP. PGP can produce ECDSA signatures through an extension RFC, but it's not a core part of OpenPGP. There is no immediate replacement, because the overwhelming majority of packages never bothered to sign with PGP (and all evidence points to the overwhelming majority of signatures never being verified). In other words, this is much closer to removing "dead" code than to killing an active feature. Longer term, the plan is to integrate Sigstore[1]-based signatures. [1]: https://www.sigstore.dev/ https://www.sigstore.dev/
jossclimb 3y ago
sigstore I hope.
- aborsy 3y ago
  I’m not sure if I understand this correctly, but, this basically seems to be a CA, with SSO-type of proof of identity, short lived certificates and transparency logs? How an OIDC identity is obtained and secured is not treated. It brings useful organization to PKI, but the problem remains. You have to delegate trust to identity providers: Google, GitHub, etc. Keybase was interesting, but the project seems semi-dead.
  - woodruffw 3y ago
    > How an OIDC identity is obtained and secured is not treated. It brings useful organization to PKI, but the problem remains. You have to delegate trust to identity providers: Google, GitHub, etc. Yes, this is a fundamental (and, IMO, reasonable) assumption in Sigstore. The trust argument for large IdPs is that they (1) have the institutional ability and resources (like incident response) to maintain their service, (2) have strong incentives to maintain and improve the overall security of their providers (billions of accounts on the Internet are bound to SSO via Google, etc.), and (3) that any failures in those providers are already catastrophic, so reducing the number of moving and potentially failing parts is a net win in terms of security.

usr1106 3y ago

Isn't that throwing out the baby with the bathwater? There seem to be non-neglible risks of installing malware from PyPI according to various headlines recently. But instead of improving security measures that don't work well they just remove them?

donaldstufft 3y ago
Removing security features that don't work is a separate concern from making security features that do work. Nobody who has done any serious work on PyPI security in the past 15 years thinks that GPG will play a part in the future of PyPI security. It's support was entirely vestigial, served no practical purpose, and never would.
- 7to2 3y ago
  [flagged]
  - gerikson 3y ago
    Note that the person you're replying to is the PyPi maintainer responsible for removing GPG from PyPi. The rationale is expanded here: https://news.ycombinator.com/item?id=36050190 https://news.ycombinator.com/item?id=36050190
    - 7to2 3y ago
      [flagged]
      - dang 3y ago
        Attacking other users will get you banned here. No more of this, please. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - dang 3y ago
    Please make your substantive points without swipes, in keeping with the site guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html. If someone else is wrong, it suffices to neutrally explain to them, and the rest of us, how they are wrong. Adding putdowns not only doesn't help, it harms the community and discredits your point, so please don't do that. Edit: please see https://news.ycombinator.com/item?id=36092913 https://news.ycombinator.com/item?id=36092913 also. You broke the site guidelines badly in this thread.

rwmj 3y ago

So they're removing PGP signatures, which certainly have some issues, and replacing them with ... nothing?

sowbug 3y ago
The research article cited in the announcement is titled "PGP signatures on PyPI: worse than useless." That's the issue. Pretending there is a security solution in place is worse than being upfront that there is none. If you look down and notice that your seatbelt is actually made out of angel hair pasta, you might drive more carefully. Hopefully you'll also get a better car.
- rwmj 3y ago
  But they're not "worse than useless", that article was wrong. PGP/GPG are without doubt problematic, they have weak points (like use of SHA-1, some keys that could not be located, and terrible UI) but they are not worse than having no traceability of the package at all between the author and PyPI.
  - bandrami 3y ago
    It's like the larger holy war against self-signed certificates in TLS. They are strictly better than plaintext but there is software that will prefer a plaintext connection to self-signed TLS.
    - Avamander 3y ago
      Unverifiable security is not "strictly better" than plaintext.
      - bandrami 3y ago
        It is. In both cases you may be MiTMed, but with plaintext that MiTM session may also be eavesdropped upon. This isn't really debateable. Unverified sig simply is strictly better.

tzs 3y ago

> In the last 3 years, about 50k signatures had been uploaded to PyPI by 1069 unique keys. Of those 1069 unique keys, about 30% of them were not discoverable on major public keyservers, making it difficult or impossible to meaningfully verify those signatures. Of the remaining 71%, nearly half of them were unable to be meaningfully verified at the time of the audit (2023-05-19) Why not include the public key in the package? 99% of the time what I want out of package signing is to know that the new version of the package I'm downloading is from the same people as the old version. I don't actually need to know who those people are...just that they are the same people as before.

hannob 3y ago
What happens if the developer looses his key? Or if it expires? pypi could show a warning that the key has changed. Which is not an actionable or helpful warning. And then everyone gets used to seeing these warnings every now and then. And you won nothing. Getting signatures to do something useful is hard.
- bombolo 3y ago
  > What happens if the developer looses his key? Or if it expires? What happens if a developer loses their google titan key that is required to login into pypi?
  - hannob 3y ago
    They either have their backup codes or there's probably a manual process the pypi team can get them their account back if they can sufficiently show they are the real developers. If you have any form of automated signature verification you basically need a concept how to handle recovery. But if this concept comes down to "trust pypi", then you really can just skip the whole thing and rely on pypi giving you the right packages and https to secure the connection).
woodruffw 3y ago
> Why not include the public key in the package? Because PyPI (or an attacker) could always substitute a new key. There's very little value in the signature and key coming from the same source: the key (and its justified identity) always need to come from a source of trust, not the source that's being verified. > 99% of the time what I want out of package signing is to know that the new version of the package I'm downloading is from the same people as the old version. I don't actually need to know who those people are...just that they are the same people as before. This might be a misunderstanding, but I don't think you actually want this: lots of large packages have multiple release managers (and contributors who come and go); you don't want to manually resolve each new human identity that appears for a package distribution. What most people actually want is a strong cryptographic attestation that the package distribution came from the same source as the thing hosting the source code, since both that service and the owner of the repository are presumed trusted. Notably, PGP is incapable of providing either of these: you only get key IDs, which are neither strong human identities nor a strong binding to a service. Key IDs might correspond to keys with email (or other identities) in them, but that's (1) not guaranteed, and (2) not a strong proof of identity (since anybody can claim any identity in a PGP key).

jwilk 3y ago

Two days ago: https://news.ycombinator.com/item?id=36021172 https://news.ycombinator.com/item?id=36021172 ("PGP signatures on PyPI: worse than useless", >150 comments)

reidrac 3y ago

I have been thinking about this in the context of Java libraries (really using Scala, but bear with me). If the repo requires a GPG signature, they could also ask for the public key of the developer making the releases (e.g. when they make the account), and they could sign it with their key at that point. Then make available the package, the signature, and the signed public key. Then I only need to trust the repo's key (in this case PyPi). Does this make any sense?

woodruffw 3y ago
> Does this make any sense? It makes sense in terms of trusting the package index, but it's inverted from the original design goal: the point of end-user signatures on package indices is to eliminate unnecessary package index trust, not reinforce it. If you already trust the package index, then mandating HTTPS and strong cryptographic digests is going to be far more effective (and secure) than some kind of PGP key attestation scheme.
- reidrac 3y ago
  The package index only hosts the packages, but doesn't release them. The dev releasing the package is who signs it. Without an easy way to verify the keys, the signatures are useless. Which is why PiPy is removing the GPG keys all together.
  - woodruffw 3y ago
    > The package index only hosts the packages, but doesn't release them. The dev releasing the package is who signs it. I know that; the GP is describing a countersigning scheme, where the package index (qua trusted entity) countersigns for the signing key, which the dev then uses to sign for their package. > Without an easy way to verify the keys, the signatures are useless. Which is why PiPy is removing the GPG keys all together. Agreed entirely; I'm the one who wrote the analysis in the linked announcement :-)

zokier 3y ago

At least you can't blame pypi for ignoring the report, and tbh I find this response time remarkably quick. It wouldn't have been far fetched to imagine someone in their position just trying to ignore/downplay/dispute this sort of reports.

deleted 3y ago
[deleted]
masklinn 3y ago
As the author of the post noted above, the pypi maintainers have been wanting to get rid of pgp for awhile. The post gave them excellent additional justification to.

WhyNotHugo 3y ago

When many developers didn't use 2FA they pushed for them to enable 2FA within a deadline. It sounds like the same approach could have been used for PyPI. E.g.: an attempt to make the feature useful before declaring it dead forever.

woodruffw 3y ago
This has very little to do with 2FA: PGP signing has been de facto dead for years on PyPI, and this change has no effect on publishing workflows: PyPI will still accept uploads that contain signatures, and just ignores them now. It's also not accurate to say that PyPI failed to make 2FA useful: it was deployed for over two years before the 2FA mandate for critical projects went into effect. That mandate also came with free hardware keys for everyone affected.
masklinn 3y ago
No. 2FA is a feature for pypi, and developers. The entire purpose of pgp sigs was external, it was for distributions to use. Distributions don’t use it, therefore it’s worthless, just just overhead and technical debt.
- LtWorf 3y ago
  Debian checks PGP signatures of releases.
  - adamckay 3y ago
    For Python packages served by PyPI?
    - donaldstufft 3y ago
      Sometimes? There's no global policy of doing it in Debian, it's up to individual package maintainers inside of Debian to enable it (it defaults to off AFAIK) and to hardcode the key that they expect the package to be signed by. In the cases that it is used, AFAIK it is only used by Debian's uscan program, which is sort of like the Debian version of Dependabot, it tells them when there is a new version of something to package. As far as I know, the process of packaging that new version is still manual, and relies on the maintainer downloading the package and packaging it, so they may or may not use the signature in that case. How useful this is, is up for debate. Many years ago when I first started taking over releasing pip, that caused the pip GPG key to change, and the reaction of the Debian maintainer at the time was to just comment out the signature bit and fall back to no signature.

bryanlyon 3y ago

I came here thinking they were removing the PGP package from PyPi, but they're just removing a barely-used signature system? I don't know why they have to remove it though. I doubt it requires much maintenance now that it's already in place. Even if only 37% of keys are verifiable, that's infinitely more than will be verifiable if they remove the PGP support.

Avamander 3y ago
> Even if only 37% of keys are verifiable, that's infinitely more than will be verifiable if they remove the PGP support. Discoverable. That does not really verify anything about the key, its identities or the supposed signer. It boils down to almost entirely to just an overcomplicated hashing system.
tedivm 3y ago
They address your comment directly in their post- > While it doesn't represent a massive operational burden to continue to support it, it does require any new features that touch the storage of files to be made aware of and capable of handling these PGP signatures, which is a non zero cost on the maintainers and contributors of PyPI.

rvz 3y ago

PGP is a solution in search of a problem. We have given it decades for it to be useful and it turns out that it is an enormous security failure. It needed to go. Sigstore [0] on the other hand makes more sense to use instead of problem. [0] https://www.sigstore.dev https://www.sigstore.dev

msm_ 3y ago
This reads like an advertisement. I routinely use GPG, and it is useful for me. It's not perfect (far from perfect, really), but it's a solution for multiple of my problems. I don't know much about the solution you promote, but as usual with many "PGP killers" it replaces one very specific application of PGP and ignores all the others. Which is ok! Doing one thing and doing it well is the Unix philosophy after all. But it's not something I have use for, and it's not a viable replacement for GPG.
- tptacek 3y ago
  If doing one thing and doing it well is the core of the Unix philosophy, PGP is (cryptographically) the antithesis of that. It's a Swiss Army Knife that does none of its tasks well by modern standards.
LtWorf 3y ago
I'll let my boss know we must stop signing our releases and having our software automatically check if the new version is legit then. We will instead switch to use some thing with a fluffy corporate website that tells absolutely nothing.

NotYourLawyer 3y ago

Interesting timeline. The Yossarian article that TFA cites and that I assumed was the impetus here was published two days ago on 5/21. But the audit was two days earlier on 5/19.

woodruffw 3y ago
I originally ran the audit on 3/27 (IIRC), and then ran it a few additional times as I fixed data quality issues in my scripts (the ones linked in the post). The last time I ran it was on 5/19-5/20, when I was finalizing the post. You can also see that I did a new release of `pgpkeydump` at around the same time, to add some more extracted datapoints. PyPI's admins have been wanting to remove PGP support for years; all I did was provide the final nudge.

zzzeek 3y ago

> Of those 1069 unique keys, about 30% of them were not discoverable on major public keyservers, making it difficult or impossible to meaningfully verify those signatures. Of the remaining 71%, nearly half of them were unable to be meaningfully verified at the time of the audit (2023-05-19) 2. so...*reject those packages*. if you use a PGP key that isn't properly available or verifiable, reject it. That way every package with a PGP key will have 100% "key is properly discoverable" rate. it's not really reasonable to just drop this feature because most packages don't use it. Packages with tens of millions of downloads (like mine) make up a small percentage of total packages, but this small number of packages makes up a huge proportion of actual downloads, and package signing is most useful for these kinds of packages. if the adoption of "proper PGP keys" were ranked by packages/ downloads rather than "packages" alone, these rates would be much different.

donaldstufft 3y ago
I don't believe they would. Looking at the top 20 packages in the last month by download (packages with hundreds of millions of downloads), only 1 of them shipped a GPG signature with their most recent release. I haven't asked the author of that one, but I do know them and I suspect they agree with the idea that it's not a valuable thing and they do it largely because it exists.
- oefrha 3y ago
  > they do it largely because it exists. That’s me. I used to upload signatures to PyPI only because it’s a thing that exists and it’s not much trouble. I’d be counted among the valid 36%, but I doubt anyone ever verified even one of the hundreds of sigs I uploaded over the years. I eventually stopped due to the pointlessness.
7to2 3y ago
That quote doesn't make any sense even if we stopped at the first part. I PGP-sign my packages and my key is not on any public key server. It's on my website. This reasoning lacks rigor and seems to only serve as an excuse to remove a feature that some pypi devs didn't like without offering an alternative for security guarantees that it provided.

eduction 3y ago

So they examined everything uploaded to PyPi with a signature over three years, including old versions, and classify those packages whose signing key is expired today, possibly years later, as "impossible to meaningfully verify." Never mind that the package may have been verifiable with a valid key for a full year or two before the key expired, and in the meantime may have been superseded by a newer version. They also say they can't "meaningfully verify" packages if the key does not have "binding identify information," by which they presumably mean automatically verifiable binding identity information, which usually means someone verified an email from keys.openpgp.org. This is a really narrow way to establish "binding identity information." For example someone who is a PyPi author and publicly links their PGP key from a (https) website on the same domain as the email on the key would not count. A well known longtime PyPi author with a well known key would not count. The ad hoc, out of band nature of how PGP keys are trusted is not remotely new - PyPi would have known from the very start of adopting PGP that many keys would not be automatically verifiable. It makes little sense to turn around now and act like this is some surprising thing. This has the smell of "we didn't want to bother supporting PGP any more because it's hard so we came up with an excuse." No need for an excuse, though: Just be honest about it and let the chips fall where they may, if you really don't want to support PGP. God knows there are valid reasons for not having the energy to deal with PGP. (FWIW I think it's a good solution for packages, for those who can navigate the tooling, but on the other hand I'm not volunteering my time to run PyPi.) P.S. There is a link in their post saying PGP has "documented issues." The specific issue described in the linked document is "packaging signing is not the holy grail" and a list of known things about PGP, like that verification of keys is ad hoc. It also concludes that there is no known better alternative.

tptacek 3y ago
There are much, much better solutions for packaging!
- c0l0 3y ago
  And which of those are in use/available at the Python Package Index?

jpgvm 3y ago

I don't understand how Java can get this right with Maven Central and co but newer languages can't. Having a slight barrier to entry which is essentially "you must learn why signing is important for users of your library and this is how to do it", a) really isn't that bad and b) doesn't result in less quality packages being uploaded c) if it acts like any sort of filter that seems to be a good thing. Maven Central isn't short of high quality packages and no high quality OSS Java libraries are missing so the filter aspect isn't culling anything important. Java, Apt, RPM, etc all have this and have absolutely gigantic numbers of packages so the argument that it's too hard really just doesn't hold water. Doing so requires reading/understanding these ~3 pages of docs: https://central.sonatype.org/publish/requirements/gpg/ https://central.sonatype.org/publish/requirements/gpg/

blibble 3y ago
> I don't understand how Java can get this right with Maven Central and co but newer languages can't. it's the magic combination of pushing their own agenda (vs. that of their users), mixed with ineptitude
donaldstufft 3y ago
I don't believe that Maven Central's use of GPG is providing a meaningful security control here, so I would dispute the idea that they're doing it "right".
- jpgvm 3y ago
  At the very least there are a) more active keys b) those keys are available on keyservers and c) it's being used by the major packages in the ecosystem correctly. i.e Spring, Jackson, Quarkus, Logback, Apache-sphere, Google-sphere, etc. So while it might not be providing meaningful security for lower-tier packages it's definitely doing it's job for top tier packages like these that are relied on by hundreds of thousands of projects.
B1FF_PSUVM 3y ago
> newer languages can't. Python (1991) is older than Java (1995) (irrelevant factoid, but still ...)

forgotmypw17 3y ago

What an amazing opportunity for someone to add a new way of integrating PGP authentication by writing two short scripts: One to compile a list of file hashes and PGP-sign them. One to validate these hashes against the provided signatures.

jxy 3y ago

I don't understand the argument. Isn't the whole point of PGP establishing some kind of chain of trust? If pypi.org has it's public key, it could sign a few major distributors's keys, and for smaller/individual packages I could either choose to always trust the same public key or don't use the package. It's not a centralized system to begin with. It's not pypi.org's responsibility to identify and verify all the keys belong to who say they belong. Pypi.org's unable to verify individual identities shouldn't impact the overall usefulness of the PGP for package distribution and verification.

lgxz 3y ago

Replace a 31% effective solution with no solution? very impressive

sacnoradhq 3y ago

So how are Python packages signed? Are they just shipping rando code without any sort of E2E assurance? FWIW, Ruby also did a piss-poor job of handling gem signing by making it both difficult and optional. How fucking hard is it to get to the level of code release assurance as Debian or Fedora? Manage GPG keys, signfest them, and enforce a policy.

7to2 3y ago

Trust on first use is absolutely a valid use of PGP signatures that is being used in many real world systems (ask me how I know). You finding that PGP isn't being used they way you think it should does not justify removing it without providing a replacement. Why on earth wasn't the community asked before you implemented this change? > Given all of this, the continued support of uploading PGP signatures to PyPI is no longer defensible. While it doesn't represent a massive operational burden to continue to support it, it does require any new features that touch the storage of files to be made aware of and capable of handling these PGP signatures, which is a non zero cost on the maintainers and contributors of PyPI. This uninformed reasoning is what's indefensible.