11 ms·
PyPI now supports digital attestations
- dlor 2y agoThis is awesome to see, and the result of many years of hard work from awesome people.
- woodruffw 2y agoI'm very excited this has come to fruition! PyPI's user docs[1] have some more details, as does the underlying PEP[2]. [1]: https://docs.pypi.org/attestations/ https://docs.pypi.org/attestations/ [2]: https://peps.python.org/pep-0740/ https://peps.python.org/pep-0740/
- ashvardanian 2y agoHere is the GitHub issue you can subscribe to for automatically generated attestations in the GitHub CI PyPi upload action: https://github.com/pypa/gh-action-pypi-publish/issues/288 https://github.com/pypa/gh-action-pypi-publish/issues/288
- woodruffw 2y agoJust a note: that issue is for connecting the already-present attestations to GitHub's attestations feature[1]. Attestations are already automatically generated and uploaded to PyPI regardless of that issue. (That issue will still be great to resolve, but just to avoid any confusion about its scope.) [1]: https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations/using-artifact-attestations-to-establish-provenance-for-builds https://docs.github.com/en/actions/security-for-github-actio...
- antononcube 2y agoIt looks like another process certain software engineers want to program and facilitate. I hope it is and stays optional.
- antononcube 2y ago[flagged]
- LtWorf 2y agoI'm pessimistic about that.
- trishankkarthik 2y agoVery cool, and congrats! The corresponding ToB blog post says the following: > Longer term, we can do even better: doing “one off” verifications means that the client has no recollection of which identities should be trusted for which distributions. To address this, installation tools need a notion of “trust on first use” for signing identities, meaning that subsequent installations can be halted and inspected by a user if the attesting identity changes (or the package becomes unattested between versions). Agree: signing is only as good as verification. However, trust-on-first-use (TOFU) is not the most secure way to map packages to attestations because nothing stops attackers who have taken over PyPI from tampering with the _unsigned_ mapping of identities to attestations in package lockfiles for new clients (certainly in containerized environments where everything could look new), and even just new versions of packages. Although [PEP 458](https://peps.python.org/pep-0458/ https://peps.python.org/pep-0458/) is about signing the Python package index, it sets the foundation for being able to securely map packages to signed in-toto _policies_, which would in turn securely map identities to attestations. I think it is worth noting how these different PEPs can work together :)
- cpburns2009 2y agoGreat, now how do you use attestations with Twine when publishing packages on PyPI outside of the Github ecosystem?
- hifromwork 2y agoYou need to rely on one of the four trusted publishers. You can't do it yourself: https://docs.pypi.org/trusted-publishers/adding-a-publisher/ https://docs.pypi.org/trusted-publishers/adding-a-publisher/
- guappa 2y agoYou don't. The whole point is that you can no longer sign anything. Microsoft signs for you. And of course the signature means "this user can push to github" and nothing more.
- krnavy 2y agoAfter 2FA, the previous PyPI buzzword that was forced on everyone, JFrog discovered a key leak that compromised everything: https://news.ycombinator.com/item?id=40941809 https://news.ycombinator.com/item?id=40941809 JFrog also discovered multiple malicious package exploits later. Now we get a Github centric new buzzword that could be replaced by trusted SHA256 sums. Python is also big on business speak like SBOM. The above key leak of course occurred after all these new security "experts" manifested themselves out of nowhere. The procedure remains the same. Download a package from the original creators, audit it, use a local repo and block PyPI.
- SethMLarson 2y agoHello! I believe I'm one of the "manifested" security experts you're hinting at :) Good security doesn't demand perfection, that's why security is both prevention and preparedness. The response from our admin was in every way beyond what you'd expect from many other orgs: prompt response (on the order of minutes), full audit of activity for the credential (none found), and full public disclosure ahead of the security researcher's report. > JFrog also discovered multiple malicious package exploits later. If you're referencing malicious packages on PyPI then yes! We want to keep PyPI freely open to all to use, and that has negative knock-on effects. Turns out that exploiting public good code repositories is quite popular, but I maintain that the impact is quite low and that our ability to respond to these sorts of attacks is also very good due to our network of security engineers and volunteers who are triaging their reports. Thanks to the work of Mike Fiedler (quarantining packages, API for reporting malicious packages, better UI for triagers) our ability to respond to malicious packages will become even better. > Now we get a Github centric new buzzword that could be replaced by trusted SHA256 sums. In a way, this feature is what you're describing but is easier to automate (therefore: good for you as a user) and is more likely to be correct because every attestation is verified by PyPI before it's made available to others (which is also good for users). The focus on GitHub Actions is because this is where many Python projects publish from, there is little incentive to create a feature that no one will use. > Python is also big on business speak like SBOM. Indeed, there is legislation in many places that will require SBOMs for all software placed in their markets so there is plenty of interest in these standards. I'm working on this myself to try to do the most we can for users while minimizing the impact this will have on upstream open source project maintainers.
- belval 2y agoI have a bit of uneasiness about how this is heavily pushing GitHub actions as the correct way to publish to PyPI. I had to check PEP740 to make sure it was not directly supported by Microsoft. > The generation and publication of attestations happens by default, and no changes are necessary for projects that meet all of these conditions: publish from GitHub Actions; via Trusted Publishing; and use the pypa/gh-action-pypi-publish action to publish. If you then click on "The manual way" it adds a big disclaimer: > STOP! You probably don't need this section; it exists only to provide some internal details about how attestation generation and uploading work. If you're an ordinary user, it is strongly recommended that you use one of the official workflows described above. Where the only official workflow is "Use GitHub Actions". I guess I am an idealist but as a maintainer this falls short of my expectations for the openness of Python and PyPI.
- woodruffw 2y ago> Where the only official workflow is "Use GitHub Actions". The standard behind this (PEP 740) supports anything that can be used with Trusted Publishing[1]. That includes GitLab, Google Cloud, ActiveState, and can include any other OIDC IdP if people make a good case for including it. It's not tied to Microsoft or GitHub in any particular way. The only reason it emphasizes GitHub Actions is because that's where the overwhelming majority of automatic publishing traffic comes from, and because it follows a similar enablement pattern as Trusted Publishing did (where we did GitHub first, followed by GitLab and other providers). [1]: https://docs.pypi.org/trusted-publishers/ https://docs.pypi.org/trusted-publishers/
- silverwind 2y agoWhy does this need to allowlist CI providers in first place? Why not publish an open interface any CI provider can integrate against?
- SethMLarson 2y agoBecause every CI/ID provider has a different set of claims and behaviors that would constitute a "secure" policy for verification. If there was one singular way to do that then we could, but there isn't yet so PyPI needs to onboard providers piecemeal. The work to add a new provider is not massive, the reason there are not tons of providers isn't because the work is hard but rather because people are voting with their feet so Github and Gitlab make sense as initial providers to support.
- SethMLarson 2y agoCongratulations to everyone involved in this work! This is an incredible building block for not just supply-chain security both inside and downstream of the PyPI ecosystem but also for data analysis of open source projects (through strong linkage back to source repositories). Thank you all <3
- jonnycomputer 2y agoSupply chain security is very important, and this seems like an important step. Seems absolutely essential that something like the Mozilla foundation, or EFF, or some other open-source friendly entity help provide such a service, instead of corralling users into companies with exploitative business models. I am in no hurry to be pushed into using Github, Gitlab or whatever else. Programmer's open source code has been monetized by these companies to feed AI LLM beasties, and it's fundamentally objectionable to me. I self-host my code using Gitea for that reason.
- Comma2976 2y agoGreat, now all that is missing is a decent packaging system for Python.
- ris 2y agoI'm not really convinced of the value of such attestations until a second party can reproduce the build themselves on their own hardware. Putting aside the fact that the mechanisms underpinning Github Actions are a mystery black box, the vast vast vast majority of github workflows are not built in a reproducible way - it's not even something that's encouraged by Github Actions' architecture, which emphasises Actions' container images that are little more than packaged installer scripts that go and download dependencies from random parts of the internet at runtime. An "attestation" makes no guarantee that one of these randomly fetched dependencies hasn't been usurped. This is not to go into the poor security record of Github Actions' permissions model, which has brought us all a number of "oh shit" moments.
- mhils 2y agoFully reproducible builds would of course be nicer from a security standpoint, but attestations have vastly lower implementation costs and scale much better while still raising the bar meaningfully.
- amelius 2y agoI'm curious what would happen if a maintainer's PC is compromised. Is there any line of defense left at that point?
- guappa 2y agoNone. Developer machine will have ssh keys and github tokens that can be used to push a commit on github, that will be built, signed, and uploaded on pypi.
- amelius 2y agoThat sounds like a gigantic attack surface then ...
- guappa 2y agoI think since when they have 2FA PyPI is less secure. Before I could learn my password and type it on twine. If my machine was stolen no upload on pypi was possible. Now it's a token file on my disk so if my machine is stolen, then token can be used to publish. Using github to publish doesn't change anything: if my machine is stolen the token needed to publish is still there, but instead of directly to pypi it will need to go via github first.
- amelius 2y agoTokens are a problem too (a yubikey might be a solution). But an attacker could simply edit the source code on the maintainer's machine directly, and it could go unnoticed.
- cpburns2009 2y agoI doubt Yubikey would help without some fancy setup. 2FA is required to sign into PyPI but that's it. When PyPI rolled it out I thought you'd have to use 2FA every time you publish. I thought they were taking security seriously. But no, you get your API token, save it to your computer, forget about it, and you can publish your packages forever. Now you can have Github automatically publish your packages. That's not any improvement to security. My Google security key is just collecting dust.
- pabs3 2y agoWonder when PyPI will be doing bootstrappable and reproducible builds. https://bootstrappable.org/ https://bootstrappable.org/ https://reproducible-builds.org/ https://reproducible-builds.org/
- stavros 2y agoGiven that it doesn't do builds, never?
- OutOfHere 2y agoAre tools like uv, rye, hatch, etc. going to facilitate this?
- HelloNurse 2y agoThey could if there was a way to create an attestation that the tool (and you) can use.
- Uptrenda 2y agoDespite crypto getting easier and easier + more main stream. You often don't see it used that often. Even in ((('"```blockchain'"```))) projects (wink emoji; skull emoji; frown; picture of a chart crashing.) I welcome these additions to PyPI. I am a proud Python Chad and always will be.
- guappa 2y agoThey even got some public money from Germany's sovreign tech fund to couple uploads with gigantic USA companies. This is probably deserving a criminal investigation since it appears the funds were probably misused? Well done guys! Good job!
- tzlander 2y agoThanks for pointing this out! In general, the sovereign tech fund should not fund Python or the PSF, since the administration is U.S. dominated and the whole organization is captured by U.S. corporations. Unpaid developers from Europe (and Asia) just serve as disposable work horses for the U.S. "elites".
- guappa 2y agoI'm just sad I didn't think to write this earlier and now it won't be seen by many.
- kps 2y agoYou can count me among those that are suspicious that this is a frog-boiling step, but it doesn't appear to me that STF money went to this, from https://www.sovereign.tech/tech/python-package-index#what-are-we-funding https://www.sovereign.tech/tech/python-package-index#what-ar... Maybe there is a case to be made for STF to fund making Codeberg (a German-headquartered organization) one of the PyPI trusted hosts. If Codeberg were supported, that would go a long way to addressing fears. And conversely, if Codeberg can't meet PyPI's bar, that suggests complete commercial capture.
- guappa 2y agoYes I agree. It'd be nice if codeberg got some funding and could be used to upload stuff in the same way github can. Then pypi wouldn't just me a means to the end of keeping people on github rather than going to open platforms.
- jeroenhd 2y agoFrom the [docs](https://docs.pypi.org/trusted-publishers/internals/ https://docs.pypi.org/trusted-publishers/internals/): > Reliability & notability: The effort necessary to integrate with a new Trusted Publisher is not exceptional, but not trivial either. In the interest of making the best use of PyPI's finite resources, we only plan to support platforms that have a reasonable level of usage among PyPI users for publishing. Additionally, we have high standards for overall reliability and security in the operation of a supported Identity Provider: in practice, this means that a home-grown or personal use IdP will not be eligible. From what I can tell, this means using self-hosted Github/Gitlab/Gitea/whatever instances is explicitly not supported for most projects. I can't really tell what "a reasonable level of usage among PyPI users" means (does the invent.kde.org qualify? gitlab.gnome.org? gitlab.postmarketos.org?) but I feel like this means "Gitlab.com and maybe gitea.com" based on my original reading. Of course by definition PyPI is already a massive single point of failure, but the focus on a few (corporate) partners to allow secure publishing feels like a mistake to me. I'm not sure what part of the cryptographic verification process restricts this API to the use of a few specific cloud providers.
- Ferret7446 2y agoI wonder why they aren't using SLSA attestations. It'd be good to have a single standard than every language having its own format.
- woodruffw 2y agoThis standard does support SLSA attestations. We’re enabling upload support for them pretty soon.
- cpburns2009 2y agoThe root point of this seems to be PyPI does not have the resources to manage user identity, and wants to outsource that component to Github, et al. That sounds fairly reasonable. But why deprecate GPG signatures? The problem with GPG signatures as I understand it is it's difficult to find the associated public key. That's fair. Why not host and allow users to add their public keys to their accounts? Wouldn't that solve the problem?
- dale_glass 2y agoGPG is an ancient bit of tech with numerous problems: * An extremely complex, byzantine packet format with security problems of its own. * Decades of backwards compatibility, which also harms security. * Extreme unfriendliness towards automation. * Way too many features. * Encouragement of bad security practices like extremely long lived keys. * Moribund and flawed ecosystem. Lots of cryptographers agree that PGP has outlived its usefulness and it's time to put it out of its misery. And really there's little need for GPG when package signing can be done more reliably and with less work without it. I was a fan of PGP since the early days, but I agree that at this point it's best to abandon it.
- cpburns2009 2y agoI'll take your word that GPG is outdated. GPG is just the one that PyPI used to support. I don't particularly care what public key signing suite is used.
- ldng 2y agoSo, let me get this clear, after PyPA enshittifying the package expérience for more than a decade, and now mess has been clean by uv, the security FUD (and let not be naive here, the power struggle) move onto enshittifying PyPI and make Python user miserable for a decade more ?