7 ms·
Are We PEP740 Yet?
- zahlman 2y ago>Using a Trusted Publisher is the easiest way to enable attestations, since they come baked in! See the PyPI user docs and official PyPA publishing action to get started. For many smaller packages in this top 360 list I could imagine this representing quite a bit of a learning curve.
- amiga386 2y agoOr it could see Microsoft tightening its proprietary grip over free software by not only generously offering gratis hosting, but now also it's a Trusted Publisher and you're not - why read those tricky docs? Move all your hosting to Microsoft today, make yourself completely dependent on it, and you'll be rewarded with a green tick!
- simonw 2y agoI think it's a little rude to imply that the people who worked on this are serving an ulterior motive.
- akira2501 2y agoIt's possible they're just naive.
- Spivak 2y agoMicrosoft for sure has an ulterior motive here, and the PyPI devs are serving it. It's not a bad thing, it's a win-win for both parties. That kind of carrot is how you get buy-in from huge companies and in return they do free labor for you that secures your software supply chain.
- woodruffw 2y agoMicrosoft was not and is not involved in any way with this work. If you look at the announcements, you'll observe that a different, large, soulless megacorporation that is one of Microsoft's primary competitors funded it. If you're going to cast aspersions about hidden motives, you might as well cast them at the right entity.
- amiga386 2y agoWould you agree that most of the large, soulless megacorporations want moats? They want extra bureaucracy or complexity or other costs, which they as a behemoth can throw people and money at, but would bury a scrappy upstart challenger. It's the same with e.g. Google and HTML standards; Microsoft and document standards; Amazon and cloud APIs; IBM/Microsoft and patents. If that's the case, it doesn't matter which megacorporation sponsors the work; it benefits all of them. I'm not saying this specific security initiative is or isn't worthwhile, that remains to be seen. But whether it was intended or not, it's a windfall to Microsoft for an advocacy website to demand PEP740 when? and the most practical way to apply it right now is Microsoft's proprietary offering. Also, if I'm reading this correctly, the docs warn you off setting up your own Trusted Publisher and steer you towards using an existing one... of which there is currently only one choice, Microsoft. That's just an endorsement with extra steps.
- woodruffw 2y agoI think, strategically, that all companies want moats. I do not think this will ultimately be anything resembling a meaningful for moat for MSFT (or GOOG), given that it's designed to interoperate with anybody who can run an OIDC IdP. As I've said about 50 times in the last day: the only reason GitHub is being used for examples and was the first one enabled is because that's where the overwhelming majority of PyPI upload traffic comes from. It's not a commitment or an endorsement; it's a strictly strategic move to help the largest uploading demographic first.
- zahlman 2y agoThankfully, the PyPI side of the hosting is done by a smaller, unrelated company (Fastly).
- simonw 2y agoI think it's pretty hard to get a Python package into the top 360 list while not picking up any maintainers who could climb that learning curve pretty quickly. I wrote my own notes on how to use Trusted Publishers here: https://til.simonwillison.net/pypi/pypi-releases-from-github https://til.simonwillison.net/pypi/pypi-releases-from-github The bigger problem is for projects that aren't hosting on GitHub and using GitHub Actions - I'm sure there are quite a few of those in the top 360. I expect that implementing attestations without using the PyPA GitHub Actions script has a much steeper learning curve, at least for the moment.
- St-Clock 2y ago> I think it's pretty hard to get a Python package into the top 360 list while not picking up any maintainers who could climb that learning curve pretty quickly. I can speak for experience. Py4J is on that list and getting maintainers is very difficult (for various reasons). Packaging is also not something that naturally attracts contributions.
- woodruffw 2y agoI suspect that most of the packages in the top 360 list are already hosted on GitHub, so this shouldn’t be a leap for many of them. This is one of the reasons we saw Trusted Publishing adopted relatively quickly: it required less work and was trivial to adopt within existing CI workflows.
- marky1991 2y agoCould someone explain why this is important? My uninformed feeling towards PEP 740 is 'who cares?'.
- rty32 2y agohttps://en.m.wikipedia.org/wiki/XZ_Utils_backdoor https://en.m.wikipedia.org/wiki/XZ_Utils_backdoor
- marky1991 2y agoBut that involved one of the developers of said package committing malicious code and it being accepted and then deployed. How would this prevent that from happening? I thought this was about ensuring the code that developers pushed is what you end up downloading.
- rty32 2y agoNo, part of the malicious code is in test data file, and the modified m4 file is not in the git repo. The package signed and published by Jia Tan is not reproducible from the source and intentionally done that way. You might want to revisit the script of xz backdoor.
- epcoa 2y agoAn absolutely irrelevant detail here. While there was an additional flourish of obfuscation of questionable prudence, the attack was not at all dependent on that. It’s a library that justifies all kinds of seemingly innocuous test data. There were plenty of creative ways to smuggle in selective backdoors to the build without resorting to a compromised tar file. The main backdoor mechanism resided in test data in the git repo, the entire compromise could have.
- hadlock 2y agoI believe this is a system where a human/system builds a package and uploads and cryptographically signs it, verifying end to end that the code uploaded to github for widget-package 3.2.1 is the code you're downloading to your laptop for widget-package 3.2.1 and there's no chance it is modified/signed by a adversarial third party
- simonw 2y agoI suggest reading this detailed article to understand why they built this: https://blog.trailofbits.com/2024/11/14/attestations-a-new-generation-of-signatures-on-pypi/ https://blog.trailofbits.com/2024/11/14/attestations-a-new-g... The implementation is interesting - it's a static page built using GitHub Actions, and the key part of the implementation is this Python function here: https://github.com/trailofbits/are-we-pep740-yet/blob/a87a8895dd238d14af50aaa2675c81060aa52846/utils.py#L31-L72 https://github.com/trailofbits/are-we-pep740-yet/blob/a87a88... If you read the code you can see that it's hitting pages like https://pypi.org/simple/pydantic/ https://pypi.org/simple/pydantic/ - which return HTML - but sending this header instead: Accept: application/vnd.pypi.simple.v1+json Then scanning through the resulting JSON looking for files that have a provenance that isn't set to null. Here's an equivalent curl + jq incantation: curl -s \ -H 'Accept: application/vnd.pypi.simple.v1+json' \ https://pypi.org/simple/pydantic/ \ | jq '.files | map(select(.provenance != null)) | length'
- Cthulhu_ 2y agoThat's the first time I've seen JSON api standard headers in the wild. There was a project where an architect indicated our APIs should be built in that fashion, but people just... disregarded it completely out of pragmatism, also because our endpoints were just pure API / JSON endpoints, never anything else. But seeing how it's used in the wild is pretty clever, same endpoint for different use cases.
- the_mitsuhiko 2y agoPython folks are a bit obsessed with weird, novel or otherwise barely adopted standards, particularly around packaging and PyPI. They also use Macaroons for tokens. It's quite interesting to see, but they rarely become particularly popular outside of that community.
- woodruffw 2y agoI can take partial credit (blame?) for going forwards with Macaroons. The reason those were selected originally is because they allow distributed permission attenuation, and the thinking was that individual users/orgs could manually attenuate their scopes as needed. In practice, that never really panned out (very few users actually attenuated their credentials). If we were to reimplement PyPI's API tokens today, I'd likely go with a more traditional design (even though all of this is a black box from a normal user's perspective - a Macaroon looks like a normal opaque token, just chunkier).
- cyrnel 2y agoWhy invest so much time and money in a feature that prevents such a small percentage of data breaches that it's not even categorized on the 2024 Verizon Data Breach Investigations Report? The vast majority of breaches are caused by credential theft, phishing, and exploiting vulnerabilities. It doesn't matter that you can cryptographically verify that a package came from a given commit if that commit has accidentally-vulnerable code, or someone just gets phished.
- some_furry 2y agoBecause Verizon's report, while a good read, isn't the end-all-be-all of threat intelligence. https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-crashed-the-world/ https://www.wired.com/story/notpetya-cyberattack-ukraine-rus... https://krebsonsecurity.com/2020/12/u-s-treasury-commerce-depts-hacked-through-solarwinds-compromise/ https://krebsonsecurity.com/2020/12/u-s-treasury-commerce-de... Software supply chain attacks are rare, but when they happen, they're usually high-impact ordeals.
- MattPalmer1086 2y agoCorrect, software supply chain attacks may be rarer than a simple SQL injection attack, but they are much harder to detect and defend against, and can have very high impact. Where I work (highly regulated financial utility), we would absolutely love to have signed attestations for the provenance of python packages. I also have personal experience of NotPetya; I produced the forensic analysis of how it compromised the whole of Maersk. Took less than 30 minutes to take out their entire global IT infrastructure, with a cost of about $400 million.
- pastage 2y agoYou need to know where these attestations were made, you need a TPM telling you that this was done on a patched untouched server, in an patched untouched VM, on a signed attested code base, with deploy scripts unchanged by anyone. We are not there as long as Jenkins and most build infrastructure is basically Remote exploit as a service it is also going to take a long time. While Petya like in Ukraine is harder the more attestations you add there is always a hole, especially when all of your infrastructure needs to run things from some proprietary vendor.
- physicsguy 2y agoPeople don’t have to use GitHub, and certainly don’t have to use GitHub Actions even if they do
- globular-toast 2y ago"How can I trust you?" "I am trusted." It's basically the same model as HTTPS. Not sure if it has a name. "Too big to fail" security? Security by fiat?
- progval 2y agoAccording to this page, urllib3 does not use trusted publishing. According to https://docs.pypi.org/project_metadata/#verified-details https://docs.pypi.org/project_metadata/#verified-details , trusted publishing and self-links are the only ways to have "verified details". However https://pypi.org/project/urllib3/ https://pypi.org/project/urllib3/ shows Changelog/Code/Issue tracker as "Verified details" even though they are not self-links. How come? urllib3 does not have a recent release that could explain https://trailofbits.github.io/are-we-pep740-yet/ https://trailofbits.github.io/are-we-pep740-yet/ lagging behind.
- darkamaul 2y agoThis page only shows if a package has been uploaded with attestations .The verified details (Changelog/Code/Issue tracker) are showing because they do use Trusted Publishing. However, they have not published a new version since the beginning of attestation support in PyPI. That's the meaning of the clock icon right to the package name. Their workflow responsible for publishing new releases [1] has support for attestations. Thus, it will turn green on this page with the next project release. [1] https://github.com/urllib3/urllib3/blob/main/.github/workflows/publish.yml#L94 https://github.com/urllib3/urllib3/blob/main/.github/workflo...
- Tknl 2y agohttps://slsa.dev/ https://slsa.dev/ gives much clearer explanations about the why of this work. Github recently started offering a SaaS sigstore implementation including support for private reps. https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations/using-artifact-attestations-to-establish-provenance-for-builds https://docs.github.com/en/actions/security-for-github-actio... Anyone working on OT should be quickly moving towards this.
- Arch-TK 2y agoSomething this doesn't answer: Can I make my package green without having to compromise my integrity by utilising proprietary git hosting?
- blenderob 2y agoI've got the same question. Anyone knows the answer please?
- darthwalsh 2y agoI read that GitLab was going to be supported too.
- Arch-TK 2y agoBut not self hosted GitLab.
- deleted 2y ago[deleted]