7 ms·

I have a bit of uneasiness about how this is heavily pushing GitHub actions as the correct way to publish to PyPI. I had to check PEP740 to make sure it was not

by belval 2y ago

I have a bit of uneasiness about how this is heavily pushing GitHub actions as the correct way to publish to PyPI. I had to check PEP740 to make sure it was not directly supported by Microsoft. > The generation and publication of attestations happens by default, and no changes are necessary for projects that meet all of these conditions: publish from GitHub Actions; via Trusted Publishing; and use the pypa/gh-action-pypi-publish action to publish. If you then click on "The manual way" it adds a big disclaimer: > STOP! You probably don't need this section; it exists only to provide some internal details about how attestation generation and uploading work. If you're an ordinary user, it is strongly recommended that you use one of the official workflows described above. Where the only official workflow is "Use GitHub Actions". I guess I am an idealist but as a maintainer this falls short of my expectations for the openness of Python and PyPI.

woodruffw 2y ago
> Where the only official workflow is "Use GitHub Actions". The standard behind this (PEP 740) supports anything that can be used with Trusted Publishing[1]. That includes GitLab, Google Cloud, ActiveState, and can include any other OIDC IdP if people make a good case for including it. It's not tied to Microsoft or GitHub in any particular way. The only reason it emphasizes GitHub Actions is because that's where the overwhelming majority of automatic publishing traffic comes from, and because it follows a similar enablement pattern as Trusted Publishing did (where we did GitHub first, followed by GitLab and other providers). [1]: https://docs.pypi.org/trusted-publishers/ https://docs.pypi.org/trusted-publishers/
- silverwind 2y ago
  Why does this need to allowlist CI providers in first place? Why not publish an open interface any CI provider can integrate against?
  - SethMLarson 2y ago
    Because every CI/ID provider has a different set of claims and behaviors that would constitute a "secure" policy for verification. If there was one singular way to do that then we could, but there isn't yet so PyPI needs to onboard providers piecemeal. The work to add a new provider is not massive, the reason there are not tons of providers isn't because the work is hard but rather because people are voting with their feet so Github and Gitlab make sense as initial providers to support.
  - woodruffw 2y ago
    Because the security benefit of Trusted Publishing via OIDC versus normal API tokens is marginal at small scales, in two senses: 1. The primary benefit of Trusted Publishing over a manual API token is knowing that the underlying OIDC IdP has an on-call staff, proper key management and rotation policies, etc. These can be guaranteed for GitHub, GitLab, etc., but they're harder to prove for one-off self-hosted CI setups. For the latter case, the user is no better off than they would be with a manual API token, which is still (and will always be) supported. 2. If the overwhelming majority of traffic comes from a single CI/CD provider, adding more code to support generic OIDC IdPs increases PyPI's attack surface for only marginal user benefit. There also is no "open interface" for PyPI to really use here: this is all built on OIDC, but each OIDC provider needs to have its unique claims mapped to something intelligible by PyPI. That step requires thoughtful, manual, per-IdP consideration to avoid security issues.
    - antononcube 2y ago
      > [...] the user is no better off than they would be with a manual API token, which is still (and will always be) supported. This is good to know. I did not see related statements in of the documents linked to this discussion, though.
      - antononcube 2y ago
        I am not sure why my comment above is downvoted -- if you know where the perpetual optionality of digital attestations is officially stated, please, provide a link.
    - freeone3000 2y ago
      I think I would be better off with API key + PGP than API key alone. And that’s being phased out?
      - woodruffw 2y ago
        You can no longer upload a PGP signature to PyPI, if that's what you mean. That was phased out last year (to virtually no complaint since nobody was actually verifying any of the signatures, much less attempting to confirm that their keys were discoverable[1]). [1]: https://blog.yossarian.net/2023/05/21/PGP-signatures-on-PyPI-worse-than-useless https://blog.yossarian.net/2023/05/21/PGP-signatures-on-PyPI...
- belval 2y ago
  I get that, that's why I didn't go "This is Embrace Extend Extinguish", but as constructive feedback I would recommend softening the language and to replace: > STOP! You probably don't need this section; In https://docs.pypi.org/attestations/producing-attestations/#the-manual-way https://docs.pypi.org/attestations/producing-attestations/#t... Perhaps also add a few of the providers you listed as well? > The only reason it emphasizes GitHub Actions is because that's where the overwhelming majority of automatic publishing traffic comes from GitHub being popular is a self-reinforcing process, if GitHub is your first class citizen for something as crucial as trusted publishing then projects on GitHub will see a higher adoption and become the de-facto "secure choice".
  - woodruffw 2y ago
    > but as constructive feedback I would recommend softening the language and to replace: I can soften it, but I think you're reading it excessively negatively: that warning is there to make sure people don't try to do the fiddly, error-prone cryptographic bits if they don't need to. It's a numerical fact that most project owners don't need that section, since most are either using manual API tokens or are publishing via GitHub Actions. > Perhaps also add a few of the providers you listed as well? They'll be added when they're enabled. Like I said in the original comment, we're using a similar enablement pattern as happened with Trusted Publishing: GitHub was enabled first because it represents the majority of publishing traffic, followed by GitLab and the others. > GitHub being popular is a self-reinforcing process, if GitHub is your first class citizen for something as crucial as trusted publishing then projects on GitHub will see a higher adoption and become the de-facto "secure choice". I agree, but I don't think this is PyPI's problem to solve. From a security perspective, PyPI should prioritize the platforms where the traffic is. (I'll note that GitLab has been supported by Trusted Publishing for a while now, and they could make the publishing workflow more of a first class citizen, the way it is on GHA.)
    - belval 2y ago
      > I agree, but I don't think this is PyPI's problem to solve. From a security perspective, PyPI should prioritize the platforms where the traffic is. To me that's a bit of a weird statement, PyPI is part of the Python foundation, making sure that the project remains true to its open-source nature is reasonable? My concern is that these type of things ultimately play out as "we are doing the right thing to limit supply chain attacks" which is good an defendable, but in ~5 years PyPI will have an announcement that they are sunsetting PyPI package upload in favor of the trusted provider system. pip (or other tooling) will add warnings whenever I install a package that is not "trusted". Maybe I am simply pessimistic. That being said we can agree to disagree, I am not part of the PSF and I did preface my first comment with "I guess I am an idealist".
- benatkin 2y ago
  That's good. IIRC the feature was launched with GitHub as the only option, which is the same state of affairs of https://jsr.io/ https://jsr.io/ right now. IMO the launch should have been delayed until there was more than one trusted publisher.
- freeone3000 2y ago
  It requires an OIDC IdP, though… with PGP, I can verify that my identity is constant, but now, I’m reliant on some chosen third-party. And it has to keep being the same one, to boot! I don’t like the lock-in this causes, and I definitely don’t like the possibility of my right to publish being revoked by a third party.
  - beng-nl 2y ago
    I’m only now learning about what OIDC IdP is (for those like me: openid connect identity provider). But from my reading, a self hosted gitlab can function as an oidc idp. That would be enough control, right?
    - chippiewill 2y ago
      You can't use a self-hosted Gitlab because you can only use a "trusted publisher". There's no hard technical reason for that. It's mostly that PyPI only want to trust certain issuers who they think will look after their signing keys responsibly.
      - intelVISA 2y ago
        Sounds like PyPI have been corrupted? Hopefully The Foundation can remind them of their mission.
        guappa 2y ago
        Good luck with that…
      - woodruffw 2y ago
        There is a technical reason for it, and it’s explained in an adjacent thread. Accepting every single small-scale IdP would result in a strictly worse security posture for PyPI as a whole, with no actual benefit to small instances (who are better off provisioning API tokens the normal way instead of using Trusted Publishing).
- guappa 2y ago
  Except that also for trusted publishing, they only allowed github in the beginning and eventually added a couple of other providers. But if you're not google or microsoft you won't be added.
  - woodruffw 2y ago
    These kinds of comments are borderline mendacious: you can observe, trivially, that 50% of the Trusted Publishers currently known to PyPI are neither Google nor Microsoft controlled[1]. If PyPI accepts two more likely ones, a full 2/3rds will unrelated to GitHub. [1]: https://docs.pypi.org/trusted-publishers/adding-a-publisher/ https://docs.pypi.org/trusted-publishers/adding-a-publisher/
    - guappa 2y ago
      Ping me when one of them will be an open source entity rather than a company.
      - woodruffw 2y ago
        https://docs.pypi.org/trusted-publishers/internals/#how-do-i-become-a-trusted-publishing-provider https://docs.pypi.org/trusted-publishers/internals/#how-do-i...
    - cpburns2009 2y ago
      Wow, you can use a whole two other providers from your list: Gitlab and ActiveState. Color me unimpressed.
    - immibis 2y ago
      Wow. I get to choose one from a total of FOUR large corporations! Amazing openness!
      - woodruffw 2y ago
        Once again: this is constrained by design. If you don’t want to use OpenID Connect, just create a token on PyPI and publish the normal way. You are not, and will never be, required to use this feature.
- zahlman 2y ago
  "Ordinary user" here. It comes across like the goal of this system is to prove to my users a) who I am; b) that I am working in cooperation with some legitimate big business. I don't understand why I should want to prove such things, nor why it's desirable for b) to even be true, nor why my users should particularly care about a) at all, whether I can prove it or not. I think my users should only care that the actual contents of the sdist match, and the built contents of the wheel correspond to, the source available at the location described in the README. Yes, byte-for-byte reproducible builds would be a problem, for those who don't develop pure Python. Part of why sdists exist is so that people who can't trust the wheel, like distro maintainers, can build things themselves. And really - why would a distro maintainer take I take "I can cryptographically prove the source came from zahlman, and trust me, this wheel corresponds to the source" from $big_company any more seriously than "trust me, this wheel corresponds to the source" directly from me? If I have to route my code through one of these companies ("Don't worry, you can work with Google instead!" is not a good response to people who don't want to work with Microsoft) to gain a compliance checkmark, and figure out how someone else's CI system works (for many years it was perfectly possible for me to write thousands of lines of code and never even have an awareness that there is such a thing as CI) so that I can actually use the publishing system, and learn what things like "OIDC IdP" are so that I can talk to the people in charge... ... then I might as well learn the internal details of how it works. And, in fact, if those people are suggesting that I shouldn't worry about such details, I count that as a red flag. I thought open source was about transparency.
  - solatic 2y ago
    There's two alternate visions: anybody can work with anybody they trust, and Only BigCo Can Protect You. We know the first falls apart because the Web of Trust fell apart. When the UI asks people "hey, this is new, here's the fingerprint, do you trust it?", nobody actually conducts review, they just mindlessly click "yes" and move on. Very, very, very few people will actually conduct full review of all changes. Even massive organizations that want to lie to themselves that they review the changes, end up just outsourcing the responsibility to someone who is willing to legally accept the risk then they go ahead and just mindlessly click "yes" anyway. There are just too many changes to keep track of and too many ways to obscure malicious code. We know the second falls apart because human beings are both the inevitable and ultimate weakest link in even the robustest security architectures. Human beings will be compromised and the blast radius on these highly centralized architectures will be immense. Pick your poison.
didntcheck 2y ago
I might be reading this wrong but doesn't it include four workflows, including Gitlab? The "manual way" section is grouped under the Github tab, which indicates to me that it's only explaining how the Github-specific workflow works? https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/ https://docs.pypi.org/trusted-publishers/creating-a-project-...
- belval 2y ago
  I was referring to the "Get Started" section that links to a producing attestation page that can be found here: https://docs.pypi.org/attestations/producing-attestations/#the-manual-way https://docs.pypi.org/attestations/producing-attestations/#t.... > Support for automatic attestation generation and publication from other Trusted Publisher environments is planned. While not recommended, maintainers can also manually generate and publish attestations.
- woodruffw 2y ago
  You're on a slightly different page: Trusted Publishing supports four different providers, but this initial enablement of attestations (which are built on top of Trusted Publishing) is for GitHub only. The plan is to enable the others in short order.
hifromwork 2y ago
Even more, the previous way was to use GPG signatures, which were recently deprecated and removed. So you don't really have a choice. >Where the only official workflow is "Use GitHub Actions". Well you can do it manually with other solutions... as long as they are one of the four trusted publishers (see "Producing attestations manually does not bypass (...) restrictions on (...) Trusted Publishers": https://docs.pypi.org/trusted-publishers/adding-a-publisher/#gitlab-cicd https://docs.pypi.org/trusted-publishers/adding-a-publisher/... This means that you literally can't do it manually, you have to rely on one of: * Github * Google Cloud * ActiveState (I'm not familiar with it) * Github.com (not just github, only that one instance) Really surprising development, IMO.
- burnt-resistor 2y ago
  It looks a lot like reinventing the wheel, but as an octagon.
ryanisnan 2y ago
I think you're being overly critical. When it says that it adds support for Trusted Publishers, it links directly to this page: https://docs.pypi.org/trusted-publishers/ https://docs.pypi.org/trusted-publishers/. This page clearly explains how this uses OIDC, and uses GitHub Actions as an example. At no point in my read did I feel like this was shilling me a microsoft product.
- guappa 2y ago
  Try to run your own gitlab instance and use this to publish. See what happens… At this moment this only supports github.
doctorpangloss 2y ago
On the one hand, you are totally right, GitHub Actions are the VS Code of automation. People choose them because they are broke, they work well enough, and the average person chooses things based on how it looks. GitHub Actions looks easy, it's cozy, it's comfy, VS Code looks like a code editor, everything has to be cozy and comfy. On the other hand, considering all of that, you can see why Python would arrive at this design. They are the only other people besides NPM who regularly have supply chain attack problems. They are seemingly completely unopinionated about how to fix their supply chain problems, while being opinionated about the lack of opinions about packaging in general. What is the end goal? Presumably one of the giant media companies, Meta, Google or Microsoft maybe, have to take over the development of runtime and PEP process, but does that even sound good?
- LtWorf 2y ago
  I think the end goal is to only allow github/google whatever accounts to publish code on Pypi, so importing whatever will be USA-sanctions safe. The burden to ban russians/koreans/iranians will be on those big companies and pypi will be able to claim they respect the rules without having the resources themselves to ban accounts from sanctioned countries. So in my opinion, in the future tooling will have some option to check if the software was uploaded via microsoft et al and can be considered unsanctioned in USA. Or they might just say that's how you upload now and that's it.
  - doctorpangloss 2y ago
    > so importing whatever will be USA-sanctions safe... You are talking about guys who can barely figure out how to structure a Python package repository. They are not doing this kind of 4D chess. And anyway, none of that makes any sense.
    - LtWorf 2y ago
      The people paying for this, not them.
  - woodruffw 2y ago
LtWorf 2y ago
Funny how I have a talk about the evolution of pypi's security at the upcoming minidebconf this weekend. I guess I'll have to update my slides :D But well as a debian developer my advice is to just use debian and completely ignore pypi, so I might be slightly biased.
- mistrial9 2y ago
  except that is impractical at best. The Debian (and Ubuntu) python packaging groups have been overwhelmed since Python2.7. no way at all does the Debian packaging ecosystem reasonably satisfy python packages.. And there is more.. the Debian system does what it must do -- ensure system Python has integrity with major software stacks and the OS use of Python. Super! that is not at all the needs of users of python. As you know, Python has passed Javascript as the most used language on Github. There is tremendous pace and breadth to several important Python uses, with their libraries. Please reconsider this position with Debian and python packaging.
  - LtWorf 2y ago
    > Please reconsider this position with Debian and python packaging. Just FYI, I'm not sending a swat team to your home to force you to follow my advice.
- zahlman 2y ago
  Does a .deb file even allow for installing for an arbitrary Python (i.e. not the system installation)? Am I supposed to find them through Apt and hope the Debian version of the package has a `python3` prefix on the name? Is that going to help if there are dependencies outside the Apt system?
ikiris 2y ago
I think y’all are being overly aggressive against what is from an outside perspective looks like a mvp targeting the majority use case. If the dev’s lower in the comments claim that other support is soon to be added doesn’t pan out then sure this is a valid argument, but it feels very premature and ignores the reality of launch expectations of most agile based teams.
immibis 2y ago
And the in-detail explanation is full of references to specific third-party services, like Fulcio and SigStore...
HdS84 2y ago
While I understand that, are more deep problem is that their main competitor (gut lab) is too busy to implement the fad of the day. Got labs cicd is already an unholy mess of scripting in yaml, and there is no good way to provide encapsulated pieces of code to gitlab. GitHub has actions which have a defined input and output, gitlab has nothing comparable. That makes it very difficult to provide functionality for gitlab.
- riffraff 2y ago
  Is it really that hard to provide a gitlab workflow? AFAIU it'd be enough to have a docker image and use that in a standalone workflow which is a few lines of yaml in both GitHub and gitlab, what'd be the problem with that?
- myworkinisgood 2y ago
  CI/CD components and steps.

by belval 2y ago

woodruffw 2y ago

> Where the only official workflow is "Use GitHub Actions". The standard behind this (PEP 740) supports anything that can be used with Trusted Publishing[1]. That includes GitLab, Google Cloud, ActiveState, and can include any other OIDC IdP if people make a good case for including it. It's not tied to Microsoft or GitHub in any particular way. The only reason it emphasizes GitHub Actions is because that's where the overwhelming majority of automatic publishing traffic comes from, and because it follows a similar enablement pattern as Trusted Publishing did (where we did GitHub first, followed by GitLab and other providers). [1]: https://docs.pypi.org/trusted-publishers/ https://docs.pypi.org/trusted-publishers/

silverwind 2y ago
Why does this need to allowlist CI providers in first place? Why not publish an open interface any CI provider can integrate against?
- SethMLarson 2y ago
  Because every CI/ID provider has a different set of claims and behaviors that would constitute a "secure" policy for verification. If there was one singular way to do that then we could, but there isn't yet so PyPI needs to onboard providers piecemeal. The work to add a new provider is not massive, the reason there are not tons of providers isn't because the work is hard but rather because people are voting with their feet so Github and Gitlab make sense as initial providers to support.
- woodruffw 2y ago
  Because the security benefit of Trusted Publishing via OIDC versus normal API tokens is marginal at small scales, in two senses: 1. The primary benefit of Trusted Publishing over a manual API token is knowing that the underlying OIDC IdP has an on-call staff, proper key management and rotation policies, etc. These can be guaranteed for GitHub, GitLab, etc., but they're harder to prove for one-off self-hosted CI setups. For the latter case, the user is no better off than they would be with a manual API token, which is still (and will always be) supported. 2. If the overwhelming majority of traffic comes from a single CI/CD provider, adding more code to support generic OIDC IdPs increases PyPI's attack surface for only marginal user benefit. There also is no "open interface" for PyPI to really use here: this is all built on OIDC, but each OIDC provider needs to have its unique claims mapped to something intelligible by PyPI. That step requires thoughtful, manual, per-IdP consideration to avoid security issues.
  - antononcube 2y ago
    > [...] the user is no better off than they would be with a manual API token, which is still (and will always be) supported. This is good to know. I did not see related statements in of the documents linked to this discussion, though.
    - antononcube 2y ago
      I am not sure why my comment above is downvoted -- if you know where the perpetual optionality of digital attestations is officially stated, please, provide a link.
  - freeone3000 2y ago
    I think I would be better off with API key + PGP than API key alone. And that’s being phased out?
    - woodruffw 2y ago
      You can no longer upload a PGP signature to PyPI, if that's what you mean. That was phased out last year (to virtually no complaint since nobody was actually verifying any of the signatures, much less attempting to confirm that their keys were discoverable[1]). [1]: https://blog.yossarian.net/2023/05/21/PGP-signatures-on-PyPI-worse-than-useless https://blog.yossarian.net/2023/05/21/PGP-signatures-on-PyPI...
belval 2y ago
I get that, that's why I didn't go "This is Embrace Extend Extinguish", but as constructive feedback I would recommend softening the language and to replace: > STOP! You probably don't need this section; In https://docs.pypi.org/attestations/producing-attestations/#the-manual-way https://docs.pypi.org/attestations/producing-attestations/#t... Perhaps also add a few of the providers you listed as well? > The only reason it emphasizes GitHub Actions is because that's where the overwhelming majority of automatic publishing traffic comes from GitHub being popular is a self-reinforcing process, if GitHub is your first class citizen for something as crucial as trusted publishing then projects on GitHub will see a higher adoption and become the de-facto "secure choice".
- woodruffw 2y ago
  > but as constructive feedback I would recommend softening the language and to replace: I can soften it, but I think you're reading it excessively negatively: that warning is there to make sure people don't try to do the fiddly, error-prone cryptographic bits if they don't need to. It's a numerical fact that most project owners don't need that section, since most are either using manual API tokens or are publishing via GitHub Actions. > Perhaps also add a few of the providers you listed as well? They'll be added when they're enabled. Like I said in the original comment, we're using a similar enablement pattern as happened with Trusted Publishing: GitHub was enabled first because it represents the majority of publishing traffic, followed by GitLab and the others. > GitHub being popular is a self-reinforcing process, if GitHub is your first class citizen for something as crucial as trusted publishing then projects on GitHub will see a higher adoption and become the de-facto "secure choice". I agree, but I don't think this is PyPI's problem to solve. From a security perspective, PyPI should prioritize the platforms where the traffic is. (I'll note that GitLab has been supported by Trusted Publishing for a while now, and they could make the publishing workflow more of a first class citizen, the way it is on GHA.)
  - belval 2y ago
    > I agree, but I don't think this is PyPI's problem to solve. From a security perspective, PyPI should prioritize the platforms where the traffic is. To me that's a bit of a weird statement, PyPI is part of the Python foundation, making sure that the project remains true to its open-source nature is reasonable? My concern is that these type of things ultimately play out as "we are doing the right thing to limit supply chain attacks" which is good an defendable, but in ~5 years PyPI will have an announcement that they are sunsetting PyPI package upload in favor of the trusted provider system. pip (or other tooling) will add warnings whenever I install a package that is not "trusted". Maybe I am simply pessimistic. That being said we can agree to disagree, I am not part of the PSF and I did preface my first comment with "I guess I am an idealist".
benatkin 2y ago
That's good. IIRC the feature was launched with GitHub as the only option, which is the same state of affairs of https://jsr.io/ https://jsr.io/ right now. IMO the launch should have been delayed until there was more than one trusted publisher.
freeone3000 2y ago
It requires an OIDC IdP, though… with PGP, I can verify that my identity is constant, but now, I’m reliant on some chosen third-party. And it has to keep being the same one, to boot! I don’t like the lock-in this causes, and I definitely don’t like the possibility of my right to publish being revoked by a third party.
- beng-nl 2y ago
  I’m only now learning about what OIDC IdP is (for those like me: openid connect identity provider). But from my reading, a self hosted gitlab can function as an oidc idp. That would be enough control, right?
  - chippiewill 2y ago
    You can't use a self-hosted Gitlab because you can only use a "trusted publisher". There's no hard technical reason for that. It's mostly that PyPI only want to trust certain issuers who they think will look after their signing keys responsibly.
    - intelVISA 2y ago
      Sounds like PyPI have been corrupted? Hopefully The Foundation can remind them of their mission.
      - guappa 2y ago
        Good luck with that…
    - woodruffw 2y ago
      There is a technical reason for it, and it’s explained in an adjacent thread. Accepting every single small-scale IdP would result in a strictly worse security posture for PyPI as a whole, with no actual benefit to small instances (who are better off provisioning API tokens the normal way instead of using Trusted Publishing).
guappa 2y ago
Except that also for trusted publishing, they only allowed github in the beginning and eventually added a couple of other providers. But if you're not google or microsoft you won't be added.
- woodruffw 2y ago
  These kinds of comments are borderline mendacious: you can observe, trivially, that 50% of the Trusted Publishers currently known to PyPI are neither Google nor Microsoft controlled[1]. If PyPI accepts two more likely ones, a full 2/3rds will unrelated to GitHub. [1]: https://docs.pypi.org/trusted-publishers/adding-a-publisher/ https://docs.pypi.org/trusted-publishers/adding-a-publisher/
  - guappa 2y ago
    Ping me when one of them will be an open source entity rather than a company.
    - woodruffw 2y ago
      https://docs.pypi.org/trusted-publishers/internals/#how-do-i-become-a-trusted-publishing-provider https://docs.pypi.org/trusted-publishers/internals/#how-do-i...
  - cpburns2009 2y ago
    Wow, you can use a whole two other providers from your list: Gitlab and ActiveState. Color me unimpressed.
  - immibis 2y ago
    Wow. I get to choose one from a total of FOUR large corporations! Amazing openness!
    - woodruffw 2y ago
      Once again: this is constrained by design. If you don’t want to use OpenID Connect, just create a token on PyPI and publish the normal way. You are not, and will never be, required to use this feature.
zahlman 2y ago
"Ordinary user" here. It comes across like the goal of this system is to prove to my users a) who I am; b) that I am working in cooperation with some legitimate big business. I don't understand why I should want to prove such things, nor why it's desirable for b) to even be true, nor why my users should particularly care about a) at all, whether I can prove it or not. I think my users should only care that the actual contents of the sdist match, and the built contents of the wheel correspond to, the source available at the location described in the README. Yes, byte-for-byte reproducible builds would be a problem, for those who don't develop pure Python. Part of why sdists exist is so that people who can't trust the wheel, like distro maintainers, can build things themselves. And really - why would a distro maintainer take I take "I can cryptographically prove the source came from zahlman, and trust me, this wheel corresponds to the source" from $big_company any more seriously than "trust me, this wheel corresponds to the source" directly from me? If I have to route my code through one of these companies ("Don't worry, you can work with Google instead!" is not a good response to people who don't want to work with Microsoft) to gain a compliance checkmark, and figure out how someone else's CI system works (for many years it was perfectly possible for me to write thousands of lines of code and never even have an awareness that there is such a thing as CI) so that I can actually use the publishing system, and learn what things like "OIDC IdP" are so that I can talk to the people in charge... ... then I might as well learn the internal details of how it works. And, in fact, if those people are suggesting that I shouldn't worry about such details, I count that as a red flag. I thought open source was about transparency.
- solatic 2y ago
  There's two alternate visions: anybody can work with anybody they trust, and Only BigCo Can Protect You. We know the first falls apart because the Web of Trust fell apart. When the UI asks people "hey, this is new, here's the fingerprint, do you trust it?", nobody actually conducts review, they just mindlessly click "yes" and move on. Very, very, very few people will actually conduct full review of all changes. Even massive organizations that want to lie to themselves that they review the changes, end up just outsourcing the responsibility to someone who is willing to legally accept the risk then they go ahead and just mindlessly click "yes" anyway. There are just too many changes to keep track of and too many ways to obscure malicious code. We know the second falls apart because human beings are both the inevitable and ultimate weakest link in even the robustest security architectures. Human beings will be compromised and the blast radius on these highly centralized architectures will be immense. Pick your poison.

didntcheck 2y ago

I might be reading this wrong but doesn't it include four workflows, including Gitlab? The "manual way" section is grouped under the Github tab, which indicates to me that it's only explaining how the Github-specific workflow works? https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/ https://docs.pypi.org/trusted-publishers/creating-a-project-...

belval 2y ago
I was referring to the "Get Started" section that links to a producing attestation page that can be found here: https://docs.pypi.org/attestations/producing-attestations/#the-manual-way https://docs.pypi.org/attestations/producing-attestations/#t.... > Support for automatic attestation generation and publication from other Trusted Publisher environments is planned. While not recommended, maintainers can also manually generate and publish attestations.
woodruffw 2y ago
You're on a slightly different page: Trusted Publishing supports four different providers, but this initial enablement of attestations (which are built on top of Trusted Publishing) is for GitHub only. The plan is to enable the others in short order.

hifromwork 2y ago

Even more, the previous way was to use GPG signatures, which were recently deprecated and removed. So you don't really have a choice. >Where the only official workflow is "Use GitHub Actions". Well you can do it manually with other solutions... as long as they are one of the four trusted publishers (see "Producing attestations manually does not bypass (...) restrictions on (...) Trusted Publishers": https://docs.pypi.org/trusted-publishers/adding-a-publisher/#gitlab-cicd https://docs.pypi.org/trusted-publishers/adding-a-publisher/... This means that you literally can't do it manually, you have to rely on one of: * Github * Google Cloud * ActiveState (I'm not familiar with it) * Github.com (not just github, only that one instance) Really surprising development, IMO.

burnt-resistor 2y ago
It looks a lot like reinventing the wheel, but as an octagon.

ryanisnan 2y ago

I think you're being overly critical. When it says that it adds support for Trusted Publishers, it links directly to this page: https://docs.pypi.org/trusted-publishers/ https://docs.pypi.org/trusted-publishers/. This page clearly explains how this uses OIDC, and uses GitHub Actions as an example. At no point in my read did I feel like this was shilling me a microsoft product.

guappa 2y ago
Try to run your own gitlab instance and use this to publish. See what happens… At this moment this only supports github.

doctorpangloss 2y ago

On the one hand, you are totally right, GitHub Actions are the VS Code of automation. People choose them because they are broke, they work well enough, and the average person chooses things based on how it looks. GitHub Actions looks easy, it's cozy, it's comfy, VS Code looks like a code editor, everything has to be cozy and comfy. On the other hand, considering all of that, you can see why Python would arrive at this design. They are the only other people besides NPM who regularly have supply chain attack problems. They are seemingly completely unopinionated about how to fix their supply chain problems, while being opinionated about the lack of opinions about packaging in general. What is the end goal? Presumably one of the giant media companies, Meta, Google or Microsoft maybe, have to take over the development of runtime and PEP process, but does that even sound good?

LtWorf 2y ago
I think the end goal is to only allow github/google whatever accounts to publish code on Pypi, so importing whatever will be USA-sanctions safe. The burden to ban russians/koreans/iranians will be on those big companies and pypi will be able to claim they respect the rules without having the resources themselves to ban accounts from sanctioned countries. So in my opinion, in the future tooling will have some option to check if the software was uploaded via microsoft et al and can be considered unsanctioned in USA. Or they might just say that's how you upload now and that's it.
- doctorpangloss 2y ago
  > so importing whatever will be USA-sanctions safe... You are talking about guys who can barely figure out how to structure a Python package repository. They are not doing this kind of 4D chess. And anyway, none of that makes any sense.
  - LtWorf 2y ago
    The people paying for this, not them.
- woodruffw 2y ago

LtWorf 2y ago

Funny how I have a talk about the evolution of pypi's security at the upcoming minidebconf this weekend. I guess I'll have to update my slides :D But well as a debian developer my advice is to just use debian and completely ignore pypi, so I might be slightly biased.

mistrial9 2y ago
except that is impractical at best. The Debian (and Ubuntu) python packaging groups have been overwhelmed since Python2.7. no way at all does the Debian packaging ecosystem reasonably satisfy python packages.. And there is more.. the Debian system does what it must do -- ensure system Python has integrity with major software stacks and the OS use of Python. Super! that is not at all the needs of users of python. As you know, Python has passed Javascript as the most used language on Github. There is tremendous pace and breadth to several important Python uses, with their libraries. Please reconsider this position with Debian and python packaging.
- LtWorf 2y ago
  > Please reconsider this position with Debian and python packaging. Just FYI, I'm not sending a swat team to your home to force you to follow my advice.
zahlman 2y ago
Does a .deb file even allow for installing for an arbitrary Python (i.e. not the system installation)? Am I supposed to find them through Apt and hope the Debian version of the package has a `python3` prefix on the name? Is that going to help if there are dependencies outside the Apt system?

ikiris 2y ago

I think y’all are being overly aggressive against what is from an outside perspective looks like a mvp targeting the majority use case. If the dev’s lower in the comments claim that other support is soon to be added doesn’t pan out then sure this is a valid argument, but it feels very premature and ignores the reality of launch expectations of most agile based teams.

immibis 2y ago

And the in-detail explanation is full of references to specific third-party services, like Fulcio and SigStore...

HdS84 2y ago

While I understand that, are more deep problem is that their main competitor (gut lab) is too busy to implement the fad of the day. Got labs cicd is already an unholy mess of scripting in yaml, and there is no good way to provide encapsulated pieces of code to gitlab. GitHub has actions which have a defined input and output, gitlab has nothing comparable. That makes it very difficult to provide functionality for gitlab.

riffraff 2y ago
Is it really that hard to provide a gitlab workflow? AFAIU it'd be enough to have a docker image and use that in a standalone workflow which is a few lines of yaml in both GitHub and gitlab, what'd be the problem with that?
myworkinisgood 2y ago
CI/CD components and steps.