25 ms·

I wonder if Apple’s trick would work for python packages. Pay a few (5? 10? 20?) bucks to become a Pypi developer/sponsor, it would also pay pypi’s operational

I wonder if Apple’s trick would work for python packages. Pay a few (5? 10? 20?) bucks to become a Pypi developer/sponsor, it would also pay pypi’s operational bills. It raises the bar for malicious actors. Additionally, if pypi provides keys to developers, pypi can also revoke certificates for developers making malicious packages. It would need a system which checks package signatures on startup of a python app, or maybe there is some other way to do that. Pip —check or some thing which then runs in pipelines, specifically meant to check for malicious packages each day. To decrease the barrier of entry on pypi, students could identify with their student number. Or pypi could work with a system where you have trusted users and packages and untrusted users. A bit like the blue checkmark, but without the negative connotation.

CameronNemo 3y ago
Universities could host gitlab instances with pypi registries built-in.
miohtama 3y ago
You can always have a trusted community member to waive the payment requirement, so anyone who demostrates genuine effort can have it for free.
- BozeWolf 3y ago
  This. Also enterprises using specific (still untrusted) packages could pay it for the developer. There must be a way to do this. But in true spirit of python the system should be as open as possible.
woodruffw 3y ago
FD: I’ve done some work on PyPI, but I am not an admin and everything below is an independent opinion/understanding. PyPI’s operational costs are, to my understanding, mostly covered: hosting is graciously provided by a sponsor, and the PSF currently funds roles for its develop, administration, and security. More funding is always good and I believe the PyPI admins are looking to enable payments through the newly released “Organizations” feature[1]. Edit: and to make it more clear: payments for Organizations would be principally aimed at corporations and other groups. > Additionally, if pypi provides keys to developers, pypi can also revoke certificates for developers making malicious packages. There are currently plans in progress to allow PyPI users to upload Sigstore[2] signatures for packages. That won’t directly address the spam issue, however — signatures will be opt-in (by necessity, due to the size of the packaging ecosystem), and no codesigning scheme can prevent a spammer from simply assuming a new identity (especially when new identities are “free,” as they normally are.) Separately, revocation itself is a nasty problem for packaging ecosystems to deal with: ecosystems with trillions of dollars of value behind them (like the Web PKI) struggle with it, so it’s not immediately clear that it would be anything other than an additional operational burden. Similarly for reputational systems: they’re difficult to operationalize without additional maintainer burden. That’s not to say that they’re necessarily bad or impractical for PyPI’s purposes; only that I’m not aware of a successful use of them in an open source packaging ecosystem. Compare, for example, PGP’s WoT failures. [1]: https://blog.pypi.org/posts/2023-04-23-introducing-pypi-organizations/ https://blog.pypi.org/posts/2023-04-23-introducing-pypi-orga... [2]: https://www.sigstore.dev/ https://www.sigstore.dev/
- XorNot 3y ago
  Web of Trust failed because it lacked motivating reasons to use it, and had a very weird idea about how trust is represented (i.e. the idea of partial trust keys meaning... What exactly?) What was missing was a role/application based idea of trust chains. i.e. what I trust to do my banking is quite different to what I trust for software reputation. Because ultimately we're operating a web of trust system now on the internet, it's just informally specified.
  - woodruffw 3y ago
    > Web of Trust failed because it lacked motivating reasons to use it, and had a very weird idea about how trust is represented (i.e. the idea of partial trust keys meaning... What exactly?) This is true, but it's also just one among several reasons that the PGP WoT failed. Some are purely implementation and design failures that wouldn't imply to a better scheme than PGP, but others are still intractable in the general case (like timely revocations and "strong set" maintenance). > Because ultimately we're operating a web of trust system now on the internet, it's just informally specified. Could you elaborate on what you mean by this? The trust scheme that the modern web is built on is a well-specified hierarchical PKI, i.e. the exact opposite of a web of trust. But it's possible I'm misunderstanding what you mean.
  - taeric 3y ago
    I don't know that I agree with this framing. Pgp could support multiple keyrings, for one. Bigger, though, is that the internet isn't a web of trust in the same vein? Take out the CA chain, and websites fail trust pretty hard. Even adding them, we wind up accepting certificate pinning on browsers, as nobody has a better option, yet. And that is building trust to web sites. The stick houses we have to trust communication is basically, "you always call us." Edit: forgot to say that partial trust is pretty close to "believed, but I wouldn't bank with this identity."
- BozeWolf 3y ago
  Didn’t know sigstore. The idea looks promising. They should work on marketing though. “Their” technology would work on any package repository, right? Also for npm and the likes.
viraptor 3y ago
That seems like a way to kill pypi as a popular service. For the 3 or so packages I have, I'd probably just change the description to pull from another location rather than pypi. It's not that I can't afford it, it's just that I don't want to end up paying for another subscription if rules change. There are also people who wouldn't be able to pay for legal reasons. It would also stop teenagers who don't need the hassle of getting parents to pay.
- wombatpm 3y ago
  I think you misinterpreted the comment. The Payment would be required for people wanting to PUSH to pypi. Developers and people PULLING from pypi would remain free. The problem is garbage and malicious submissions.
  - chatmasta 3y ago
    I don't think the comment you're replying to misunderstood the comment. I think the author was saying that they would update the readme of their projects (which they _push_ to a registry, and for which they do not want to pay) to instruct people to pull the package from somewhere else.
    - wombatpm 3y ago
      You can do that, but it certainly doesn’t help you with project discovery. If you want only distribute via GitHub that’s fine. But as a developer I’d need a compelling reason to use such a project vs one distributed via pypi
      - viraptor 3y ago
        That's ok. Many projects don't care about discovery. (As in being popular/brand name, rather than being possible to find if someone needs that specific thing) And even those that do can rely on people searching for "python package for frobnicating" and finding it somewhere else. I'm publishing my code for the benefit of others, not mine, so if someone writes the same thing and pays to get it published, that's fine with me. > I’d need a compelling reason to use such a project vs one distributed via pypi Apart from trivial projects, there's not that many alternatives you can swap out. Your choices may be "use such project or write your own".
crabbone 3y ago
Hahaha no. You physically cannot do it. Read PEP-508. Python can install packages from anywhere. Even if you start with PyPI, the dependency can be specified as hosted on a random unrelated resource. And, on principle, I wouldn't pay PyPI anything. I want them gone, I don't want my money to feed a bunch of incompetent people who make my life miserable. So, if they were to implement this idea, I'd be hosting my packages on GitHub or Gitlab etc. and have dependencies link to those sources.
- hluska 3y ago
  You seem pleasant.
CommitSyn 3y ago
As long as devs could be identified properly (ID selfie + ID closeup + small payment with card in same name) and students do the same with a smaller amount, I think that could work. Refund of protection deposit when you close your account. System of honor based on account age. The problem is it's incredibly hard to properly ID people. If fraudsters can get an ID+card they can make a convincing fake selfie ID photo from social media pictures. Plus accounts can just be stolen.
pmeira 3y ago
I maintain a few niche (electric power systems) packages, and I wouldn't mind a one-time or yearly fee, or a fee per project created. I say this as a Brazilian who lived in the middle of nowhere and managed to have a website in the 90's as a teen. If a monetary fee is not desirable, some other hurdle/challenge would probably work fine. Recently I've seen someone on Reddit trying to automate the creation of PyPI projects through GitHub Actions. The person was complaining that the first deployment couldn't use an API key for that project since it didn't exist. So I'm not surprised some people are trying to do the same for malicious purposes. The PyPI front page lists 455k projects. If you search for "test", you'll see there's a lot of throwaway projects (note that test.pypi.org is a thing). I'm mostly an EE researcher and I'm not sure students need a low barrier to entry to PyPI, since pip and other tools support installing from GitHub without too much hassle and there are also other non-PyPI package indices. Student packages/projects tend to be abandoned soon after graduation. An archived repo (with a license...), on GitHub or somewhere else, sounds more reasonable and also has more visibility that could end in code reuse someday (through the service's own search and search engines in general). I'd love to understand why so many people repeat this meme that student and teens need trivial access to production infra like PyPI. So, I'd say being too inclusive, allowing fully unrestricted trivial creation of projects is kinda foolish. There needs to be some extra step, be it a fee, identity confirmation, manual moderation/approval, or something else. I'm sure the PyPA devs/maintainers have ideas.
- woodruffw 3y ago
  > Recently I've seen someone on Reddit trying to automate the creation of PyPI projects through GitHub Actions. The person was complaining that the first deployment couldn't use an API key for that project since it didn't exist. So I'm not surprised some people are trying to do the same for malicious purposes. Sorry for the tangent, but: you can do this now! If you use trusted publishing, you can register a "pending publisher" for a project that doesn't exist yet. When the trusted publisher (like GitHub Actions) is used, it'll create the project[1]. All of this is supported transparently by the official publishing action for GitHub Actions[2]. [1]: https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/ https://docs.pypi.org/trusted-publishers/creating-a-project-... [2]: https://github.com/pypa/gh-action-pypi-publish https://github.com/pypa/gh-action-pypi-publish
NegativeK 3y ago
Start asking maintainers for SBOMs (this is rhetorical; please don't) and you'll find out how disinterested they are in doing even more work for others. They understand that there's a problem, but don't have the will to deal with extra work when they're already short on time. Open source projects tend to be maintained by a tiny number of people who are scratching their own itch, not signing up for more barriers.

CameronNemo 3y ago

Universities could host gitlab instances with pypi registries built-in.

miohtama 3y ago

You can always have a trusted community member to waive the payment requirement, so anyone who demostrates genuine effort can have it for free.

BozeWolf 3y ago
This. Also enterprises using specific (still untrusted) packages could pay it for the developer. There must be a way to do this. But in true spirit of python the system should be as open as possible.

woodruffw 3y ago

FD: I’ve done some work on PyPI, but I am not an admin and everything below is an independent opinion/understanding. PyPI’s operational costs are, to my understanding, mostly covered: hosting is graciously provided by a sponsor, and the PSF currently funds roles for its develop, administration, and security. More funding is always good and I believe the PyPI admins are looking to enable payments through the newly released “Organizations” feature[1]. Edit: and to make it more clear: payments for Organizations would be principally aimed at corporations and other groups. > Additionally, if pypi provides keys to developers, pypi can also revoke certificates for developers making malicious packages. There are currently plans in progress to allow PyPI users to upload Sigstore[2] signatures for packages. That won’t directly address the spam issue, however — signatures will be opt-in (by necessity, due to the size of the packaging ecosystem), and no codesigning scheme can prevent a spammer from simply assuming a new identity (especially when new identities are “free,” as they normally are.) Separately, revocation itself is a nasty problem for packaging ecosystems to deal with: ecosystems with trillions of dollars of value behind them (like the Web PKI) struggle with it, so it’s not immediately clear that it would be anything other than an additional operational burden. Similarly for reputational systems: they’re difficult to operationalize without additional maintainer burden. That’s not to say that they’re necessarily bad or impractical for PyPI’s purposes; only that I’m not aware of a successful use of them in an open source packaging ecosystem. Compare, for example, PGP’s WoT failures. [1]: https://blog.pypi.org/posts/2023-04-23-introducing-pypi-organizations/ https://blog.pypi.org/posts/2023-04-23-introducing-pypi-orga... [2]: https://www.sigstore.dev/ https://www.sigstore.dev/

XorNot 3y ago
Web of Trust failed because it lacked motivating reasons to use it, and had a very weird idea about how trust is represented (i.e. the idea of partial trust keys meaning... What exactly?) What was missing was a role/application based idea of trust chains. i.e. what I trust to do my banking is quite different to what I trust for software reputation. Because ultimately we're operating a web of trust system now on the internet, it's just informally specified.
- woodruffw 3y ago
  > Web of Trust failed because it lacked motivating reasons to use it, and had a very weird idea about how trust is represented (i.e. the idea of partial trust keys meaning... What exactly?) This is true, but it's also just one among several reasons that the PGP WoT failed. Some are purely implementation and design failures that wouldn't imply to a better scheme than PGP, but others are still intractable in the general case (like timely revocations and "strong set" maintenance). > Because ultimately we're operating a web of trust system now on the internet, it's just informally specified. Could you elaborate on what you mean by this? The trust scheme that the modern web is built on is a well-specified hierarchical PKI, i.e. the exact opposite of a web of trust. But it's possible I'm misunderstanding what you mean.
- taeric 3y ago
  I don't know that I agree with this framing. Pgp could support multiple keyrings, for one. Bigger, though, is that the internet isn't a web of trust in the same vein? Take out the CA chain, and websites fail trust pretty hard. Even adding them, we wind up accepting certificate pinning on browsers, as nobody has a better option, yet. And that is building trust to web sites. The stick houses we have to trust communication is basically, "you always call us." Edit: forgot to say that partial trust is pretty close to "believed, but I wouldn't bank with this identity."
BozeWolf 3y ago
Didn’t know sigstore. The idea looks promising. They should work on marketing though. “Their” technology would work on any package repository, right? Also for npm and the likes.

viraptor 3y ago

That seems like a way to kill pypi as a popular service. For the 3 or so packages I have, I'd probably just change the description to pull from another location rather than pypi. It's not that I can't afford it, it's just that I don't want to end up paying for another subscription if rules change. There are also people who wouldn't be able to pay for legal reasons. It would also stop teenagers who don't need the hassle of getting parents to pay.

wombatpm 3y ago
I think you misinterpreted the comment. The Payment would be required for people wanting to PUSH to pypi. Developers and people PULLING from pypi would remain free. The problem is garbage and malicious submissions.
- chatmasta 3y ago
  I don't think the comment you're replying to misunderstood the comment. I think the author was saying that they would update the readme of their projects (which they _push_ to a registry, and for which they do not want to pay) to instruct people to pull the package from somewhere else.
  - wombatpm 3y ago
    You can do that, but it certainly doesn’t help you with project discovery. If you want only distribute via GitHub that’s fine. But as a developer I’d need a compelling reason to use such a project vs one distributed via pypi
    - viraptor 3y ago
      That's ok. Many projects don't care about discovery. (As in being popular/brand name, rather than being possible to find if someone needs that specific thing) And even those that do can rely on people searching for "python package for frobnicating" and finding it somewhere else. I'm publishing my code for the benefit of others, not mine, so if someone writes the same thing and pays to get it published, that's fine with me. > I’d need a compelling reason to use such a project vs one distributed via pypi Apart from trivial projects, there's not that many alternatives you can swap out. Your choices may be "use such project or write your own".

crabbone 3y ago

Hahaha no. You physically cannot do it. Read PEP-508. Python can install packages from anywhere. Even if you start with PyPI, the dependency can be specified as hosted on a random unrelated resource. And, on principle, I wouldn't pay PyPI anything. I want them gone, I don't want my money to feed a bunch of incompetent people who make my life miserable. So, if they were to implement this idea, I'd be hosting my packages on GitHub or Gitlab etc. and have dependencies link to those sources.

hluska 3y ago
You seem pleasant.

CommitSyn 3y ago

As long as devs could be identified properly (ID selfie + ID closeup + small payment with card in same name) and students do the same with a smaller amount, I think that could work. Refund of protection deposit when you close your account. System of honor based on account age. The problem is it's incredibly hard to properly ID people. If fraudsters can get an ID+card they can make a convincing fake selfie ID photo from social media pictures. Plus accounts can just be stolen.

pmeira 3y ago

I maintain a few niche (electric power systems) packages, and I wouldn't mind a one-time or yearly fee, or a fee per project created. I say this as a Brazilian who lived in the middle of nowhere and managed to have a website in the 90's as a teen. If a monetary fee is not desirable, some other hurdle/challenge would probably work fine. Recently I've seen someone on Reddit trying to automate the creation of PyPI projects through GitHub Actions. The person was complaining that the first deployment couldn't use an API key for that project since it didn't exist. So I'm not surprised some people are trying to do the same for malicious purposes. The PyPI front page lists 455k projects. If you search for "test", you'll see there's a lot of throwaway projects (note that test.pypi.org is a thing). I'm mostly an EE researcher and I'm not sure students need a low barrier to entry to PyPI, since pip and other tools support installing from GitHub without too much hassle and there are also other non-PyPI package indices. Student packages/projects tend to be abandoned soon after graduation. An archived repo (with a license...), on GitHub or somewhere else, sounds more reasonable and also has more visibility that could end in code reuse someday (through the service's own search and search engines in general). I'd love to understand why so many people repeat this meme that student and teens need trivial access to production infra like PyPI. So, I'd say being too inclusive, allowing fully unrestricted trivial creation of projects is kinda foolish. There needs to be some extra step, be it a fee, identity confirmation, manual moderation/approval, or something else. I'm sure the PyPA devs/maintainers have ideas.

woodruffw 3y ago
> Recently I've seen someone on Reddit trying to automate the creation of PyPI projects through GitHub Actions. The person was complaining that the first deployment couldn't use an API key for that project since it didn't exist. So I'm not surprised some people are trying to do the same for malicious purposes. Sorry for the tangent, but: you can do this now! If you use trusted publishing, you can register a "pending publisher" for a project that doesn't exist yet. When the trusted publisher (like GitHub Actions) is used, it'll create the project[1]. All of this is supported transparently by the official publishing action for GitHub Actions[2]. [1]: https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/ https://docs.pypi.org/trusted-publishers/creating-a-project-... [2]: https://github.com/pypa/gh-action-pypi-publish https://github.com/pypa/gh-action-pypi-publish

NegativeK 3y ago

Start asking maintainers for SBOMs (this is rhetorical; please don't) and you'll find out how disinterested they are in doing even more work for others. They understand that there's a problem, but don't have the will to deal with extra work when they're already short on time. Open source projects tend to be maintained by a tiny number of people who are scratching their own itch, not signing up for more barriers.