4 ms·

I think maven central has the right idea, but it could be generalized. Require people submitting packages to prove they own a domain and use the domain as the

I think maven central has the right idea, but it could be generalized. Require people submitting packages to prove they own a domain and use the domain as the namespace representing the user. We could then use SRV records or TXT records to specify where a domain’s packages are authoritatively hosted and then have conventions to find a signing key for a given domain that can be used to verify cached packages in non-authoritative hosts.

woodruffw 3y ago
Again: namespacing isn't the problem (or a totalizing solution) here. I admire Maven's approach on a conceptual level, but it leaves a lot to be desired on a technical level: it effectively gatekeeps the entire packaging ecosystem on the ability to purchase and maintain a domain name, which (1) is unnecessarily exclusionary, and (2) assumes trust in a namespace (DNS) that was never really designed to be authenticated or a source of stable, permanent identifiers (domains change hands all the times, including for hostile reasons). A modern version of Maven's scheme would bootstrap on well-known URIs over HTTPS instead, but even this would still represent a significant imposition on PyPI uploaders.
- fiddlerwoaroof 3y ago
  I think the goal is not to have "pypi uploaders" but instead have a discovery system designed to enable safe caching of artifacts and automatic discovery of where an artifact is hosted. The exclusionary aspects can be mitigated via somewhere like github offering package hosting for their users. $10/yr for a domain name also isn't all that much.
  - woodruffw 3y ago
    I don't know how that's panned out for the Java ecosystem, but distributed hosting is something that PyPI moved away from (with a great deal of pain and heartburn) years ago. Previous versions of PyPI were a "pure" index, with distribution hosted being provided by individual projects. This made package installation exactly as reliable and secure as the least reliable and least secure host.
    - fiddlerwoaroof 3y ago
      You can setup a system like DNS, where downstream systems can cache verifiable versions of the packages hosted at the authoritative source to help here. If the lock file for the project records signatures or similar identifiers for the project’s dependencies, the exact source of the bits matters a lot less, reducing the reliance on package hosts remaining up indefinitely.
- blibble 3y ago
  > which (1) is unnecessarily exclusionary meanwhile pypa spent months making it so the only trusted publisher is microsoft
  - woodruffw 3y ago
    This is misleading: the trusted publisher is GitHub's OIDC IdP, which happens to be the single most common publishing source for PyPI (as well as the first IdP to give us all the features we needed). It has absolutely nothing to do with Microsoft; I don't think Microsoft (as a discrete corporate entity) was even aware of the work. The feature was intentionally built to be extensible and to support additional publishers. I know this because I'm the one who made it extensible, and who's working on supporting them[1][2]. [1]: https://github.com/pypi/warehouse/issues/13551 https://github.com/pypi/warehouse/issues/13551 [2]: https://github.com/pypi/warehouse/issues/13575 https://github.com/pypi/warehouse/issues/13575
- ryan29 3y ago
  > Again: namespacing isn't the problem (or a totalizing solution) here. The current system works better for impersonators and squatters than it does for me as a legitimate participant, so, in my opinion, that's at least part of the problem. Domain validated namespaces go a long way towards solving that. > it effectively gatekeeps the entire packaging ecosystem on the ability to purchase and maintain a domain name There's no reason there can't be a default namespace similar to what Bluesky did. By default you get @example.bsky.social and can optionally switch to a custom domain. The main thing that Bluesky got wrong is they recycle your original handle when you switch to a custom domain and I think it should be (optionally) kept as an alias to prevent impersonation. > assumes trust in a namespace (DNS) that was never really designed to be authenticated It's trusted enough for MS365, Google Workspace, ACME (protocol), and basically everything on the internet at this point. PyPI.org is using a domain validated SSL certificate and (AFAIK) distributes unsigned packages. I view that as about the same amount of risk as using domain validation for namespaces. > or a source of stable, permanent identifiers (domains change hands all the times, including for hostile reasons). Domains changing ownership can be handled reasonably well by treating the domain as a mutable pointer to an immutable namespace. The immutable namespace would be the source of truth that would be trusted by clients and would point back to the domain so they reference each other. If either pointer changes the client can warn the user and let them decide how to handle it. By using the immutable namespace after following the mutable (domain) pointer once, you eliminate a lot of the value of stealing a domain to take over a namespace. If you flag domains that have recently started pointing to a different immutable namespace (ie: a new domain or owner), it acts as a cautionary warning so users know they shouldn't be blindly trusting the namespace. Dealing with banned domains would likely be a pain point, but I don't think banning an account changes much in terms of human factors involved. On the domain side of things I'd probably maintain a public list of permanently banned domains, at least to start with, and see what problems that creates before trying to find solutions. Non-negotiable, permanent banning might even be ok. For me personally, domain validated namespaces would be a huge improvement. I'm sick of the gold rush and having to chase my handle on every platform. Even if I get it, I'm still indistinguishable from a bad actor on most platforms because I have very little activity and nothing popular. At least with domain verified namespaces someone who doesn't know me can go from pypi.org/@example.com to github.com/@example.com or example.com and be nearly guaranteed it's me (ignoring domain hijacking). That makes it easier to make a judgment call on whether or not to trust me because most of my activity will show up at, or be linked to from, those places. Plus, as a package becomes more popular, the recognizably of the owner's handle also grows. With a single, domain validated namespace, that's only one definitive thing for people to have to recognize which cuts down on the cognitive load of trying to figure out if mismatched namespaces are legitimate or someone trying to engineer an attack. I think the current system for namespaces is a big chunk of the problem because it's non-authoritative in the context of knowing who owns a namespace. By switching to domain validated namespaces a lot of that ambiguity disappears because you know that example.com, pypi.org/@example.com, github.com/@example.com, twitter.com/@example.com, etc. are all the same person or organization. That means that as soon as you know you can trust them on one of those platforms, you can trust them everywhere else because, as long as the UI flags new domain pointers to combat domain hijacking, you've basically removed impersonation and squatting as a potential threat. So, maybe it's not perfect, but, in my opinion, it adds enough value that it's worth considering as an option. I think it's great on Bluesky because I find it so much easier to determine whether or not people are legitimately who they say they are and not some prankster or impersonator. For example, I'm following @tailscale.com on Bluesky and I know it's them.
- Symbiote 3y ago
  Maven supports multiple package repositories, and it's not unusual to see small packages in a self-hosted repo. That could be a self-hosted HTTP server, S3 or similar, or GitHub's package repo. The hostname of the repository needn't match the name space of the packages it hosts.

woodruffw 3y ago

Again: namespacing isn't the problem (or a totalizing solution) here. I admire Maven's approach on a conceptual level, but it leaves a lot to be desired on a technical level: it effectively gatekeeps the entire packaging ecosystem on the ability to purchase and maintain a domain name, which (1) is unnecessarily exclusionary, and (2) assumes trust in a namespace (DNS) that was never really designed to be authenticated or a source of stable, permanent identifiers (domains change hands all the times, including for hostile reasons). A modern version of Maven's scheme would bootstrap on well-known URIs over HTTPS instead, but even this would still represent a significant imposition on PyPI uploaders.

fiddlerwoaroof 3y ago
I think the goal is not to have "pypi uploaders" but instead have a discovery system designed to enable safe caching of artifacts and automatic discovery of where an artifact is hosted. The exclusionary aspects can be mitigated via somewhere like github offering package hosting for their users. $10/yr for a domain name also isn't all that much.
- woodruffw 3y ago
  I don't know how that's panned out for the Java ecosystem, but distributed hosting is something that PyPI moved away from (with a great deal of pain and heartburn) years ago. Previous versions of PyPI were a "pure" index, with distribution hosted being provided by individual projects. This made package installation exactly as reliable and secure as the least reliable and least secure host.
  - fiddlerwoaroof 3y ago
    You can setup a system like DNS, where downstream systems can cache verifiable versions of the packages hosted at the authoritative source to help here. If the lock file for the project records signatures or similar identifiers for the project’s dependencies, the exact source of the bits matters a lot less, reducing the reliance on package hosts remaining up indefinitely.
blibble 3y ago
> which (1) is unnecessarily exclusionary meanwhile pypa spent months making it so the only trusted publisher is microsoft
- woodruffw 3y ago
  This is misleading: the trusted publisher is GitHub's OIDC IdP, which happens to be the single most common publishing source for PyPI (as well as the first IdP to give us all the features we needed). It has absolutely nothing to do with Microsoft; I don't think Microsoft (as a discrete corporate entity) was even aware of the work. The feature was intentionally built to be extensible and to support additional publishers. I know this because I'm the one who made it extensible, and who's working on supporting them[1][2]. [1]: https://github.com/pypi/warehouse/issues/13551 https://github.com/pypi/warehouse/issues/13551 [2]: https://github.com/pypi/warehouse/issues/13575 https://github.com/pypi/warehouse/issues/13575
ryan29 3y ago
> Again: namespacing isn't the problem (or a totalizing solution) here. The current system works better for impersonators and squatters than it does for me as a legitimate participant, so, in my opinion, that's at least part of the problem. Domain validated namespaces go a long way towards solving that. > it effectively gatekeeps the entire packaging ecosystem on the ability to purchase and maintain a domain name There's no reason there can't be a default namespace similar to what Bluesky did. By default you get @example.bsky.social and can optionally switch to a custom domain. The main thing that Bluesky got wrong is they recycle your original handle when you switch to a custom domain and I think it should be (optionally) kept as an alias to prevent impersonation. > assumes trust in a namespace (DNS) that was never really designed to be authenticated It's trusted enough for MS365, Google Workspace, ACME (protocol), and basically everything on the internet at this point. PyPI.org is using a domain validated SSL certificate and (AFAIK) distributes unsigned packages. I view that as about the same amount of risk as using domain validation for namespaces. > or a source of stable, permanent identifiers (domains change hands all the times, including for hostile reasons). Domains changing ownership can be handled reasonably well by treating the domain as a mutable pointer to an immutable namespace. The immutable namespace would be the source of truth that would be trusted by clients and would point back to the domain so they reference each other. If either pointer changes the client can warn the user and let them decide how to handle it. By using the immutable namespace after following the mutable (domain) pointer once, you eliminate a lot of the value of stealing a domain to take over a namespace. If you flag domains that have recently started pointing to a different immutable namespace (ie: a new domain or owner), it acts as a cautionary warning so users know they shouldn't be blindly trusting the namespace. Dealing with banned domains would likely be a pain point, but I don't think banning an account changes much in terms of human factors involved. On the domain side of things I'd probably maintain a public list of permanently banned domains, at least to start with, and see what problems that creates before trying to find solutions. Non-negotiable, permanent banning might even be ok. For me personally, domain validated namespaces would be a huge improvement. I'm sick of the gold rush and having to chase my handle on every platform. Even if I get it, I'm still indistinguishable from a bad actor on most platforms because I have very little activity and nothing popular. At least with domain verified namespaces someone who doesn't know me can go from pypi.org/@example.com to github.com/@example.com or example.com and be nearly guaranteed it's me (ignoring domain hijacking). That makes it easier to make a judgment call on whether or not to trust me because most of my activity will show up at, or be linked to from, those places. Plus, as a package becomes more popular, the recognizably of the owner's handle also grows. With a single, domain validated namespace, that's only one definitive thing for people to have to recognize which cuts down on the cognitive load of trying to figure out if mismatched namespaces are legitimate or someone trying to engineer an attack. I think the current system for namespaces is a big chunk of the problem because it's non-authoritative in the context of knowing who owns a namespace. By switching to domain validated namespaces a lot of that ambiguity disappears because you know that example.com, pypi.org/@example.com, github.com/@example.com, twitter.com/@example.com, etc. are all the same person or organization. That means that as soon as you know you can trust them on one of those platforms, you can trust them everywhere else because, as long as the UI flags new domain pointers to combat domain hijacking, you've basically removed impersonation and squatting as a potential threat. So, maybe it's not perfect, but, in my opinion, it adds enough value that it's worth considering as an option. I think it's great on Bluesky because I find it so much easier to determine whether or not people are legitimately who they say they are and not some prankster or impersonator. For example, I'm following @tailscale.com on Bluesky and I know it's them.
Symbiote 3y ago
Maven supports multiple package repositories, and it's not unusual to see small packages in a self-hosted repo. That could be a self-hosted HTTP server, S3 or similar, or GitHub's package repo. The hostname of the repository needn't match the name space of the packages it hosts.