4 ms·

The solution is to design package managers around the uniform resource identifier: a way to locate online assets that is mostly (ignoring DNS) decentralised and

The solution is to design package managers around the uniform resource identifier: a way to locate online assets that is mostly (ignoring DNS) decentralised and better than having one org own all the packages.

hosh 1y ago
That sounds like a neat idea. Do you have a proposal for that? Would it be compatible with specifying urls (such as git repos)?
- hiharryhere 1y ago
  Bundler already does this. # From a specific branch gem 'my_gem', git: 'https://github.com/user/my_gem.git', branch: 'development' # From a specific tag gem 'my_gem', git: 'https://github.com/user/my_gem.git', tag: 'v1.2.3' # From a specific commit (ref) gem 'my_gem', git: 'https://github.com/user/my_gem.git', ref: 'a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0'
  - hosh 1y ago
    Yes, I know bundler does that. But I thought we were talking about urn instead of uri. Seems I was mistaken.
cortesoft 1y ago
You can absolutely use bundler and gem without touching the rubygems servers. You can point to an alternate rubygems host (including one you run yourself), point to a git repo, or a local gem file source
rmoriz 1y ago
This resembles the "monolith" vs "micro-services" discussion. If you spread the packages over thousands of domains, hosts, providers, reliability will be horrible. And it's uncontrollable. In theory, RubyGems could run code analyzers on all uploads to detected malware. Good look if you just haven an index of repositories/packages hosted elsewhere.
- nenenejej 1y ago
  Step 2: store a copy of the library in your repo. Sounds nut? We used to do this with .dlls in sourcesafe and was fine. boring.
notatallshaw 1y ago
Taking PyPI as a central place of packages, it is known that their bandwidth bill would be $1.8+M per month (https://dustingram.com/articles/2021/04/14/powering-the-python-package-index-in-2021/ https://dustingram.com/articles/2021/04/14/powering-the-pyth...) were it not for Fastly giving them a 100% discount. Are there any reliable decentralized package distribution systems operating at within 2 orders of magnitude of that scale? How do they handle administrative issues such as malicious packages or name squatting? Standards updates? Enforcement of correct metadata? And all the other common things package indexes need to handle. I'm clearly skeptical, but would be very interested in any real world success stories.
- hellcow 1y ago
  Go does this, and I’d say it’s been highly successful.
- nextaccountic 1y ago
  There is, the web. The web distributes code directly to end users at a much larger scale. To distribute the bandwidth costs, the web is federated: to depend on a script you refer to its url, and whoever hosts this url foots the bill. Deno is a Javascript implementation for the backend that attempts to mimic this pattern (it later introduced a more npm-like centralized repository, but afaik it's optional). Deno is of course less popular than Python, but its url-centered model can really scale imo.
  - notatallshaw 1y ago
    > There is, the web. The web distributes code directly to end users at a much larger scale. To distribute the bandwidth costs, the web is federated: to depend on a script you refer to its url, and whoever hosts this url foots the bill. But the Web is notorious for the problems I listed, you end up with standards around not following standards. It leaves almost all the responsibility on the client tool (browser or whatever) to do validation to stop malicious sites, name squatting, accepting and "fixing" poorly constructed metadata etc. > Deno is a Javascript implementation for the backend that attempts to mimic this pattern (it later introduced a more npm-like centralized repository, but afaik it's optional). Deno is of course less popular than Python, but its url-centered model can really scale imo. I was not familiar with Deno, I've done some shallow reading on this now and it's certainly interesting. I don't know enough about the JavaScript world to make a comment on the pros or cons. But I don't think can work for Python, as transitive dependencies would immediately conflict as soon as dependencies required a different version of the same transitive dependency. And the guarantee of Python packaging is you only have a single version of a library installed in an environment, while it can cause some dependency solver headache, it also solves a lot of problems as it makes it safe to pass around objects.

hosh 1y ago

That sounds like a neat idea. Do you have a proposal for that? Would it be compatible with specifying urls (such as git repos)?

hiharryhere 1y ago
Bundler already does this. # From a specific branch gem 'my_gem', git: 'https://github.com/user/my_gem.git', branch: 'development' # From a specific tag gem 'my_gem', git: 'https://github.com/user/my_gem.git', tag: 'v1.2.3' # From a specific commit (ref) gem 'my_gem', git: 'https://github.com/user/my_gem.git', ref: 'a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0'
- hosh 1y ago
  Yes, I know bundler does that. But I thought we were talking about urn instead of uri. Seems I was mistaken.

cortesoft 1y ago

You can absolutely use bundler and gem without touching the rubygems servers. You can point to an alternate rubygems host (including one you run yourself), point to a git repo, or a local gem file source

rmoriz 1y ago

This resembles the "monolith" vs "micro-services" discussion. If you spread the packages over thousands of domains, hosts, providers, reliability will be horrible. And it's uncontrollable. In theory, RubyGems could run code analyzers on all uploads to detected malware. Good look if you just haven an index of repositories/packages hosted elsewhere.

nenenejej 1y ago
Step 2: store a copy of the library in your repo. Sounds nut? We used to do this with .dlls in sourcesafe and was fine. boring.

notatallshaw 1y ago

Taking PyPI as a central place of packages, it is known that their bandwidth bill would be $1.8+M per month (https://dustingram.com/articles/2021/04/14/powering-the-python-package-index-in-2021/ https://dustingram.com/articles/2021/04/14/powering-the-pyth...) were it not for Fastly giving them a 100% discount. Are there any reliable decentralized package distribution systems operating at within 2 orders of magnitude of that scale? How do they handle administrative issues such as malicious packages or name squatting? Standards updates? Enforcement of correct metadata? And all the other common things package indexes need to handle. I'm clearly skeptical, but would be very interested in any real world success stories.

hellcow 1y ago
Go does this, and I’d say it’s been highly successful.
nextaccountic 1y ago
There is, the web. The web distributes code directly to end users at a much larger scale. To distribute the bandwidth costs, the web is federated: to depend on a script you refer to its url, and whoever hosts this url foots the bill. Deno is a Javascript implementation for the backend that attempts to mimic this pattern (it later introduced a more npm-like centralized repository, but afaik it's optional). Deno is of course less popular than Python, but its url-centered model can really scale imo.
- notatallshaw 1y ago
  > There is, the web. The web distributes code directly to end users at a much larger scale. To distribute the bandwidth costs, the web is federated: to depend on a script you refer to its url, and whoever hosts this url foots the bill. But the Web is notorious for the problems I listed, you end up with standards around not following standards. It leaves almost all the responsibility on the client tool (browser or whatever) to do validation to stop malicious sites, name squatting, accepting and "fixing" poorly constructed metadata etc. > Deno is a Javascript implementation for the backend that attempts to mimic this pattern (it later introduced a more npm-like centralized repository, but afaik it's optional). Deno is of course less popular than Python, but its url-centered model can really scale imo. I was not familiar with Deno, I've done some shallow reading on this now and it's certainly interesting. I don't know enough about the JavaScript world to make a comment on the pros or cons. But I don't think can work for Python, as transitive dependencies would immediately conflict as soon as dependencies required a different version of the same transitive dependency. And the guarantee of Python packaging is you only have a single version of a library installed in an environment, while it can cause some dependency solver headache, it also solves a lot of problems as it makes it safe to pass around objects.