5 ms·

I think true innovation here would be forgoing a centralised package registry and just using good old file systems. Go modules have the the right idea, pull a p

I think true innovation here would be forgoing a centralised package registry and just using good old file systems. Go modules have the the right idea, pull a package from a server which responds to basic HTTP requests or even a file system. If you want some smart searching capability, or generated docs then create a proxy which packages are downloaded through which allows you to index them. Just take things back to basics. You shouldn't have to publish a package on some centralised registry, you should just be able to import a package from anywhere.

davexunit 2y ago
Go programs are a nightmare to package for linux distros due to the way they handle dependencies so I would strongly discourage any other language from copying the Go model.
- jitl 2y ago
  This says more about Linux distros creaky 90s era design than it says about Go’s model.
  - crabbone 2y ago
    Not really. With all the problems of packaging software for Linux, I'll take that any time over Go. The whole "spirit" of how Go authors approached every infrastructure task they had was to start as simple as possible and grow as necessary, in small increments. Unfortunately, incrementalism doesn't work in this domain. It's very hard / virtually impossible to go back and undo bad decisions once they become public. Some steps will require sweeping changes all across the board and cannot be performed in one increment. Go packaging mostly works because of the extra restrictions imposed on themselves by most package authors. But, if you were to stray off the beaten path, you'll discover that the system is unprepared to deal with your case. Go is the opposite of thoughtful and insightful design. It wins at first because its fast and easy to do, and once you are hooked, you will have to work extra hard to deal with the difficult parts yourself.
    - jitl 2y ago
      I guess I view Debian et al similarly to how you view Go: distros like Debian have been incrementally iterating on a simple design since the 90s and their package systems have become a Katamari Damacy ball of mud around a core designed for having a single agreed-upon version of each C library shared by all software in the universe. Now to reliably distribute Linux software, it’s very popular to distribute the software along with the entire userspace the software compiled against in the form of a Docker container.
      - crabbone 2y ago
        > it’s very popular to distribute the software along with the entire userspace OK, there are two ways to interpret the word "popular". * As in "a presidential candidate enjoyed popular support". * As in "ski is a popular sport in some European countries". So, let me tell you this: distributing software with the entire userspace is not popular, if you use the first interpretation of the word. Users hate developers who distribute software like this, this is what they call "bloatware", "not a team-player" etc. I, personally, despise developers who do this, because, from my perspective, these are the low-skill developers who do less to get equal pay at my expense (as a user). But, of course, it's popular to be a bad developer, in the same way how it's popular to steal bicycles -- low effort, high reward, if you ignore the disappointment of someone else.
ManBeardPc 2y ago
Bazel also allows to pull dependencies from different sources via Bzlmod. 1. define dependency like bazel_dep(name = "protobuf", version = "3.19.0") 2. define repositories where to look for it A repository is just a file structure like /modules/$MODULE/$VERSION + some info about the module. Can be on a HTTP server or just on your local file system. https://bazel.build/external/overview#bzlmod https://bazel.build/external/overview#bzlmod I hope more tools adopt something similar. Or maybe a single package manager for everything, so we can finally build cross-language software without having to debug why tool A can't find stuff from B.
- fire_lake 2y ago
  Bazel mod requires a registry of packages in a git repo, so it’s quite centralized.
  - ManBeardPc 2y ago
    From the documentation: https://bazel.build/external/registry https://bazel.build/external/registry > Bzlmod discovers dependencies by requesting their information from Bazel registries: databases of Bazel modules. Currently, Bzlmod only supports index registries — local directories or static HTTP servers following a specific format. It just uses the Bazel Central Registry (BCR) by default. You can specify your own via the --registry flag and then it uses them instead. It is possible to specify multiple registries at the same time, so you can mix the official, company internal and on your local filesystem at the same time.
wwwigham 2y ago
You can use file:// or git:// versioned dependencies in normal npm `package.json`s today. People just don't, outside some edge cases (local packages, nightly versions) because the centralized registry has upsides for discoverability and maintenance. There's also private registries, where you can setup a .npmrc to pull different namespaces from different registries. But if you want, you can totally be that guy who only publishes their packages to their presumably self-hosted repo - it works.
- Macha 2y ago
  The same is also true of Python, and once used to be the majority option, which was around the time that the "Python packaging is terrible" impression was starting. Since like 2015, pypi has banned it in hosted packages, but you can do it for private packages all you want.
pjmlp 2y ago
Go is exactly how one should not do it, hardcoding SCM URLs on source code, only allowing for source code distribution, and forcing everyone else to jump through hoops to make it work otherwise.
lifthrasiir 2y ago
The fact that Go eventually introduced the central proxy server highlights one of the biggest issues with this approach though: the vast majority of use cases do prefer centralized systems.
- crabbone 2y ago
  The major advantages of centralized systems in my mind are: * No need to deal with duplicates (what if two indices provide the same version of a package but with different dependencies?) * Oversight (the ability of most popular package managers to "side-load" from Git repo / URL / etc. "illegitimate" source has been a bane of my existence for a while). Being accepted into some "central" index means at least some (albeit usually not much...) degree of oversight. Being able to load from anywhere means exposure to spoofing. Also, unfortunately, lots of package management systems in common use can be "subverted" to be used in a decentralized way. And, if it were up to me, I'd rather not have this ability at all than have to try to defend against negative consequences.
deleted 2y ago
[deleted]
crabmusket 2y ago
Deno literally did this[1] in order to be browser-compatible[2], and everybody whinged about it. I happen to love that I can just import an HTTPS URL inside Deno code. It's really nice for playing around with libraries, writing little scripts that pull in dependencies, etc. [1] https://docs.deno.com/runtime/manual/basics/modules/#remote-import https://docs.deno.com/runtime/manual/basics/modules/#remote-... [2] https://html.spec.whatwg.org/multipage/webappapis.html#module-specifier-resolution https://html.spec.whatwg.org/multipage/webappapis.html#modul...
- tionis 2y ago
  I love this approach in Deno! I did write some nice scripts using this!

davexunit 2y ago

Go programs are a nightmare to package for linux distros due to the way they handle dependencies so I would strongly discourage any other language from copying the Go model.

jitl 2y ago
This says more about Linux distros creaky 90s era design than it says about Go’s model.
- crabbone 2y ago
  Not really. With all the problems of packaging software for Linux, I'll take that any time over Go. The whole "spirit" of how Go authors approached every infrastructure task they had was to start as simple as possible and grow as necessary, in small increments. Unfortunately, incrementalism doesn't work in this domain. It's very hard / virtually impossible to go back and undo bad decisions once they become public. Some steps will require sweeping changes all across the board and cannot be performed in one increment. Go packaging mostly works because of the extra restrictions imposed on themselves by most package authors. But, if you were to stray off the beaten path, you'll discover that the system is unprepared to deal with your case. Go is the opposite of thoughtful and insightful design. It wins at first because its fast and easy to do, and once you are hooked, you will have to work extra hard to deal with the difficult parts yourself.
  - jitl 2y ago
    I guess I view Debian et al similarly to how you view Go: distros like Debian have been incrementally iterating on a simple design since the 90s and their package systems have become a Katamari Damacy ball of mud around a core designed for having a single agreed-upon version of each C library shared by all software in the universe. Now to reliably distribute Linux software, it’s very popular to distribute the software along with the entire userspace the software compiled against in the form of a Docker container.
    - crabbone 2y ago
      > it’s very popular to distribute the software along with the entire userspace OK, there are two ways to interpret the word "popular". * As in "a presidential candidate enjoyed popular support". * As in "ski is a popular sport in some European countries". So, let me tell you this: distributing software with the entire userspace is not popular, if you use the first interpretation of the word. Users hate developers who distribute software like this, this is what they call "bloatware", "not a team-player" etc. I, personally, despise developers who do this, because, from my perspective, these are the low-skill developers who do less to get equal pay at my expense (as a user). But, of course, it's popular to be a bad developer, in the same way how it's popular to steal bicycles -- low effort, high reward, if you ignore the disappointment of someone else.

ManBeardPc 2y ago

Bazel also allows to pull dependencies from different sources via Bzlmod. 1. define dependency like bazel_dep(name = "protobuf", version = "3.19.0") 2. define repositories where to look for it A repository is just a file structure like /modules/$MODULE/$VERSION + some info about the module. Can be on a HTTP server or just on your local file system. https://bazel.build/external/overview#bzlmod https://bazel.build/external/overview#bzlmod I hope more tools adopt something similar. Or maybe a single package manager for everything, so we can finally build cross-language software without having to debug why tool A can't find stuff from B.

fire_lake 2y ago
Bazel mod requires a registry of packages in a git repo, so it’s quite centralized.
- ManBeardPc 2y ago
  From the documentation: https://bazel.build/external/registry https://bazel.build/external/registry > Bzlmod discovers dependencies by requesting their information from Bazel registries: databases of Bazel modules. Currently, Bzlmod only supports index registries — local directories or static HTTP servers following a specific format. It just uses the Bazel Central Registry (BCR) by default. You can specify your own via the --registry flag and then it uses them instead. It is possible to specify multiple registries at the same time, so you can mix the official, company internal and on your local filesystem at the same time.

wwwigham 2y ago

You can use file:// or git:// versioned dependencies in normal npm `package.json`s today. People just don't, outside some edge cases (local packages, nightly versions) because the centralized registry has upsides for discoverability and maintenance. There's also private registries, where you can setup a .npmrc to pull different namespaces from different registries. But if you want, you can totally be that guy who only publishes their packages to their presumably self-hosted repo - it works.

Macha 2y ago
The same is also true of Python, and once used to be the majority option, which was around the time that the "Python packaging is terrible" impression was starting. Since like 2015, pypi has banned it in hosted packages, but you can do it for private packages all you want.

pjmlp 2y ago

Go is exactly how one should not do it, hardcoding SCM URLs on source code, only allowing for source code distribution, and forcing everyone else to jump through hoops to make it work otherwise.

lifthrasiir 2y ago

The fact that Go eventually introduced the central proxy server highlights one of the biggest issues with this approach though: the vast majority of use cases do prefer centralized systems.

crabbone 2y ago
The major advantages of centralized systems in my mind are: * No need to deal with duplicates (what if two indices provide the same version of a package but with different dependencies?) * Oversight (the ability of most popular package managers to "side-load" from Git repo / URL / etc. "illegitimate" source has been a bane of my existence for a while). Being accepted into some "central" index means at least some (albeit usually not much...) degree of oversight. Being able to load from anywhere means exposure to spoofing. Also, unfortunately, lots of package management systems in common use can be "subverted" to be used in a decentralized way. And, if it were up to me, I'd rather not have this ability at all than have to try to defend against negative consequences.

deleted 2y ago

[deleted]

crabmusket 2y ago

Deno literally did this[1] in order to be browser-compatible[2], and everybody whinged about it. I happen to love that I can just import an HTTPS URL inside Deno code. It's really nice for playing around with libraries, writing little scripts that pull in dependencies, etc. [1] https://docs.deno.com/runtime/manual/basics/modules/#remote-import https://docs.deno.com/runtime/manual/basics/modules/#remote-... [2] https://html.spec.whatwg.org/multipage/webappapis.html#module-specifier-resolution https://html.spec.whatwg.org/multipage/webappapis.html#modul...

tionis 2y ago
I love this approach in Deno! I did write some nice scripts using this!