7 ms·

Have you used Go or Rust package ecosystems? My experience is that the older gen languages you mention had to invent package management, made lots of understan

Have you used Go or Rust package ecosystems? My experience is that the older gen languages you mention had to invent package management, made lots of understandable mistakes and now are in a backwards compat hellscape. Rust and Go built their packaging story with the benefit of lessons learned from those other systems, and in my experience the difference is night and day.

omgtehlion 4y ago
Idk about Go, but Rust’s cargo seems nice, clean yet powerful. That was my impression some time ago. But last week I attempted to compile a couple of (not very big) tools from cargo. And it ended up downloading hundreds of dependencies and gigabytes of packages. Looks like node_modules.jpg all over again :(
- verdverm 4y ago
  > Idk about Go I wrote a post highlighting Go's mod system: https://verdverm.com/go-mods/ https://verdverm.com/go-mods/ imo, it is the best designed dependency system I know of. One of the nice things is that Go uses a shared module cache so there is only one copy on your computer when multiple projects use the same dependency@version
  - omgtehlion 4y ago
    I think, this is not a problem with package manager per se. But with extremities of coding culture. On one side of the spectrum is “NIH” and reinvent everything yourself. On the other side is: let’s pull left-pad from package, because packages are good, we need MORE PACKAGES. The best solution always lies somewhere in the middle. But finding this “middle” (and adhering to this approach) is the hard part.
    - marcus_holmes 4y ago
      I think I'm on the NIH side of things. Not because I like inventing things (well, partially), but because of the security problem [0]. I don't see it being talked about here, either. Our current system of pulling in packages made by random people on the internet is going to burn us. We assume that everyone who creates a package is an honest, reliable, developer who will not inject malicious code into their package. This assumption is similar to the assumptions we made around SMTP, HTTP, DNS, and every other internet protocol. Turns out we were wrong and surprise! you can't trust people on the internet. I'm not sure if we can solve this with package managers. But package managers are part of the culture that has created this problem, and are probably a reasonable starting point to try and address it. [0]:https://david-gilbertson.medium.com/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5 https://david-gilbertson.medium.com/im-harvesting-credit-car...
  - Groxx 4y ago
    I really like a lot of the distribution- and safety-related decisions Go modules made. Domain names and the proxy+sumdb are wonderfully clear, flexible, and scalable, and I think we'll see copies of it in many future languages. The rest of the stuff around modules, like crippled constraints, zero control over contents (which they have changed!), and completely non-existent "x is available, upgrade" or release tooling: constant, unnecessary pain, and it'll be inflicting serious damage on the ecosystem for many years to come.
    - verdverm 4y ago
      > crippled constraints Do you mean ranges on dep versions? The way it is currently, the version you set is the minimum, and the algo finds the highest minimum set across all deps and uses that. If ranges were introduced, you'd end up with an NP hard problem and need a SAT solver for your deps again > Release tooling What are you looking for here? Libraries only need to push a git tag, binaries do require a bit of work, but Goreleaser fills that pretty nicely. It would seem hard to standard where binaries would be pushed > completely non-existent "x is available, upgrade" https://go.dev/doc/modules/managing-dependencies#discovering_updates https://go.dev/doc/modules/managing-dependencies#discovering... "go list -m -u all"
      - Groxx 4y ago
        I'm continually confused why SAT solving is seen as a bad thing. It automatically solves common real-world problems in less than a second even in absurdly extreme cases, and a couple milliseconds normally - why would you avoid it?
        verdverm 4y ago
        Did I say SAT was bad? Go's algorithm is much simpler and does not need a lockfile while still giving deterministic results. Ranged deps without a lockfile cannot. There is benefit to two people running the same command and getting the same dependency versions. Most projects do not start with a lockfile, so it is quite easy to have different versions when running getting started commands. Another example, if I install two ranged dependencies in both orders, will I get the same final deps@version list? Why choose SAT over simpler deterministic algos?
- junon 4y ago
  As someone who contributed somewhat extensively to the node_modules problem early on, Cargo is definitely better than the JS ecosystem in this regard. Further, another major difference is that you don't need those dependencies after you've built. You can blow them away. Doing that with node is not as straightforward, and in many cases, not possible.
pjmlp 4y ago
Doesn't look like it, see module drama in Go, while Rust is having an npm like ecosystem of tiny crates. Plus none of them handle binary library distribution as some of the packing models that came before them.
- tomcam 4y ago
  Can you give me a one-liner about module drama in Go?
  - pjmlp 4y ago
    First they weren't supported, so the community created various ways of dealing it including a kind of Google's blessed implementation, then Google decided to create their own official way, then there was the transition time which I guess not everyone has done, having SCM urls as imports is just bad, and how GOPROXY DoS some SCM repos outside of Github.
    - Nzen 4y ago
      [0] https://go.dev/blog/appengine-gopath https://go.dev/blog/appengine-gopath gopath / workspaces original [1] https://go.dev/blog/migrating-to-go-modules https://go.dev/blog/migrating-to-go-modules current system [2] https://news.ycombinator.com/item?id=34310674 https://news.ycombinator.com/item?id=34310674 goproxy agressively polling sourcehut It's worth recognizing that go isn't alone in dns-based namespacing : java's maven/gradle use the same strategy: https://repo1.maven.org/maven2/gov/nih/imagej/imagej/1.47/ https://repo1.maven.org/maven2/gov/nih/imagej/imagej/1.47/
      - tomcam 4y ago
        Fantastic links, thanks! I like the DNS-based way so much I am still trying to tease apart its disadvantages.
        mhio 4y ago
        Most DNS registrations won't match the lifetime of the software.
xiphias2 4y ago
Npm has lots of not understandable mistakes: it should already be as good with ES6+TS as other systems. Import should just work everywhere. We should have left Commonjs a long time ago, while keeping backwards compatibility. At the same time what I see with the node+npm system is that everything is just ,,it just doesn't work by default''. Having 10 other package managers doesn't work either, they are faster, but don't solve this problem.
- eurasiantiger 4y ago
  Just use .mjs extension for your code, add extensions to your imports and everything does just work.
  - noahtallen 4y ago
    Except in typescript, where you have to import a file with the .js extension, despite the actual file being .ts: https://github.com/microsoft/TypeScript/issues/42151 https://github.com/microsoft/TypeScript/issues/42151, https://github.com/microsoft/TypeScript/issues/49083 https://github.com/microsoft/TypeScript/issues/49083, https://github.com/microsoft/TypeScript/issues/16577 https://github.com/microsoft/TypeScript/issues/16577 This one issue (IMO) is preventing parts of the ecosystem from switching to esm.
    - mhio 4y ago
      How does the js extension stop a project from targeting ESM?
tsimionescu 4y ago
Go package management is a mess of hacks. It doesn't even have a repository, instead relying on source control systems to do the actual work, with special hacks for each source control system to define the artifacts that can be downloaded (e.g. using Git tags with some magic format, or Perforce commit Metadata). It requires you to physically move all code to a new folder in your version control if you want to increase the major version number. It requires users of your code to update all of their import statements throughout their code whenever you move your hosting. It relies on DNS for package identity. It takes arcane magic to support multiple Go modules in the same repo. I can go on, but it's a terrible hodge-podge of systems. It works nicely for simple cases (consuming libraries off Github), but it's awful when you go into details. And it's not even used by its creators - since Google has a monorepo and they actually use their internal universal build tool to just compile everything from source.
- gwd 4y ago
  > It relies on DNS for package identity. The flip side of this is that it never has to worry about naming collisions or namespacing: Your public package name must be a URL you control. Additionally, there is no requirement for a centralized package facility to be run. The Golang project is currently running pkg.go.dev, but that's only been in the last few years; and if they decided to get rid of it, it wouldn't significantly impact the development environment. Finally, the current system makes "typo-squatting attacks" harder to do. Consider the popular golang package github.com/mattn/go-sqlite3. The only way to "typosquat" the package is to typosquat somewhere up the dependency tree; e.g., by creating github.com/matn/go-sqlite3 or something. You can't typosquat github.com/mattn/sqlite3, or github.com/mattn/go-sqlite, because you don't own those namespaces; whereas with non-DNS-based package systems, the package would be called `go-sqlite3`, and `sqlite3` or `go-sqlite` would be much easier to typosquat. All those things I find really valuable; and honestly it's something I wish the Rust ecosystem had picked up. > It requires users of your code to update all of their import statements throughout their code whenever you move your hosting. This is a necessary cost of the item above. It can be somewhat annoying, but I believe this can be done with a one-line change to the go.mod. I'd much rather occasionally deal with this. > It requires you to physically move all code to a new folder in your version control if you want to increase the major version number. And the benefit of this is that legacy code will continue to compile into the future. I do tend to find this annoying, but it was explicit trade-off that was decided back when they were developing their packaging system. Packaging is a hard problem, with lots of trade-offs; I think Go has done a pretty good job. One way in which Go and Rust have it easier than Python or Node is that the former only have to deal with developers; the latter have to deal with both developers and users, whose requirements are often at odds with one another.
folex 4y ago
Being in the Rust full time for last 2 or 3 years: it is quite a pain to setup a release process for big Rust workspace. Version incrementing, packaging wasms, dancing around code generation – all doable, but not standardized. There's a release-please to automate all that, but it's not an easy task to set it up in all of your repos. Besides, if in addition to Rust projects, you have projects in other languages like JavaScript, then you have to do it twice and struggle with understanding all of the package management systems provided by all languages you have. A single swiss-army-knife package manager would be amazing.
dboreham 4y ago
Go and Rust don't have packages. They have tooling to pull library code from git and build it in-place to be linked into a local project. That's not (really) packaging.
- steveklabnik 4y ago
  Cargo does not “pull library code from git” unless you expressly ask for it to. And given that packages have to depend on other packages, and cannot depend on a git repository, that feature is mostly useful for testing bug fixes, private repos for leaf packages, stuff like that.
  - tremon 4y ago
    I thought Rust doesn't have binary libraries? If you declare a Rust dependency, cargo pulls its source and builds it together with your own code? I assume that's what the GP meant, anyway.
    - steveklabnik 4y ago
      Source code is stored in an S3 bucket owned by crates.io, and not from any form of source control, including git. That the entire code of all dependencies is hosted in one place is an important differentiator of crates.io vs what Go does, which is why this is relevant in this context. Crates.io is centralized, with all of the advantages and disadvantages that that brings, and Go is decentralized, with all of the advantages and disadvantages that brings.
SuperSandro2000 4y ago
But it is a pain for a distro to upgrade a vulnerable crate dependency

omgtehlion 4y ago

Idk about Go, but Rust’s cargo seems nice, clean yet powerful. That was my impression some time ago. But last week I attempted to compile a couple of (not very big) tools from cargo. And it ended up downloading hundreds of dependencies and gigabytes of packages. Looks like node_modules.jpg all over again :(

verdverm 4y ago
> Idk about Go I wrote a post highlighting Go's mod system: https://verdverm.com/go-mods/ https://verdverm.com/go-mods/ imo, it is the best designed dependency system I know of. One of the nice things is that Go uses a shared module cache so there is only one copy on your computer when multiple projects use the same dependency@version
- omgtehlion 4y ago
  I think, this is not a problem with package manager per se. But with extremities of coding culture. On one side of the spectrum is “NIH” and reinvent everything yourself. On the other side is: let’s pull left-pad from package, because packages are good, we need MORE PACKAGES. The best solution always lies somewhere in the middle. But finding this “middle” (and adhering to this approach) is the hard part.
  - marcus_holmes 4y ago
    I think I'm on the NIH side of things. Not because I like inventing things (well, partially), but because of the security problem [0]. I don't see it being talked about here, either. Our current system of pulling in packages made by random people on the internet is going to burn us. We assume that everyone who creates a package is an honest, reliable, developer who will not inject malicious code into their package. This assumption is similar to the assumptions we made around SMTP, HTTP, DNS, and every other internet protocol. Turns out we were wrong and surprise! you can't trust people on the internet. I'm not sure if we can solve this with package managers. But package managers are part of the culture that has created this problem, and are probably a reasonable starting point to try and address it. [0]:https://david-gilbertson.medium.com/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5 https://david-gilbertson.medium.com/im-harvesting-credit-car...
- Groxx 4y ago
  I really like a lot of the distribution- and safety-related decisions Go modules made. Domain names and the proxy+sumdb are wonderfully clear, flexible, and scalable, and I think we'll see copies of it in many future languages. The rest of the stuff around modules, like crippled constraints, zero control over contents (which they have changed!), and completely non-existent "x is available, upgrade" or release tooling: constant, unnecessary pain, and it'll be inflicting serious damage on the ecosystem for many years to come.
  - verdverm 4y ago
    > crippled constraints Do you mean ranges on dep versions? The way it is currently, the version you set is the minimum, and the algo finds the highest minimum set across all deps and uses that. If ranges were introduced, you'd end up with an NP hard problem and need a SAT solver for your deps again > Release tooling What are you looking for here? Libraries only need to push a git tag, binaries do require a bit of work, but Goreleaser fills that pretty nicely. It would seem hard to standard where binaries would be pushed > completely non-existent "x is available, upgrade" https://go.dev/doc/modules/managing-dependencies#discovering_updates https://go.dev/doc/modules/managing-dependencies#discovering... "go list -m -u all"
    - Groxx 4y ago
      I'm continually confused why SAT solving is seen as a bad thing. It automatically solves common real-world problems in less than a second even in absurdly extreme cases, and a couple milliseconds normally - why would you avoid it?
      - verdverm 4y ago
        Did I say SAT was bad? Go's algorithm is much simpler and does not need a lockfile while still giving deterministic results. Ranged deps without a lockfile cannot. There is benefit to two people running the same command and getting the same dependency versions. Most projects do not start with a lockfile, so it is quite easy to have different versions when running getting started commands. Another example, if I install two ranged dependencies in both orders, will I get the same final deps@version list? Why choose SAT over simpler deterministic algos?
junon 4y ago
As someone who contributed somewhat extensively to the node_modules problem early on, Cargo is definitely better than the JS ecosystem in this regard. Further, another major difference is that you don't need those dependencies after you've built. You can blow them away. Doing that with node is not as straightforward, and in many cases, not possible.

pjmlp 4y ago

Doesn't look like it, see module drama in Go, while Rust is having an npm like ecosystem of tiny crates. Plus none of them handle binary library distribution as some of the packing models that came before them.

tomcam 4y ago
Can you give me a one-liner about module drama in Go?
- pjmlp 4y ago
  First they weren't supported, so the community created various ways of dealing it including a kind of Google's blessed implementation, then Google decided to create their own official way, then there was the transition time which I guess not everyone has done, having SCM urls as imports is just bad, and how GOPROXY DoS some SCM repos outside of Github.
  - Nzen 4y ago
    [0] https://go.dev/blog/appengine-gopath https://go.dev/blog/appengine-gopath gopath / workspaces original [1] https://go.dev/blog/migrating-to-go-modules https://go.dev/blog/migrating-to-go-modules current system [2] https://news.ycombinator.com/item?id=34310674 https://news.ycombinator.com/item?id=34310674 goproxy agressively polling sourcehut It's worth recognizing that go isn't alone in dns-based namespacing : java's maven/gradle use the same strategy: https://repo1.maven.org/maven2/gov/nih/imagej/imagej/1.47/ https://repo1.maven.org/maven2/gov/nih/imagej/imagej/1.47/
    - tomcam 4y ago
      Fantastic links, thanks! I like the DNS-based way so much I am still trying to tease apart its disadvantages.
      - mhio 4y ago
        Most DNS registrations won't match the lifetime of the software.

xiphias2 4y ago

Npm has lots of not understandable mistakes: it should already be as good with ES6+TS as other systems. Import should just work everywhere. We should have left Commonjs a long time ago, while keeping backwards compatibility. At the same time what I see with the node+npm system is that everything is just ,,it just doesn't work by default''. Having 10 other package managers doesn't work either, they are faster, but don't solve this problem.

eurasiantiger 4y ago
Just use .mjs extension for your code, add extensions to your imports and everything does just work.
- noahtallen 4y ago
  Except in typescript, where you have to import a file with the .js extension, despite the actual file being .ts: https://github.com/microsoft/TypeScript/issues/42151 https://github.com/microsoft/TypeScript/issues/42151, https://github.com/microsoft/TypeScript/issues/49083 https://github.com/microsoft/TypeScript/issues/49083, https://github.com/microsoft/TypeScript/issues/16577 https://github.com/microsoft/TypeScript/issues/16577 This one issue (IMO) is preventing parts of the ecosystem from switching to esm.
  - mhio 4y ago
    How does the js extension stop a project from targeting ESM?

tsimionescu 4y ago

Go package management is a mess of hacks. It doesn't even have a repository, instead relying on source control systems to do the actual work, with special hacks for each source control system to define the artifacts that can be downloaded (e.g. using Git tags with some magic format, or Perforce commit Metadata). It requires you to physically move all code to a new folder in your version control if you want to increase the major version number. It requires users of your code to update all of their import statements throughout their code whenever you move your hosting. It relies on DNS for package identity. It takes arcane magic to support multiple Go modules in the same repo. I can go on, but it's a terrible hodge-podge of systems. It works nicely for simple cases (consuming libraries off Github), but it's awful when you go into details. And it's not even used by its creators - since Google has a monorepo and they actually use their internal universal build tool to just compile everything from source.

gwd 4y ago
> It relies on DNS for package identity. The flip side of this is that it never has to worry about naming collisions or namespacing: Your public package name must be a URL you control. Additionally, there is no requirement for a centralized package facility to be run. The Golang project is currently running pkg.go.dev, but that's only been in the last few years; and if they decided to get rid of it, it wouldn't significantly impact the development environment. Finally, the current system makes "typo-squatting attacks" harder to do. Consider the popular golang package github.com/mattn/go-sqlite3. The only way to "typosquat" the package is to typosquat somewhere up the dependency tree; e.g., by creating github.com/matn/go-sqlite3 or something. You can't typosquat github.com/mattn/sqlite3, or github.com/mattn/go-sqlite, because you don't own those namespaces; whereas with non-DNS-based package systems, the package would be called `go-sqlite3`, and `sqlite3` or `go-sqlite` would be much easier to typosquat. All those things I find really valuable; and honestly it's something I wish the Rust ecosystem had picked up. > It requires users of your code to update all of their import statements throughout their code whenever you move your hosting. This is a necessary cost of the item above. It can be somewhat annoying, but I believe this can be done with a one-line change to the go.mod. I'd much rather occasionally deal with this. > It requires you to physically move all code to a new folder in your version control if you want to increase the major version number. And the benefit of this is that legacy code will continue to compile into the future. I do tend to find this annoying, but it was explicit trade-off that was decided back when they were developing their packaging system. Packaging is a hard problem, with lots of trade-offs; I think Go has done a pretty good job. One way in which Go and Rust have it easier than Python or Node is that the former only have to deal with developers; the latter have to deal with both developers and users, whose requirements are often at odds with one another.

folex 4y ago

Being in the Rust full time for last 2 or 3 years: it is quite a pain to setup a release process for big Rust workspace. Version incrementing, packaging wasms, dancing around code generation – all doable, but not standardized. There's a release-please to automate all that, but it's not an easy task to set it up in all of your repos. Besides, if in addition to Rust projects, you have projects in other languages like JavaScript, then you have to do it twice and struggle with understanding all of the package management systems provided by all languages you have. A single swiss-army-knife package manager would be amazing.

dboreham 4y ago

Go and Rust don't have packages. They have tooling to pull library code from git and build it in-place to be linked into a local project. That's not (really) packaging.

steveklabnik 4y ago
Cargo does not “pull library code from git” unless you expressly ask for it to. And given that packages have to depend on other packages, and cannot depend on a git repository, that feature is mostly useful for testing bug fixes, private repos for leaf packages, stuff like that.
- tremon 4y ago
  I thought Rust doesn't have binary libraries? If you declare a Rust dependency, cargo pulls its source and builds it together with your own code? I assume that's what the GP meant, anyway.
  - steveklabnik 4y ago
    Source code is stored in an S3 bucket owned by crates.io, and not from any form of source control, including git. That the entire code of all dependencies is hosted in one place is an important differentiator of crates.io vs what Go does, which is why this is relevant in this context. Crates.io is centralized, with all of the advantages and disadvantages that that brings, and Go is decentralized, with all of the advantages and disadvantages that brings.

SuperSandro2000 4y ago

But it is a pain for a distro to upgrade a vulnerable crate dependency