5 ms·

Package management is the bane of nearly every language/technology Nobody has "solved" it, and I don't think that there will ever be one (never say never, thou

Package management is the bane of nearly every language/technology Nobody has "solved" it, and I don't think that there will ever be one (never say never, though, right?) For Go we rely on developers of libraries to adhere to the semver versioning scheme accurately, and we cannot "pin" versions (a personal bugbear of mine) There is a couple of workarounds - using SHAs not unlike the git commit hash to provide a pseudo version, and, vendoring (which is a cache of known dependencies - which brings with it cache management problems) I had the misfortune of having to use Python with a virtual env on the weekend - it did not end well, and reminded me why I migrated away from Python. Look at Perl (cpan) Java (maven, gradle) Ruby (gems) Go (dep, glide, vgo, modules) Rust (cargo) Node (npm, yarn, etc) OSes too Redhat (yum, rpm, etc) Debian (apt) Ubuntu (snap - god why????) And so on

corndoge 5mo ago
Nix solved it. Languages could choose to adopt Nix as their packaging system.
- tadfisher 5mo ago
  It did and didn't. Nix tools for building language-specific packages almost always wrap the language build tool/package manager. This can be easy or hard, depending on how onerous the build tool is for vendoring libraries. What Nix and build tools need to agree on is a specification or protocol for "building a software dependency tree". Like, I should be able to say 'builder = cargo' in a Nix derivation and Cargo should be able to pick up everything it needs from the build environment. Alas, there is simply far too much tied up in nixpkg's stdenv for this to be viable, so we have magic stdenv builder behavior via hooks when a build tool is included in nativeBuildInputs.
  - awesome_dude 5mo ago
    I think one of the key problems too is that a system level dependency is managed by people dedicated to ensuring the chaotic nature of the package they are responsible for conforms to the way the OS they are maintaining for has proscribed. There's no real way to do that at a language level - we cannot have "Go has determined the package you are trying to fix has not met the versioning requirements proscribed so you cannot submit the patch to fix it" What language dependencies do is what OSes would think of as "unofficial versioning" that is, an OS will let you install and run an unofficial version of some lib (we've all been there, right, multiple versions of some core library because one doesn't work with whatever you are trying to install), but they will not manage it at all.
  - corndoge 5mo ago
    Thanks for writing this, I learned something
- gitaarik 5mo ago
  In theory, but not in practice
never_inline 5mo ago
> we cannot "pin" versions you can? that's why go.sum exists. you can also use the replace directive for more advanced scenarios.
- awesome_dude 5mo ago
  No - go.sum alerts you to the change - it doesn't prevent it. replace directives are ok, but you need to look at why workspaces were invented to get an idea of their shortcomings (hint: people used to have a replace directive locally that they would accidentally push and that would break other peoples builds)
mxey 5mo ago
Actually with Go modules you are always pinning dependencies. What’s in your go.mod is what is used. If your go.mod needs to be updated because a dependency wants to bring in a newer version of a transient dependency, the go.mod has to be modified (by the go command, not by you)
- awesome_dude 5mo ago
  I don't think you understand the term "pinning" go mod tidy will update your go modules whenever it feels it needs to and there's nothing you can do to stop it. The workaround is vendoring, where you control the versions in a cache.
  - deleted 5mo ago
    [deleted]
  - kbolino 5mo ago
    There is something you can do to stop it actually. You can use a replace directive, specifying that a module is replaced by itself at a fixed version. See e.g. https://stackoverflow.com/a/77412524/814422 https://stackoverflow.com/a/77412524/814422 It is worth noting though that, even without such pinning, `go mod tidy` does not update versions willy-nilly. [edit: the following is inaccurate, see grandchild comment] It only syncs go.mod with what is already being used by the build process. In other words, if you see `go mod tidy` change a version, it means that you haven't tidied the file since making other changes to it, and the listing in go.mod was stale with respect to the resolved set of transitive dependencies actually being used.
    - mxey 5mo ago
      > It only syncs go.mod with what is already being used by the build process If dependencies are incomplete, Go will fail to compile and tell you to run go mod tidy to fix it.
      - kbolino 5mo ago
dxdm 5mo ago
> I had the misfortune of having to use Python with a virtual env on the weekend - it did not end well, and reminded me why I migrated away from Python. I see this sentiment a lot, and it doesn't match my experience at all. In my decade-old bubble of using Python professionally, I've never had an issue with virtualenvs. The few issues I might've had with dependency resolution must be so far in the past that I don't remember. But that's not strictly about virtualenvs. Likewise, pip could be clunky, but we don't have to deal with it anymore. My niche is mostly backend. Other Python niches must be considerably worse in this regard.
- awesome_dude 5mo ago
  I used Python for a decade (professionally), gave up on it once I started using Go (professionally) in earnest - about 8 or 9 years ago. I never liked virtual envs, having to remember where they were, what their names were, and what was installed into each one was a pain point for me. This weekend I was trying to learn some AWS stuffs, and I cloned the official repo of example code which was Python. I followed the directions exactly and ... boom Python versioning issues... inside the freaking venv Who needs that? Why do I need to spend the better part of a couple of hours debugging a versioning problem? (FTR The problem turned out to be the repo was hardcoded to 3.8 and my local Python was 3.9.. or something along those lines - you are welcome to correct me, but that's what I remember of a painful waste of my time) With Go I have backward compatibility guarantees - usually (there have been instances in the past where the backward guarantee have been broken AND the build process got broken hard for modules, with the claim that it was external and therefore not subject to the same guarantees) > I see this sentiment a lot, and it doesn't match my experience at all. My old HCI professor used to tell me - if users are complaining (or producing workarounds like post-it notes on their monitors) - regardless of how clean or elegant you think the system is - it's not. You're saying you see people complain about it a lot - therefore it's a genuine problem.
  - andrewflnr 5mo ago
    > having to remember where they were, what their names were, and what was installed into each one was a pain point for me. It's at the project root, it's named 'venv', and its contents are described by requirements.txt. > You're saying you see people complain about it a lot - therefore it's a genuine problem. Debatable as a principle, but applicable enough here I suppose. Still, I'm not saying the problems aren't real, but what I (and probably most of us virtualenv users) are saying is that there's a pretty broad swathe of projects where you don't encounter them. It's just fine. You install your packages and use your packages and that's the whole story. I guess if you have a hard dependency on a particular version of python, it's going to be harder, but... why? That's already niche in my book. If you're saying the AWS repo was pinned to a particular version of python, I'm going to blame that on Amazon frankly. That's definitely bizarre. Edit: Were you looking at this? https://github.com/boto/boto3 https://github.com/boto/boto3 Definitely more complicated than a typical greenfield virtualenv-able project, with some python version restrictions.

corndoge 5mo ago

Nix solved it. Languages could choose to adopt Nix as their packaging system.

tadfisher 5mo ago
It did and didn't. Nix tools for building language-specific packages almost always wrap the language build tool/package manager. This can be easy or hard, depending on how onerous the build tool is for vendoring libraries. What Nix and build tools need to agree on is a specification or protocol for "building a software dependency tree". Like, I should be able to say 'builder = cargo' in a Nix derivation and Cargo should be able to pick up everything it needs from the build environment. Alas, there is simply far too much tied up in nixpkg's stdenv for this to be viable, so we have magic stdenv builder behavior via hooks when a build tool is included in nativeBuildInputs.
- awesome_dude 5mo ago
  I think one of the key problems too is that a system level dependency is managed by people dedicated to ensuring the chaotic nature of the package they are responsible for conforms to the way the OS they are maintaining for has proscribed. There's no real way to do that at a language level - we cannot have "Go has determined the package you are trying to fix has not met the versioning requirements proscribed so you cannot submit the patch to fix it" What language dependencies do is what OSes would think of as "unofficial versioning" that is, an OS will let you install and run an unofficial version of some lib (we've all been there, right, multiple versions of some core library because one doesn't work with whatever you are trying to install), but they will not manage it at all.
- corndoge 5mo ago
  Thanks for writing this, I learned something
gitaarik 5mo ago
In theory, but not in practice

never_inline 5mo ago

> we cannot "pin" versions you can? that's why go.sum exists. you can also use the replace directive for more advanced scenarios.

awesome_dude 5mo ago
No - go.sum alerts you to the change - it doesn't prevent it. replace directives are ok, but you need to look at why workspaces were invented to get an idea of their shortcomings (hint: people used to have a replace directive locally that they would accidentally push and that would break other peoples builds)

mxey 5mo ago

Actually with Go modules you are always pinning dependencies. What’s in your go.mod is what is used. If your go.mod needs to be updated because a dependency wants to bring in a newer version of a transient dependency, the go.mod has to be modified (by the go command, not by you)

awesome_dude 5mo ago
I don't think you understand the term "pinning" go mod tidy will update your go modules whenever it feels it needs to and there's nothing you can do to stop it. The workaround is vendoring, where you control the versions in a cache.
- deleted 5mo ago
  [deleted]
- kbolino 5mo ago
  There is something you can do to stop it actually. You can use a replace directive, specifying that a module is replaced by itself at a fixed version. See e.g. https://stackoverflow.com/a/77412524/814422 https://stackoverflow.com/a/77412524/814422 It is worth noting though that, even without such pinning, `go mod tidy` does not update versions willy-nilly. [edit: the following is inaccurate, see grandchild comment] It only syncs go.mod with what is already being used by the build process. In other words, if you see `go mod tidy` change a version, it means that you haven't tidied the file since making other changes to it, and the listing in go.mod was stale with respect to the resolved set of transitive dependencies actually being used.
  - mxey 5mo ago
    > It only syncs go.mod with what is already being used by the build process If dependencies are incomplete, Go will fail to compile and tell you to run go mod tidy to fix it.
    - kbolino 5mo ago

dxdm 5mo ago

> I had the misfortune of having to use Python with a virtual env on the weekend - it did not end well, and reminded me why I migrated away from Python. I see this sentiment a lot, and it doesn't match my experience at all. In my decade-old bubble of using Python professionally, I've never had an issue with virtualenvs. The few issues I might've had with dependency resolution must be so far in the past that I don't remember. But that's not strictly about virtualenvs. Likewise, pip could be clunky, but we don't have to deal with it anymore. My niche is mostly backend. Other Python niches must be considerably worse in this regard.

awesome_dude 5mo ago
I used Python for a decade (professionally), gave up on it once I started using Go (professionally) in earnest - about 8 or 9 years ago. I never liked virtual envs, having to remember where they were, what their names were, and what was installed into each one was a pain point for me. This weekend I was trying to learn some AWS stuffs, and I cloned the official repo of example code which was Python. I followed the directions exactly and ... boom Python versioning issues... inside the freaking venv Who needs that? Why do I need to spend the better part of a couple of hours debugging a versioning problem? (FTR The problem turned out to be the repo was hardcoded to 3.8 and my local Python was 3.9.. or something along those lines - you are welcome to correct me, but that's what I remember of a painful waste of my time) With Go I have backward compatibility guarantees - usually (there have been instances in the past where the backward guarantee have been broken AND the build process got broken hard for modules, with the claim that it was external and therefore not subject to the same guarantees) > I see this sentiment a lot, and it doesn't match my experience at all. My old HCI professor used to tell me - if users are complaining (or producing workarounds like post-it notes on their monitors) - regardless of how clean or elegant you think the system is - it's not. You're saying you see people complain about it a lot - therefore it's a genuine problem.
- andrewflnr 5mo ago
  > having to remember where they were, what their names were, and what was installed into each one was a pain point for me. It's at the project root, it's named 'venv', and its contents are described by requirements.txt. > You're saying you see people complain about it a lot - therefore it's a genuine problem. Debatable as a principle, but applicable enough here I suppose. Still, I'm not saying the problems aren't real, but what I (and probably most of us virtualenv users) are saying is that there's a pretty broad swathe of projects where you don't encounter them. It's just fine. You install your packages and use your packages and that's the whole story. I guess if you have a hard dependency on a particular version of python, it's going to be harder, but... why? That's already niche in my book. If you're saying the AWS repo was pinned to a particular version of python, I'm going to blame that on Amazon frankly. That's definitely bizarre. Edit: Were you looking at this? https://github.com/boto/boto3 https://github.com/boto/boto3 Definitely more complicated than a typical greenfield virtualenv-able project, with some python version restrictions.