6 ms·

Go's old $GOPATH story for development and dependencies

60 points by ingve 2y ago

stairlane 2y ago
I struggle to see the authors point. His last sentence: > It could have worked, at least in theory, in a world where preserving API compatibility (in a broad sense) is much more common than it clearly is (or isn't) in this one. Is he implying that using vendor packages is more clear that using version tags in a go.mod? That would certainly be the first I’ve ever heard of someone preferring dependency management via vendor packages over a go.mod & ‘go get’. Personally I absolutely love the way go manages dependencies. I have way more confidence that my old go programs will run with minimal dependency related issues compared to programs written in something like python.
- mdaniel 2y ago
  There is a small asterisk about "more clear" in that if one vendors the deps, they are guaranteed to be available, and not subject to github outages, which seems to be 99.5% of all go deps in the world It for sure is easier to reason about a declarative text file than 18 quadrillion .go files in a subdirectory, but using go.mod does come with tradeoffs
  - Yasuraka 2y ago
    Go uses $GOPROXY which points to https://proxy.golang.org/ https://proxy.golang.org/ by default with direct sources as fallback, very reliable
- siebenmann 2y ago
  My badly communicated overall point is that I don't think it's right to say that Go started without any thought about dependency management. Instead, the Go developers had a theory for how it would work (with $GOPATH creating workspaces), but in practice their theory didn't work out (for various reasons). For me, this makes the evolution of Go modules much more interesting, because we can get a window into what didn't work. (I'm the author of the linked-to entry. I wrote the entry because my impression is that a lot of modern Go programmers don't have this view of pre-module Go, and especially Go when you had to set $GOPATH and it was expected that you'd change it, instead of a default $HOME/go that you used almost all the time.)
  - quectophoton 2y ago
    > but in practice their theory didn't work out (for various reasons) For me, one of the reasons I disliked $GOPATH is that source code was not in $GOPATH; it was in `$GOPATH/src`. The fact that they are very strongly against relative imports also didn't help, even if I get why they do it.
    - demi56 2y ago
      From my experience the reason $GOPATH has pkg, bin src directory was they wanted to limit spread of affecting the filesystem, I could zip my go directory put it in another computer and still works as good as before, I hated Python because it couldn’t do this even pyenv have high chances of it not working correctly if changed to a different distro and most times I don’t wanna download 500M of anaconda everytime I swap.
  - gwd 2y ago
    It's certainly an interesting angle. It might have been more clear if the documentation and how-to's had explicitly used that sort of terminology: e.g., saying that step 1 of creating a project would be to create a new workspace directory for all the dependencies, then create a package directory (with a git tree) inside that. There are certainly nice things that the current Go module system buys; but one thing I miss is that under the old system, if one of the packages wasn't working the way you expected, the code was right there already; all you had to do was to go to that directory and start editing it, because it was already cloned and ready to go. The current thing where you have to clone the repo locally, add a "go.work" to replace just that package locally to do experimentation, and then remove the "go.work" afterwards isn't terrible, but it's just that extra bit of annoying busy-work. But being able to downgrade by simply changing the version in go.mod is certainly a big improvement, as is having the hashes to make supply chain attacks more difficult.
    - peter_l_downs 2y ago
      You can directly edit the downloaded code when using modules, you don’t need your go.work flow at all. I often do it while debugging particularly weird bugs or to better understand library code I’m using.
- dfawcus 2y ago
  > Is he implying that using vendor packages is more clear that using version tags in a go.mod? That would certainly be the first I’ve ever heard of someone preferring dependency management via vendor packages over a go.mod & ‘go get’. Isn't it the case that Debian builds of go 'packages' (i.e. making a .deb package') are using the pre modules mechanisms? If so, then they would seem to prefer it for some reason.
  - guilherme-puida 2y ago
    > Isn't it the case that Debian builds of go 'packages' (i.e. making a .deb package') are using the pre modules mechanisms? As far as I know, yes: https://salsa.debian.org/go-team/packages/dh-golang/-/blob/debian/sid/lib/Debian/Debhelper/Buildsystem/golang.pm?ref_type=heads#L361 https://salsa.debian.org/go-team/packages/dh-golang/-/blob/d...
  - vetrom 2y ago
    They prefer it for a number of reasons, for better or worse. The big reasons I usually call out are: - packages should be buildable without calling out to any network resources. (this is the primary reason Debian dh-make-golang disables 1.11 style gomod support) - Debian specifically avoids vendored copies of code. This does create functionality and miantenance challenges, but it also produces a leaner installed code base. One again, for better or worse, this is a standing part of Debian package policy and warts from this particular choice stand out all over, not just golang. (newer Debian python, for example outright disables pip for non-venv python) There are other nits and warts that become apparent, but I think those all waterfall out from the two points I've called out here. (The Debian package policy in question: https://www.debian.org/doc/debian-policy/ch-source.html#embedded-code-copies https://www.debian.org/doc/debian-policy/ch-source.html#embe...)
leaveyou 2y ago
sorry for the slight off topic but I still don't understand what is the advantage of having the domain hard coded in the name of the module go mod init example.com/hello and also hardcoded in all the import statements import "example.com/hello" what if you move the code to other domain ? you need to change the module and all the references to it instead of only the references I find it quite cumbersome/counter intuitive even after all these years.
- ionut-maxim 2y ago
  I think that the domain is hard coded so you can import the module from any git repo and not depend on a central repository.
- _ph_ 2y ago
  One big disadvantage of using the domain name in the module specification is, that as a package provider, you are kind of permanentely chained to the hosting provider of your choice, e.g. github.com. Moving your package would destroy its "identity". It is bad enough how much of a monopoly and tie-in github today has (probably the reason Microsoft bought it) and a language environment shouldn't contribute or even aplify that role.
  - foldr 2y ago
    The alternative (as with e.g. npm) is being tied to the hosting provider for the package ecosystem. If you want to, you can use your own domain name for your Go package URL and then link it to a repo hosted elsewhere. There's really no namespace on the internet that you have a more permanent claim on than a domain that you yourself have registered.
    - KronisLV 2y ago
      In the case of npm you could also use something like Sonatype Nexus, push your packages to your own npm repository and install them with --registry or something like that: https://stackoverflow.com/a/35624145 https://stackoverflow.com/a/35624145 Further discussion: https://stackoverflow.com/questions/40288843/installing-npm-packages-from-multiple-registries https://stackoverflow.com/questions/40288843/installing-npm-... It's just that most people don't do that, same as how most people will just store their code in GitHub.
tomphoolery 2y ago
As someone who used Go since before v1, I can say that Go's approach to dependency management was basically "we don't want to touch it until we know we have the right idea". At the time, dependency management solutions were pretty primitive to what we have today...there was no caching, and if the central server where you got your dependencies was down, you were shit out of luck. So this was really not a bad decision for the language maintainers if they wanted to keep their sanity and not subject their user base to "legacy on arrival" software. By depending on URLs and using a `$GOPATH`, this problem was solved from a localized perspective...as a developer, I could vendor in packages, not check them into Git, and compile the program. It worked. However, the biggest problem with depending on packages in Go back then was that it was difficult to communicate to other people on your project the exact version of the dependency that you wanted them to install. For projects in which you were the only developer, this wasn't an issue...but as soon as you started to use Go in a team setting, it became a real blocker toward getting your work done. Go developers needed _some_ way to communicate what version of each package they wanted to install, and a bunch of solutions popped up to help with that. But they were all still bound by the `$GOPATH` constraint and such. Although it took a lot longer than many predicted, I'm still pretty happy with how Go approached dependency management at the end of the day. Generally, all I have to do is import a dependency from a known URL and my editor/Go modules will take care of the installation work. This is way better than the JS world, in which if I want to just sketch something out I have to actually install the dependencies I need otherwise TypeScript will yell at me. With Go, it all seems to happen automatically.
- richbell 2y ago
  > At the time, dependency management solutions were pretty primitive to what we have today... What has fundamentally changed since then, in your opinion?
User23 2y ago
Go’s initial development priorities were ordered for Google’s needs. The described problems weren’t major issues at Google because they store or at least used to store all of their code in a single repo.
- azornathogron 2y ago
  Not only that, but Google doesn't use language-provided build systems anyway because it's important (to enable many other useful dev tools and practices) that everything is built using blaze (the thing externally released as bazel). So really, Go's decisions about modules, vendoring, source layout, and the "go" command itself are all kind of irrelevant within Google. (The underlying Go compiler is used of course)
- p_l 2y ago
  Golang's approach in the days of GOPATH fit very well with its native environment, which was Plan 9, where mounting remote sources as /n/... wasn't unusual.
Waterluvian 2y ago
The one issue I had with Go when poking at it years ago was that it seemed to have fairly strict opinions on directory structure. It felt like if your project wants to use Go it had better be about Go. It was immediately in conflict with the other code I had going on in the library and where on my disk I wanted to put things.
- fsmv 2y ago
  This has been fixed with the newer modules system. You do have to name the folders the same as your packages but you can put the folders anywhere you want now.
  - haiku2077 2y ago
    That naming requirement is only a requirement _within_ the source code. The folder containing go.mod can have any name.
- MereInterest 2y ago
  Not only that, but those extremely strict opinions on directory structure extend to your entire home directory. As an example, I wanted to install "gopls", the language server for go. There was a command that could install the dev tools, but it also generated a $HOME/sdk. With such a generic name in my home directory, there's not a chance that in 6 months time, I'd remember what it was for, why it was there, and whether it is safe to delete. No amount of changing $GOROOT could convince it to write into a different directory, nor was the go language server available through any other installer.
  - evilduck 2y ago
    Containerizing every Go project helped me mitigate Go's opinions on my home directory but needing to do so in the first place was irritating.
    - MereInterest 2y ago
      That was my next step, but I had already spent an afternoon on it, so I ended up just making do without and using grep. (Partly because docker has it's own set of obnoxious quirks, as it assumes everything can and should be run as root. And rootless docker cannot coexist on a system with over-permissioned docker. I really need to switch over to podman at some point.)
cesarb 2y ago
> The whole thing is fairly similar to a materialized Python virtual environment. The feeling I got about GOPATH, back when I first encountered it, was not that it should be used like a Python virtual environment, where you'd have a separate one for each of your projects. What I understood back then was that you were supposed to set up GOPATH only once for your user, in your shell's startup scripts, and all of your projects would have to live within that directory, in the rigid structure the Go tools dictated. The default for GOPATH added later was just to avoid confusing newbies who aren't used to adding environment variables to their shell startup scripts.
darioush 2y ago
The current go dependency management is lacking: - If you want to make a change in one of your dependencies, you have to fork the repository, and in the forked repository you have to textually rename the package to the new name. This makes for abysmal maintenance and unneeded merge conflicts if you want to maintain parity with upstream code. - There is the go replace directive, but this does not transitively apply to dependencies of the module that declares the go replace directive. - If your patch gets into the upstream repository, now you have to undo the forking (again via a large textual rename). - If a dependency of your code and your code both depend on the same package, you are forced to take one version per binary that gets compiled. This is just plain absurd and leads to situations where you cannot bump the dependencies independently. If you have a tree of these dependencies, you must update each dependency in the order that respects the dependency tree. This sort of defeats the purpose of specifying and locking the dependency version. - Overall, go was designed for a mono-repo in a company (Google) that does not version their software (everything runs at tip), and it shows in any type of effort that attempts to re-use software in non-trivial fashion, with distributed development that happens at different rates in different repositories.
- sunshinekitty 2y ago
  The last point is simply not true, notably Google does not use the go build or dependency tooling, instead using blaze. Blaze handles the aforementioned issues as well.

60 points by ingve 2y ago

I struggle to see the authors point. His last sentence: > It could have worked, at least in theory, in a world where preserving API compatibility (in a broad sense) is much more common than it clearly is (or isn't) in this one. Is he implying that using vendor packages is more clear that using version tags in a go.mod? That would certainly be the first I’ve ever heard of someone preferring dependency management via vendor packages over a go.mod & ‘go get’. Personally I absolutely love the way go manages dependencies. I have way more confidence that my old go programs will run with minimal dependency related issues compared to programs written in something like python.

mdaniel 2y ago
There is a small asterisk about "more clear" in that if one vendors the deps, they are guaranteed to be available, and not subject to github outages, which seems to be 99.5% of all go deps in the world It for sure is easier to reason about a declarative text file than 18 quadrillion .go files in a subdirectory, but using go.mod does come with tradeoffs
- Yasuraka 2y ago
  Go uses $GOPROXY which points to https://proxy.golang.org/ https://proxy.golang.org/ by default with direct sources as fallback, very reliable
siebenmann 2y ago
My badly communicated overall point is that I don't think it's right to say that Go started without any thought about dependency management. Instead, the Go developers had a theory for how it would work (with $GOPATH creating workspaces), but in practice their theory didn't work out (for various reasons). For me, this makes the evolution of Go modules much more interesting, because we can get a window into what didn't work. (I'm the author of the linked-to entry. I wrote the entry because my impression is that a lot of modern Go programmers don't have this view of pre-module Go, and especially Go when you had to set $GOPATH and it was expected that you'd change it, instead of a default $HOME/go that you used almost all the time.)
- quectophoton 2y ago
  > but in practice their theory didn't work out (for various reasons) For me, one of the reasons I disliked $GOPATH is that source code was not in $GOPATH; it was in `$GOPATH/src`. The fact that they are very strongly against relative imports also didn't help, even if I get why they do it.
  - demi56 2y ago
    From my experience the reason $GOPATH has pkg, bin src directory was they wanted to limit spread of affecting the filesystem, I could zip my go directory put it in another computer and still works as good as before, I hated Python because it couldn’t do this even pyenv have high chances of it not working correctly if changed to a different distro and most times I don’t wanna download 500M of anaconda everytime I swap.
- gwd 2y ago
  It's certainly an interesting angle. It might have been more clear if the documentation and how-to's had explicitly used that sort of terminology: e.g., saying that step 1 of creating a project would be to create a new workspace directory for all the dependencies, then create a package directory (with a git tree) inside that. There are certainly nice things that the current Go module system buys; but one thing I miss is that under the old system, if one of the packages wasn't working the way you expected, the code was right there already; all you had to do was to go to that directory and start editing it, because it was already cloned and ready to go. The current thing where you have to clone the repo locally, add a "go.work" to replace just that package locally to do experimentation, and then remove the "go.work" afterwards isn't terrible, but it's just that extra bit of annoying busy-work. But being able to downgrade by simply changing the version in go.mod is certainly a big improvement, as is having the hashes to make supply chain attacks more difficult.
  - peter_l_downs 2y ago
    You can directly edit the downloaded code when using modules, you don’t need your go.work flow at all. I often do it while debugging particularly weird bugs or to better understand library code I’m using.
dfawcus 2y ago
> Is he implying that using vendor packages is more clear that using version tags in a go.mod? That would certainly be the first I’ve ever heard of someone preferring dependency management via vendor packages over a go.mod & ‘go get’. Isn't it the case that Debian builds of go 'packages' (i.e. making a .deb package') are using the pre modules mechanisms? If so, then they would seem to prefer it for some reason.
- guilherme-puida 2y ago
  > Isn't it the case that Debian builds of go 'packages' (i.e. making a .deb package') are using the pre modules mechanisms? As far as I know, yes: https://salsa.debian.org/go-team/packages/dh-golang/-/blob/debian/sid/lib/Debian/Debhelper/Buildsystem/golang.pm?ref_type=heads#L361 https://salsa.debian.org/go-team/packages/dh-golang/-/blob/d...
- vetrom 2y ago
  They prefer it for a number of reasons, for better or worse. The big reasons I usually call out are: - packages should be buildable without calling out to any network resources. (this is the primary reason Debian dh-make-golang disables 1.11 style gomod support) - Debian specifically avoids vendored copies of code. This does create functionality and miantenance challenges, but it also produces a leaner installed code base. One again, for better or worse, this is a standing part of Debian package policy and warts from this particular choice stand out all over, not just golang. (newer Debian python, for example outright disables pip for non-venv python) There are other nits and warts that become apparent, but I think those all waterfall out from the two points I've called out here. (The Debian package policy in question: https://www.debian.org/doc/debian-policy/ch-source.html#embedded-code-copies https://www.debian.org/doc/debian-policy/ch-source.html#embe...)

leaveyou 2y ago

sorry for the slight off topic but I still don't understand what is the advantage of having the domain hard coded in the name of the module go mod init example.com/hello and also hardcoded in all the import statements import "example.com/hello" what if you move the code to other domain ? you need to change the module and all the references to it instead of only the references I find it quite cumbersome/counter intuitive even after all these years.

ionut-maxim 2y ago
I think that the domain is hard coded so you can import the module from any git repo and not depend on a central repository.
_ph_ 2y ago
One big disadvantage of using the domain name in the module specification is, that as a package provider, you are kind of permanentely chained to the hosting provider of your choice, e.g. github.com. Moving your package would destroy its "identity". It is bad enough how much of a monopoly and tie-in github today has (probably the reason Microsoft bought it) and a language environment shouldn't contribute or even aplify that role.
- foldr 2y ago
  The alternative (as with e.g. npm) is being tied to the hosting provider for the package ecosystem. If you want to, you can use your own domain name for your Go package URL and then link it to a repo hosted elsewhere. There's really no namespace on the internet that you have a more permanent claim on than a domain that you yourself have registered.
  - KronisLV 2y ago
    In the case of npm you could also use something like Sonatype Nexus, push your packages to your own npm repository and install them with --registry or something like that: https://stackoverflow.com/a/35624145 https://stackoverflow.com/a/35624145 Further discussion: https://stackoverflow.com/questions/40288843/installing-npm-packages-from-multiple-registries https://stackoverflow.com/questions/40288843/installing-npm-... It's just that most people don't do that, same as how most people will just store their code in GitHub.

tomphoolery 2y ago

As someone who used Go since before v1, I can say that Go's approach to dependency management was basically "we don't want to touch it until we know we have the right idea". At the time, dependency management solutions were pretty primitive to what we have today...there was no caching, and if the central server where you got your dependencies was down, you were shit out of luck. So this was really not a bad decision for the language maintainers if they wanted to keep their sanity and not subject their user base to "legacy on arrival" software. By depending on URLs and using a `$GOPATH`, this problem was solved from a localized perspective...as a developer, I could vendor in packages, not check them into Git, and compile the program. It worked. However, the biggest problem with depending on packages in Go back then was that it was difficult to communicate to other people on your project the exact version of the dependency that you wanted them to install. For projects in which you were the only developer, this wasn't an issue...but as soon as you started to use Go in a team setting, it became a real blocker toward getting your work done. Go developers needed _some_ way to communicate what version of each package they wanted to install, and a bunch of solutions popped up to help with that. But they were all still bound by the `$GOPATH` constraint and such. Although it took a lot longer than many predicted, I'm still pretty happy with how Go approached dependency management at the end of the day. Generally, all I have to do is import a dependency from a known URL and my editor/Go modules will take care of the installation work. This is way better than the JS world, in which if I want to just sketch something out I have to actually install the dependencies I need otherwise TypeScript will yell at me. With Go, it all seems to happen automatically.

richbell 2y ago
> At the time, dependency management solutions were pretty primitive to what we have today... What has fundamentally changed since then, in your opinion?

User23 2y ago

Go’s initial development priorities were ordered for Google’s needs. The described problems weren’t major issues at Google because they store or at least used to store all of their code in a single repo.

azornathogron 2y ago
Not only that, but Google doesn't use language-provided build systems anyway because it's important (to enable many other useful dev tools and practices) that everything is built using blaze (the thing externally released as bazel). So really, Go's decisions about modules, vendoring, source layout, and the "go" command itself are all kind of irrelevant within Google. (The underlying Go compiler is used of course)
p_l 2y ago
Golang's approach in the days of GOPATH fit very well with its native environment, which was Plan 9, where mounting remote sources as /n/... wasn't unusual.

Waterluvian 2y ago

The one issue I had with Go when poking at it years ago was that it seemed to have fairly strict opinions on directory structure. It felt like if your project wants to use Go it had better be about Go. It was immediately in conflict with the other code I had going on in the library and where on my disk I wanted to put things.

fsmv 2y ago
This has been fixed with the newer modules system. You do have to name the folders the same as your packages but you can put the folders anywhere you want now.
- haiku2077 2y ago
  That naming requirement is only a requirement _within_ the source code. The folder containing go.mod can have any name.
MereInterest 2y ago
Not only that, but those extremely strict opinions on directory structure extend to your entire home directory. As an example, I wanted to install "gopls", the language server for go. There was a command that could install the dev tools, but it also generated a $HOME/sdk. With such a generic name in my home directory, there's not a chance that in 6 months time, I'd remember what it was for, why it was there, and whether it is safe to delete. No amount of changing $GOROOT could convince it to write into a different directory, nor was the go language server available through any other installer.
- evilduck 2y ago
  Containerizing every Go project helped me mitigate Go's opinions on my home directory but needing to do so in the first place was irritating.
  - MereInterest 2y ago
    That was my next step, but I had already spent an afternoon on it, so I ended up just making do without and using grep. (Partly because docker has it's own set of obnoxious quirks, as it assumes everything can and should be run as root. And rootless docker cannot coexist on a system with over-permissioned docker. I really need to switch over to podman at some point.)

cesarb 2y ago

> The whole thing is fairly similar to a materialized Python virtual environment. The feeling I got about GOPATH, back when I first encountered it, was not that it should be used like a Python virtual environment, where you'd have a separate one for each of your projects. What I understood back then was that you were supposed to set up GOPATH only once for your user, in your shell's startup scripts, and all of your projects would have to live within that directory, in the rigid structure the Go tools dictated. The default for GOPATH added later was just to avoid confusing newbies who aren't used to adding environment variables to their shell startup scripts.

darioush 2y ago

The current go dependency management is lacking: - If you want to make a change in one of your dependencies, you have to fork the repository, and in the forked repository you have to textually rename the package to the new name. This makes for abysmal maintenance and unneeded merge conflicts if you want to maintain parity with upstream code. - There is the go replace directive, but this does not transitively apply to dependencies of the module that declares the go replace directive. - If your patch gets into the upstream repository, now you have to undo the forking (again via a large textual rename). - If a dependency of your code and your code both depend on the same package, you are forced to take one version per binary that gets compiled. This is just plain absurd and leads to situations where you cannot bump the dependencies independently. If you have a tree of these dependencies, you must update each dependency in the order that respects the dependency tree. This sort of defeats the purpose of specifying and locking the dependency version. - Overall, go was designed for a mono-repo in a company (Google) that does not version their software (everything runs at tip), and it shows in any type of effort that attempts to re-use software in non-trivial fashion, with distributed development that happens at different rates in different repositories.

sunshinekitty 2y ago
The last point is simply not true, notably Google does not use the go build or dependency tooling, instead using blaze. Blaze handles the aforementioned issues as well.