12 ms·

Introduction to Go Modules

252 points by robteix 8y ago

kromem 8y ago
The v2 library suffix on the package path seems rather hackish. Feels like it would have been better to use a different delineating character instead of a slash to make it abundantly clear at a glance that it's effectively a major version tag and not a subpackage. Other than that, seems relatively straightforward. Hope this is the last "new and completely different" attempt to tackle dependency management in Go.
- cube2222 8y ago
  It's kinda both. The point is, that major versions should (by the philosophy of vgo) effectively be different packages and be handled like different packages. Check out Russ's blog post series: https://research.swtch.com/vgo https://research.swtch.com/vgo
  - nanimo 8y ago
    I personally like that philosophy, but I wonder if it's in Go's best interests to ignore the desires of the people who don't like, or perhaps aren't even aware of, that reasoning.
- BillinghamJ 8y ago
  Particularly the fact that it only applies after the second version. If we're going to require /v2, /v3, etc - shouldn't we also require /v1 for the first? Also it's unclear how initial development (as per server) is to be handled. If the first few versions are v0.1.0, v0.2.0, etc. - how are these handled on the import paths. What if a package URL actually contains "/v2" at the end, but this isn't actually referring to the version and should be used when fetching the package. Seems very odd. Agree that it should have been more like "@2". What if you want to include two different minor versions, for example, as you can include two different major ones. Is that possible/allowed?
  - cube2222 8y ago
    Requiring v1 would break backwards compatibility. If you create a new library, feel free to do that. The point is to treat different major versions as different packages, enough obviously lets you use multiple major versions in your project. v0 and v1 are treated as the same package, you're supposed to use v0 as long as you're adapting to vgo / potentially doing breaking changes and use v1 when you know you'll keep a stability guarantee. As I mentioned in the sibling comment, it's all described in Russ's blog post series, long, but well worth the read: https://research.swtch.com/vgo https://research.swtch.com/vgo
  - pests 8y ago
    > If the first few versions are v0.1.0, v0.2.0, etc. - how are these handled on the import paths. This was answered in the post. It's handled in the mod file and not via import path. Import path only changes when there is breaking changes, ie: a major bump and a new /v# for the import path. Upgrade minor or patch to most recent: go get -u Upgrade patch to most recent: go get -u=patch Upgrade to specific version: go get package@version
- Athas 8y ago
  I agree. It seems to be based on having quite specific information about the structure of the package path based on the service it references. For instance, I assume that github.com/user/v2 will refer to version 0 or 1 of the 'v2' repository. When I wrote my own vgo-inspired package manager[0] I changed the handling of major versions in package paths such that they are separated by an @-sign. While github.com/user/v2@v3 still looks silly, I think it is less confusing. [0]: https://futhark.readthedocs.io/en/latest/package-management.html https://futhark.readthedocs.io/en/latest/package-management....
- crabmusket 8y ago
  I was very surprised at the way they decided to go for `v2`. It seems kind counter to the Go philosophy so far of simplicity, predictability, etc. Now I can't just look at the imports list and apply a very simple mental rule to know what names are in scope - there's this hacky exception which magically changes the rule for very specific name schemes.
  - nemo1618 8y ago
    If you're implying that "import foo" always puts foo in scope, that's never been true -- the package name doesn't need to match the last element of the path. Granted, you can count on it being true in 90% of cases, but I think "/v2" will quickly become unsurprising, the same way we expect "import go-foo" to add the name foo, not go-foo.
rukenshia 8y ago
Do I have the ability to somehow specify "use git+ssh for this dependency" with the new modules system? Right now it seems near impossible with go to do that other than manually cloning the repositories into the correct path. We can't host our things publicly and have to use SSH to clone the repositories at my company. It is especially frustrating in our CI/CD process if we need to manually clone our packages for setting everything up.
- ereyes01 8y ago
  I currently use submodules inside of vendor/ and set each dependency's remote to the SSH URL. You can then simply use git clone --recursive to get everything, including via SSH. That being said, I share your hope that this can remain smooth in the new Go modules implementation. $GOPATH is always a hurdle for new-comers to wrap their heads around in my experience.
  - rukenshia 8y ago
    Definitely sounds like an upgrade to our current approach, thank you!
- thwarted 8y ago
  That's one thing that frustrated me about dep. go get is deficient in this area too. I understand the need to namespace packages, but requiring them to be hosted (or have metatags on a page) at the location the import path specifies in order to be able to pull them down is insanity. To make matters worse, dep tried to stuff too much of a DSL into the package specification on the command line. example.com/path/pkg@hashish made it impossible to specify git@example.com/path/pkg as the location because the parser wasn't robust enough, and the package location parser wasn't/isn't smart enough to honor ssh://git@example.com/path/path as a way to be explicit about how you wanted this done. dep did work for our use case if you edited the toml file directly, once I made a 2 character change to a regular expression in v0.3.0. We use dep and stopped upgrading with that version; I'm hoping go modules make non-public repos easier, but I'm not holding my breath.
- BillinghamJ 8y ago
Jyaif 8y ago
"Go modules is a first step in potentially eliminating $GOPATH entirely at some point." Hooray!
- ainar-g 8y ago
  Ech. I will prefer GOPATH over the hell that are C/C++ include paths any day of the week. How many are there on a modern Linux system? Six? Holy hell, it's eight! $ gcc -xc++ -E -v - #include <...> search starts here: /usr/include/c++/7 /usr/include/x86_64-linux-gnu/c++/7 /usr/include/c++/7/backward /usr/lib/gcc/x86_64-linux-gnu/7/include /usr/local/include /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed /usr/include/x86_64-linux-gnu /usr/include
  - sseth 8y ago
    Unfortunately, the GOPATH structure makes it harder to mix golang projects with other languages when you need to. And it does not support package versions. I think the time has come to say goodbye to GOPATH. Does not mean of course that include paths are better.
  - masklinn 8y ago
    I'm not sure I see the difference. $GOPATH can be a list, all this tells you is that C and C++ libraries come from various sources (system, compiler, package manager, …), you could have the exact same crap in your system-provided GOPATH if there was such a thing.
    - majewsky 8y ago
      Well, you have a system-provided GOPATH, in GOROOT (on my system, the stdlib source is in /usr/lib/go/src). But you just surprised me: I never knew that GOPATH is a list.
dom96 8y ago
Isn't it a little odd that these are called modules when they sound very much like packages? In many languages, a module is just a source code file that can be imported. Why is Go redefining this term?
- adisbladis 8y ago
  Go already has a concept called 'package' (think Java package).
- masklinn 8y ago
  Go's terminology matches java, and is the exact reverse of your comments: a package is a namespace, a module is a bundle of 1..n namespaces (packages).
  - paulddraper 8y ago
    Java "packages" are namespaces, Java "modules" are modules, and Maven/Ivy "modules" are packages.
    - _old_dude_ 8y ago
      Maven modules are a bit of a mess, it's a set of packages but a package can be split into different Maven modules. I agree with the GP that go package/module are similar to Java package/module
      - snuxoll 8y ago
        Maven modules aren’t necessarily packages; you can have POM only maven artifacts for things like BOM’s, for example. Thankfully the whole split-package nonsense will eventually go away, since the JPMS doesn’t allow a package to reside in multiple modules.
      - paulddraper 8y ago
        Java packages (namespaces) can be split across Maven modules (packages). That doesn't seem weird to me. Lots of languages can split their namespaces across packages.
tamalsaha001 8y ago
Lot of thanks to the author for this post. I have 2 questions: 1. How does one use a branch of a dependency in go.mod? (scenario: local dev where api & impl repos are separated) 2. How does one use branch from a fork in go.mod? (scenario: fixed bug in someone else's repo and now want to use my fork)
- ainar-g 8y ago
  1. IIRC you should be able to do go get example.com/repo@branch-1 2. Something like go mod edit -replace example.com/repo@version=example.com/fork@version It seems like the second command doesn't understand branches, so you'll need to specify a "pseudo-version" like "v0.0.0-20171125154426-754f7301a386". Or maybe my go1.11 is just out of date.
BillinghamJ 8y ago
Is there a recommended process or guide on the best way to switch from dep to modules?
- cube2222 8y ago
  vgo mod can read your lock or vendor file (it supports most of the go vendoring solutions) when initting.
  - robteix 8y ago
    Happens automatically when you run 'go mod init' in a directory containing Gopkg.lock. Caught me by surprise the first time as I had no idea it did that :)
e12e 8y ago
So, if I want to use the new code in v2 for my v1 legacy consumers, would I let v1 (or a new v1.1) depend on v2, and wrap the new hi-method passing in a default parameter of "en" for language? Or is my only "supported" option to push consumers to the new major version - with the possible bit-rot of the old code? [ed: that is manual backport fixes/improvements to a v1.x version]
- ainar-g 8y ago
  Russ has explicitly written that yes, redefining v1 in terms of v2 is not only the way to go, but in the future it might help automatise the switch to v2 with the "go fix" command. See https://research.swtch.com/vgo-import https://research.swtch.com/vgo-import, section "Automatic API Updates".
bobjordan 8y ago
Recently found a command line library written in go that was so useful to me in a python webapp I'm building, I decided to just wrap the go app and call it from a python subprocess. It is a bit slow to do it this way, but premature optimization, and all that. Anyhow, the bottom line is due to this I was forced to get a handle on Go this week so I could at least build the go library and make a few changes as needed. Wow, was I absolutely shocked to find out current package management in this go app is just pulling libraries from master on github. It really made me feel like go is far behind python in maturity. Pulling all these dependency libs from master definitely does not feel production ready. Maybe this will help resolve things.
- aczerepinski 8y ago
  I don't think your impression of the current state of dependency management in Go is accurate. Pretty much everybody uses this: https://golang.github.io/dep/docs/Gopkg.lock.html https://golang.github.io/dep/docs/Gopkg.lock.html
  - the_duke 8y ago
    Some do, but many don't.
    - yahyaheee 8y ago
      Every major stable library uses a form of package management. Anyone pulling from master is a new go developer
      - wereHamster 8y ago
        The fact that new go developers do that tells a lot about the state of documentation, tutorials, tooling etc. A language and its ecosystem should encourage best practices from the start. Not as some kind of late afterthought.
        yahyaheee 8y ago
sheeshkebab 8y ago
Why can’t google just maintain a npm.js style repo for go packages (together with a package manager program)? And a proper simple to read manifest file for a go programmer to list out external dependencies with their versions I really don’t see these git based hacks can improve situation much.
- Yokohiii 8y ago
  Because a centralized repo is a disaster.
  - wlll 8y ago
    As someone who has programmed a number of languages and currently mostly programs either Ruby or Go I have to say I would greatly prefer a Rubygems/bundler type system to the one Go seems to be heading for.
  - ofrzeta 8y ago
    Why? It has worked out quite well for pretty much every other ecosystem like CPAN, PyPI, Maven etc. don't you think?
    - Yokohiii 8y ago
      It worked out for the time being, but npm destroyed the illusion. I don't even see the problem. A package manager could rely completely on git with the advantage to not have an single authority that isn't giving a flying fuck about quality and security of the hosted packages. Yes it forces you to dig out a good package, evaluate it and look at the track record of the devs. Someone has to do it, but it is currently hardly happening.
  - BillinghamJ 8y ago
    Have you ever noticed that 99%+ of Go libraries are pulled from Github?
    - Yokohiii 8y ago
pests 8y ago
Does vgo miss a nuance in the semantic versioning of 0.x.x branches in that any bump to minor or patch allows a breaking change? Specifically point 4 of the spec. Some projects or libraries have a huge psychological barrier to releasing a 1.0.0. React comes to mind when they finally went from 0.14.x to 15.0.0. --- 4. Major version zero (0.y.z) is for initial development. Anything may change at any time. The public API should not be considered stable. https://semver.org/#spec-item-4 https://semver.org/#spec-item-4
- cube2222 8y ago
  No, you're supposed to stop breaking when you're in 1.x.x
  - pests 8y ago
    How can you say this? I linked the spec.
    - cube2222 8y ago
      I meant vgo didn't miss the nuicance :)
- fpoling 8y ago
  Go developers made a reasonable choice. Unstable API is just another way to say that there is no API. And with zero API any change is compatible and cannot break anything. So there is no need for v0.
  - cesarb 8y ago
    If there is no API, how can the library be used? Every function a library exposes is part of the library's API, so for a library to have no API, it cannot expose any function.
    - deleted 8y ago
      [deleted]
kalekold 8y ago
Go modules are a step in the right direction but still doesn't protect you for when a developer deletes their repository!

252 points by robteix 8y ago

The v2 library suffix on the package path seems rather hackish. Feels like it would have been better to use a different delineating character instead of a slash to make it abundantly clear at a glance that it's effectively a major version tag and not a subpackage. Other than that, seems relatively straightforward. Hope this is the last "new and completely different" attempt to tackle dependency management in Go.

cube2222 8y ago
It's kinda both. The point is, that major versions should (by the philosophy of vgo) effectively be different packages and be handled like different packages. Check out Russ's blog post series: https://research.swtch.com/vgo https://research.swtch.com/vgo
- nanimo 8y ago
  I personally like that philosophy, but I wonder if it's in Go's best interests to ignore the desires of the people who don't like, or perhaps aren't even aware of, that reasoning.
BillinghamJ 8y ago
Particularly the fact that it only applies after the second version. If we're going to require /v2, /v3, etc - shouldn't we also require /v1 for the first? Also it's unclear how initial development (as per server) is to be handled. If the first few versions are v0.1.0, v0.2.0, etc. - how are these handled on the import paths. What if a package URL actually contains "/v2" at the end, but this isn't actually referring to the version and should be used when fetching the package. Seems very odd. Agree that it should have been more like "@2". What if you want to include two different minor versions, for example, as you can include two different major ones. Is that possible/allowed?
- cube2222 8y ago
  Requiring v1 would break backwards compatibility. If you create a new library, feel free to do that. The point is to treat different major versions as different packages, enough obviously lets you use multiple major versions in your project. v0 and v1 are treated as the same package, you're supposed to use v0 as long as you're adapting to vgo / potentially doing breaking changes and use v1 when you know you'll keep a stability guarantee. As I mentioned in the sibling comment, it's all described in Russ's blog post series, long, but well worth the read: https://research.swtch.com/vgo https://research.swtch.com/vgo
- pests 8y ago
  > If the first few versions are v0.1.0, v0.2.0, etc. - how are these handled on the import paths. This was answered in the post. It's handled in the mod file and not via import path. Import path only changes when there is breaking changes, ie: a major bump and a new /v# for the import path. Upgrade minor or patch to most recent: go get -u Upgrade patch to most recent: go get -u=patch Upgrade to specific version: go get package@version
Athas 8y ago
I agree. It seems to be based on having quite specific information about the structure of the package path based on the service it references. For instance, I assume that github.com/user/v2 will refer to version 0 or 1 of the 'v2' repository. When I wrote my own vgo-inspired package manager[0] I changed the handling of major versions in package paths such that they are separated by an @-sign. While github.com/user/v2@v3 still looks silly, I think it is less confusing. [0]: https://futhark.readthedocs.io/en/latest/package-management.html https://futhark.readthedocs.io/en/latest/package-management....
crabmusket 8y ago
I was very surprised at the way they decided to go for `v2`. It seems kind counter to the Go philosophy so far of simplicity, predictability, etc. Now I can't just look at the imports list and apply a very simple mental rule to know what names are in scope - there's this hacky exception which magically changes the rule for very specific name schemes.
- nemo1618 8y ago
  If you're implying that "import foo" always puts foo in scope, that's never been true -- the package name doesn't need to match the last element of the path. Granted, you can count on it being true in 90% of cases, but I think "/v2" will quickly become unsurprising, the same way we expect "import go-foo" to add the name foo, not go-foo.

rukenshia 8y ago

Do I have the ability to somehow specify "use git+ssh for this dependency" with the new modules system? Right now it seems near impossible with go to do that other than manually cloning the repositories into the correct path. We can't host our things publicly and have to use SSH to clone the repositories at my company. It is especially frustrating in our CI/CD process if we need to manually clone our packages for setting everything up.

ereyes01 8y ago
I currently use submodules inside of vendor/ and set each dependency's remote to the SSH URL. You can then simply use git clone --recursive to get everything, including via SSH. That being said, I share your hope that this can remain smooth in the new Go modules implementation. $GOPATH is always a hurdle for new-comers to wrap their heads around in my experience.
- rukenshia 8y ago
  Definitely sounds like an upgrade to our current approach, thank you!
thwarted 8y ago
That's one thing that frustrated me about dep. go get is deficient in this area too. I understand the need to namespace packages, but requiring them to be hosted (or have metatags on a page) at the location the import path specifies in order to be able to pull them down is insanity. To make matters worse, dep tried to stuff too much of a DSL into the package specification on the command line. example.com/path/pkg@hashish made it impossible to specify git@example.com/path/pkg as the location because the parser wasn't robust enough, and the package location parser wasn't/isn't smart enough to honor ssh://git@example.com/path/path as a way to be explicit about how you wanted this done. dep did work for our use case if you edited the toml file directly, once I made a 2 character change to a regular expression in v0.3.0. We use dep and stopped upgrading with that version; I'm hoping go modules make non-public repos easier, but I'm not holding my breath.
BillinghamJ 8y ago

Jyaif 8y ago

"Go modules is a first step in potentially eliminating $GOPATH entirely at some point." Hooray!

ainar-g 8y ago
Ech. I will prefer GOPATH over the hell that are C/C++ include paths any day of the week. How many are there on a modern Linux system? Six? Holy hell, it's eight! $ gcc -xc++ -E -v - #include <...> search starts here: /usr/include/c++/7 /usr/include/x86_64-linux-gnu/c++/7 /usr/include/c++/7/backward /usr/lib/gcc/x86_64-linux-gnu/7/include /usr/local/include /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed /usr/include/x86_64-linux-gnu /usr/include
- sseth 8y ago
  Unfortunately, the GOPATH structure makes it harder to mix golang projects with other languages when you need to. And it does not support package versions. I think the time has come to say goodbye to GOPATH. Does not mean of course that include paths are better.
- masklinn 8y ago
  I'm not sure I see the difference. $GOPATH can be a list, all this tells you is that C and C++ libraries come from various sources (system, compiler, package manager, …), you could have the exact same crap in your system-provided GOPATH if there was such a thing.
  - majewsky 8y ago
    Well, you have a system-provided GOPATH, in GOROOT (on my system, the stdlib source is in /usr/lib/go/src). But you just surprised me: I never knew that GOPATH is a list.

dom96 8y ago

Isn't it a little odd that these are called modules when they sound very much like packages? In many languages, a module is just a source code file that can be imported. Why is Go redefining this term?

adisbladis 8y ago
Go already has a concept called 'package' (think Java package).
masklinn 8y ago
Go's terminology matches java, and is the exact reverse of your comments: a package is a namespace, a module is a bundle of 1..n namespaces (packages).
- paulddraper 8y ago
  Java "packages" are namespaces, Java "modules" are modules, and Maven/Ivy "modules" are packages.
  - _old_dude_ 8y ago
    Maven modules are a bit of a mess, it's a set of packages but a package can be split into different Maven modules. I agree with the GP that go package/module are similar to Java package/module
    - snuxoll 8y ago
      Maven modules aren’t necessarily packages; you can have POM only maven artifacts for things like BOM’s, for example. Thankfully the whole split-package nonsense will eventually go away, since the JPMS doesn’t allow a package to reside in multiple modules.
    - paulddraper 8y ago
      Java packages (namespaces) can be split across Maven modules (packages). That doesn't seem weird to me. Lots of languages can split their namespaces across packages.

tamalsaha001 8y ago

Lot of thanks to the author for this post. I have 2 questions: 1. How does one use a branch of a dependency in go.mod? (scenario: local dev where api & impl repos are separated) 2. How does one use branch from a fork in go.mod? (scenario: fixed bug in someone else's repo and now want to use my fork)

ainar-g 8y ago
1. IIRC you should be able to do go get example.com/repo@branch-1 2. Something like go mod edit -replace example.com/repo@version=example.com/fork@version It seems like the second command doesn't understand branches, so you'll need to specify a "pseudo-version" like "v0.0.0-20171125154426-754f7301a386". Or maybe my go1.11 is just out of date.

BillinghamJ 8y ago

Is there a recommended process or guide on the best way to switch from dep to modules?

cube2222 8y ago
vgo mod can read your lock or vendor file (it supports most of the go vendoring solutions) when initting.
- robteix 8y ago
  Happens automatically when you run 'go mod init' in a directory containing Gopkg.lock. Caught me by surprise the first time as I had no idea it did that :)

e12e 8y ago

So, if I want to use the new code in v2 for my v1 legacy consumers, would I let v1 (or a new v1.1) depend on v2, and wrap the new hi-method passing in a default parameter of "en" for language? Or is my only "supported" option to push consumers to the new major version - with the possible bit-rot of the old code? [ed: that is manual backport fixes/improvements to a v1.x version]

ainar-g 8y ago
Russ has explicitly written that yes, redefining v1 in terms of v2 is not only the way to go, but in the future it might help automatise the switch to v2 with the "go fix" command. See https://research.swtch.com/vgo-import https://research.swtch.com/vgo-import, section "Automatic API Updates".

bobjordan 8y ago

Recently found a command line library written in go that was so useful to me in a python webapp I'm building, I decided to just wrap the go app and call it from a python subprocess. It is a bit slow to do it this way, but premature optimization, and all that. Anyhow, the bottom line is due to this I was forced to get a handle on Go this week so I could at least build the go library and make a few changes as needed. Wow, was I absolutely shocked to find out current package management in this go app is just pulling libraries from master on github. It really made me feel like go is far behind python in maturity. Pulling all these dependency libs from master definitely does not feel production ready. Maybe this will help resolve things.

aczerepinski 8y ago
I don't think your impression of the current state of dependency management in Go is accurate. Pretty much everybody uses this: https://golang.github.io/dep/docs/Gopkg.lock.html https://golang.github.io/dep/docs/Gopkg.lock.html
- the_duke 8y ago
  Some do, but many don't.
  - yahyaheee 8y ago
    Every major stable library uses a form of package management. Anyone pulling from master is a new go developer
    - wereHamster 8y ago
      The fact that new go developers do that tells a lot about the state of documentation, tutorials, tooling etc. A language and its ecosystem should encourage best practices from the start. Not as some kind of late afterthought.
      - yahyaheee 8y ago

sheeshkebab 8y ago

Why can’t google just maintain a npm.js style repo for go packages (together with a package manager program)? And a proper simple to read manifest file for a go programmer to list out external dependencies with their versions I really don’t see these git based hacks can improve situation much.

Yokohiii 8y ago
Because a centralized repo is a disaster.
- wlll 8y ago
  As someone who has programmed a number of languages and currently mostly programs either Ruby or Go I have to say I would greatly prefer a Rubygems/bundler type system to the one Go seems to be heading for.
- ofrzeta 8y ago
  Why? It has worked out quite well for pretty much every other ecosystem like CPAN, PyPI, Maven etc. don't you think?
  - Yokohiii 8y ago
    It worked out for the time being, but npm destroyed the illusion. I don't even see the problem. A package manager could rely completely on git with the advantage to not have an single authority that isn't giving a flying fuck about quality and security of the hosted packages. Yes it forces you to dig out a good package, evaluate it and look at the track record of the devs. Someone has to do it, but it is currently hardly happening.
- BillinghamJ 8y ago
  Have you ever noticed that 99%+ of Go libraries are pulled from Github?
  - Yokohiii 8y ago

pests 8y ago

Does vgo miss a nuance in the semantic versioning of 0.x.x branches in that any bump to minor or patch allows a breaking change? Specifically point 4 of the spec. Some projects or libraries have a huge psychological barrier to releasing a 1.0.0. React comes to mind when they finally went from 0.14.x to 15.0.0. --- 4. Major version zero (0.y.z) is for initial development. Anything may change at any time. The public API should not be considered stable. https://semver.org/#spec-item-4 https://semver.org/#spec-item-4

cube2222 8y ago
No, you're supposed to stop breaking when you're in 1.x.x
- pests 8y ago
  How can you say this? I linked the spec.
  - cube2222 8y ago
    I meant vgo didn't miss the nuicance :)
fpoling 8y ago
Go developers made a reasonable choice. Unstable API is just another way to say that there is no API. And with zero API any change is compatible and cannot break anything. So there is no need for v0.
- cesarb 8y ago
  If there is no API, how can the library be used? Every function a library exposes is part of the library's API, so for a library to have no API, it cannot expose any function.
  - deleted 8y ago
    [deleted]

kalekold 8y ago

Go modules are a step in the right direction but still doesn't protect you for when a developer deletes their repository!