15 ms·

Go Modules in 2019

238 points by baijum 8y ago

pjmlp 8y ago
Nice overview, however a small correction to the statement about Maven repositories. Decentralized support has been a thing since the last decade.
- emersion 8y ago
  What do you mean by "decentralized"? Do you mean someone can add a third-party repository and download packages from there? (Then it's been the case for other environments too) I think decentralized is meant as "there's no central repository" here.
  - pjmlp 8y ago
    Same applies to Maven, in fact many enterprises forbid access to Maven central, while curating an internal one.
    - Merovius 8y ago
      > Same applies to Maven, in fact many enterprises forbid access to Maven central "Maven central" implies there is a central repository though.
      - TheDong 8y ago
        There are two different meanings of 'central' in play here, 'center' as in a town square, and 'center' as in the centralized component of a system. It does not imply that there is a central repository when using 'central' to mean 'a unique, difficult to replicate, or otherwise somehow special component in a system' as in 'centralization'. Central in the case of 'Maven Central' means something closer to 'default' or 'town square' or such. Saying that 'maven central' implies maven isn't decentralized would be the same as saying "Most go projects are on github, so go can't be hosted in a decentralized way because github is 'golang central'".
        Merovius 8y ago
        > Saying that 'maven central' implies maven isn't decentralized would be the same as saying "Most go projects are on github, so go can't be hosted in a decentralized way because github is 'golang central'". It's not, though. Because github isn't the default. You have to explicitly specify that you want a project from github every time you use it. The difference might be minor from a technological standpoint, but it means a lot in terms of how centralized things become culturally.
        pjmlp 8y ago
        Just like enterprises have their official .m2/settings.xml file pointing to the actual default Maven repository.
jrockway 8y ago
Does anyone know what the story is for binaries generated by things your project depends on? npm versions them and lets you invoke them from the project with "npx"; I wish something like this existed for go. (protoc-gen-go is the main thing I want; if your global version gets out of sync with the version in go.modules, the generated protobuf doesn't compile.) I am glad to see that the go team is working on cleaning up the tooling situation. I used gocode, then started using modules, found that the primary version was unmaintained, and had to switch to a different fork. I believe I also needed a different version of goimports for a while. Having all this tooling unified into the langserver maintained alongside go sounds wonderful. I hope other languages do the same thing.
- dub 8y ago
  For tools in golang like goimports the best practice at the moment is to have a tools.go with a "tools" build tag. See https://github.com/golang/go/issues/25922 https://github.com/golang/go/issues/25922 For non-go tools like the protoc compiler I don't think the golang tooling provides any help. A heavyweight option would be a build system like bazel
- laumars 8y ago
  On a vaguely related note, is there any decent documentation for writing a lang server. I've seen the official documentation by Microsoft and while it does seem a detailed reference, it's not great for someone who doesn't even know where to start in writing one.
  - sbinet 8y ago
    this is being worked on: - https://github.com/golang/tools/blob/master/cmd/golsp/main.go https://github.com/golang/tools/blob/master/cmd/golsp/main.g...
    - laumars 8y ago
      I don’t want to create one for Go. It’s for a hobby language I’ve created. I only mentioned it because the op discussed Go’s language server.
plg 8y ago
I really want to get into Go, I like that it's opinionated, I like that it's compiled, I like that it's garbage-collected, I like that coroutines are built-in. My primary use case is scientific computing, both data processing and interactive visualization. I know Julia is an option as well. For reasons I don't want to bother getting into I dislike Python. Thoughts?
- petre 8y ago
  Dlang is better suited for that use case. Have you looked at mir?
- llimllib 8y ago
  It's not well-suited to scientific computing at all. I use and like Go for writing network servers and ETL processes. In a scientific context though, the type system is awkward in the extreme, there is essentially no library of modules, and the interactive visualization story is nonexistent. Python, R, Julia, even C++ would be better options IMO. (I'll clarify again: I like Go! I just think it's not well suited for this context)
  - stcredzero 8y ago
    If one wants to be a pioneer and be one of the people who starts the movement, now is the time to jump in and make a name for yourself.
    - DannyBee 8y ago
      This is absolutely true. It's also a lot of work :)
    - jerf 8y ago
      I'd suggest "now" is actually a terrible time to try to make Go do scientific computing. With generics all-but-guaranteed to be incoming, but also not here yet, a lot of what you would do is going to be superceded in the forseeable future, on the same sort of time scale as your library's expected completion date. I'd say that even once Go has generics, that will simply take it from being an awful scientific programming language to a mediocre one. I don't really understand why there's a few people who seem to think it's a good idea to try to do their scientific computing in Go when there are so many better options for that that already exist. Post-generics, though, if you really insist, it will probably be the case that Go can be upgraded from "mediocre" to "tolerable" with a lot of library work. (Part of the "mediocre" is lacking libraries. That aspect can be fixed.) But it'll be hard to start that work without running generic code. Even if you assume the current documentation is the final specs, you still won't be able to guess the performance implications of anything you'd be blindly writing, and performance is very important for this sort of code.
treve 8y ago
One of the thing that's always been a bit off-putting about the Go community and leadership is that it seemed that when I came across things that I perceived as flaws in the tooling or language, I often felt told off and was made to feel that it's me who is wrong for a variety of reasons. Examples of this include GOPATH, the package infrastructure, error handling, lack of generics. Rob Pike disagrees so I must be wrong. It's kind of satisfying to see that the "Go way" wasn't the best way after all and that some of these things are actively being addressed. I hope it makes the community a bit more welcoming as well.
- cgag 8y ago
  Honestly I got the impression no one ever liked gopath.
  - revskill 8y ago
    Because setting environment variables is something most devs like to avoid.
    - cgag 8y ago
      No, because I have like 50 projects in ~/src and suddenly have to keep some of my code in ~/src/go/src/GitHub.com/cgag and that annoys me.
      - cyphar 8y ago
        I have tried several times to fix this problem in the past 5 or 6 years, once with symlinks (which would break quite often -- it's unsurprising that Plan 9 developers wouldn't care too much for handling symlinks correctly :P). The current way I handle it is that $HOME/.local is my GOPATH, and $HOME/src links to $HOME/.local/src. So you can just have $HOME/src/github.com/bar/foo -- which is less ideal than just $HOME/src/foo, but it's something at least.
      - revel 8y ago
        it's not just that large numbers of projects are a hassle, they make it harder to develop projects in isolation
candiodari 8y ago
Maybe it's just me but I hate this option. Are we really, really trying to optimize the downloading of source code ? All 5 megabytes of it ? Why would you do that ? I regularly find myself in a position that this system would make impossible: I need a few custom changes in public available libraries. An extra method. A bugfix. What have you. This makes that impossible using the default method. What we want is consistent builds. That means a build that happens, with the same source, with the same ... every time, always, on everybody's workstation.
- icholy 8y ago
  Vendoring will still be supported.
  - calcifer 8y ago
    How will that work with modules though? As in, how do you vendor a single library (that you want to make a fix to) while still building with module support? Right now, "module mode" is mutually exclusive with "vendor mode" and I don't think there are any plans to change that.
    - EdiX 8y ago
      > Right now, "module mode" is mutually exclusive with "vendor mode" and I don't think there are any plans to change that. This is not true. If you pass -mod=vendor, or place it in $GOFLAGS, go will build something outside of GOPATH, in module mode, using the vendor directory exclusively. I wish it was the default, but only because it encourages people to not use vendor directories (which is a bad habit), not because it can't be worked around easily.
      - calcifer 8y ago
        > in module mode, using the vendor directory exclusively But that's not what I'm asking. I want "go build" (with some flags) to use my fork (vendored if needed) for one library, and pick up all other dependencies from the module cache as usual. How do I that? Because this is already possible in a vendor-only world; I just edit the code in vendor.
dana321 8y ago
A centralized module index will be nice, i tend to end up searching github.com which leaves out all the other sites or locations that could have a module that solves my problem.
- xyzzy_plugh 8y ago
  I very much do not want this. godoc.org has decent search and I've never had any complaints.
  - Merovius 8y ago
    Note that there's a difference between "central registry" and "central index". In particular, I don't see any downsides from what they are changing - godoc.org is already, for all intents and purposes, a central index, so I don't see how anything changes in that regard.
teacpde 8y ago
So the mirror will serve both the package and signed hashes, what if the package has been indexed in the central index but not yet signed by the notary?
- emersion 8y ago
  Maybe asking a hash to the notary will actually make the notary fetch and hash the package if it doesn't know about it yet? (Related: https://news.ycombinator.com/item?id=18717766 https://news.ycombinator.com/item?id=18717766)
  - teacpde 8y ago
    That works, but slows down the mirror? I guess it is okay since it should only happen once.
spullara 8y ago
It is amazing that it took them this long to realize they were entirely wrong about how dependencies should be distributed. Since it was obvious from day 1 to most people from other ecosystems maybe they shouldn't have so much NIH. We'll likely get exceptions and generics soon as well. What a waste of time.
- emersion 8y ago
  The way dependencies are distributed (ie. downloaded) won't change a lot, and as they say the distributed model is very important. Maybe you mean how dependencies are installed by the go command? (As a side note, I don't think there are any plans for exceptions -- for good)
  - spullara 8y ago
    They will be globally indexed, signed and versioned rather than downloaded at HEAD from a random repository. I'd say that is pretty different. Also, https://go.googlesource.com/proposal/+/master/design/go2draft.md https://go.googlesource.com/proposal/+/master/design/go2draf...
- tracker1 8y ago
  It's also important to get a number of other things worked out and discuss possibilities. The go process in terms of using a baseline directory with git sources seems to be a very pragmatic approach to start from. I started playing with node before npm was in the box. There were some competing ideas and eventually one won out. I think that having a system in place with the language may be a better option than a company with its' own motivations and needs separated from the language/runtime/platform. As to generics, I think you may well find generics in the future. I feel that most of the resistance was in order to better support core language features. I can't think of any languages (I'm no expert) that started with generics support, so I'd be surprised if this wasn't a go 2.x goal. For exceptions, I think that the go solution works. Similar to the callback interfaces in node, it puts errors in your face, which isn't a bad thing. I mainly contrast this with node, as it's another language/platform that has grown a LOT but also relatively recent. You can compare the progress of node, go and others to say python2/3, C#, Java and others. I think go progress has been great by comparison, and pragmatic choices have ruled out.
emersion 8y ago
It seems like there's a mistake in the diagram. The "notary → mirror" arrow should be replaced with a "notary → go command" one, because the go command shouldn't trust the mirror when it comes to cryptographic hashes.
- pokstad 8y ago
  I think the mirror can pass through the public signature provided by the notary. That cannot be spoofed if you have a trust chain for the notary to ensure the mirror has not tampered with the module.
- teacpde 8y ago
  The hashes are signed by the notary, go command can get the hashes from anywhere and be able to verified they are signed by the notary.
Boulth 8y ago
> One of the most important parts of the original design for go get was that it was decentralized That's one of the things I like in go most. Having decentralized packages via domains and URLs and then index them (godoc.org works very well), it mirrors the design of the Web. In contrast npm, Rust and others that are tightly coupled to one site feels like Google AMP - centralizing and hosting everything in one place.
- almostdeadguy 8y ago
  Cargo (in the nightly channel, but eventually will become stable) and npm can both use alternative package indexes and vendored dependencies, so they are not tightly coupled. Having a standardized package index as a database is what allows efficient dependency solving, which Go hasn't considered providing up until recently. > that anyone should be able to publish their code on any server, in contrast to central registries such as Perl’s CPAN, Java’s Maven, or Node’s NPM. Placing domain names at the start of the go get import space reused an existing decentralized system and avoided needing to solve anew the problems of deciding who can use which names. It also allowed companies to import code on private servers alongside code from public servers. lol statements like this are infuriating. Go is no more decentralized than those languages, unless you consider git hosting something people in standard practice do on their own (survey some of the top Go dependencies and see if that holds true). They simply lack a package index, but the cited language package managers both have those and allow you to host your own.

Go Modules in 2019

238 points by baijum 8y ago

Nice overview, however a small correction to the statement about Maven repositories. Decentralized support has been a thing since the last decade.

emersion 8y ago
What do you mean by "decentralized"? Do you mean someone can add a third-party repository and download packages from there? (Then it's been the case for other environments too) I think decentralized is meant as "there's no central repository" here.
- pjmlp 8y ago
  Same applies to Maven, in fact many enterprises forbid access to Maven central, while curating an internal one.
  - Merovius 8y ago
    > Same applies to Maven, in fact many enterprises forbid access to Maven central "Maven central" implies there is a central repository though.
    - TheDong 8y ago
      There are two different meanings of 'central' in play here, 'center' as in a town square, and 'center' as in the centralized component of a system. It does not imply that there is a central repository when using 'central' to mean 'a unique, difficult to replicate, or otherwise somehow special component in a system' as in 'centralization'. Central in the case of 'Maven Central' means something closer to 'default' or 'town square' or such. Saying that 'maven central' implies maven isn't decentralized would be the same as saying "Most go projects are on github, so go can't be hosted in a decentralized way because github is 'golang central'".
      - Merovius 8y ago
        > Saying that 'maven central' implies maven isn't decentralized would be the same as saying "Most go projects are on github, so go can't be hosted in a decentralized way because github is 'golang central'". It's not, though. Because github isn't the default. You have to explicitly specify that you want a project from github every time you use it. The difference might be minor from a technological standpoint, but it means a lot in terms of how centralized things become culturally.
        pjmlp 8y ago
        Just like enterprises have their official .m2/settings.xml file pointing to the actual default Maven repository.

jrockway 8y ago

Does anyone know what the story is for binaries generated by things your project depends on? npm versions them and lets you invoke them from the project with "npx"; I wish something like this existed for go. (protoc-gen-go is the main thing I want; if your global version gets out of sync with the version in go.modules, the generated protobuf doesn't compile.) I am glad to see that the go team is working on cleaning up the tooling situation. I used gocode, then started using modules, found that the primary version was unmaintained, and had to switch to a different fork. I believe I also needed a different version of goimports for a while. Having all this tooling unified into the langserver maintained alongside go sounds wonderful. I hope other languages do the same thing.

dub 8y ago
For tools in golang like goimports the best practice at the moment is to have a tools.go with a "tools" build tag. See https://github.com/golang/go/issues/25922 https://github.com/golang/go/issues/25922 For non-go tools like the protoc compiler I don't think the golang tooling provides any help. A heavyweight option would be a build system like bazel
laumars 8y ago
On a vaguely related note, is there any decent documentation for writing a lang server. I've seen the official documentation by Microsoft and while it does seem a detailed reference, it's not great for someone who doesn't even know where to start in writing one.
- sbinet 8y ago
  this is being worked on: - https://github.com/golang/tools/blob/master/cmd/golsp/main.go https://github.com/golang/tools/blob/master/cmd/golsp/main.g...
  - laumars 8y ago
    I don’t want to create one for Go. It’s for a hobby language I’ve created. I only mentioned it because the op discussed Go’s language server.

plg 8y ago

I really want to get into Go, I like that it's opinionated, I like that it's compiled, I like that it's garbage-collected, I like that coroutines are built-in. My primary use case is scientific computing, both data processing and interactive visualization. I know Julia is an option as well. For reasons I don't want to bother getting into I dislike Python. Thoughts?

petre 8y ago
Dlang is better suited for that use case. Have you looked at mir?
llimllib 8y ago
It's not well-suited to scientific computing at all. I use and like Go for writing network servers and ETL processes. In a scientific context though, the type system is awkward in the extreme, there is essentially no library of modules, and the interactive visualization story is nonexistent. Python, R, Julia, even C++ would be better options IMO. (I'll clarify again: I like Go! I just think it's not well suited for this context)
- stcredzero 8y ago
  If one wants to be a pioneer and be one of the people who starts the movement, now is the time to jump in and make a name for yourself.
  - DannyBee 8y ago
    This is absolutely true. It's also a lot of work :)
  - jerf 8y ago
    I'd suggest "now" is actually a terrible time to try to make Go do scientific computing. With generics all-but-guaranteed to be incoming, but also not here yet, a lot of what you would do is going to be superceded in the forseeable future, on the same sort of time scale as your library's expected completion date. I'd say that even once Go has generics, that will simply take it from being an awful scientific programming language to a mediocre one. I don't really understand why there's a few people who seem to think it's a good idea to try to do their scientific computing in Go when there are so many better options for that that already exist. Post-generics, though, if you really insist, it will probably be the case that Go can be upgraded from "mediocre" to "tolerable" with a lot of library work. (Part of the "mediocre" is lacking libraries. That aspect can be fixed.) But it'll be hard to start that work without running generic code. Even if you assume the current documentation is the final specs, you still won't be able to guess the performance implications of anything you'd be blindly writing, and performance is very important for this sort of code.

treve 8y ago

One of the thing that's always been a bit off-putting about the Go community and leadership is that it seemed that when I came across things that I perceived as flaws in the tooling or language, I often felt told off and was made to feel that it's me who is wrong for a variety of reasons. Examples of this include GOPATH, the package infrastructure, error handling, lack of generics. Rob Pike disagrees so I must be wrong. It's kind of satisfying to see that the "Go way" wasn't the best way after all and that some of these things are actively being addressed. I hope it makes the community a bit more welcoming as well.

cgag 8y ago
Honestly I got the impression no one ever liked gopath.
- revskill 8y ago
  Because setting environment variables is something most devs like to avoid.
  - cgag 8y ago
    No, because I have like 50 projects in ~/src and suddenly have to keep some of my code in ~/src/go/src/GitHub.com/cgag and that annoys me.
    - cyphar 8y ago
      I have tried several times to fix this problem in the past 5 or 6 years, once with symlinks (which would break quite often -- it's unsurprising that Plan 9 developers wouldn't care too much for handling symlinks correctly :P). The current way I handle it is that $HOME/.local is my GOPATH, and $HOME/src links to $HOME/.local/src. So you can just have $HOME/src/github.com/bar/foo -- which is less ideal than just $HOME/src/foo, but it's something at least.
    - revel 8y ago
      it's not just that large numbers of projects are a hassle, they make it harder to develop projects in isolation

candiodari 8y ago

Maybe it's just me but I hate this option. Are we really, really trying to optimize the downloading of source code ? All 5 megabytes of it ? Why would you do that ? I regularly find myself in a position that this system would make impossible: I need a few custom changes in public available libraries. An extra method. A bugfix. What have you. This makes that impossible using the default method. What we want is consistent builds. That means a build that happens, with the same source, with the same ... every time, always, on everybody's workstation.

icholy 8y ago
Vendoring will still be supported.
- calcifer 8y ago
  How will that work with modules though? As in, how do you vendor a single library (that you want to make a fix to) while still building with module support? Right now, "module mode" is mutually exclusive with "vendor mode" and I don't think there are any plans to change that.
  - EdiX 8y ago
    > Right now, "module mode" is mutually exclusive with "vendor mode" and I don't think there are any plans to change that. This is not true. If you pass -mod=vendor, or place it in $GOFLAGS, go will build something outside of GOPATH, in module mode, using the vendor directory exclusively. I wish it was the default, but only because it encourages people to not use vendor directories (which is a bad habit), not because it can't be worked around easily.
    - calcifer 8y ago
      > in module mode, using the vendor directory exclusively But that's not what I'm asking. I want "go build" (with some flags) to use my fork (vendored if needed) for one library, and pick up all other dependencies from the module cache as usual. How do I that? Because this is already possible in a vendor-only world; I just edit the code in vendor.

dana321 8y ago

A centralized module index will be nice, i tend to end up searching github.com which leaves out all the other sites or locations that could have a module that solves my problem.

xyzzy_plugh 8y ago
I very much do not want this. godoc.org has decent search and I've never had any complaints.
- Merovius 8y ago
  Note that there's a difference between "central registry" and "central index". In particular, I don't see any downsides from what they are changing - godoc.org is already, for all intents and purposes, a central index, so I don't see how anything changes in that regard.

teacpde 8y ago

So the mirror will serve both the package and signed hashes, what if the package has been indexed in the central index but not yet signed by the notary?

emersion 8y ago
Maybe asking a hash to the notary will actually make the notary fetch and hash the package if it doesn't know about it yet? (Related: https://news.ycombinator.com/item?id=18717766 https://news.ycombinator.com/item?id=18717766)
- teacpde 8y ago
  That works, but slows down the mirror? I guess it is okay since it should only happen once.

spullara 8y ago

It is amazing that it took them this long to realize they were entirely wrong about how dependencies should be distributed. Since it was obvious from day 1 to most people from other ecosystems maybe they shouldn't have so much NIH. We'll likely get exceptions and generics soon as well. What a waste of time.

emersion 8y ago
The way dependencies are distributed (ie. downloaded) won't change a lot, and as they say the distributed model is very important. Maybe you mean how dependencies are installed by the go command? (As a side note, I don't think there are any plans for exceptions -- for good)
- spullara 8y ago
  They will be globally indexed, signed and versioned rather than downloaded at HEAD from a random repository. I'd say that is pretty different. Also, https://go.googlesource.com/proposal/+/master/design/go2draft.md https://go.googlesource.com/proposal/+/master/design/go2draf...
tracker1 8y ago
It's also important to get a number of other things worked out and discuss possibilities. The go process in terms of using a baseline directory with git sources seems to be a very pragmatic approach to start from. I started playing with node before npm was in the box. There were some competing ideas and eventually one won out. I think that having a system in place with the language may be a better option than a company with its' own motivations and needs separated from the language/runtime/platform. As to generics, I think you may well find generics in the future. I feel that most of the resistance was in order to better support core language features. I can't think of any languages (I'm no expert) that started with generics support, so I'd be surprised if this wasn't a go 2.x goal. For exceptions, I think that the go solution works. Similar to the callback interfaces in node, it puts errors in your face, which isn't a bad thing. I mainly contrast this with node, as it's another language/platform that has grown a LOT but also relatively recent. You can compare the progress of node, go and others to say python2/3, C#, Java and others. I think go progress has been great by comparison, and pragmatic choices have ruled out.

emersion 8y ago

It seems like there's a mistake in the diagram. The "notary → mirror" arrow should be replaced with a "notary → go command" one, because the go command shouldn't trust the mirror when it comes to cryptographic hashes.

pokstad 8y ago
I think the mirror can pass through the public signature provided by the notary. That cannot be spoofed if you have a trust chain for the notary to ensure the mirror has not tampered with the module.
teacpde 8y ago
The hashes are signed by the notary, go command can get the hashes from anywhere and be able to verified they are signed by the notary.

Boulth 8y ago

> One of the most important parts of the original design for go get was that it was decentralized That's one of the things I like in go most. Having decentralized packages via domains and URLs and then index them (godoc.org works very well), it mirrors the design of the Web. In contrast npm, Rust and others that are tightly coupled to one site feels like Google AMP - centralizing and hosting everything in one place.

almostdeadguy 8y ago
Cargo (in the nightly channel, but eventually will become stable) and npm can both use alternative package indexes and vendored dependencies, so they are not tightly coupled. Having a standardized package index as a database is what allows efficient dependency solving, which Go hasn't considered providing up until recently. > that anyone should be able to publish their code on any server, in contrast to central registries such as Perl’s CPAN, Java’s Maven, or Node’s NPM. Placing domain names at the start of the go get import space reused an existing decentralized system and avoided needing to solve anew the problems of deciding who can use which names. It also allowed companies to import code on private servers alongside code from public servers. lol statements like this are infuriating. Go is no more decentralized than those languages, unless you consider git hosting something people in standard practice do on their own (survey some of the top Go dependencies and see if that holds true). They simply lack a package index, but the cited language package managers both have those and allow you to host your own.