10 ms·

Go Packaging Proposal Process

171 points by zalmoxes 10y ago

leonardinius 10y ago
I wonder why I was able to edit document on first open. :o Seems to be solved quite quickly.
- zellyn 10y ago
  Heh. Thanks for the warning. I just changed my document (mentioned at the bottom) to comment-only too.
mfer 10y ago
Hi folks, My name is on that list and I work on one of the Go package managers (Glide - it's like npm, crates, composer, bundler, etc... but for Go... https://glide.sh https://glide.sh). If you have questions or input those who are working on this problem are listening.
- scrollaway 10y ago
  [This question is purely informational and doesn't directly relate to golang, but I want to ask someone who has been involved in domain-specific package management.] Why is the protocol side of package management not a solved problem already? At least Go reuses the git protocol to its advantage. But the general problem of package management applies to a lot of things. "There is a package with a name and a description. That package has versions, which can be greater/lesser than others. Those versions may have deltas from one-another. They contain one or more files. Those files must have their integrity checked. etc, etc..." -> Why does every single language reimplement that? Nevermind languages actually. Why does everyone reimplement that? Firefox/Chrome extensions, Android/ios app stores, a bazillion different game addons and app-specific stores... We have a mostly-identical problem being solved a million different ways. pypi, npmjs, glide, they should be running the same server software. It'd be a first step towards standardization and avoid a bunch of mistakes repeated every time a new solution comes out. Is anyone even trying to do something about this? (and before anyone says "do it yourself", a package management protocol is something I used to work on, not anymore. It keeps getting worse - it's not just a technical problem, also a political one.)
  - zalmoxes 10y ago
    Sam Boyer, one of the other people in the document wrote a nice article that dives into the complexity of dependency management. https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527#.pdtxz22hv https://medium.com/@sdboyer/so-you-want-to-write-a-package-m... Discussion https://news.ycombinator.com/item?id=11088125 https://news.ycombinator.com/item?id=11088125 Sam is currently working on a library for Go https://github.com/sdboyer/gps https://github.com/sdboyer/gps
    - zellyn 10y ago
      In fact, here's Sam and wycats (of Cargo) talking about how a generic library (obviously Go is not the best choice for that) would be nice: https://twitter.com/sdboyer/status/755036174016806912 https://twitter.com/sdboyer/status/755036174016806912
  - mfer 10y ago
    This is a good question. I have a few thoughts. First, different languages have subtleties that can make a different in package management. For example, the way nested dependencies can work in JavaScript, Go, and PHP are all different. Or, the way you can scan a codebase to determine things is difference. It would be great if different languages had virtually the same interface. That way metadata and interaction could be about the same. Like diving a car from different manufacturers. In many ways the different package managers are similar enough for programming languages were sorta there (but not all that well). Second, there is the cultural problem. Many languages are designed on islands from others. Reinventing things, trying different things, and wanting to be their own special flower. Sometimes this is good because a new language is not like the others and that can be a good thing. Other times you reinvent a fairly common wheel. I'm not aware of anyone solving this in a common way.
  - Drup 10y ago
    At least for the dependency solving part, a generic exchange format has emerged: CUDF (http://www.mancoosi.org/cudf/ http://www.mancoosi.org/cudf/). Several efficient solvers have been implemented that supports this input format. You can read about it more in http://opam.ocaml.org/doc/Specifying_Solver_Preferences.html#Yestherearedifferentversionsoftheuserpreferencelanguage http://opam.ocaml.org/doc/Specifying_Solver_Preferences.html... [1]: https://www.youtube.com/watch?v=E-gtFnbHcv0&list=UUP9g4dLR7xt6KzCYntNqYcw https://www.youtube.com/watch?v=E-gtFnbHcv0&list=UUP9g4dLR7x... [2]: http://ocaml.org/meetings/ocaml/2014/ocaml2014_17.pdf http://ocaml.org/meetings/ocaml/2014/ocaml2014_17.pdf This is at least used in OPAM (the OCaml package manager) and is also available for apt. If you are implementing a SAT solver for your package manager nowadays, you are doing it wrong. :)
- zzzcpan 10y ago
  Don't ignore nix/guix. Personally I don't think we need yet another package manager with painful external dependencies, non-reproducible builds, conflicts, lack of rollbacks, etc. We have that with "go get". What we need is to actually solve all those thing. Make sure there are no fragile external dependencies, but everything is in the package manager, even gcc, binutils. You get the idea.
  - mfer 10y ago
    Go package management needs to work with Git, Bzr, Hg, and Svn as it does today. Do you have suggestions on building a tool cross platform that works with packages in all of these that doesn't have external dependencies?
    - zzzcpan 10y ago
      It doesn't matter where the code is stored, certainly not for nix.
      - mfer 10y ago
        I'm sorry, I don't understand. `go get` or any package manager needs to be able to fetch the code remotely. Without interacting with the external binaries would mean the Go tool would need to have custom code to retrieve the codebase and interact with the VCS metadata such as tags. Are you suggesting custom Go code for that instead of using dependencies like git?
        zzzcpan 10y ago
        No, git would have to be a package in the same package manager, as a build dependency. Think OS package manager, but limited to what developer needs, i.e. just software, no services, no superuser privileges, etc.
        Vendan 10y ago
        Talk about feature creep! It sounds like you want a package manager and ecosystem that can install half an OS, just so you don't have to fuss with installing git on your own...
- ahacker15 10y ago
  Hi (I've tried Glide but it slowness and few bugs made me move to Govendor. Nothing personal, it just didn't work out that well for me.) A few questions: - Is the intention desincentive the use vendoring? - Is the intention to make a single solution, and then deprecate existing tools (Glide, Govendor and like)? - Do you have the intention to make a central registry for Go packages? Or the solution will use GitHub and like, as it does today? It would be not backward compatible, but the document don't make this clear.
  - mfer 10y ago
    I think the intention is to go figure it out. I don't know that we can say what it will be just yet.
- grosbisou 10y ago
  Dumb question but what are the reasons a solution like cargo or bundle (never used glide) would be rejected by the go committee?
  - themihai 10y ago
    I think cargo does too much. For example it has a bunch of meta information which could be extracted from the source code(i.e package documentation), the imports must be manually maintained in a config file(the Go way is to extract them from the source code automatically and snapshot the version). Shotly said I feel cargo is bloated, verbose and it wouldn't fit well with the Go philosophy(minimalist, convention over configuration). Why do you need both cargo.toml and cargo.lock? I also doubt Cargo would fix issues not fixed by the existing vendoring tools(e.g libraries that expose common types of different versions)
- SEJeff 10y ago
  What are some of the proposed solutions other than vendoring, which tends to be very difficult for large distributions such as Fedora. I was just reading a few days ago about some of the extreme challenges Fedora is having packaging the golang ecosystem.
  - mfer 10y ago
    Can you point me to those Fedora discussions? I'd be curious to learn more. I understand how operating system package managers deal with things differently than application developers. I've heard of problems with the way PHP apps are handling their dependencies these days and how Linux distro package managers aren't happy about it. I'm curious to know more in this case.
    - SEJeff 10y ago
      This email was from July 25th: https://lists.fedoraproject.org/archives/list/golang@lists.fedoraproject.org/thread/RD6TA5X3HR3XCPBLF7Z3E5D34PO6NEDW/ https://lists.fedoraproject.org/archives/list/golang@lists.f... But there have been several regarding the challenges Fedora has had as a large distribution while trying to package many Go apps, each with potentially hard requirements on different versions of the same golang package to import. Most of the conversation about it is in the golang sig (special interest group) that I've linked to their mail. May I suggest you subscribe, send an email out introducing yourself, and ask what issues we are facing? I suspect it would be extremely well received having someone working on a language specific solution asking on the Fedora go sig mailinglist. Just a thought.
howeyc 10y ago
I'm exited to see this progressing. I'm grateful for anyone who serves on the committee, as it could be a thankless job (some people may not like the agreed upon tool/process, but hopefully a small group that doesn't feel entirely disenfranchised). I really hope whatever they come up with is welcomed by the community and projects migrate to using it. Basically becoming the "Standard Go Way" to do things.
zellyn 10y ago
btw, I also wrote up a brief "State of Go Packaging" doc you might be interested in reading: https://docs.google.com/document/d/1M4Ldgtxr9vC8wpyTldQe5oUyGN9eb1CeUcJT3N5XaPk https://docs.google.com/document/d/1M4Ldgtxr9vC8wpyTldQe5oUy...
mmkx 10y ago
This is needed badly. I am a huge Golang advocate, but I refuse to use the "github.com" package naming/folder setup mess. It's just not elegant.
- jvehent 10y ago
  Projects I care about use a domain I control. It's not that hard to setup: https://jve.linuxwall.info/blog/index.php?post/2015/08/26/Hosting_Go_code_on_Github_with_custom_import_path https://jve.linuxwall.info/blog/index.php?post/2015/08/26/Ho... We even do it with just an S3 bucket for go.mozilla.org: https://go.mozilla.org/ https://go.mozilla.org/
  - mfer 10y ago
    I'm a fan of that as well. Had not considered using S3 for that. Do you know of a good writeup for handling "?go-get=1" paths differently from html paths in S3? I should likely know this but a quick pointer would be helpful.
    - jvehent 10y ago
      I have not looked into this. The S3 method is kinda experimental. It works for first-level packages but fails for nested packages. We need to look into setting up redirects for those. The best way to do this yourself is to run an instance of https://github.com/niemeyer/gopkg https://github.com/niemeyer/gopkg, but that's not #serverless anymore ;)
- dlsniper 10y ago
  You spelled Go incorrectly.
the_duke 10y ago
Looong overdue... With Go being a new language (1.0 in 2012), there is no excuse for not having had a solid pkg solution in 1.0. It's one of the things holding Go back, in my opinion, producing makeshift solutions like Glide and gopkg.in, which are only minimally adopted. I hope they also specify some mandatory versioning scheme (like npm with semver), because there are so many Go packages out there which don't even do any versioning!
- zzzcpan 10y ago
  But the solutions don't offer much over "go get", what would be the point to use them?
  - the_duke 10y ago
    I thoroughly disagree. The Golang ecosystem is in an incredibly weak state when you compare it with other modern languages. Major issues: -) Due to the lack of a proper package system, there's also no central pkg repository, which really hurts discoverabilty and growth of the ecosystem a lot. Just compare it to solutions like NPM for node, https://crates.io https://crates.io for Rust, Hackage for Haskell, etc... Github + https://godoc.org/ https://godoc.org/ is not a replacement for a proper package repository. -) Like I mentioned, way, way too many Go packages don't follow a proper release process with versioning and release notes. AT ALL. You often need to mirror repos for your own project, and closely track what upstream projects are doing. git clone + go get for your big projects just doesn't work. This is incredibly problematic when developing big projects. A proper pkg solution should enforce versioning and release best practices.
    - jvehent 10y ago
      Godoc is searchable: https://godoc.org/?q=json+web+token https://godoc.org/?q=json+web+token
      - mfer 10y ago
        How do you find the main vs the forks of a package? What about useful metadata to help you make a decision? Compare godoc.org projects to https://crates.io/crates/bitflags https://crates.io/crates/bitflags or https://packagist.org/packages/masterminds/html5 https://packagist.org/packages/masterminds/html5. Look at the metadata to help you make a decision. Just the docs for the tip of master are far from what people expect.
kaeshiwaza 10y ago
I hope you will find a "Go solution". I mean stupid simple (for most of the time).
- nameless912 10y ago
  Actually though, I'm hoping for exactly this. One of the things I love about Golang is that it's opinionated; gofmt forces your code to conform to a uniform style, the linters are top-notch, and the go command line tool builds your code in exactly one way. Some of the decisions made (the lack of generics sticks out) are frustrating at times, but the language authors having opinions stops many developer quibbles from becoming a problem. Now it's time to add packaging to this list. When we use go at work, everyone conforms to the same coding standards, everyone uses the same documentation tools, everyone deals with the same build system, and everyone (for the most part) writes cleaner, more uniform code because of it. I get why some developers hate Golang: it forces lots of its own opinions on how to do things on you, but in exchange it removes almost completely the bikeshed problem; everyone has to paint their bike shed the same color, so we can move onto the meat of solving the problems. I personally have been fine with go get and the vendoring tools people have created because we're using almost entirely internal libraries so getting the latest version of the master branch is usually the right choice, but I understand the need for a better package management solution. I just hope, like the parent commenter, that the go package solution is as opinionated as the rest of the language; give us an easy, sensible default so that the tooling gets the hell out of my way. I hate working on C++ and Java projects at my job because of how heavy the tooling often is (CMake and Maven, respectively), and that's just for the build system; forget autocomplete or linting or refactoring support unless you're in a commercial IDE. On the other hand, Go is simple enough that emacs is powerful enough to take care of every IDE feature I could ever want. Hell, the command line and nano is plenty to edit Go code, because the build system works exactly one way, and that is so incredibly important to me. So yeah, I want to see a good solution. But I will take a less complete solution over one that adds more complexity.
aikah 10y ago
Go ecosystem is a complete mess. I understand what Go maintainers tried to do, but Go isn't like Gnome lib,or Linux core lib where dependencies are stable. And the "just vendor" stuff pushed by some Go pundits is stupid. Clutter or Gtk don't vendor Gdk or Glib. So Go right now is about everybody reinventing the wheel, because you can't write reproducible build with 3rd party packages thus create an ecosystem with Go get. The solution ? deprecate Go get in favor of a new tool that supports at least fetching tags for packages. Project based build Tools (like Gb) work for apps, they don't work for libraries. Glide and co have a big problem : everybody is using a different package manager that is incompatible with others. The question isn't being solved yet because to solve it seriously it would require to change how the language works at first place.
- BarkMore 10y ago
  I understand that the tools will need to be changed, but what needs to be changed in the language? Because import paths are strings interpreted by the tools and not the language, there seems to be good separation between the language and issues of source code organization, packaging, etc.
- 0xmohit 10y ago
  > The question isn't being solved yet because to solve it seriously it would require to change how the language works at first place. And Go 1 compatibility promise [0] would prevent any change. In theory, you could define a GOPATH to install custom packages. You install half a dozen or so packages, and it's not trivial to determine what all you installed. A Go update arrives. Now some of those packages fail to update. Either spend hours debugging, reporting bugs, or rm -rf $GOPATH and reinstall everything. The language is wonderful in so many ways, but a number of seemingly silly things make it annoying. [0] https://golang.org/doc/go1compat https://golang.org/doc/go1compat
  - yiyus 10y ago
    > And Go 1 compatibility promise [0] would prevent any change. For example? I cannot see how the language itself could be the problem here.
0xdeadbeefbabe 10y ago
So what's wrong with vendoring all the deps and version controlling everything? Edit: Seriously, I'm not being a jerk.
- mfer 10y ago
  vendoring is dependency storage. Management, especially when you have transitive dependencies (deps of deps), is more than storage. We need to solve the non-storage related problems. Are you familiar with the diamond dependency problem? That's one example.
  - 0xdeadbeefbabe 10y ago
    I'm not familiar with the name, so I looked it up thank you. Program A depends on B and C which both depend on the stdlib (D). That seems pretty common. I actually have this going on now, but so far so good. I suppose D' (the new version of D) might break B while improving C and then I'd have a problem, right?
niftich 10y ago
I'm new to Go and I read some documents [1][2][3] on what 'vendoring' means but I still don't understand. What problem does it solve? Why is it needed? Is there an 'explain for a 5-year-old' version, or better yet, an 'explain for someone who is used to other packaging systems that may not be entirely elegant but work pretty okay most of the time, like Java's Maven or Node's npm' [1] https://engineeredweb.com/blog/2015/go-1.5-vendor-handling/ https://engineeredweb.com/blog/2015/go-1.5-vendor-handling/ [2] https://github.com/golang/go/wiki/PackageManagementTools https://github.com/golang/go/wiki/PackageManagementTools [3] https://docs.google.com/document/d/1rMoZ0rmpxw6dShalHnkuPtia6NkmB024MUa4T3SJeh8 https://docs.google.com/document/d/1rMoZ0rmpxw6dShalHnkuPtia...
- grey-area 10y ago
  It solves the problem of reproducible builds when using third party code. You can record state (v 1.3 of pkg foo) or check in code (vendor). It only really gets complex on big projects with lots of dependencies which themselves have dependencies.
  - niftich 10y ago
    After more searching, I found a StackOverflow question asking the same thing [1], and the answer appears to be: "Vendoring is the act of making your own copy of the 3rd party packages your project is using. Those copies are traditionally placed inside each project and then saved in the project repository." Being familiar with both Maven and npm, in those communities, this act of checking dependencies into source control is a controversial and hotly-debated topic, but in those package managers is more often than not (please bear with my point) considered an anti-pattern, because both of them provide solutions [2][3][4], like a project's declarative list of dependencies along with a baseline (but often, overridable) conflicting-version-resolution algorithm. Of course, the more I understand the issue, the more questions I have. Why does Go have multiple, competing package managers? Some, like Glide [5] (without judgements about all the others), encourage the type of declarative version management I quoted. Why does Go even ship with built-in package fetcher 'go get'? Why would anyone ever build a package fetcher for a publicly-released ecosystem that has no concept of versions or commit-IDs or content-adressed-hashes, or some other mechanism of tracking state to something that's as inherently mutable as a software package? I have no desire to point fingers, and I don't want to encourage bikeshedding, and I don't want to turn this sub-thread into an in-band debate, as I'm certain it's been done (over and over again) in the past, I just simply feel that 'go get' is fundamentally broken in its understanding of how people in the wild actually write software. I'm suddenly beginning to understand why there are concerted efforts [6][7][8] to address this issue in a community-led, officially-sanctioned way. [1] http://stackoverflow.com/questions/35109393/ http://stackoverflow.com/questions/35109393/ [2] http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html http://maven.apache.org/guides/introduction/introduction-to-... [3] https://github.com/ning/maven-dependency-versions-check-plugin https://github.com/ning/maven-dependency-versions-check-plug... [4] http://bytearcher.com/articles/semver-explained-why-theres-a-caret-in-my-package-json/ http://bytearcher.com/articles/semver-explained-why-theres-a... [5] https://github.com/Masterminds/glide https://github.com/Masterminds/glide [6] https://docs.google.com/document/d/18tNd8r5DV0yluCR7tPvkMTsWD_lYcRO7NhpNSDymRr8 https://docs.google.com/document/d/18tNd8r5DV0yluCR7tPvkMTsW... [7] https://docs.google.com/document/d/1M4Ldgtxr9vC8wpyTldQe5oUyGN9eb1CeUcJT3N5XaPk https://docs.google.com/document/d/1M4Ldgtxr9vC8wpyTldQe5oUy... [8] https://github.com/sdboyer/gps/wiki/Rationale https://github.com/sdboyer/gps/wiki/Rationale
randomsofr 10y ago
I hate using the url for the namespace.
- themihai 10y ago
  Do you prefer :: instead of / ? Btw go doesn't use URLs, just domain and path (e.g randomsofr.io/pkg/subpkg )
twblalock 10y ago
I never understood why Go conflates where something is (its Github url) with what something is (a specific version of a specific bit of code, what Maven calls an "artifact," and what lots of other solutions call a "package," or "module," or "gem", which has a version and never changes). This creates a lot of problems that simply don't need to be there: 1. Versioning dependencies is pretty much impossible unless you make your own copy of them, and keep them updated. Good dependency management solutions do not require you to "vendor" your dependencies. They enable reproducible builds by keeping every version of an artifact around. 2. Github could change its URL schema, and most Go projects would stop compiling. 3. The maintainer could delete the Github repo. 4. Source control, specifically Github, is tightly coupled with dependency management, and that should not be necessary. These are not unsolved problems. Maven solved every single one of these problems years ago. I don't expect people to use Maven to build Go applications, but clearly there is an opportunity to use Maven's good ideas and build a better dependency management system for Go. I have to say, in defense of Maven, that it is very robust: artifacts are mirrored in many places, including in your own local repo; versioning works well, including availability of older versions; and nobody can screw you by deleting package or changing its Github url. My company uses Maven for Node.js as well as Java, because we've had so many problems with NPM. I should also point out that the expectation that the core Go team should develop a dependency management system is a bit odd. The dependency management systems for Java, Ruby, Python, Node.js, and Perl were all developed by the community, not the core team. Some of them were later absorbed by the core team, but the point still stands.
- slrz 10y ago
  You, too, seem to misunderstand the concept of a Go import path. It's a string that identifies a package. It's not a URL. To provide for sane namespacing, people tend to use DNS names and URL paths to name their packages, just like they do with Java package names. There's no requirement that there's any correlation between the package import path and where you fetch that package's code from. Thus, your statement: 2. Github could change its URL schema, and most Go projects would stop compiling. is wrong. No single Go project that compiled before would now fail to do so. The common build tools use the import path to determine a path on the local filesystem where source or object code may be found. Neither a change of Github's URL schema, nor a repo deletion by some maintainer would cause your build to fail. The thing that would break is of course the re-fetching of source code from that repo, but that's kinda obvious, don't you think? You can't clone from a deleted Git repo. A common solution to that problem (language-independent) is to mirror the code somewhere else. Now, there's one single thing in the standard Go tooling universe that interacts with remote Git repos. It's 'go get', a small convenience wrapper around several VCS commands. It allows you to clone a remote repo to the right place on your disk while using the same syntax for Git, Mercurial and SVN alike. That's basically all it does. Yet, some people expect it to be some kind of elaborate package management system. It's not.
peterwwillis 10y ago
If you want to model a real package manager, don't do a comparison of existing tools, all of which have flaws. Start with the feature set of RPM, and then ask a professional software packager + developer what is missing. In order to have a feature-complete package manager (that is, it does everything you will concievably need) you need at least the following: - Unique file/object tracking - Upgrade & Rollback - Dynamic string interpolation (file paths, for example) - Identification of architecture and OS - ACL and extended permission support - Tracking of both source and binary dependencies & versions - Before & after script hooks for every single operation - Cryptographic signing & verification - Support for multiple central & distributed repositories This is, of course, completely different from a build system, but has a few features in common. You may not need a package manager at all; you may just need a feature-complete build system, which is how you end up with CPAN. (There is no Perl package manager, though PPM attempts one, and Carton is just a bundler)

171 points by zalmoxes 10y ago

leonardinius 10y ago

I wonder why I was able to edit document on first open. :o Seems to be solved quite quickly.

zellyn 10y ago
Heh. Thanks for the warning. I just changed my document (mentioned at the bottom) to comment-only too.

mfer 10y ago

Hi folks, My name is on that list and I work on one of the Go package managers (Glide - it's like npm, crates, composer, bundler, etc... but for Go... https://glide.sh https://glide.sh). If you have questions or input those who are working on this problem are listening.

scrollaway 10y ago
[This question is purely informational and doesn't directly relate to golang, but I want to ask someone who has been involved in domain-specific package management.] Why is the protocol side of package management not a solved problem already? At least Go reuses the git protocol to its advantage. But the general problem of package management applies to a lot of things. "There is a package with a name and a description. That package has versions, which can be greater/lesser than others. Those versions may have deltas from one-another. They contain one or more files. Those files must have their integrity checked. etc, etc..." -> Why does every single language reimplement that? Nevermind languages actually. Why does everyone reimplement that? Firefox/Chrome extensions, Android/ios app stores, a bazillion different game addons and app-specific stores... We have a mostly-identical problem being solved a million different ways. pypi, npmjs, glide, they should be running the same server software. It'd be a first step towards standardization and avoid a bunch of mistakes repeated every time a new solution comes out. Is anyone even trying to do something about this? (and before anyone says "do it yourself", a package management protocol is something I used to work on, not anymore. It keeps getting worse - it's not just a technical problem, also a political one.)
- zalmoxes 10y ago
  Sam Boyer, one of the other people in the document wrote a nice article that dives into the complexity of dependency management. https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527#.pdtxz22hv https://medium.com/@sdboyer/so-you-want-to-write-a-package-m... Discussion https://news.ycombinator.com/item?id=11088125 https://news.ycombinator.com/item?id=11088125 Sam is currently working on a library for Go https://github.com/sdboyer/gps https://github.com/sdboyer/gps
  - zellyn 10y ago
    In fact, here's Sam and wycats (of Cargo) talking about how a generic library (obviously Go is not the best choice for that) would be nice: https://twitter.com/sdboyer/status/755036174016806912 https://twitter.com/sdboyer/status/755036174016806912
- mfer 10y ago
  This is a good question. I have a few thoughts. First, different languages have subtleties that can make a different in package management. For example, the way nested dependencies can work in JavaScript, Go, and PHP are all different. Or, the way you can scan a codebase to determine things is difference. It would be great if different languages had virtually the same interface. That way metadata and interaction could be about the same. Like diving a car from different manufacturers. In many ways the different package managers are similar enough for programming languages were sorta there (but not all that well). Second, there is the cultural problem. Many languages are designed on islands from others. Reinventing things, trying different things, and wanting to be their own special flower. Sometimes this is good because a new language is not like the others and that can be a good thing. Other times you reinvent a fairly common wheel. I'm not aware of anyone solving this in a common way.
- Drup 10y ago
  At least for the dependency solving part, a generic exchange format has emerged: CUDF (http://www.mancoosi.org/cudf/ http://www.mancoosi.org/cudf/). Several efficient solvers have been implemented that supports this input format. You can read about it more in http://opam.ocaml.org/doc/Specifying_Solver_Preferences.html#Yestherearedifferentversionsoftheuserpreferencelanguage http://opam.ocaml.org/doc/Specifying_Solver_Preferences.html... [1]: https://www.youtube.com/watch?v=E-gtFnbHcv0&list=UUP9g4dLR7xt6KzCYntNqYcw https://www.youtube.com/watch?v=E-gtFnbHcv0&list=UUP9g4dLR7x... [2]: http://ocaml.org/meetings/ocaml/2014/ocaml2014_17.pdf http://ocaml.org/meetings/ocaml/2014/ocaml2014_17.pdf This is at least used in OPAM (the OCaml package manager) and is also available for apt. If you are implementing a SAT solver for your package manager nowadays, you are doing it wrong. :)
zzzcpan 10y ago
Don't ignore nix/guix. Personally I don't think we need yet another package manager with painful external dependencies, non-reproducible builds, conflicts, lack of rollbacks, etc. We have that with "go get". What we need is to actually solve all those thing. Make sure there are no fragile external dependencies, but everything is in the package manager, even gcc, binutils. You get the idea.
- mfer 10y ago
  Go package management needs to work with Git, Bzr, Hg, and Svn as it does today. Do you have suggestions on building a tool cross platform that works with packages in all of these that doesn't have external dependencies?
  - zzzcpan 10y ago
    It doesn't matter where the code is stored, certainly not for nix.
    - mfer 10y ago
      I'm sorry, I don't understand. `go get` or any package manager needs to be able to fetch the code remotely. Without interacting with the external binaries would mean the Go tool would need to have custom code to retrieve the codebase and interact with the VCS metadata such as tags. Are you suggesting custom Go code for that instead of using dependencies like git?
      - zzzcpan 10y ago
        No, git would have to be a package in the same package manager, as a build dependency. Think OS package manager, but limited to what developer needs, i.e. just software, no services, no superuser privileges, etc.
        Vendan 10y ago
        Talk about feature creep! It sounds like you want a package manager and ecosystem that can install half an OS, just so you don't have to fuss with installing git on your own...
ahacker15 10y ago
Hi (I've tried Glide but it slowness and few bugs made me move to Govendor. Nothing personal, it just didn't work out that well for me.) A few questions: - Is the intention desincentive the use vendoring? - Is the intention to make a single solution, and then deprecate existing tools (Glide, Govendor and like)? - Do you have the intention to make a central registry for Go packages? Or the solution will use GitHub and like, as it does today? It would be not backward compatible, but the document don't make this clear.
- mfer 10y ago
  I think the intention is to go figure it out. I don't know that we can say what it will be just yet.
grosbisou 10y ago
Dumb question but what are the reasons a solution like cargo or bundle (never used glide) would be rejected by the go committee?
- themihai 10y ago
  I think cargo does too much. For example it has a bunch of meta information which could be extracted from the source code(i.e package documentation), the imports must be manually maintained in a config file(the Go way is to extract them from the source code automatically and snapshot the version). Shotly said I feel cargo is bloated, verbose and it wouldn't fit well with the Go philosophy(minimalist, convention over configuration). Why do you need both cargo.toml and cargo.lock? I also doubt Cargo would fix issues not fixed by the existing vendoring tools(e.g libraries that expose common types of different versions)
SEJeff 10y ago
What are some of the proposed solutions other than vendoring, which tends to be very difficult for large distributions such as Fedora. I was just reading a few days ago about some of the extreme challenges Fedora is having packaging the golang ecosystem.
- mfer 10y ago
  Can you point me to those Fedora discussions? I'd be curious to learn more. I understand how operating system package managers deal with things differently than application developers. I've heard of problems with the way PHP apps are handling their dependencies these days and how Linux distro package managers aren't happy about it. I'm curious to know more in this case.
  - SEJeff 10y ago
    This email was from July 25th: https://lists.fedoraproject.org/archives/list/golang@lists.fedoraproject.org/thread/RD6TA5X3HR3XCPBLF7Z3E5D34PO6NEDW/ https://lists.fedoraproject.org/archives/list/golang@lists.f... But there have been several regarding the challenges Fedora has had as a large distribution while trying to package many Go apps, each with potentially hard requirements on different versions of the same golang package to import. Most of the conversation about it is in the golang sig (special interest group) that I've linked to their mail. May I suggest you subscribe, send an email out introducing yourself, and ask what issues we are facing? I suspect it would be extremely well received having someone working on a language specific solution asking on the Fedora go sig mailinglist. Just a thought.

howeyc 10y ago

I'm exited to see this progressing. I'm grateful for anyone who serves on the committee, as it could be a thankless job (some people may not like the agreed upon tool/process, but hopefully a small group that doesn't feel entirely disenfranchised). I really hope whatever they come up with is welcomed by the community and projects migrate to using it. Basically becoming the "Standard Go Way" to do things.

zellyn 10y ago

btw, I also wrote up a brief "State of Go Packaging" doc you might be interested in reading: https://docs.google.com/document/d/1M4Ldgtxr9vC8wpyTldQe5oUyGN9eb1CeUcJT3N5XaPk https://docs.google.com/document/d/1M4Ldgtxr9vC8wpyTldQe5oUy...

mmkx 10y ago

This is needed badly. I am a huge Golang advocate, but I refuse to use the "github.com" package naming/folder setup mess. It's just not elegant.

jvehent 10y ago
Projects I care about use a domain I control. It's not that hard to setup: https://jve.linuxwall.info/blog/index.php?post/2015/08/26/Hosting_Go_code_on_Github_with_custom_import_path https://jve.linuxwall.info/blog/index.php?post/2015/08/26/Ho... We even do it with just an S3 bucket for go.mozilla.org: https://go.mozilla.org/ https://go.mozilla.org/
- mfer 10y ago
  I'm a fan of that as well. Had not considered using S3 for that. Do you know of a good writeup for handling "?go-get=1" paths differently from html paths in S3? I should likely know this but a quick pointer would be helpful.
  - jvehent 10y ago
    I have not looked into this. The S3 method is kinda experimental. It works for first-level packages but fails for nested packages. We need to look into setting up redirects for those. The best way to do this yourself is to run an instance of https://github.com/niemeyer/gopkg https://github.com/niemeyer/gopkg, but that's not #serverless anymore ;)
dlsniper 10y ago
You spelled Go incorrectly.

the_duke 10y ago

Looong overdue... With Go being a new language (1.0 in 2012), there is no excuse for not having had a solid pkg solution in 1.0. It's one of the things holding Go back, in my opinion, producing makeshift solutions like Glide and gopkg.in, which are only minimally adopted. I hope they also specify some mandatory versioning scheme (like npm with semver), because there are so many Go packages out there which don't even do any versioning!

zzzcpan 10y ago
But the solutions don't offer much over "go get", what would be the point to use them?
- the_duke 10y ago
  I thoroughly disagree. The Golang ecosystem is in an incredibly weak state when you compare it with other modern languages. Major issues: -) Due to the lack of a proper package system, there's also no central pkg repository, which really hurts discoverabilty and growth of the ecosystem a lot. Just compare it to solutions like NPM for node, https://crates.io https://crates.io for Rust, Hackage for Haskell, etc... Github + https://godoc.org/ https://godoc.org/ is not a replacement for a proper package repository. -) Like I mentioned, way, way too many Go packages don't follow a proper release process with versioning and release notes. AT ALL. You often need to mirror repos for your own project, and closely track what upstream projects are doing. git clone + go get for your big projects just doesn't work. This is incredibly problematic when developing big projects. A proper pkg solution should enforce versioning and release best practices.
  - jvehent 10y ago
    Godoc is searchable: https://godoc.org/?q=json+web+token https://godoc.org/?q=json+web+token
    - mfer 10y ago
      How do you find the main vs the forks of a package? What about useful metadata to help you make a decision? Compare godoc.org projects to https://crates.io/crates/bitflags https://crates.io/crates/bitflags or https://packagist.org/packages/masterminds/html5 https://packagist.org/packages/masterminds/html5. Look at the metadata to help you make a decision. Just the docs for the tip of master are far from what people expect.

kaeshiwaza 10y ago

I hope you will find a "Go solution". I mean stupid simple (for most of the time).

nameless912 10y ago
Actually though, I'm hoping for exactly this. One of the things I love about Golang is that it's opinionated; gofmt forces your code to conform to a uniform style, the linters are top-notch, and the go command line tool builds your code in exactly one way. Some of the decisions made (the lack of generics sticks out) are frustrating at times, but the language authors having opinions stops many developer quibbles from becoming a problem. Now it's time to add packaging to this list. When we use go at work, everyone conforms to the same coding standards, everyone uses the same documentation tools, everyone deals with the same build system, and everyone (for the most part) writes cleaner, more uniform code because of it. I get why some developers hate Golang: it forces lots of its own opinions on how to do things on you, but in exchange it removes almost completely the bikeshed problem; everyone has to paint their bike shed the same color, so we can move onto the meat of solving the problems. I personally have been fine with go get and the vendoring tools people have created because we're using almost entirely internal libraries so getting the latest version of the master branch is usually the right choice, but I understand the need for a better package management solution. I just hope, like the parent commenter, that the go package solution is as opinionated as the rest of the language; give us an easy, sensible default so that the tooling gets the hell out of my way. I hate working on C++ and Java projects at my job because of how heavy the tooling often is (CMake and Maven, respectively), and that's just for the build system; forget autocomplete or linting or refactoring support unless you're in a commercial IDE. On the other hand, Go is simple enough that emacs is powerful enough to take care of every IDE feature I could ever want. Hell, the command line and nano is plenty to edit Go code, because the build system works exactly one way, and that is so incredibly important to me. So yeah, I want to see a good solution. But I will take a less complete solution over one that adds more complexity.

aikah 10y ago

Go ecosystem is a complete mess. I understand what Go maintainers tried to do, but Go isn't like Gnome lib,or Linux core lib where dependencies are stable. And the "just vendor" stuff pushed by some Go pundits is stupid. Clutter or Gtk don't vendor Gdk or Glib. So Go right now is about everybody reinventing the wheel, because you can't write reproducible build with 3rd party packages thus create an ecosystem with Go get. The solution ? deprecate Go get in favor of a new tool that supports at least fetching tags for packages. Project based build Tools (like Gb) work for apps, they don't work for libraries. Glide and co have a big problem : everybody is using a different package manager that is incompatible with others. The question isn't being solved yet because to solve it seriously it would require to change how the language works at first place.

BarkMore 10y ago
I understand that the tools will need to be changed, but what needs to be changed in the language? Because import paths are strings interpreted by the tools and not the language, there seems to be good separation between the language and issues of source code organization, packaging, etc.
0xmohit 10y ago
> The question isn't being solved yet because to solve it seriously it would require to change how the language works at first place. And Go 1 compatibility promise [0] would prevent any change. In theory, you could define a GOPATH to install custom packages. You install half a dozen or so packages, and it's not trivial to determine what all you installed. A Go update arrives. Now some of those packages fail to update. Either spend hours debugging, reporting bugs, or rm -rf $GOPATH and reinstall everything. The language is wonderful in so many ways, but a number of seemingly silly things make it annoying. [0] https://golang.org/doc/go1compat https://golang.org/doc/go1compat
- yiyus 10y ago
  > And Go 1 compatibility promise [0] would prevent any change. For example? I cannot see how the language itself could be the problem here.

0xdeadbeefbabe 10y ago

So what's wrong with vendoring all the deps and version controlling everything? Edit: Seriously, I'm not being a jerk.

mfer 10y ago
vendoring is dependency storage. Management, especially when you have transitive dependencies (deps of deps), is more than storage. We need to solve the non-storage related problems. Are you familiar with the diamond dependency problem? That's one example.
- 0xdeadbeefbabe 10y ago
  I'm not familiar with the name, so I looked it up thank you. Program A depends on B and C which both depend on the stdlib (D). That seems pretty common. I actually have this going on now, but so far so good. I suppose D' (the new version of D) might break B while improving C and then I'd have a problem, right?

niftich 10y ago

I'm new to Go and I read some documents [1][2][3] on what 'vendoring' means but I still don't understand. What problem does it solve? Why is it needed? Is there an 'explain for a 5-year-old' version, or better yet, an 'explain for someone who is used to other packaging systems that may not be entirely elegant but work pretty okay most of the time, like Java's Maven or Node's npm' [1] https://engineeredweb.com/blog/2015/go-1.5-vendor-handling/ https://engineeredweb.com/blog/2015/go-1.5-vendor-handling/ [2] https://github.com/golang/go/wiki/PackageManagementTools https://github.com/golang/go/wiki/PackageManagementTools [3] https://docs.google.com/document/d/1rMoZ0rmpxw6dShalHnkuPtia6NkmB024MUa4T3SJeh8 https://docs.google.com/document/d/1rMoZ0rmpxw6dShalHnkuPtia...

grey-area 10y ago
It solves the problem of reproducible builds when using third party code. You can record state (v 1.3 of pkg foo) or check in code (vendor). It only really gets complex on big projects with lots of dependencies which themselves have dependencies.
- niftich 10y ago
  After more searching, I found a StackOverflow question asking the same thing [1], and the answer appears to be: "Vendoring is the act of making your own copy of the 3rd party packages your project is using. Those copies are traditionally placed inside each project and then saved in the project repository." Being familiar with both Maven and npm, in those communities, this act of checking dependencies into source control is a controversial and hotly-debated topic, but in those package managers is more often than not (please bear with my point) considered an anti-pattern, because both of them provide solutions [2][3][4], like a project's declarative list of dependencies along with a baseline (but often, overridable) conflicting-version-resolution algorithm. Of course, the more I understand the issue, the more questions I have. Why does Go have multiple, competing package managers? Some, like Glide [5] (without judgements about all the others), encourage the type of declarative version management I quoted. Why does Go even ship with built-in package fetcher 'go get'? Why would anyone ever build a package fetcher for a publicly-released ecosystem that has no concept of versions or commit-IDs or content-adressed-hashes, or some other mechanism of tracking state to something that's as inherently mutable as a software package? I have no desire to point fingers, and I don't want to encourage bikeshedding, and I don't want to turn this sub-thread into an in-band debate, as I'm certain it's been done (over and over again) in the past, I just simply feel that 'go get' is fundamentally broken in its understanding of how people in the wild actually write software. I'm suddenly beginning to understand why there are concerted efforts [6][7][8] to address this issue in a community-led, officially-sanctioned way. [1] http://stackoverflow.com/questions/35109393/ http://stackoverflow.com/questions/35109393/ [2] http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html http://maven.apache.org/guides/introduction/introduction-to-... [3] https://github.com/ning/maven-dependency-versions-check-plugin https://github.com/ning/maven-dependency-versions-check-plug... [4] http://bytearcher.com/articles/semver-explained-why-theres-a-caret-in-my-package-json/ http://bytearcher.com/articles/semver-explained-why-theres-a... [5] https://github.com/Masterminds/glide https://github.com/Masterminds/glide [6] https://docs.google.com/document/d/18tNd8r5DV0yluCR7tPvkMTsWD_lYcRO7NhpNSDymRr8 https://docs.google.com/document/d/18tNd8r5DV0yluCR7tPvkMTsW... [7] https://docs.google.com/document/d/1M4Ldgtxr9vC8wpyTldQe5oUyGN9eb1CeUcJT3N5XaPk https://docs.google.com/document/d/1M4Ldgtxr9vC8wpyTldQe5oUy... [8] https://github.com/sdboyer/gps/wiki/Rationale https://github.com/sdboyer/gps/wiki/Rationale

randomsofr 10y ago

I hate using the url for the namespace.

themihai 10y ago
Do you prefer :: instead of / ? Btw go doesn't use URLs, just domain and path (e.g randomsofr.io/pkg/subpkg )

twblalock 10y ago

I never understood why Go conflates where something is (its Github url) with what something is (a specific version of a specific bit of code, what Maven calls an "artifact," and what lots of other solutions call a "package," or "module," or "gem", which has a version and never changes). This creates a lot of problems that simply don't need to be there: 1. Versioning dependencies is pretty much impossible unless you make your own copy of them, and keep them updated. Good dependency management solutions do not require you to "vendor" your dependencies. They enable reproducible builds by keeping every version of an artifact around. 2. Github could change its URL schema, and most Go projects would stop compiling. 3. The maintainer could delete the Github repo. 4. Source control, specifically Github, is tightly coupled with dependency management, and that should not be necessary. These are not unsolved problems. Maven solved every single one of these problems years ago. I don't expect people to use Maven to build Go applications, but clearly there is an opportunity to use Maven's good ideas and build a better dependency management system for Go. I have to say, in defense of Maven, that it is very robust: artifacts are mirrored in many places, including in your own local repo; versioning works well, including availability of older versions; and nobody can screw you by deleting package or changing its Github url. My company uses Maven for Node.js as well as Java, because we've had so many problems with NPM. I should also point out that the expectation that the core Go team should develop a dependency management system is a bit odd. The dependency management systems for Java, Ruby, Python, Node.js, and Perl were all developed by the community, not the core team. Some of them were later absorbed by the core team, but the point still stands.

slrz 10y ago
You, too, seem to misunderstand the concept of a Go import path. It's a string that identifies a package. It's not a URL. To provide for sane namespacing, people tend to use DNS names and URL paths to name their packages, just like they do with Java package names. There's no requirement that there's any correlation between the package import path and where you fetch that package's code from. Thus, your statement: 2. Github could change its URL schema, and most Go projects would stop compiling. is wrong. No single Go project that compiled before would now fail to do so. The common build tools use the import path to determine a path on the local filesystem where source or object code may be found. Neither a change of Github's URL schema, nor a repo deletion by some maintainer would cause your build to fail. The thing that would break is of course the re-fetching of source code from that repo, but that's kinda obvious, don't you think? You can't clone from a deleted Git repo. A common solution to that problem (language-independent) is to mirror the code somewhere else. Now, there's one single thing in the standard Go tooling universe that interacts with remote Git repos. It's 'go get', a small convenience wrapper around several VCS commands. It allows you to clone a remote repo to the right place on your disk while using the same syntax for Git, Mercurial and SVN alike. That's basically all it does. Yet, some people expect it to be some kind of elaborate package management system. It's not.

peterwwillis 10y ago

If you want to model a real package manager, don't do a comparison of existing tools, all of which have flaws. Start with the feature set of RPM, and then ask a professional software packager + developer what is missing. In order to have a feature-complete package manager (that is, it does everything you will concievably need) you need at least the following: - Unique file/object tracking - Upgrade & Rollback - Dynamic string interpolation (file paths, for example) - Identification of architecture and OS - ACL and extended permission support - Tracking of both source and binary dependencies & versions - Before & after script hooks for every single operation - Cryptographic signing & verification - Support for multiple central & distributed repositories This is, of course, completely different from a build system, but has a few features in common. You may not need a package manager at all; you may just need a feature-complete build system, which is how you end up with CPAN. (There is no Perl package manager, though PPM attempts one, and Carton is just a bundler)