6 ms·

Hi folks, My name is on that list and I work on one of the Go package managers (Glide - it's like npm, crates, composer, bundler, etc... but for Go... https://g

by mfer 10y ago

Hi folks, My name is on that list and I work on one of the Go package managers (Glide - it's like npm, crates, composer, bundler, etc... but for Go... https://glide.sh https://glide.sh). If you have questions or input those who are working on this problem are listening.

scrollaway 10y ago
[This question is purely informational and doesn't directly relate to golang, but I want to ask someone who has been involved in domain-specific package management.] Why is the protocol side of package management not a solved problem already? At least Go reuses the git protocol to its advantage. But the general problem of package management applies to a lot of things. "There is a package with a name and a description. That package has versions, which can be greater/lesser than others. Those versions may have deltas from one-another. They contain one or more files. Those files must have their integrity checked. etc, etc..." -> Why does every single language reimplement that? Nevermind languages actually. Why does everyone reimplement that? Firefox/Chrome extensions, Android/ios app stores, a bazillion different game addons and app-specific stores... We have a mostly-identical problem being solved a million different ways. pypi, npmjs, glide, they should be running the same server software. It'd be a first step towards standardization and avoid a bunch of mistakes repeated every time a new solution comes out. Is anyone even trying to do something about this? (and before anyone says "do it yourself", a package management protocol is something I used to work on, not anymore. It keeps getting worse - it's not just a technical problem, also a political one.)
- zalmoxes 10y ago
  Sam Boyer, one of the other people in the document wrote a nice article that dives into the complexity of dependency management. https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527#.pdtxz22hv https://medium.com/@sdboyer/so-you-want-to-write-a-package-m... Discussion https://news.ycombinator.com/item?id=11088125 https://news.ycombinator.com/item?id=11088125 Sam is currently working on a library for Go https://github.com/sdboyer/gps https://github.com/sdboyer/gps
  - zellyn 10y ago
    In fact, here's Sam and wycats (of Cargo) talking about how a generic library (obviously Go is not the best choice for that) would be nice: https://twitter.com/sdboyer/status/755036174016806912 https://twitter.com/sdboyer/status/755036174016806912
- mfer 10y ago
  This is a good question. I have a few thoughts. First, different languages have subtleties that can make a different in package management. For example, the way nested dependencies can work in JavaScript, Go, and PHP are all different. Or, the way you can scan a codebase to determine things is difference. It would be great if different languages had virtually the same interface. That way metadata and interaction could be about the same. Like diving a car from different manufacturers. In many ways the different package managers are similar enough for programming languages were sorta there (but not all that well). Second, there is the cultural problem. Many languages are designed on islands from others. Reinventing things, trying different things, and wanting to be their own special flower. Sometimes this is good because a new language is not like the others and that can be a good thing. Other times you reinvent a fairly common wheel. I'm not aware of anyone solving this in a common way.
- Drup 10y ago
  At least for the dependency solving part, a generic exchange format has emerged: CUDF (http://www.mancoosi.org/cudf/ http://www.mancoosi.org/cudf/). Several efficient solvers have been implemented that supports this input format. You can read about it more in http://opam.ocaml.org/doc/Specifying_Solver_Preferences.html#Yestherearedifferentversionsoftheuserpreferencelanguage http://opam.ocaml.org/doc/Specifying_Solver_Preferences.html... [1]: https://www.youtube.com/watch?v=E-gtFnbHcv0&list=UUP9g4dLR7xt6KzCYntNqYcw https://www.youtube.com/watch?v=E-gtFnbHcv0&list=UUP9g4dLR7x... [2]: http://ocaml.org/meetings/ocaml/2014/ocaml2014_17.pdf http://ocaml.org/meetings/ocaml/2014/ocaml2014_17.pdf This is at least used in OPAM (the OCaml package manager) and is also available for apt. If you are implementing a SAT solver for your package manager nowadays, you are doing it wrong. :)
- Rafert 10y ago
  There's Molinillo used in CocoaPods and Bundler: https://github.com/CocoaPods/Molinillo https://github.com/CocoaPods/Molinillo
  - mfer 10y ago
    When I wrote "Seeing How The Other Side Lives, A Package Manager Overview For Go Developers"[1] I noted CocoaPods. Was not aware of Molinillo though. Thanks for sharing. [1] http://engineeredweb.com/blog/2015/pkg-mgr-overview/ http://engineeredweb.com/blog/2015/pkg-mgr-overview/
- shurcooL 10y ago
  > That package has versions This is a statement that doesn't always apply. For example, some packages don't have versions, they only have a single version: the latest version. It's a moving target, but it's there. Older versions are just an unimportant and unsupported implementation side-effect. Of course, many packages do have versions, but this one factor makes a big difference on how you look at vendoring.
  - peq 10y ago
    Without versions you don't have package management, you have a download script.
    - thawkins 10y ago
      Agreed, and you also cant guarentee reproducable builds. Builds that are dependent on when you compile them, even if no changes have been made due to ephemeral dependances are a nightmare.
  - vec 10y ago
    First, your described software still has versions. Each daily snapshot or git commit or whatever is a new version. There's no reason you can't put an auto-incrementing version number on each release and call that your version number. You can even stick a "0." on the front of it and be SemVer complant. Second, just because some people do this doesn't mean it's a good idea. If you're writing a full application and just want to distribute it, do whatever you want with your versioning (although using a dependency management tool to distribute your webapp or CLI tool is probably not a great idea in the first place), but if you are writing something to be included in another project please use a sane versioning system. Anything else is selfish and disrespectful of your users.
    - bassislife 10y ago
      In Go, each version corresponds to a given import/Pkg path. If you want to change version (major), you need to create a new package. The advantage is that you do not have to download a manifest such as package.json or whatever. It also provides a constraint on library authors to provide a backward compatible API for their published package. Eventually, I guess we could use go/types to enforce/check the API backward compatibility requirement. The main issues for now are: * make sure that packages importing a same vendored one are able to use the same latest minor version in use. * allow a project based approach to go get for people who need reproducible builds (or alternatively allow for multiple $GOPATH and make being able to switch between them easy)
- bbatha 10y ago
  > At least Go reuses the git protocol to its advantage. I think this is a mistake. Packaging should come from a central server (that can be overriden with a mirror). Allowing downloads from adhoc git repositories is a real pain. First, it makes mirroring the package repository impossible. Second, it makes it difficult for companies to disallow certain packages in a centralized fashion. Third, it makes distro packaging a royal pain. And finally Git simply wasn't built for the this kind of work, compressed tarballs happen to work great for this purpose. Git's mutable history also presents other challenges for pinning to specific versions of packages. Simply put, sometimes a central authority has advantages. That all said, vendoring alleviates most of these problems for apps and some of them for libraries. Reusing tools does make it a little easier to get started with too. I'd hate to see these features removed in whatever go get becomes in the long term.
zzzcpan 10y ago
Don't ignore nix/guix. Personally I don't think we need yet another package manager with painful external dependencies, non-reproducible builds, conflicts, lack of rollbacks, etc. We have that with "go get". What we need is to actually solve all those thing. Make sure there are no fragile external dependencies, but everything is in the package manager, even gcc, binutils. You get the idea.
- mfer 10y ago
  Go package management needs to work with Git, Bzr, Hg, and Svn as it does today. Do you have suggestions on building a tool cross platform that works with packages in all of these that doesn't have external dependencies?
  - zzzcpan 10y ago
    It doesn't matter where the code is stored, certainly not for nix.
    - mfer 10y ago
      I'm sorry, I don't understand. `go get` or any package manager needs to be able to fetch the code remotely. Without interacting with the external binaries would mean the Go tool would need to have custom code to retrieve the codebase and interact with the VCS metadata such as tags. Are you suggesting custom Go code for that instead of using dependencies like git?
      - zzzcpan 10y ago
        No, git would have to be a package in the same package manager, as a build dependency. Think OS package manager, but limited to what developer needs, i.e. just software, no services, no superuser privileges, etc.
        Vendan 10y ago
        Talk about feature creep! It sounds like you want a package manager and ecosystem that can install half an OS, just so you don't have to fuss with installing git on your own...
ahacker15 10y ago
Hi (I've tried Glide but it slowness and few bugs made me move to Govendor. Nothing personal, it just didn't work out that well for me.) A few questions: - Is the intention desincentive the use vendoring? - Is the intention to make a single solution, and then deprecate existing tools (Glide, Govendor and like)? - Do you have the intention to make a central registry for Go packages? Or the solution will use GitHub and like, as it does today? It would be not backward compatible, but the document don't make this clear.
- mfer 10y ago
  I think the intention is to go figure it out. I don't know that we can say what it will be just yet.
grosbisou 10y ago
Dumb question but what are the reasons a solution like cargo or bundle (never used glide) would be rejected by the go committee?
- themihai 10y ago
  I think cargo does too much. For example it has a bunch of meta information which could be extracted from the source code(i.e package documentation), the imports must be manually maintained in a config file(the Go way is to extract them from the source code automatically and snapshot the version). Shotly said I feel cargo is bloated, verbose and it wouldn't fit well with the Go philosophy(minimalist, convention over configuration). Why do you need both cargo.toml and cargo.lock? I also doubt Cargo would fix issues not fixed by the existing vendoring tools(e.g libraries that expose common types of different versions)
SEJeff 10y ago
What are some of the proposed solutions other than vendoring, which tends to be very difficult for large distributions such as Fedora. I was just reading a few days ago about some of the extreme challenges Fedora is having packaging the golang ecosystem.
- mfer 10y ago
  Can you point me to those Fedora discussions? I'd be curious to learn more. I understand how operating system package managers deal with things differently than application developers. I've heard of problems with the way PHP apps are handling their dependencies these days and how Linux distro package managers aren't happy about it. I'm curious to know more in this case.
  - SEJeff 10y ago
    This email was from July 25th: https://lists.fedoraproject.org/archives/list/golang@lists.fedoraproject.org/thread/RD6TA5X3HR3XCPBLF7Z3E5D34PO6NEDW/ https://lists.fedoraproject.org/archives/list/golang@lists.f... But there have been several regarding the challenges Fedora has had as a large distribution while trying to package many Go apps, each with potentially hard requirements on different versions of the same golang package to import. Most of the conversation about it is in the golang sig (special interest group) that I've linked to their mail. May I suggest you subscribe, send an email out introducing yourself, and ask what issues we are facing? I suspect it would be extremely well received having someone working on a language specific solution asking on the Fedora go sig mailinglist. Just a thought.

by mfer 10y ago

scrollaway 10y ago

[This question is purely informational and doesn't directly relate to golang, but I want to ask someone who has been involved in domain-specific package management.] Why is the protocol side of package management not a solved problem already? At least Go reuses the git protocol to its advantage. But the general problem of package management applies to a lot of things. "There is a package with a name and a description. That package has versions, which can be greater/lesser than others. Those versions may have deltas from one-another. They contain one or more files. Those files must have their integrity checked. etc, etc..." -> Why does every single language reimplement that? Nevermind languages actually. Why does everyone reimplement that? Firefox/Chrome extensions, Android/ios app stores, a bazillion different game addons and app-specific stores... We have a mostly-identical problem being solved a million different ways. pypi, npmjs, glide, they should be running the same server software. It'd be a first step towards standardization and avoid a bunch of mistakes repeated every time a new solution comes out. Is anyone even trying to do something about this? (and before anyone says "do it yourself", a package management protocol is something I used to work on, not anymore. It keeps getting worse - it's not just a technical problem, also a political one.)

zalmoxes 10y ago
Sam Boyer, one of the other people in the document wrote a nice article that dives into the complexity of dependency management. https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527#.pdtxz22hv https://medium.com/@sdboyer/so-you-want-to-write-a-package-m... Discussion https://news.ycombinator.com/item?id=11088125 https://news.ycombinator.com/item?id=11088125 Sam is currently working on a library for Go https://github.com/sdboyer/gps https://github.com/sdboyer/gps
- zellyn 10y ago
  In fact, here's Sam and wycats (of Cargo) talking about how a generic library (obviously Go is not the best choice for that) would be nice: https://twitter.com/sdboyer/status/755036174016806912 https://twitter.com/sdboyer/status/755036174016806912
mfer 10y ago
This is a good question. I have a few thoughts. First, different languages have subtleties that can make a different in package management. For example, the way nested dependencies can work in JavaScript, Go, and PHP are all different. Or, the way you can scan a codebase to determine things is difference. It would be great if different languages had virtually the same interface. That way metadata and interaction could be about the same. Like diving a car from different manufacturers. In many ways the different package managers are similar enough for programming languages were sorta there (but not all that well). Second, there is the cultural problem. Many languages are designed on islands from others. Reinventing things, trying different things, and wanting to be their own special flower. Sometimes this is good because a new language is not like the others and that can be a good thing. Other times you reinvent a fairly common wheel. I'm not aware of anyone solving this in a common way.
Drup 10y ago
At least for the dependency solving part, a generic exchange format has emerged: CUDF (http://www.mancoosi.org/cudf/ http://www.mancoosi.org/cudf/). Several efficient solvers have been implemented that supports this input format. You can read about it more in http://opam.ocaml.org/doc/Specifying_Solver_Preferences.html#Yestherearedifferentversionsoftheuserpreferencelanguage http://opam.ocaml.org/doc/Specifying_Solver_Preferences.html... [1]: https://www.youtube.com/watch?v=E-gtFnbHcv0&list=UUP9g4dLR7xt6KzCYntNqYcw https://www.youtube.com/watch?v=E-gtFnbHcv0&list=UUP9g4dLR7x... [2]: http://ocaml.org/meetings/ocaml/2014/ocaml2014_17.pdf http://ocaml.org/meetings/ocaml/2014/ocaml2014_17.pdf This is at least used in OPAM (the OCaml package manager) and is also available for apt. If you are implementing a SAT solver for your package manager nowadays, you are doing it wrong. :)
Rafert 10y ago
There's Molinillo used in CocoaPods and Bundler: https://github.com/CocoaPods/Molinillo https://github.com/CocoaPods/Molinillo
- mfer 10y ago
  When I wrote "Seeing How The Other Side Lives, A Package Manager Overview For Go Developers"[1] I noted CocoaPods. Was not aware of Molinillo though. Thanks for sharing. [1] http://engineeredweb.com/blog/2015/pkg-mgr-overview/ http://engineeredweb.com/blog/2015/pkg-mgr-overview/
shurcooL 10y ago
> That package has versions This is a statement that doesn't always apply. For example, some packages don't have versions, they only have a single version: the latest version. It's a moving target, but it's there. Older versions are just an unimportant and unsupported implementation side-effect. Of course, many packages do have versions, but this one factor makes a big difference on how you look at vendoring.
- peq 10y ago
  Without versions you don't have package management, you have a download script.
  - thawkins 10y ago
    Agreed, and you also cant guarentee reproducable builds. Builds that are dependent on when you compile them, even if no changes have been made due to ephemeral dependances are a nightmare.
- vec 10y ago
  First, your described software still has versions. Each daily snapshot or git commit or whatever is a new version. There's no reason you can't put an auto-incrementing version number on each release and call that your version number. You can even stick a "0." on the front of it and be SemVer complant. Second, just because some people do this doesn't mean it's a good idea. If you're writing a full application and just want to distribute it, do whatever you want with your versioning (although using a dependency management tool to distribute your webapp or CLI tool is probably not a great idea in the first place), but if you are writing something to be included in another project please use a sane versioning system. Anything else is selfish and disrespectful of your users.
  - bassislife 10y ago
    In Go, each version corresponds to a given import/Pkg path. If you want to change version (major), you need to create a new package. The advantage is that you do not have to download a manifest such as package.json or whatever. It also provides a constraint on library authors to provide a backward compatible API for their published package. Eventually, I guess we could use go/types to enforce/check the API backward compatibility requirement. The main issues for now are: * make sure that packages importing a same vendored one are able to use the same latest minor version in use. * allow a project based approach to go get for people who need reproducible builds (or alternatively allow for multiple $GOPATH and make being able to switch between them easy)
bbatha 10y ago
> At least Go reuses the git protocol to its advantage. I think this is a mistake. Packaging should come from a central server (that can be overriden with a mirror). Allowing downloads from adhoc git repositories is a real pain. First, it makes mirroring the package repository impossible. Second, it makes it difficult for companies to disallow certain packages in a centralized fashion. Third, it makes distro packaging a royal pain. And finally Git simply wasn't built for the this kind of work, compressed tarballs happen to work great for this purpose. Git's mutable history also presents other challenges for pinning to specific versions of packages. Simply put, sometimes a central authority has advantages. That all said, vendoring alleviates most of these problems for apps and some of them for libraries. Reusing tools does make it a little easier to get started with too. I'd hate to see these features removed in whatever go get becomes in the long term.

zzzcpan 10y ago

Don't ignore nix/guix. Personally I don't think we need yet another package manager with painful external dependencies, non-reproducible builds, conflicts, lack of rollbacks, etc. We have that with "go get". What we need is to actually solve all those thing. Make sure there are no fragile external dependencies, but everything is in the package manager, even gcc, binutils. You get the idea.

mfer 10y ago
Go package management needs to work with Git, Bzr, Hg, and Svn as it does today. Do you have suggestions on building a tool cross platform that works with packages in all of these that doesn't have external dependencies?
- zzzcpan 10y ago
  It doesn't matter where the code is stored, certainly not for nix.
  - mfer 10y ago
    I'm sorry, I don't understand. `go get` or any package manager needs to be able to fetch the code remotely. Without interacting with the external binaries would mean the Go tool would need to have custom code to retrieve the codebase and interact with the VCS metadata such as tags. Are you suggesting custom Go code for that instead of using dependencies like git?
    - zzzcpan 10y ago
      No, git would have to be a package in the same package manager, as a build dependency. Think OS package manager, but limited to what developer needs, i.e. just software, no services, no superuser privileges, etc.
      - Vendan 10y ago
        Talk about feature creep! It sounds like you want a package manager and ecosystem that can install half an OS, just so you don't have to fuss with installing git on your own...

ahacker15 10y ago

Hi (I've tried Glide but it slowness and few bugs made me move to Govendor. Nothing personal, it just didn't work out that well for me.) A few questions: - Is the intention desincentive the use vendoring? - Is the intention to make a single solution, and then deprecate existing tools (Glide, Govendor and like)? - Do you have the intention to make a central registry for Go packages? Or the solution will use GitHub and like, as it does today? It would be not backward compatible, but the document don't make this clear.

mfer 10y ago
I think the intention is to go figure it out. I don't know that we can say what it will be just yet.

grosbisou 10y ago

Dumb question but what are the reasons a solution like cargo or bundle (never used glide) would be rejected by the go committee?

themihai 10y ago
I think cargo does too much. For example it has a bunch of meta information which could be extracted from the source code(i.e package documentation), the imports must be manually maintained in a config file(the Go way is to extract them from the source code automatically and snapshot the version). Shotly said I feel cargo is bloated, verbose and it wouldn't fit well with the Go philosophy(minimalist, convention over configuration). Why do you need both cargo.toml and cargo.lock? I also doubt Cargo would fix issues not fixed by the existing vendoring tools(e.g libraries that expose common types of different versions)

SEJeff 10y ago

What are some of the proposed solutions other than vendoring, which tends to be very difficult for large distributions such as Fedora. I was just reading a few days ago about some of the extreme challenges Fedora is having packaging the golang ecosystem.

mfer 10y ago
Can you point me to those Fedora discussions? I'd be curious to learn more. I understand how operating system package managers deal with things differently than application developers. I've heard of problems with the way PHP apps are handling their dependencies these days and how Linux distro package managers aren't happy about it. I'm curious to know more in this case.
- SEJeff 10y ago
  This email was from July 25th: https://lists.fedoraproject.org/archives/list/golang@lists.fedoraproject.org/thread/RD6TA5X3HR3XCPBLF7Z3E5D34PO6NEDW/ https://lists.fedoraproject.org/archives/list/golang@lists.f... But there have been several regarding the challenges Fedora has had as a large distribution while trying to package many Go apps, each with potentially hard requirements on different versions of the same golang package to import. Most of the conversation about it is in the golang sig (special interest group) that I've linked to their mail. May I suggest you subscribe, send an email out introducing yourself, and ask what issues we are facing? I suspect it would be extremely well received having someone working on a language specific solution asking on the Fedora go sig mailinglist. Just a thought.