9 ms·

Code.google.com was shut down, Go packaging is broken

99 points by dz0ny 11y ago

lolc 11y ago
"This change drops support for code.google.com and fixes test." How was packaging broken?
- deleted 11y ago
  [deleted]
- dz0ny 11y ago
  Some apps/packages still use packages hosted on code.google.com. Basicly you have to update dependency tree for packages that were hosted there. If you vendor packages then you are not immediately affected but it still poses a problem in long term.
- aikah 11y ago
  There is no central repository system in Go. No Maven, no rubygems, no packagist , ... code.google.com/foo/bar package is the physical address were go get command will fetch the package. If that address is broken then the code is broken. Go doesn't have a package manager either. Go Get is certainly not a package manager. So when you use a third party package you have to fork it somewhere. The problem is that 2 libraries with the same exact code are different packages if their address is different. So sometimes it becomes a bit insane to work with third party libraries if people start forking them for safety. I consider this whole package url thing as a design mistake. But most of the Go community sees no problem with it.
lukaslalinsky 11y ago
I never really understood why would they map package names to the sites that host the sources. It was obvious something like this was going to happen and it's happening on smaller scale all the time. Projects move.
- Gigablah 11y ago
  And even if projects stay at the same URL, their folder structures would change.
  - danieldk 11y ago
    In the Go packaging scheme, they shouldn't. The idea is that a URL is API-stable. Changing the folder structure changes the API (package names). For this reason, you have wrappers like http://gopkg.in/ http://gopkg.in/ , which permit some freedom in the form of git tags. (I fully agree that Go packaging is very sensitive to API changes and incidents like sites disappearing altogether.)
- manojlds 11y ago
  Less impactful, but docker images have to be tagged with the docker registry URL they will be pushed to.
  - lukaslalinsky 11y ago
    That's another thing I always disliked. I'd have preferred a configurable prefix, or something like Git remotes.
- cfstras 11y ago
  The idea was that people should never make API-breaking changes for URLs and instead offer only new interfaces or make a new package at a different URL. Funny coincidence that google itself broke that idea.
CSDude 11y ago
I hate go packaging. I want to fork a package for my private purposes, maintain it in my hosted repo or somewhere private, but no, I have to change all the required import paths to my repo path. Or I have to provide it in vendor folder, or have custom gopath folders, but then go get does not work, I have to git clone and put it in some weird old repo's path. You could use relative paths, or package names but no, you had to use site urls so you could make it harder for anyone that does not work open source or private. I have not found a better solution for forking a repo and working it on my own besides vendor folder solution.
- cfstras 11y ago
  https://github.com/tools/godep https://github.com/tools/godep should solve all these issues. Yes, it vendors packages. But it happens completely automatically and you can switch to your own forks at will.
- mseepgood 11y ago
  > I have to change all the required import paths to my repo path GO15VENDOREXPERIMENT=1 will be default behaviour in Go 1.6. So you don't have to rewrite the import paths anymore. > Or I have to provide it in vendor folder What's wrong with that?
  - zimbatm 11y ago
    > What's wrong with that? Vendoring external dependencies adds a lot of noise to your git history. Especially with github's UI it's easy to miss an important change because of skipping trough the vendor changes or getting the "too many changes to display" message.
    - anonx 11y ago
      > Vendoring external dependencies adds a lot of noise to your git history. This is my concern as well. Is it possible to use git submodules in vendor directory? Looks like it is possible [1]. [1]: https://github.com/golang/go/issues/12573 https://github.com/golang/go/issues/12573
petke 11y ago
Google shuts down yet another service developers rely on. Its getting quite a bad track record. Trust is not so easily rebuilt. I think more developers will avoid depending on google services in the future if they have another choice. Edit. What happens to the thousands of webpages that link to jquery or google fonts on google.code. This must break lots of things ...
- fishnchips 11y ago
  Services come and go, Google or else. Unlike Reader, code.google.com is not something I will personally miss.
- MatthewWilkes 11y ago
  Google has googleapis.com for hosted libraries and fonts, which is free to use. I doubt many people were hotlinking directly to google code repos. If they were, I have little sympathy. It's entirely inappropriate to co-opt a repository hosting service as a CDN.
- EvanPlaice 11y ago
  Google code has been shutdown since last May. AFAIK, Google uses code to host internal projects as well as open source projects. Apparently some users were spamming and/or hosting illegal content for download. Instead of spending resources fixing what is essentially a free service to the public, they just shut it down and converted all repos to read-only mode. At this point, I pretty much assume that any service that Google offers for free sans advertising exists to attract free beta testers. Eventually, it'll be converted to either a paid service or internal-only use. Source: lots of time wasted migrating projects to GitHub.
- dspillett 11y ago
  To be fair they did give ten months notice so people had time to manage the change, and what we are talking about here isn't "correct" use for that service anyway (it was for source and issue management, not library hosting or a more general CDN).
  - 11y ago
twblalock 11y ago
I think the only realistic solution to this is to vendor your go dependencies. If Github ever changes its URL schema, a whole bunch of stuff is going to break. The Go maintainers have conflated what a library is and where a library is. That's an important distinction that should not be glossed over like this.
- _asummers 11y ago
  Java does this pretty well, at least if you use Gradle, I think. You specify the location of the repository in the repositories { ... } block and then specify (with maven naming scheme) dependencies { compile "group:name:version" } where group might be something like org.apache, name might be apache-commons and version might be 2.5.
  - lmm 11y ago
    It's nothing Gradle-specific - as you say it's the Maven scheme and used by almost all JVM dependency managers. Note also there's a convention of using a domain you control in reverse order as the groupId (i.e. com.mycompany:myproject:1.0) which makes collisions extremely unlikely. Finally it's very easy in the maven ecosystem to run a "proxy repository" for your organization that indefinitely caches all the artifacts you depend on, thus ensuring that your builds will always be reproducible.
    - merb 11y ago
      the only aweful thing is deploying to the central repository
      - LoSboccacc 11y ago
        why? I have used sonatype and it's pretty much log in, promote to production.
        lmm 11y ago
        It's a very painful process to get set up - no doubt it's easy after the first time. I've ultimately found it easier to go via bintray: http://blog.bintray.com/2014/02/11/bintray-as-pain-free-gateway-to-maven-central/ http://blog.bintray.com/2014/02/11/bintray-as-pain-free-gate... . On the positive side, some of the complexity comes through enforcing some standards about what packages go into central - in particular you must have a properly declared license, developer contact information, and GPG sign your releases. Which are things other language ecosystems would do well to enforce.
jacobr 11y ago
It will be great fun when Github is shut down. Many npm packages, jspm, and more are using hard coded links to repositories or even paths in repositories.
- fortytw2 11y ago
  Do you foresee GitHub shutting down in the next 5 years? I certainly don't.
  - bachmeier 11y ago
    That's not the only way there will be trouble. A more likely scenario is "We've been going over the numbers. To keep our costs down, we are making the following changes to our support of open source projects..." I'd be surprised if that didn't happen in some capacity in the next five years.
  - chippy 11y ago
    Five years is a long time in tech. I think it's more likely that we will see same thing happen with Sourceforge and Google Code - a better thing comes along and people move to it. This is something which is seen over and over again. Its not that loved active services and sites get turned off, it's that the userbase moves somewhere else first.
  - afandian 11y ago
    Did you foresee what happened to SourceForge?
    - maaarghk 11y ago
      Well... maybe? SourceForge was always a really terrible experience for me, using it in say 2004/05. Loads of ads, interstitials and other general bad behaviour. It's not exactly surprising.
      - castell 11y ago
        SourceForge used to be okay though already looked a bit dated, back then. Slashdot and SF were owned by the same company. Though adding adware to binary downloads is downright incorrect and alarming. SF doesn't even host the binary files themselves, they just provide an interface for various open file mirrors like HEAnet. HEAnet and co should block SF and we would need an open community website that links to the mirrors to replace SF.
akerro 11y ago
Uff just in time I moved some projects I relay on from CG to Gitlab... developers didn't bother.
vruiz 11y ago
Please note that the linked URL is just a commit to remove support from code.google.com in go get, and add a warning to users trying to go get from it. Also remember that while not very widely used, Go does provide support against this problems via custom URLs and canonical import paths [0]. And most importantly, whatever your programming language, remember to vendor your dependencies. [0] https://texlution.com/post/golang-canonical-import-paths/ https://texlution.com/post/golang-canonical-import-paths/
4ad 11y ago
I've flagged this editorialized title (Code.google.com was shut down, Go packaging is broken). Apart from being against the rules, it's intentionally misleading. It gives the false impression that Google Code shutting down somehow caused go get functionality to fail in general, which is false. Go get works as always; of course you can't go get go packages on Google Code, because Google Code doesn't exist anymore! The link just points to a change that fixes some trivial Go test which didn't impact anything else; something that people working on Go have to deal with, not something users of Go have to deal with. As for the news about Google Code itself, that was announced almost an year ago. The Go project itself migrated to git well before that, and of course, every Go package that's maintained also migrated off Google Code in the meantime. This is (misleading) non-news.
- lolc 11y ago
  Thanks for your clarifications.
0xFFC 11y ago
This is exactly why I am trying to avoid Google's product as much as possible and suggest people avoid them as much as possible. They want just shove the internet to your throat. With all of their product.
golergka 11y ago
I just got into Go, and I'm a fan a lot of language design decisions. But a lot of configuration leaves me wondering: is it me who doesn't understand something, or did the same folks who designed such a good language actually screwed up configuration stuff? No way to specify a version of library or language I'm working with is already a little bit annoying, although I haven't actually discovered problems because of it. But the $GOPATH just drives me nuts: why the hell would the language infrastructure work out of assumption that all my projects that are based on a single language are contained in a single folder? On the contrary, I often have projects that include several different languages and frameworks in them, and I'm usually keeping client and server in the same repo. Language is a part of a project, not the other way around. I ended up with this hackery in my server Makefile: build: GOPATH=$(shell pwd) go install main Overall, just as Go itself is transparent and simple, it's `import` statement and build infrastructure is not. "Convention over configuration" may be a good approach for simple web apps, but when this convention just doesn't fit your workflow, it brings more bad than good.
- Cthulhu_ 11y ago
  The idea with using $GOPATH is due to go having its own packaging tool; in other systems, you often see something similar (Maven for Java is probably the largest one), i.e. a directory structure somewhere. The $GOPATH makes it a lot easier to contribute on other packages though, you can just jump in there because it's right next to your own project/packages.
  - golergka 11y ago
    Well, in that department I think that NPM got it better than anything else — although I dislike Node.js and don't have a lot of experience with it apart from a couple of hobby projects.
    - ajitjor 11y ago
      You can use gvm for go and it of course setups GOROOT and GOPATH
- 11y ago
donatj 11y ago
'go get' is a highly useful bag of cats. The fact that code.google.com needed to be hard coded rubs me in such the wrong way for what should be a generalized tool. I wish instead of magic self resolving paths, it just took git repo ssh/http paths.
eternalban 11y ago
Someone needs to let the Gophers know about this little concept known as 'indirection'.
csense 11y ago
The "website going out of business destroyed my content" problem is one that needs to be addressed. One promising idea is content-addressable storage, like IPFS-hosted repos -- anyone can self-host at a stable URL. (I don't know if IPFS has a good solution yet to the problem branches solve in git, namely having a named, auto-advancing pointer to the content address of the latest version.)
- robto 11y ago
  They do! IIRC, the originator of the content can have a cryptographically signed hash that is a pointer to the latest version. That way everyone who is interested pins the originator's hash and automatically get directed to the newest content.
- ProAm 11y ago
  Just host it yourself. People act like managing a server is akin to brain surgery these days. The 'cloud' is great, but this is what you have to plan for and deal with if you don't want to do it yourself.
tmaly 11y ago
I cannot blame Google, they ditched the Reader and various other free services. the Code service may not have been a core thing for the company. Github on the other hand does have a core focus on hosting code. I would expect them to be in this business as long as it is profitable for them.
msie 11y ago
Isn't any packaging system reliant on a server and ultimately fragile if you don't vendor? I don't want to see an overengineered packaging system. Go is open source, has anyone looked at improving "go get"?
Lethalman 11y ago
How Nix solved the issue: 1. Each go package has an attribute name in nix, like anything else, that is not tied to its url. 2. Each go package has an url where to fetch the source from, which is not the import path to use the package. 3. Each go package has a number of old import url aliases (e.g. "code.google.com/...") 4. Whenever a package X depends on Y with an old import url, then X is rewritten to use the new url path of Y using govers. So the problem is solved at our packaging level. This is not to say it's the best solution, but certainly a good alternative solution.

99 points by dz0ny 11y ago

lolc 11y ago

"This change drops support for code.google.com and fixes test." How was packaging broken?

deleted 11y ago
[deleted]
dz0ny 11y ago
Some apps/packages still use packages hosted on code.google.com. Basicly you have to update dependency tree for packages that were hosted there. If you vendor packages then you are not immediately affected but it still poses a problem in long term.
aikah 11y ago
There is no central repository system in Go. No Maven, no rubygems, no packagist , ... code.google.com/foo/bar package is the physical address were go get command will fetch the package. If that address is broken then the code is broken. Go doesn't have a package manager either. Go Get is certainly not a package manager. So when you use a third party package you have to fork it somewhere. The problem is that 2 libraries with the same exact code are different packages if their address is different. So sometimes it becomes a bit insane to work with third party libraries if people start forking them for safety. I consider this whole package url thing as a design mistake. But most of the Go community sees no problem with it.

lukaslalinsky 11y ago

I never really understood why would they map package names to the sites that host the sources. It was obvious something like this was going to happen and it's happening on smaller scale all the time. Projects move.

Gigablah 11y ago
And even if projects stay at the same URL, their folder structures would change.
- danieldk 11y ago
  In the Go packaging scheme, they shouldn't. The idea is that a URL is API-stable. Changing the folder structure changes the API (package names). For this reason, you have wrappers like http://gopkg.in/ http://gopkg.in/ , which permit some freedom in the form of git tags. (I fully agree that Go packaging is very sensitive to API changes and incidents like sites disappearing altogether.)
manojlds 11y ago
Less impactful, but docker images have to be tagged with the docker registry URL they will be pushed to.
- lukaslalinsky 11y ago
  That's another thing I always disliked. I'd have preferred a configurable prefix, or something like Git remotes.
cfstras 11y ago
The idea was that people should never make API-breaking changes for URLs and instead offer only new interfaces or make a new package at a different URL. Funny coincidence that google itself broke that idea.

CSDude 11y ago

I hate go packaging. I want to fork a package for my private purposes, maintain it in my hosted repo or somewhere private, but no, I have to change all the required import paths to my repo path. Or I have to provide it in vendor folder, or have custom gopath folders, but then go get does not work, I have to git clone and put it in some weird old repo's path. You could use relative paths, or package names but no, you had to use site urls so you could make it harder for anyone that does not work open source or private. I have not found a better solution for forking a repo and working it on my own besides vendor folder solution.

cfstras 11y ago
https://github.com/tools/godep https://github.com/tools/godep should solve all these issues. Yes, it vendors packages. But it happens completely automatically and you can switch to your own forks at will.
mseepgood 11y ago
> I have to change all the required import paths to my repo path GO15VENDOREXPERIMENT=1 will be default behaviour in Go 1.6. So you don't have to rewrite the import paths anymore. > Or I have to provide it in vendor folder What's wrong with that?
- zimbatm 11y ago
  > What's wrong with that? Vendoring external dependencies adds a lot of noise to your git history. Especially with github's UI it's easy to miss an important change because of skipping trough the vendor changes or getting the "too many changes to display" message.
  - anonx 11y ago
    > Vendoring external dependencies adds a lot of noise to your git history. This is my concern as well. Is it possible to use git submodules in vendor directory? Looks like it is possible [1]. [1]: https://github.com/golang/go/issues/12573 https://github.com/golang/go/issues/12573

petke 11y ago

Google shuts down yet another service developers rely on. Its getting quite a bad track record. Trust is not so easily rebuilt. I think more developers will avoid depending on google services in the future if they have another choice. Edit. What happens to the thousands of webpages that link to jquery or google fonts on google.code. This must break lots of things ...

fishnchips 11y ago
Services come and go, Google or else. Unlike Reader, code.google.com is not something I will personally miss.
MatthewWilkes 11y ago
Google has googleapis.com for hosted libraries and fonts, which is free to use. I doubt many people were hotlinking directly to google code repos. If they were, I have little sympathy. It's entirely inappropriate to co-opt a repository hosting service as a CDN.
EvanPlaice 11y ago
Google code has been shutdown since last May. AFAIK, Google uses code to host internal projects as well as open source projects. Apparently some users were spamming and/or hosting illegal content for download. Instead of spending resources fixing what is essentially a free service to the public, they just shut it down and converted all repos to read-only mode. At this point, I pretty much assume that any service that Google offers for free sans advertising exists to attract free beta testers. Eventually, it'll be converted to either a paid service or internal-only use. Source: lots of time wasted migrating projects to GitHub.
dspillett 11y ago
To be fair they did give ten months notice so people had time to manage the change, and what we are talking about here isn't "correct" use for that service anyway (it was for source and issue management, not library hosting or a more general CDN).
- 11y ago

twblalock 11y ago

I think the only realistic solution to this is to vendor your go dependencies. If Github ever changes its URL schema, a whole bunch of stuff is going to break. The Go maintainers have conflated what a library is and where a library is. That's an important distinction that should not be glossed over like this.

_asummers 11y ago
Java does this pretty well, at least if you use Gradle, I think. You specify the location of the repository in the repositories { ... } block and then specify (with maven naming scheme) dependencies { compile "group:name:version" } where group might be something like org.apache, name might be apache-commons and version might be 2.5.
- lmm 11y ago
  It's nothing Gradle-specific - as you say it's the Maven scheme and used by almost all JVM dependency managers. Note also there's a convention of using a domain you control in reverse order as the groupId (i.e. com.mycompany:myproject:1.0) which makes collisions extremely unlikely. Finally it's very easy in the maven ecosystem to run a "proxy repository" for your organization that indefinitely caches all the artifacts you depend on, thus ensuring that your builds will always be reproducible.
  - merb 11y ago
    the only aweful thing is deploying to the central repository
    - LoSboccacc 11y ago
      why? I have used sonatype and it's pretty much log in, promote to production.
      - lmm 11y ago
        It's a very painful process to get set up - no doubt it's easy after the first time. I've ultimately found it easier to go via bintray: http://blog.bintray.com/2014/02/11/bintray-as-pain-free-gateway-to-maven-central/ http://blog.bintray.com/2014/02/11/bintray-as-pain-free-gate... . On the positive side, some of the complexity comes through enforcing some standards about what packages go into central - in particular you must have a properly declared license, developer contact information, and GPG sign your releases. Which are things other language ecosystems would do well to enforce.

jacobr 11y ago

It will be great fun when Github is shut down. Many npm packages, jspm, and more are using hard coded links to repositories or even paths in repositories.

fortytw2 11y ago
Do you foresee GitHub shutting down in the next 5 years? I certainly don't.
- bachmeier 11y ago
  That's not the only way there will be trouble. A more likely scenario is "We've been going over the numbers. To keep our costs down, we are making the following changes to our support of open source projects..." I'd be surprised if that didn't happen in some capacity in the next five years.
- chippy 11y ago
  Five years is a long time in tech. I think it's more likely that we will see same thing happen with Sourceforge and Google Code - a better thing comes along and people move to it. This is something which is seen over and over again. Its not that loved active services and sites get turned off, it's that the userbase moves somewhere else first.
- afandian 11y ago
  Did you foresee what happened to SourceForge?
  - maaarghk 11y ago
    Well... maybe? SourceForge was always a really terrible experience for me, using it in say 2004/05. Loads of ads, interstitials and other general bad behaviour. It's not exactly surprising.
    - castell 11y ago
      SourceForge used to be okay though already looked a bit dated, back then. Slashdot and SF were owned by the same company. Though adding adware to binary downloads is downright incorrect and alarming. SF doesn't even host the binary files themselves, they just provide an interface for various open file mirrors like HEAnet. HEAnet and co should block SF and we would need an open community website that links to the mirrors to replace SF.

akerro 11y ago

Uff just in time I moved some projects I relay on from CG to Gitlab... developers didn't bother.

vruiz 11y ago

Please note that the linked URL is just a commit to remove support from code.google.com in go get, and add a warning to users trying to go get from it. Also remember that while not very widely used, Go does provide support against this problems via custom URLs and canonical import paths [0]. And most importantly, whatever your programming language, remember to vendor your dependencies. [0] https://texlution.com/post/golang-canonical-import-paths/ https://texlution.com/post/golang-canonical-import-paths/

4ad 11y ago

I've flagged this editorialized title (Code.google.com was shut down, Go packaging is broken). Apart from being against the rules, it's intentionally misleading. It gives the false impression that Google Code shutting down somehow caused go get functionality to fail in general, which is false. Go get works as always; of course you can't go get go packages on Google Code, because Google Code doesn't exist anymore! The link just points to a change that fixes some trivial Go test which didn't impact anything else; something that people working on Go have to deal with, not something users of Go have to deal with. As for the news about Google Code itself, that was announced almost an year ago. The Go project itself migrated to git well before that, and of course, every Go package that's maintained also migrated off Google Code in the meantime. This is (misleading) non-news.

lolc 11y ago
Thanks for your clarifications.

0xFFC 11y ago

This is exactly why I am trying to avoid Google's product as much as possible and suggest people avoid them as much as possible. They want just shove the internet to your throat. With all of their product.

golergka 11y ago

I just got into Go, and I'm a fan a lot of language design decisions. But a lot of configuration leaves me wondering: is it me who doesn't understand something, or did the same folks who designed such a good language actually screwed up configuration stuff? No way to specify a version of library or language I'm working with is already a little bit annoying, although I haven't actually discovered problems because of it. But the $GOPATH just drives me nuts: why the hell would the language infrastructure work out of assumption that all my projects that are based on a single language are contained in a single folder? On the contrary, I often have projects that include several different languages and frameworks in them, and I'm usually keeping client and server in the same repo. Language is a part of a project, not the other way around. I ended up with this hackery in my server Makefile: build: GOPATH=$(shell pwd) go install main Overall, just as Go itself is transparent and simple, it's `import` statement and build infrastructure is not. "Convention over configuration" may be a good approach for simple web apps, but when this convention just doesn't fit your workflow, it brings more bad than good.

Cthulhu_ 11y ago
The idea with using $GOPATH is due to go having its own packaging tool; in other systems, you often see something similar (Maven for Java is probably the largest one), i.e. a directory structure somewhere. The $GOPATH makes it a lot easier to contribute on other packages though, you can just jump in there because it's right next to your own project/packages.
- golergka 11y ago
  Well, in that department I think that NPM got it better than anything else — although I dislike Node.js and don't have a lot of experience with it apart from a couple of hobby projects.
  - ajitjor 11y ago
    You can use gvm for go and it of course setups GOROOT and GOPATH
11y ago

donatj 11y ago

'go get' is a highly useful bag of cats. The fact that code.google.com needed to be hard coded rubs me in such the wrong way for what should be a generalized tool. I wish instead of magic self resolving paths, it just took git repo ssh/http paths.

eternalban 11y ago

Someone needs to let the Gophers know about this little concept known as 'indirection'.

csense 11y ago

The "website going out of business destroyed my content" problem is one that needs to be addressed. One promising idea is content-addressable storage, like IPFS-hosted repos -- anyone can self-host at a stable URL. (I don't know if IPFS has a good solution yet to the problem branches solve in git, namely having a named, auto-advancing pointer to the content address of the latest version.)

robto 11y ago
They do! IIRC, the originator of the content can have a cryptographically signed hash that is a pointer to the latest version. That way everyone who is interested pins the originator's hash and automatically get directed to the newest content.
ProAm 11y ago
Just host it yourself. People act like managing a server is akin to brain surgery these days. The 'cloud' is great, but this is what you have to plan for and deal with if you don't want to do it yourself.

tmaly 11y ago

I cannot blame Google, they ditched the Reader and various other free services. the Code service may not have been a core thing for the company. Github on the other hand does have a core focus on hosting code. I would expect them to be in this business as long as it is profitable for them.

msie 11y ago

Isn't any packaging system reliant on a server and ultimately fragile if you don't vendor? I don't want to see an overengineered packaging system. Go is open source, has anyone looked at improving "go get"?

Lethalman 11y ago

How Nix solved the issue: 1. Each go package has an attribute name in nix, like anything else, that is not tied to its url. 2. Each go package has an url where to fetch the source from, which is not the import path to use the package. 3. Each go package has a number of old import url aliases (e.g. "code.google.com/...") 4. Whenever a package X depends on Y with an old import url, then X is rewritten to use the new url path of Y using govers. So the problem is solved at our packaging level. This is not to say it's the best solution, but certainly a good alternative solution.