9 ms·

The Modules Mirror and the Checksum Database are governed by this specific privacy policy: https://proxy.golang.org/privacy https://proxy.golang.org/privacy "g

The Modules Mirror and the Checksum Database are governed by this specific privacy policy: https://proxy.golang.org/privacy https://proxy.golang.org/privacy "go env -w GOPRIVATE=*.corp.example.com" was added to make it as easy as possible to configure private modules. If the environment is not set up, just the name of the module will reach the Google services, it will not be published, and an error will be returned. Anything else would have make it possible to bypass the protections of the new Checksum Database (https://golang.org/design/25530-sumdb https://golang.org/design/25530-sumdb), a state-of-the-art authentication system that provides auditable proof that the operator is behaving honestly. For an example of why the Mirror is valuable, beyond the 10x performance gains (depending on number of dependencies and connection latency) and security gains from not using git on the client, look no further than git.apache.org, which has been down for days breaking a number of builds (https://status.apache.org/incidents/63030p4241xj https://status.apache.org/incidents/63030p4241xj). I really think this architecture brings together the best of the decentralized module ecosystem that Go had (with its lack of regularly compromised registry accounts, and no mismatch between code and package origin) and the centralized registries of other languages (with their better performance and reliability). Particularly proud of how the Checksum Database provides even more security than the centralized registries (because it's cryptographically accountable for its actions, while registries can sign anything they want) without requiring authors to manage keys, register an account, or change their workflow at all. [Note that I work on the Go team, and co-designed the Checksum Database. But of course I don't speak for the company.] EDIT to add: I should also mention that the sumdb allows the use of any untrusted proxy, easily configured with "go env -w GOPROXY", for users that can't or don't want to use the Google provided one. The proxy can also opt-in to mirroring the Checksum Database, in which case no user requests at all need to reach the Google services, while still providing all the integrity guarantees that proxy.golang.org users benefit from.

mfer 7y ago
First, I think proxies have their place. Please don't mistake my comment for any issue with proxies themselves. Second, the don't need Git part is a bit of misdirection. Other package manager (Composer for PHP comes to mind but there are others, too) can pull the source at a version right for GitHub. This is much faster than using Git and people have been doing it for some time. Typically it's pulled down as a tarball. You don't need a proxy to have git out of the loop for downloads or install time stuff. Third, when it came to making tradeoffs one was made here. I don't know if it was explicit or not. Protections from the global checksum database as a default were put ahead of privacy as a default. I wonder what other designs could have kept a certain amount of the global checksum protection while keeping privacy. I came up with a few idea while considering my response here. Fourth, to really speed up performance in things like CI it's much faster to cache the modules locally. This is something used across programming languages for dependencies and most of the major CI systems have this documented. I looked at using GoCenter months ago but didn't opt for it because caching in CI was so much faster. We were looking at things holistically rather than just with a single tool. As I said in the blog post, I appreciate the privacy policy being up front. It is the first link on the proxy website. And, I appreciate the configuration to customize the proxy. A lot of this has to do with the default. Most people won't change it. A lot of people are going to leak proprietary information to Google without realizing it. Everyone here can decide if that is a good or bad thing for themselves. Hopefully, they will learn the config to customize things if they want.
- TheDong 7y ago
  > You don't need a proxy to have git out of the loop for downloads or install time stuff. You do with go because go uses git tags for version information and must parse the `go.mod` file at various versions. Go also supports git repositories hosted in places other than github, where there's no standardized way to fetch a single file at a tag (the go.sum file), or to perform a suitable search for a versoin and its metadata. > This [pulling github archives] is much faster than using Git That's only true if you only ever look at a single version of a library. If you have a 20MB git repo at v1.2.3, and ~100KB of files change for the v1.2.4 patch release, using the zip/tar archive means you have to download 40MB total to use it and update it to the new patch. Using the git repository in the first place means the update can only download 20MB since the new `fetch` will be effectively zero data, assuming you cache git stuff well.
rgbrenner 7y ago
First programming language that I've had to read and understand a privacy policy to use.. and consider that it may change in the future. The idea that the language I'm programming in now reports anything back to google is distressing.. and I say that as someone who has been programming in Go for about 7 years. What right does google have to collect usage information from modules hosted on github (or elsewhere)? I checkout the code for the modules, review it, and commit it into my own repository. I don't need you to speed up the rare git checkout of a module. Not that I think some minor speed improvement is worth my privacy. You talk about decentralization... while centralizing the entire thing around Google. You talk about git.apache.org being down, while you're willing to bring down the entire ecosystem across the board with your own special server. This "feature" should have an option to disable it.. at the very least. This is the first language in my 23 years programming that reports my usage back to the language authors. It's unprecedented. And the truth is, if this is ok to you, then there's really no limit what the code I write today, may report back to google tomorrow.
- TheDong 7y ago
  > This is the first language in my 23 years programming that reports my usage back to the language authors I suppose that means you haven't used any of: 1. nodejs, which reports your usage back to npm Inc with the exact same amount of detail (names of dependencies, ip of caller) (and stores much more, since it publishes 'downloads per month', etc) 2. rust, which does the same with crates.io 3. perl, which does that with cpan 4. python, which does that with pypi 5. ... etc All of those languages have a central registry of packages that has the same level of detail in the metadata it can potentially collect as the go proxy does. > This "feature" should have an option to disable it.. at the very least. It does. Multiple options. You can run your own proxy, disable it entirely, or opt out of the go mod experiment and never run go get, but rather vendor by hand with `git clone` or whatever. > And the truth is, if this is ok to you, then there's really no limit what the code I write today, may report back to google tomorrow. Very much a slippery slope. They've communicated clearly how to disable it and what it does, it's in-line with what other language ecosystems do in terms of metadata reported to a central package repository, and I think it's very easy for someone to be okay with this, but to not be okay with any step beyond this line.
gjs278 7y ago
thanks but I don't want it

mfer 7y ago

First, I think proxies have their place. Please don't mistake my comment for any issue with proxies themselves. Second, the don't need Git part is a bit of misdirection. Other package manager (Composer for PHP comes to mind but there are others, too) can pull the source at a version right for GitHub. This is much faster than using Git and people have been doing it for some time. Typically it's pulled down as a tarball. You don't need a proxy to have git out of the loop for downloads or install time stuff. Third, when it came to making tradeoffs one was made here. I don't know if it was explicit or not. Protections from the global checksum database as a default were put ahead of privacy as a default. I wonder what other designs could have kept a certain amount of the global checksum protection while keeping privacy. I came up with a few idea while considering my response here. Fourth, to really speed up performance in things like CI it's much faster to cache the modules locally. This is something used across programming languages for dependencies and most of the major CI systems have this documented. I looked at using GoCenter months ago but didn't opt for it because caching in CI was so much faster. We were looking at things holistically rather than just with a single tool. As I said in the blog post, I appreciate the privacy policy being up front. It is the first link on the proxy website. And, I appreciate the configuration to customize the proxy. A lot of this has to do with the default. Most people won't change it. A lot of people are going to leak proprietary information to Google without realizing it. Everyone here can decide if that is a good or bad thing for themselves. Hopefully, they will learn the config to customize things if they want.

TheDong 7y ago
> You don't need a proxy to have git out of the loop for downloads or install time stuff. You do with go because go uses git tags for version information and must parse the `go.mod` file at various versions. Go also supports git repositories hosted in places other than github, where there's no standardized way to fetch a single file at a tag (the go.sum file), or to perform a suitable search for a versoin and its metadata. > This [pulling github archives] is much faster than using Git That's only true if you only ever look at a single version of a library. If you have a 20MB git repo at v1.2.3, and ~100KB of files change for the v1.2.4 patch release, using the zip/tar archive means you have to download 40MB total to use it and update it to the new patch. Using the git repository in the first place means the update can only download 20MB since the new `fetch` will be effectively zero data, assuming you cache git stuff well.

rgbrenner 7y ago

First programming language that I've had to read and understand a privacy policy to use.. and consider that it may change in the future. The idea that the language I'm programming in now reports anything back to google is distressing.. and I say that as someone who has been programming in Go for about 7 years. What right does google have to collect usage information from modules hosted on github (or elsewhere)? I checkout the code for the modules, review it, and commit it into my own repository. I don't need you to speed up the rare git checkout of a module. Not that I think some minor speed improvement is worth my privacy. You talk about decentralization... while centralizing the entire thing around Google. You talk about git.apache.org being down, while you're willing to bring down the entire ecosystem across the board with your own special server. This "feature" should have an option to disable it.. at the very least. This is the first language in my 23 years programming that reports my usage back to the language authors. It's unprecedented. And the truth is, if this is ok to you, then there's really no limit what the code I write today, may report back to google tomorrow.

TheDong 7y ago
> This is the first language in my 23 years programming that reports my usage back to the language authors I suppose that means you haven't used any of: 1. nodejs, which reports your usage back to npm Inc with the exact same amount of detail (names of dependencies, ip of caller) (and stores much more, since it publishes 'downloads per month', etc) 2. rust, which does the same with crates.io 3. perl, which does that with cpan 4. python, which does that with pypi 5. ... etc All of those languages have a central registry of packages that has the same level of detail in the metadata it can potentially collect as the go proxy does. > This "feature" should have an option to disable it.. at the very least. It does. Multiple options. You can run your own proxy, disable it entirely, or opt out of the go mod experiment and never run go get, but rather vendor by hand with `git clone` or whatever. > And the truth is, if this is ok to you, then there's really no limit what the code I write today, may report back to google tomorrow. Very much a slippery slope. They've communicated clearly how to disable it and what it does, it's in-line with what other language ecosystems do in terms of metadata reported to a central package repository, and I think it's very easy for someone to be okay with this, but to not be okay with any step beyond this line.

gjs278 7y ago

thanks but I don't want it