6 ms·

I started poking around and noticed that Go 1.13 now defaults to the Golang Proxy to fetch modules. This means a proxy, governed by the Google Privacy Policy, i

by mfer 7y ago

I started poking around and noticed that Go 1.13 now defaults to the Golang Proxy to fetch modules. This means a proxy, governed by the Google Privacy Policy, is now capturing everyone's module usage by default. Unless you change settings this includes proprietary/corp stuff. https://codeengineered.com/blog/2019/go-mod-proxy-psa/ https://codeengineered.com/blog/2019/go-mod-proxy-psa/

onei 7y ago
Why do you need a proxy to fetch the modules in the first place? It seems a little odd for a package manager.
- mfer 7y ago
  The jFrog folks provided a good example with GoCenter. It was optimized for the performance use case. If you used their proxy, released at the end of last year I think, the time to pull a dependency was significantly less than to pull it from GitHub. This leads to things like faster CI builds.
  - onei 7y ago
    Ah, that makes more sense. I was imagining it as a routing thing rather than caching the packages.
    - DuskStar 7y ago
      Also, the dependency resolution really just relies on `go.mod` and `go.sum` - having to download the entire git tree just to access different versions of those files is a bit inefficient. (It also makes it harder for 3rd parties to serve different versions of a dependency to different people - and easier for Google to do the same)
- matahwoosh 7y ago
  It's so that you can have a seamless module version checksum [1]. The checksum in proxy ensures that you don't "trust on first use" whenever you update a module. [1] https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md https://go.googlesource.com/proposal/+/master/design/25530-s...
- sangnoir 7y ago
  Write-only caches are a good way to avoid left-pad-type debacles[1] 1. https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/ https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/
privateSFacct 7y ago
I think they chose a default that was designed to provide a better user experience. If you can't even let the google know what your dependencies / stack is then I think you can run your own as well. Would be interesting to test the security at some of these companies where this is managements worry. How do these companies worry about dependency leaks but then leave huge S3 / remote exploitable web holes totally open? If google as an adversary is your worry: What about all the folks your org emails with that use google in various ways (including perhaps their email stack that they've turned over to google?). How many users are using google single sign on lots of places? How many of their users are using android with telemetry? Browse websites using google chrome? Browse websites that themselves use google analytics. Google / Apple / AWS / Microsoft are at the root of trust for a lot of what happens online these days.
- mfer 7y ago
  With different companies there are different agreements. For example, when a company stores something in a private S3 bucket that company has an agreement between it and AWS on who has access to that bucket and for what reasons. The agreements allow companies to know and document risk, SLA, and many other things. Many companies have not turned their email over to Google. Many companies are in competition or at least coopertition with Google. Many are trying to get a competitive advantage these days. All of this was a big enough deal for Mozilla that they got a special contract with Google before using Google Analytics that says Google cannot use the data collected on Mozilla sites for other purposes. Some won't worry or care about the implications of this. There are a bunch who will. It is not an all or nothing thing. It depends on your business if you are building stuff for a business.
FiloSottile 7y ago
The Modules Mirror and the Checksum Database are governed by this specific privacy policy: https://proxy.golang.org/privacy https://proxy.golang.org/privacy "go env -w GOPRIVATE=*.corp.example.com" was added to make it as easy as possible to configure private modules. If the environment is not set up, just the name of the module will reach the Google services, it will not be published, and an error will be returned. Anything else would have make it possible to bypass the protections of the new Checksum Database (https://golang.org/design/25530-sumdb https://golang.org/design/25530-sumdb), a state-of-the-art authentication system that provides auditable proof that the operator is behaving honestly. For an example of why the Mirror is valuable, beyond the 10x performance gains (depending on number of dependencies and connection latency) and security gains from not using git on the client, look no further than git.apache.org, which has been down for days breaking a number of builds (https://status.apache.org/incidents/63030p4241xj https://status.apache.org/incidents/63030p4241xj). I really think this architecture brings together the best of the decentralized module ecosystem that Go had (with its lack of regularly compromised registry accounts, and no mismatch between code and package origin) and the centralized registries of other languages (with their better performance and reliability). Particularly proud of how the Checksum Database provides even more security than the centralized registries (because it's cryptographically accountable for its actions, while registries can sign anything they want) without requiring authors to manage keys, register an account, or change their workflow at all. [Note that I work on the Go team, and co-designed the Checksum Database. But of course I don't speak for the company.] EDIT to add: I should also mention that the sumdb allows the use of any untrusted proxy, easily configured with "go env -w GOPROXY", for users that can't or don't want to use the Google provided one. The proxy can also opt-in to mirroring the Checksum Database, in which case no user requests at all need to reach the Google services, while still providing all the integrity guarantees that proxy.golang.org users benefit from.
- mfer 7y ago
  First, I think proxies have their place. Please don't mistake my comment for any issue with proxies themselves. Second, the don't need Git part is a bit of misdirection. Other package manager (Composer for PHP comes to mind but there are others, too) can pull the source at a version right for GitHub. This is much faster than using Git and people have been doing it for some time. Typically it's pulled down as a tarball. You don't need a proxy to have git out of the loop for downloads or install time stuff. Third, when it came to making tradeoffs one was made here. I don't know if it was explicit or not. Protections from the global checksum database as a default were put ahead of privacy as a default. I wonder what other designs could have kept a certain amount of the global checksum protection while keeping privacy. I came up with a few idea while considering my response here. Fourth, to really speed up performance in things like CI it's much faster to cache the modules locally. This is something used across programming languages for dependencies and most of the major CI systems have this documented. I looked at using GoCenter months ago but didn't opt for it because caching in CI was so much faster. We were looking at things holistically rather than just with a single tool. As I said in the blog post, I appreciate the privacy policy being up front. It is the first link on the proxy website. And, I appreciate the configuration to customize the proxy. A lot of this has to do with the default. Most people won't change it. A lot of people are going to leak proprietary information to Google without realizing it. Everyone here can decide if that is a good or bad thing for themselves. Hopefully, they will learn the config to customize things if they want.
typical182 7y ago
I've used Rust a bit, but not enough to know the details of how cargo and crates.io approach managing collected information. There seems to be an open issue and WIP PR with more details: Issue: https://github.com/rust-lang/crates.io/issues/955 https://github.com/rust-lang/crates.io/issues/955 PR: https://github.com/rust-lang/www.rust-lang.org/pull/919/files#diff-39965d15e6e89e238a4d6b3e05733277 https://github.com/rust-lang/www.rust-lang.org/pull/919/file... The cargo manifest also supports things like: The publish field (optional) The publish field can be used to prevent a package from being published to a package registry (like crates.io) by mistake, for instance to keep a package private in a company. [package] # ... publish = false But I would be curious if someone more knowledgeable than myself would be interested in giving a quick summary of the approach?
- zeeboo 7y ago
  Publish isn't necessary for the Go version, since the model is on-demand pull driven. godoc.org is the same way: there's no way for private code to end up there unless your private code is actually public. The privacy concern is entirely that Google may see the name of some module you attempted to download.
rvz 7y ago
> This means a proxy, governed by the Google Privacy Policy, is now capturing everyone's module usage by default. No surprise there, was even called a 'privacy nut' on the neighboring Android 10 thread for highlighting the nature of Google's 'privacy' stance. So essentially they turn on tracking / analytics on by default for module usage without consent or any notice except for the lengthy privacy policy. This reminds me of when Microsoft VC++ embedded analytics in the developer binary when using the compiler. [0] But then again, Google can do what it pleases since Golang is authored by Googlers. [0] https://www.reddit.com/r/cpp/comments/4ibauu/visual_studio_adding_telemetry_function_calls_to/ https://www.reddit.com/r/cpp/comments/4ibauu/visual_studio_a...
- workaway 7y ago
  > "without consent or any notice except for the lengthy privacy policy" There's also there in release notes: > "The GOPROXY environment variable may now be set to a comma-separated list of proxy URLs or the special token direct, and its default value is now https://proxy.golang.org https://proxy.golang.org "
sangnoir 7y ago
Had you read the linked release notes, you wouldn't have had to poke around to find that out: it's not some hidden thing. The second paragraph starts off with: As of Go 1.13, the go command by default downloads and authenticates modules using the Go module mirror and Go checksum database run by Google.
ben_jones 7y ago
I think this is a perfect example of a completely rational implementation that given all the facts almost all developers would agree to - except because of Google's reputation for pervasive tracking, abruptly sunsetting popular products, and general we-know-better approach to software, will become a point of flak for the core Go team. To be abundantly clear, and because a lot of Google employees read this, the we-know-better comment isn't a jab - because often Google engineers DO know better, but it does occasionally mix with i.e. poor communication and create frustration for the many developers who use Google services and tools.
shkkmo 7y ago
I don't really get what the risk to proprietary information? The only thing that leaks for private repos would be the URL and version number, right? While opt-out for this not ideal, this seems like a minor privacy risk relative to the security risks ameliorated by enforcing checksums by default. I guess an option which solves would be a mandatory prompt about proxy usage unless there is an explicit environment variable set?

by mfer 7y ago

onei 7y ago

Why do you need a proxy to fetch the modules in the first place? It seems a little odd for a package manager.

mfer 7y ago
The jFrog folks provided a good example with GoCenter. It was optimized for the performance use case. If you used their proxy, released at the end of last year I think, the time to pull a dependency was significantly less than to pull it from GitHub. This leads to things like faster CI builds.
- onei 7y ago
  Ah, that makes more sense. I was imagining it as a routing thing rather than caching the packages.
  - DuskStar 7y ago
    Also, the dependency resolution really just relies on `go.mod` and `go.sum` - having to download the entire git tree just to access different versions of those files is a bit inefficient. (It also makes it harder for 3rd parties to serve different versions of a dependency to different people - and easier for Google to do the same)
matahwoosh 7y ago
It's so that you can have a seamless module version checksum [1]. The checksum in proxy ensures that you don't "trust on first use" whenever you update a module. [1] https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md https://go.googlesource.com/proposal/+/master/design/25530-s...
sangnoir 7y ago
Write-only caches are a good way to avoid left-pad-type debacles[1] 1. https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/ https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/

privateSFacct 7y ago

I think they chose a default that was designed to provide a better user experience. If you can't even let the google know what your dependencies / stack is then I think you can run your own as well. Would be interesting to test the security at some of these companies where this is managements worry. How do these companies worry about dependency leaks but then leave huge S3 / remote exploitable web holes totally open? If google as an adversary is your worry: What about all the folks your org emails with that use google in various ways (including perhaps their email stack that they've turned over to google?). How many users are using google single sign on lots of places? How many of their users are using android with telemetry? Browse websites using google chrome? Browse websites that themselves use google analytics. Google / Apple / AWS / Microsoft are at the root of trust for a lot of what happens online these days.

mfer 7y ago
With different companies there are different agreements. For example, when a company stores something in a private S3 bucket that company has an agreement between it and AWS on who has access to that bucket and for what reasons. The agreements allow companies to know and document risk, SLA, and many other things. Many companies have not turned their email over to Google. Many companies are in competition or at least coopertition with Google. Many are trying to get a competitive advantage these days. All of this was a big enough deal for Mozilla that they got a special contract with Google before using Google Analytics that says Google cannot use the data collected on Mozilla sites for other purposes. Some won't worry or care about the implications of this. There are a bunch who will. It is not an all or nothing thing. It depends on your business if you are building stuff for a business.

FiloSottile 7y ago

The Modules Mirror and the Checksum Database are governed by this specific privacy policy: https://proxy.golang.org/privacy https://proxy.golang.org/privacy "go env -w GOPRIVATE=*.corp.example.com" was added to make it as easy as possible to configure private modules. If the environment is not set up, just the name of the module will reach the Google services, it will not be published, and an error will be returned. Anything else would have make it possible to bypass the protections of the new Checksum Database (https://golang.org/design/25530-sumdb https://golang.org/design/25530-sumdb), a state-of-the-art authentication system that provides auditable proof that the operator is behaving honestly. For an example of why the Mirror is valuable, beyond the 10x performance gains (depending on number of dependencies and connection latency) and security gains from not using git on the client, look no further than git.apache.org, which has been down for days breaking a number of builds (https://status.apache.org/incidents/63030p4241xj https://status.apache.org/incidents/63030p4241xj). I really think this architecture brings together the best of the decentralized module ecosystem that Go had (with its lack of regularly compromised registry accounts, and no mismatch between code and package origin) and the centralized registries of other languages (with their better performance and reliability). Particularly proud of how the Checksum Database provides even more security than the centralized registries (because it's cryptographically accountable for its actions, while registries can sign anything they want) without requiring authors to manage keys, register an account, or change their workflow at all. [Note that I work on the Go team, and co-designed the Checksum Database. But of course I don't speak for the company.] EDIT to add: I should also mention that the sumdb allows the use of any untrusted proxy, easily configured with "go env -w GOPROXY", for users that can't or don't want to use the Google provided one. The proxy can also opt-in to mirroring the Checksum Database, in which case no user requests at all need to reach the Google services, while still providing all the integrity guarantees that proxy.golang.org users benefit from.

mfer 7y ago
First, I think proxies have their place. Please don't mistake my comment for any issue with proxies themselves. Second, the don't need Git part is a bit of misdirection. Other package manager (Composer for PHP comes to mind but there are others, too) can pull the source at a version right for GitHub. This is much faster than using Git and people have been doing it for some time. Typically it's pulled down as a tarball. You don't need a proxy to have git out of the loop for downloads or install time stuff. Third, when it came to making tradeoffs one was made here. I don't know if it was explicit or not. Protections from the global checksum database as a default were put ahead of privacy as a default. I wonder what other designs could have kept a certain amount of the global checksum protection while keeping privacy. I came up with a few idea while considering my response here. Fourth, to really speed up performance in things like CI it's much faster to cache the modules locally. This is something used across programming languages for dependencies and most of the major CI systems have this documented. I looked at using GoCenter months ago but didn't opt for it because caching in CI was so much faster. We were looking at things holistically rather than just with a single tool. As I said in the blog post, I appreciate the privacy policy being up front. It is the first link on the proxy website. And, I appreciate the configuration to customize the proxy. A lot of this has to do with the default. Most people won't change it. A lot of people are going to leak proprietary information to Google without realizing it. Everyone here can decide if that is a good or bad thing for themselves. Hopefully, they will learn the config to customize things if they want.

typical182 7y ago

I've used Rust a bit, but not enough to know the details of how cargo and crates.io approach managing collected information. There seems to be an open issue and WIP PR with more details: Issue: https://github.com/rust-lang/crates.io/issues/955 https://github.com/rust-lang/crates.io/issues/955 PR: https://github.com/rust-lang/www.rust-lang.org/pull/919/files#diff-39965d15e6e89e238a4d6b3e05733277 https://github.com/rust-lang/www.rust-lang.org/pull/919/file... The cargo manifest also supports things like: The publish field (optional) The publish field can be used to prevent a package from being published to a package registry (like crates.io) by mistake, for instance to keep a package private in a company. [package] # ... publish = false But I would be curious if someone more knowledgeable than myself would be interested in giving a quick summary of the approach?

zeeboo 7y ago
Publish isn't necessary for the Go version, since the model is on-demand pull driven. godoc.org is the same way: there's no way for private code to end up there unless your private code is actually public. The privacy concern is entirely that Google may see the name of some module you attempted to download.

rvz 7y ago

> This means a proxy, governed by the Google Privacy Policy, is now capturing everyone's module usage by default. No surprise there, was even called a 'privacy nut' on the neighboring Android 10 thread for highlighting the nature of Google's 'privacy' stance. So essentially they turn on tracking / analytics on by default for module usage without consent or any notice except for the lengthy privacy policy. This reminds me of when Microsoft VC++ embedded analytics in the developer binary when using the compiler. [0] But then again, Google can do what it pleases since Golang is authored by Googlers. [0] https://www.reddit.com/r/cpp/comments/4ibauu/visual_studio_adding_telemetry_function_calls_to/ https://www.reddit.com/r/cpp/comments/4ibauu/visual_studio_a...

workaway 7y ago
> "without consent or any notice except for the lengthy privacy policy" There's also there in release notes: > "The GOPROXY environment variable may now be set to a comma-separated list of proxy URLs or the special token direct, and its default value is now https://proxy.golang.org https://proxy.golang.org "

sangnoir 7y ago

Had you read the linked release notes, you wouldn't have had to poke around to find that out: it's not some hidden thing. The second paragraph starts off with: As of Go 1.13, the go command by default downloads and authenticates modules using the Go module mirror and Go checksum database run by Google.

ben_jones 7y ago

I think this is a perfect example of a completely rational implementation that given all the facts almost all developers would agree to - except because of Google's reputation for pervasive tracking, abruptly sunsetting popular products, and general we-know-better approach to software, will become a point of flak for the core Go team. To be abundantly clear, and because a lot of Google employees read this, the we-know-better comment isn't a jab - because often Google engineers DO know better, but it does occasionally mix with i.e. poor communication and create frustration for the many developers who use Google services and tools.

shkkmo 7y ago

I don't really get what the risk to proprietary information? The only thing that leaks for private repos would be the URL and version number, right? While opt-out for this not ideal, this seems like a minor privacy risk relative to the security risks ameliorated by enforcing checksums by default. I guess an option which solves would be a mandatory prompt about proxy usage unless there is an explicit environment variable set?