5 ms·

Is there a justifiable need for the proxy? It seems like we’re just adding a centralized point of failure, the lack thereof something I have absolutely touted a

by donatj 7y ago

Is there a justifiable need for the proxy? It seems like we’re just adding a centralized point of failure, the lack thereof something I have absolutely touted as a feature. For verification we already have the go.sum file validating the hash. We’ve got vendoring if you’re concerned about dependencies disappearing. The touted speed feature seems like a non-issue? The absolute largest Go package I’ve ever pulled took maybe a minute to grab its dependencies, and then I had them and never had to again. The cynical side of me feels like this is a way for Google to collect analytics on the Go ecosystem. The slightly less cynical part of me thinks people from other ecosystems demanded it due to cargo culting the idea. The fact that it makes it more difficult to use private modules is pretty irritating as well as I am going to need to make sure all our developers and build systems are aware when we move to 1.13. Having to set GOPRIVATE on each machine is absurd. It should be definable in the go.mod so it can be known via git.

DelightOne 7y ago
A package repository can go down, vanish, versions can be deleted.
- donatj 7y ago
  Which as I mentioned is why we have `go mod vendor` if that’s a concern.
  - nindalf 7y ago
    Some people don't like vendoring and would prefer keeping their dependencies separate.
jrockway 7y ago
My experience from building docker images of Go programs from Github repositories is that Github very aggressively rate limits you. More than one "go build ..." per 10 minutes and it just randomly won't send you the repository contents anymore. So a module proxy was a must. (You could also vendor dependencies, which is my preference, but my team was strongly opposed to vendoring. So we used Athens.)
- donatj 7y ago
  Are you pulling as an authenticated GitHub user? The rate limit for an authenticated user is 5000 pulls per hour, unauthenticated is 60.
  - jrockway 7y ago
    No I am not. I don't even know how to make "go build" authenticate to Github. (It's not my code I'm pulling, it's the random libraries my code depends on.) This is why the proxy exists.
    - nhooyr 7y ago
      If git is authenticated with GitHub for you, go build is as well.
    - thefounder 7y ago
      "go build" doesn't fetch the dependencies, does it? Go uses the local settings (i.e .netrc file in your home directory)
      - ownagefool 7y ago
        go build will fetch the dependencies if you don't have them. However, they're stored in the $GOPATH, which now defaults to ~/go I believe. Therefore, chances are you already have them.
- nhooyr 7y ago
  This has been a problem on CI if your CI doesn't support caching dependencies.
justinclift 7y ago
> ... this is a way for Google to collect analytics on the Go ecosystem. Yeah. It's extremely tone deaf of the Go dev team to pull this crap. Note - setting `export GOPROXY=direct` in your .bashrc (on nix Bash) should* stop the info collection. Unless a "feature" is added to stop that working too.
- skybrian 7y ago
  Making the feature optional is a pretty normal way of catering to the minority of people who are more risk adverse and won't use something like this. Giving up entirely is not.
  - justinclift 7y ago
    Optional yeah, but enabled by default.
elithrar 7y ago
Paragraphs 3, 4 & 5 of TFA speak directly to the benefits of a module proxy and mirror w.r.t. module availability & performance.
- donatj 7y ago
  Which is what I’m attempting to refute the need for. These are solved and non-problems respectively.
  - lokar 7y ago
    The point about go.mod only protecting you from changes after you first use it is valid.
  - skybrian 7y ago
    You're saying things work fine for you, but it doesn't follow that they work fine for everyone else. Maybe there are larger Go modules out there than the largest one you've personally built? Or people with slower network access to Github? (Consider China and Australia.) Or maybe they're anticipating that there will be larger Go projects in the future?
    - aloknnikhil 7y ago
      Most Go projects are hosted on Github. Github is backed by a global CDN and can handle significantly large projects. I don't think the focus of this project was to address slow network access. It was primarily driven by the need to analyze dependency usage and perhaps add redundancy to some extent.
      - FiloSottile 7y ago
        Fetching a module from GitHub often requires cloning the repository, when all you needed was the contents of a specific version, or even just the go.mod to do version resolution. The proxy protocol provides the artifacts you need, for sometimes a 10x bandwidth saving. (Also, lots of security issues in "go get" came from using git to fetch untrusted repositories.)
zellyn 7y ago
I don't have the link handy, but I remember someone posting very impressive speed improvements. More important though, to those of us writing and/or stewarding Go code at companies that deal with financial information, these kind of safeguards are absolutely necessary. You can be cynical about Google's motivations, sure, but doing this to track you harder is not consistent with the conversation and approach they've taken with these features.
- secure 7y ago
  That might have been me: https://twitter.com/zekjur/status/1154934952477093888 https://twitter.com/zekjur/status/1154934952477093888 > Oh, wow! Setting GOPROXY=https://proxy.golang.org https://proxy.golang.org results in a 7x speed up for module installation on the hotel WiFi: from 17.7s (no proxy) to 2.3s (http://proxy.golang.org http://proxy.golang.org)
paxys 7y ago
GitHub is already the central point of failure for the vast majority of the ecosystem, and that is run by a company that has no incentive to care about Go.
- pjmlp 7y ago
  Guess who supports VS Code Go plugin and has sponsored the work on Go debugging tools like Delve?
earthboundkid 7y ago
Marking packages as private in Go.mod is a good idea. Bring it up on the issue tracker.
nindalf 7y ago
> The cynical side of me feels like this is a way for Google to collect analytics on the Go ecosystem > other ecosystems demanded it due to cargo culting the idea And there it is, the hallmark of a typical comment on a thread like this * Cynical * Expresses concern about data being collected even though this data can't possibly be used for anything bad. (If you disagree, go to crates.io and tell me what concerns you have). * Expresses disapproval of a large corporation even though they haven't done anything wrong here. * Pithy dismissal of other programmers by calling them names ("cargo cult"). This change improves speed of downloads by 6-7x on poor connections and 3x on good connections. Clearly this is helpful for situations like CI which typically involve clean builds. What's the issue here exactly? * Furious attack on a free product that's completely optional to use. If you want to keep vendoring your dependencies, go ahead and do that. Why are you angry that some people are going to use this proxy?
- eeZah7Ux 7y ago
  That's a good example of cargo culting. (Also the downvotes this message is getting. Nobody can dare to question The Google.)
  - heyoni 7y ago
    What’s cargo culting exactly? Seriously asking
    - JoeAltmaier 7y ago
      https://en.wikipedia.org/wiki/Cargo_cult https://en.wikipedia.org/wiki/Cargo_cult
      - heyoni 7y ago
        Thank you sir!
- donatj 7y ago
  "Furious attack". For serious? I gave a light hearted criticism of adding a single lynchpin to the entire Go ecosystem. I think that's a legitimate source of concern. I was by no means pithy, I was by no means being dismissive. Thinking you need something just because you had it elsewhere is the definition of cargo culting - and is something the Go developers have otherwise largely avoided. Just because something is faster doesn't make it better or worth it, particularly when it comes with tradeoffs and is already really fast - which is my argument - that the tradeoffs aren't worth the benefit.
homologate 7y ago
The default behavior is to fall back to fetching from source control directly if the proxy fails. https://tip.golang.org/doc/go1.13 https://tip.golang.org/doc/go1.13
SamWhited 7y ago
Not to mention that I used to be able to get my own metrics (eg. by tailing access logs on the vanity import path server, or checking stats on my version control service or server, etc.) now only Google has that information and module owners are left in the dark about how many times their code is being hit. They also aren't being very forward thinking. The Go team keeps saying "we promise no Google exec is doing anything nefarious with that" but we don't know who will be the execs next year, or 10 years from now or what they'll do with the power the Go team just gave them by being so short sighted.

by donatj 7y ago

DelightOne 7y ago

A package repository can go down, vanish, versions can be deleted.

donatj 7y ago
Which as I mentioned is why we have `go mod vendor` if that’s a concern.
- nindalf 7y ago
  Some people don't like vendoring and would prefer keeping their dependencies separate.

jrockway 7y ago

My experience from building docker images of Go programs from Github repositories is that Github very aggressively rate limits you. More than one "go build ..." per 10 minutes and it just randomly won't send you the repository contents anymore. So a module proxy was a must. (You could also vendor dependencies, which is my preference, but my team was strongly opposed to vendoring. So we used Athens.)

donatj 7y ago
Are you pulling as an authenticated GitHub user? The rate limit for an authenticated user is 5000 pulls per hour, unauthenticated is 60.
- jrockway 7y ago
  No I am not. I don't even know how to make "go build" authenticate to Github. (It's not my code I'm pulling, it's the random libraries my code depends on.) This is why the proxy exists.
  - nhooyr 7y ago
    If git is authenticated with GitHub for you, go build is as well.
  - thefounder 7y ago
    "go build" doesn't fetch the dependencies, does it? Go uses the local settings (i.e .netrc file in your home directory)
    - ownagefool 7y ago
      go build will fetch the dependencies if you don't have them. However, they're stored in the $GOPATH, which now defaults to ~/go I believe. Therefore, chances are you already have them.
nhooyr 7y ago
This has been a problem on CI if your CI doesn't support caching dependencies.

justinclift 7y ago

> ... this is a way for Google to collect analytics on the Go ecosystem. Yeah. It's extremely tone deaf of the Go dev team to pull this crap. Note - setting `export GOPROXY=direct` in your .bashrc (on nix Bash) should* stop the info collection. Unless a "feature" is added to stop that working too.

skybrian 7y ago
Making the feature optional is a pretty normal way of catering to the minority of people who are more risk adverse and won't use something like this. Giving up entirely is not.
- justinclift 7y ago
  Optional yeah, but enabled by default.

elithrar 7y ago

Paragraphs 3, 4 & 5 of TFA speak directly to the benefits of a module proxy and mirror w.r.t. module availability & performance.

donatj 7y ago
Which is what I’m attempting to refute the need for. These are solved and non-problems respectively.
- lokar 7y ago
  The point about go.mod only protecting you from changes after you first use it is valid.
- skybrian 7y ago
  You're saying things work fine for you, but it doesn't follow that they work fine for everyone else. Maybe there are larger Go modules out there than the largest one you've personally built? Or people with slower network access to Github? (Consider China and Australia.) Or maybe they're anticipating that there will be larger Go projects in the future?
  - aloknnikhil 7y ago
    Most Go projects are hosted on Github. Github is backed by a global CDN and can handle significantly large projects. I don't think the focus of this project was to address slow network access. It was primarily driven by the need to analyze dependency usage and perhaps add redundancy to some extent.
    - FiloSottile 7y ago
      Fetching a module from GitHub often requires cloning the repository, when all you needed was the contents of a specific version, or even just the go.mod to do version resolution. The proxy protocol provides the artifacts you need, for sometimes a 10x bandwidth saving. (Also, lots of security issues in "go get" came from using git to fetch untrusted repositories.)

zellyn 7y ago

I don't have the link handy, but I remember someone posting very impressive speed improvements. More important though, to those of us writing and/or stewarding Go code at companies that deal with financial information, these kind of safeguards are absolutely necessary. You can be cynical about Google's motivations, sure, but doing this to track you harder is not consistent with the conversation and approach they've taken with these features.

secure 7y ago
That might have been me: https://twitter.com/zekjur/status/1154934952477093888 https://twitter.com/zekjur/status/1154934952477093888 > Oh, wow! Setting GOPROXY=https://proxy.golang.org https://proxy.golang.org results in a 7x speed up for module installation on the hotel WiFi: from 17.7s (no proxy) to 2.3s (http://proxy.golang.org http://proxy.golang.org)

paxys 7y ago

GitHub is already the central point of failure for the vast majority of the ecosystem, and that is run by a company that has no incentive to care about Go.

pjmlp 7y ago
Guess who supports VS Code Go plugin and has sponsored the work on Go debugging tools like Delve?

earthboundkid 7y ago

Marking packages as private in Go.mod is a good idea. Bring it up on the issue tracker.

nindalf 7y ago

> The cynical side of me feels like this is a way for Google to collect analytics on the Go ecosystem > other ecosystems demanded it due to cargo culting the idea And there it is, the hallmark of a typical comment on a thread like this * Cynical * Expresses concern about data being collected even though this data can't possibly be used for anything bad. (If you disagree, go to crates.io and tell me what concerns you have). * Expresses disapproval of a large corporation even though they haven't done anything wrong here. * Pithy dismissal of other programmers by calling them names ("cargo cult"). This change improves speed of downloads by 6-7x on poor connections and 3x on good connections. Clearly this is helpful for situations like CI which typically involve clean builds. What's the issue here exactly? * Furious attack on a free product that's completely optional to use. If you want to keep vendoring your dependencies, go ahead and do that. Why are you angry that some people are going to use this proxy?

eeZah7Ux 7y ago
That's a good example of cargo culting. (Also the downvotes this message is getting. Nobody can dare to question The Google.)
- heyoni 7y ago
  What’s cargo culting exactly? Seriously asking
  - JoeAltmaier 7y ago
    https://en.wikipedia.org/wiki/Cargo_cult https://en.wikipedia.org/wiki/Cargo_cult
    - heyoni 7y ago
      Thank you sir!
donatj 7y ago
"Furious attack". For serious? I gave a light hearted criticism of adding a single lynchpin to the entire Go ecosystem. I think that's a legitimate source of concern. I was by no means pithy, I was by no means being dismissive. Thinking you need something just because you had it elsewhere is the definition of cargo culting - and is something the Go developers have otherwise largely avoided. Just because something is faster doesn't make it better or worth it, particularly when it comes with tradeoffs and is already really fast - which is my argument - that the tradeoffs aren't worth the benefit.

homologate 7y ago

The default behavior is to fall back to fetching from source control directly if the proxy fails. https://tip.golang.org/doc/go1.13 https://tip.golang.org/doc/go1.13

SamWhited 7y ago

Not to mention that I used to be able to get my own metrics (eg. by tailing access logs on the vanity import path server, or checking stats on my version control service or server, etc.) now only Google has that information and module owners are left in the dark about how many times their code is being hit. They also aren't being very forward thinking. The Go team keeps saying "we promise no Google exec is doing anything nefarious with that" but we don't know who will be the execs next year, or 10 years from now or what they'll do with the power the Go team just gave them by being so short sighted.