3 ms·

Why not make the proxy an opt-in fallback by default? I like my fails to be catastrophic. I don't want to touch any host I don't have to. Ever. And doesn't it l

Why not make the proxy an opt-in fallback by default? I like my fails to be catastrophic. I don't want to touch any host I don't have to. Ever. And doesn't it leak metadata if you aren't careful? I remember a Go page to request data removal. A package named github.com/microsoft/clippy already makes my skin crawl.

jrockway 6y ago
Not using a proxy by default causes people a lot of pain -- it's the common case to not care that the network is in use if it makes your builds faster and more reliable. If you don't want to accidentally leak things to the network, that's kind of your OS's or IT department's jobs. The tools are available and they will work with things that aren't explicit about needing the network. (What I'm saying is, if this concerns you, Go is one worry of many.) I have a concrete example about why module caching is good. I have an example repository that shows what happens when upstreams exercise poor release hygiene. Clone github.com/jrockway/evil-module-user. Run "go run main.go". (It's not actually evil, it just prints a message.) You will find that with the Go module proxy turned on, it compiles and prints "This is a good module!". With the proxy turned off, though, you will find that the checksum in go.sum doesn't match for the upstream github.com/jrockway/evil-module, and that evil-module has been silently compromised to become evil. What happened here is that evil-module released v0.0.1 and the module cache cached it. Then someone edited the code, and forced pushed the v0.0.1 tag to the repository, pointing at the commit. When go get reaches out to Github to clone github.com/jrockway/evil-module@v0.0.1, it gets code that doesn't match the checksum in github.com/jrockway/evil-module-user's go.sum, because it's not the v0.0.1 that the developer of github.com/jrockway/evil-module-user used when writing the code. Obviously this example is intentionally contrived to demonstrate this exact problem. But when I first started using Go modules early on, this happened all the time. Different engineers got different "version 0.0.1" from many repositories, and didn't notice there was a problem until CI. They then loudly complained that go modules suck, when in fact the problem is that upstream authors were very fast and loose about retagging releases. They are sending your release system code that you've never even seen! That's good that the module system detects that. But annoying to fix :) The problem all went away when Google started caching the modules -- you get the old version of the code, but at least everyone gets the same version. (As an author, you can always release v0.0.2 to bust the cache.) I actually thought until I investigated this in great detail recently that everyone had just embraced immutable version tags and were being smart about releases, as they learned more about Go modules. Turned out I was wrong -- people are as bad as ever, but they don't break your CI build because there is now a centralized cache that gives your workstation, your coworkers, and your CI system the same code. The upstream modules authors aren't doing that. They are force-pushing with abandon.
- yencabulator 6y ago
  goproxy and gosumdb are two separate services, you can get the TOFU benefits without goproxy, and a checksum database would be possible to query without even revealing what import path you are talking about (just like browser safe browsing).
  - jrockway 6y ago
    I think the proxy is a bigger benefit than the checksum database. For the CI use case, it lets you get your modules from a server that's under less load than github (which has to support writes). Your go.sum already contains the checksums, so the sum database is not relevant there, but the module cache adds a big reliability and speed boost to your workflow. The checksum database is of course extremely important (you want to get a known-good copy the first time you use a module), but I stand by my assertion that it's beneficial for both to be turned on by default. Yes, it's a small privacy leak. But not one that is particularly personally embarrassing or useful to adversaries. So I think the go team made the right call.
enneff 6y ago
If you don’t want to use the proxy you can turn it off, trivially. M For the overwhelming majority of users the proxy is a huge benefit (installing and updating dependencies is way faster and the checksum database provides security benefits). Note that most other packaging ecosystems use centralised services, where you don’t even have the option of merely turning off the proxy.
- yencabulator 6y ago
  I know you know this, but gosumdb and goproxy are two very different entities, consumed separately. Let's not muddy conversations about the default centralized goproxy with the existence of a global TOFU service. gosumdb would be perfectly possible to query without even revealing a module import path you're asking for (just like browser-side safe browsing can do).

jrockway 6y ago

Not using a proxy by default causes people a lot of pain -- it's the common case to not care that the network is in use if it makes your builds faster and more reliable. If you don't want to accidentally leak things to the network, that's kind of your OS's or IT department's jobs. The tools are available and they will work with things that aren't explicit about needing the network. (What I'm saying is, if this concerns you, Go is one worry of many.) I have a concrete example about why module caching is good. I have an example repository that shows what happens when upstreams exercise poor release hygiene. Clone github.com/jrockway/evil-module-user. Run "go run main.go". (It's not actually evil, it just prints a message.) You will find that with the Go module proxy turned on, it compiles and prints "This is a good module!". With the proxy turned off, though, you will find that the checksum in go.sum doesn't match for the upstream github.com/jrockway/evil-module, and that evil-module has been silently compromised to become evil. What happened here is that evil-module released v0.0.1 and the module cache cached it. Then someone edited the code, and forced pushed the v0.0.1 tag to the repository, pointing at the commit. When go get reaches out to Github to clone github.com/jrockway/evil-module@v0.0.1, it gets code that doesn't match the checksum in github.com/jrockway/evil-module-user's go.sum, because it's not the v0.0.1 that the developer of github.com/jrockway/evil-module-user used when writing the code. Obviously this example is intentionally contrived to demonstrate this exact problem. But when I first started using Go modules early on, this happened all the time. Different engineers got different "version 0.0.1" from many repositories, and didn't notice there was a problem until CI. They then loudly complained that go modules suck, when in fact the problem is that upstream authors were very fast and loose about retagging releases. They are sending your release system code that you've never even seen! That's good that the module system detects that. But annoying to fix :) The problem all went away when Google started caching the modules -- you get the old version of the code, but at least everyone gets the same version. (As an author, you can always release v0.0.2 to bust the cache.) I actually thought until I investigated this in great detail recently that everyone had just embraced immutable version tags and were being smart about releases, as they learned more about Go modules. Turned out I was wrong -- people are as bad as ever, but they don't break your CI build because there is now a centralized cache that gives your workstation, your coworkers, and your CI system the same code. The upstream modules authors aren't doing that. They are force-pushing with abandon.

yencabulator 6y ago
goproxy and gosumdb are two separate services, you can get the TOFU benefits without goproxy, and a checksum database would be possible to query without even revealing what import path you are talking about (just like browser safe browsing).
- jrockway 6y ago
  I think the proxy is a bigger benefit than the checksum database. For the CI use case, it lets you get your modules from a server that's under less load than github (which has to support writes). Your go.sum already contains the checksums, so the sum database is not relevant there, but the module cache adds a big reliability and speed boost to your workflow. The checksum database is of course extremely important (you want to get a known-good copy the first time you use a module), but I stand by my assertion that it's beneficial for both to be turned on by default. Yes, it's a small privacy leak. But not one that is particularly personally embarrassing or useful to adversaries. So I think the go team made the right call.

enneff 6y ago

If you don’t want to use the proxy you can turn it off, trivially. M For the overwhelming majority of users the proxy is a huge benefit (installing and updating dependencies is way faster and the checksum database provides security benefits). Note that most other packaging ecosystems use centralised services, where you don’t even have the option of merely turning off the proxy.

yencabulator 6y ago
I know you know this, but gosumdb and goproxy are two very different entities, consumed separately. Let's not muddy conversations about the default centralized goproxy with the existence of a global TOFU service. gosumdb would be perfectly possible to query without even revealing a module import path you're asking for (just like browser-side safe browsing can do).