12 ms·
Go Module Mirror and Checksum Database Launched
- cube2222 7y agoFor anybody wondering if vgo is production ready, we've been using it since it came out in go 1.11 and so far didn't have any problems. Existing workflows don't break because there's the vendor command and dependency management is instantaneous. You can also roll it out gradually as it doesn't have any problems depending on non-module repositories (using commit hashes and dates) There's a great blog post series too if you're interested in the internals of go modules: https://research.swtch.com/vgo https://research.swtch.com/vgo (it's also one of the good counterexamples to people claiming go ignored x years of PL research and history)
- eknkc 7y agoI found the editor support to be still somewhat hit and miss though. VsCode go extension still does not work as good on module enabled projects for example. Otherwise we did not face any issues either.
- cube2222 7y agoOh yeah, that's something I heard about from my coworkers. Goland had support for go modules since the 1.11 prerelease as far as I remember, so I hadn't experienced it myself. However, most tooling works if you use go mod vendor prior to using it, that was the case for me at least when using linters and similar stuff.
- deforciant 7y agoI also got this problem when using modules with go mod but then I did "go mod vendor" and all extensions are now happy again for that project! :)
- calcifer 7y agoThat's a great reason to try out Goland, which has had excellent module support since day 1.
- weberc2 7y agoIf you can afford that $100-200/year subscription.
- heyoni 7y agoBut that’s the point isn’t it? Goland clearly goes above and beyond what free and open source can provide, so don’t feel like you’re getting ripped off paying for an IDE.
- pjmlp 7y agoWhich most professionals are able to.
- earthboundkid 7y agoI mean, if you're a professional Go developer, then yes, you can? You could even expense it or write it off.
- calcifer 7y ago> You could even expense it or write it off. If you only use it for your employer's code, sure. Otherwise it's a license violation. Personally, I think it's better to buy an individual license, which you can use for your day job as well as long as you are not compensated for it.
- sethammons 7y agoI bought it and stopped using it. It does work with modules, but I don't like the IDE itself and went back to vs code even though it breaks on me regularly. The UI/UX is just too foreign for me in GoLand. I had to search Google to find out how to search the project. Everything I wanted to do was non-discoverable for me.
- konart 7y agoCMD+SHIFT+F just like any other editor (vscode, sublime etc) out there? Also you have action manu on CMD+SHIFT+A that will give you any command IDE can offer. It's as discoverable as vscode if not more (you can even use IDE Features Trainer plugin if you want to)
- fourseventy 7y agoevery 6 months I spend half a day trying to get gocode and whatnot to work with my IDE and still can't get a reliable fix. Go modules really messed up a lot of dev tooling
- federicotdn 7y agoI had the same problem with Emacs, couldn't get it to work reliably.
- badrequest 7y agowith the addition of the `gopls` tool (https://github.com/golang/go/wiki/gopls https://github.com/golang/go/wiki/gopls), this shouldn't be nearly as much of a problem in the coming years.
- skybrian 7y agoShould anyone be using vgo anymore? The regular go command supports modules now.
- cube2222 7y agoIndeed, I meant go modules saying vgo.
- LukeShu 7y ago> For anybody wondering if vgo is production ready, we've been using it since it came out in go 1.11 and so far didn't have any problems. As a 2nd datapoint: We've been using it since 1.11.2, and have had problems, but have had no problems since 1.12.
- donatj 7y agoIs there a justifiable need for the proxy? It seems like we’re just adding a centralized point of failure, the lack thereof something I have absolutely touted as a feature. For verification we already have the go.sum file validating the hash. We’ve got vendoring if you’re concerned about dependencies disappearing. The touted speed feature seems like a non-issue? The absolute largest Go package I’ve ever pulled took maybe a minute to grab its dependencies, and then I had them and never had to again. The cynical side of me feels like this is a way for Google to collect analytics on the Go ecosystem. The slightly less cynical part of me thinks people from other ecosystems demanded it due to cargo culting the idea. The fact that it makes it more difficult to use private modules is pretty irritating as well as I am going to need to make sure all our developers and build systems are aware when we move to 1.13. Having to set GOPRIVATE on each machine is absurd. It should be definable in the go.mod so it can be known via git.
- DelightOne 7y agoA package repository can go down, vanish, versions can be deleted.
- jrockway 7y agoMy experience from building docker images of Go programs from Github repositories is that Github very aggressively rate limits you. More than one "go build ..." per 10 minutes and it just randomly won't send you the repository contents anymore. So a module proxy was a must. (You could also vendor dependencies, which is my preference, but my team was strongly opposed to vendoring. So we used Athens.)
- deleted 7y ago[deleted]
- w8rbt 7y agoThis is probably a dumb question, but here goes... I'm testing go modules with an internal project that uses a private github.com repo. It depends on other public go packages and modules is nice to manage those but the general public will probably never see or want to use this code. Are modules in private repos impacted by this in any way? Would they show-up in the index somehow?
- donatj 7y agoYou have to use the GOPRIVATE env variable to define what not to use the proxy for. https://tip.golang.org/cmd/go/#hdr-Module_configuration_for_non_public_modules https://tip.golang.org/cmd/go/#hdr-Module_configuration_for_...
- w8rbt 7y agoThank you for this. It seems they've thought of almost everything we do.
- ainar-g 7y agoSee here[1]. From what I've gathered, you should set GOPRIVATE appropriately, otherwise the Go Module Mirror(s) and the Go Checksum Database(s) may know that you have a private repo called “github.com/you/very-private”. It could also probably reason about your dependencies a little bit. [1]: https://tip.golang.org/cmd/go/#hdr-Module_configuration_for_non_public_modules https://tip.golang.org/cmd/go/#hdr-Module_configuration_for_...
- w8rbt 7y agoYes, I don't mind if it shows up in the index. I have nothing to hide. That just seems wasteful and clutterish. If too much of that occurs, the index could be littered with it. Sort of like old PGP keys that never go away.
- earthboundkid 7y agoIn 10,000 years, the Go code will be lost, but bored future people will reconstruct it from its hash through shear brute force searching. More seriously, if the hashes turn out to have serious security flaws, someone may be able to reconstruct the file someday.
- joppy 7y ago"Below is an example of such a tree"... and there is a picture of a binary tree, with no indication as to how the tree is used. I appreciate trying to explain the interesting technology behind authenticating modules, but this diagram appears to explain nothing at all.
- Verath 7y agoThere was a talk at gophercon that I belive goes into more detail about how this works and the cryptography behind it (I only skimmed it so far) https://youtu.be/KqTySYYhPUE https://youtu.be/KqTySYYhPUE
- bndw 7y agoDoes anyone know if the Go Proxy server using the original download protocol[0] defined by vgo? GET baseURL/module/@v/list fetches a list of all known versions, one per line. GET baseURL/module/@v/version.info fetches JSON-formatted metadata about that version. GET baseURL/module/@v/version.mod fetches the go.mod file for that version. GET baseURL/module/@v/version.zip fetches the zip file for that version. [0] https://research.swtch.com/vgo-module https://research.swtch.com/vgo-module
- secure 7y agoYes, see also https://youtu.be/WDbbIS7m9bU?t=298 https://youtu.be/WDbbIS7m9bU?t=298 (sorry, slide deck not yet available it seems)
- philips 7y agoI like the simple plain text signing format for the API calls. For example: https://sum.golang.org/lookup/go.etcd.io/etcd@v0.4.0 https://sum.golang.org/lookup/go.etcd.io/etcd@v0.4.0 Is signed with: — sum.golang.org Az3grobHchAJWrV4M34o1kLnZV4vrGSfFA+2Q9VClbmWqBjsnN4GzK1xB1RaYGSo0jIjWH9GDcR3Tja5sadw2ESoKwg= The format is documented here: https://godoc.org/golang.org/x/mod/sumdb/note https://godoc.org/golang.org/x/mod/sumdb/note Also, as an aside, I am building a transparency tool for arbitrary binary downloads with the rget project: https://github.com/merklecounty/rget https://github.com/merklecounty/rget
- benhoyt 7y agoThe format feels ad-hoc to me. These are intended almost exclusively for machine-reading, so why not JSON? One of the proxy endpoints (version metadata) returns JSON, so why not this one? If I were writing the client, I'd much prefer to parse a structured JSON object than that text file. So I'm curious why this choice -- I'm probably missing something.
- rsc 7y agoIn a security-critical context, I would much rather use a format where every byte is specified than delegate to something with as much flexibility as JSON. I mean, I guess we could have used a JSON object like {"Msg": "text", "Sig": "signature"} but there's so much opportunity for surprises there and so little benefit to JSON.
- rsc 7y agoI really like rget, btw. I posted a trivial little command-line demo of a client and server for some others who were thinking about a transparency tool. It implements the cacheable GET API we designed for the checksum database but obviously it applies to arbitrary key-value pairs. It might be a better foundation than stuffing things into CT logs (or not, you do get some interesting infrastructure that way). https://rsc.io/tlogdb https://rsc.io/tlogdb
- philips 7y ago
- programd 7y agoVendor. Everything. Always. I appreciate the fact that Google is investing in infrastructure to make the Go ecosystem robust, I really do. Having said that I encourage everyone to do a simple experiment. Check out your code into some random directory. Copy the directory to a USB drive and walk it over to an air gapped machine (no wi-fi, no ethernet, clean dev environment installed). Copy the directory to the box, make some small code change and try to build your binary. If your build fails you're doing it wrong. In real life one of the following will happen[1]: - Your network connection to the Internet will be down - Your network connection to the corporate network will be down - The services you depend on (Github, Google proxy, Docker hub) will be down - The services you depend on will throttle you - The services you depend on will serve you wrong or corrupt data - The services you depend on will be offline for maintenance - The certificates on the service you depend on were not renewed in time - The account you use on services you depend on will be inexplicably suspended Murphy's law being what it is, some or all of the above will happen when your SaaS is on fire, your customers are screaming, and you're trying to roll out a fix. Meanwhile your company is losing approximately your personal lifetime earnings in revenue every half an hour. Vendor. Everything. Always. (and have backups, lots of backups) [1] Based on highly educational career experience
- ndarwincorn 7y agoI don't see any of those as reasons to commit dependencies to the same VCS repo your source control lives in. Your first point is particularly laughable. If your network connection to the internet is down and you're a SaaS provider, you're probably going to need to fix that before you can roll out a fix to your hosted software. That said, they're all good points to consider when building out a deployment pipeline, and in general mirroring your dependencies is a great idea. But committing a vendor directory is a poor solution/mitigation for most of those risks.
- TheSwordsman 7y agoI'm not sure a solution that solves 100% of the problems, with minimal to no side effects, is a poor solution.
- eduren 7y agoThis looks great. I'm interested in the possibility of running a private mirror. Does anybody know if they've release the source for proxy.golang.org? I can't seem to find it in the golang github org.
- tjpnz 7y agohttps://proxy.golang.org/privacy https://proxy.golang.org/privacy A Google privacy policy that's actually reasonable.
- deleted 7y ago[deleted]
- tyri_kai_psomi 7y agoAlso I would like to bring up since JFrog's module proxy, GoCenter was publicly hosted first, they will have more than likely been first to host a module version. That means you are now open to the situation where someone uploaded v1.0 of a module to Go Center, and v1.0 of a module to Google's Mirror, and they contain different code. I am not sure how the Go team plans to handle this, but the only obvious solution would be for Google's proxy to reach out to GoCenter first to see if they have a copy of a specific version, download and cache, and serve that. Does anyone know if that happens?
- justinclift 7y agoWhy would two versions - specifically tied to a given commit - of say "v1.2.3" contain different code?
- deleted 7y ago[deleted]
- tyri_kai_psomi 7y agoWell you mentioned it in your comment, tags are not specific commits. Go Modules standardize around SemVer, and fallback to hashes only when it needs to. The reality is anyone can git push -f and change what a tag points to currently. So the scenario goes like this: 1. Someone requests a module from GoCenter that is currently not in it's caches. 2. Gocenter downloads the requested version v1.2.3, and stores that. 3. Bad/Careless/Ignorant/Evil (depending on circumstances, I guess) git push -f's 4. Module is requested in Google's module proxy, which now loads a "v1.2.3" into it's local caches, but it contains different code now than "v1.2.3" of GoCenter Outside of git push -f, there are other hacky techniques you can use to "update" the code a tag points to.
- justinclift 7y agoHmmm, maybe my understanding of tags is incorrect. When I tag something in git, that is for a specific commit isn't it?