22 ms·
Sourcehut will blacklist the Go module mirror
- lwagreputation 4y agoMhmmm. Yeah I wish the default were to not use a proxy. Though to be fair I'm not sure exactly what the performance implications would be.
- xena 4y agoFrom what I understand, the proxy also helps people make sure that an upstream deleting their GitHub repos doesn't result in builds breaking on new machines that don't have it cached locally. Imagine the problems that could happen if someone new joins your team, runs `go build` and then one of the vital dependencies 404s. The other problem is that it's Google so their perception of "not much traffic" is "biblical floods" to other people.
- _wolfie_ 4y agoBut that is what should happen. The build should break. So that someone can fix it.
- cp9 4y agothe blast radius of that outcome is another leftpad incident. the go team decided that they don't want that to happen
- psanford 4y agoWhy? If some open source maintainer goes off the rails and deletes all their packages, why should that break my builds? I still have a valid license to the code, I don't really care that a maintainer rage quit 4 dependencies down from my application. I certainly don't want to have to scramble to deal with that.
- luckylion 4y agoDo they change the behavior if the repo was dropped for legal issues, and no valid licenses could have been obtained because the repo owner didn't have one in the first place?
- psanford 4y agoThe go proxy won't save modules that don't have a permissive license.
- dpkirchner 4y agoThis is an argument for maintaining a local cache of necessary build dependencies, not to rely on a third party.
- psanford 4y agoBefore the proxy service that is what people did. Now we don't need to because the proxy service handles that automatically. You can run your own proxy service if you want. There's a large benefit to the go community as a whole for there to be a shared default proxy service.
- cesarb 4y ago> the proxy also helps people make sure that an upstream deleting their GitHub repos doesn't result in builds breaking This is a consequence of what IMO is another bad decision by the Go team: having packages be obtained from (and identified by) random github repos, instead of one or more central repositories like Maven Central for Java or crates.io for Rust. The proxy ends up being nothing more than an ad-hoc, informally-specified, bug-ridden re-implementation of half of a central repository, to paraphrase Greenspun's tenth rule.
- Macha 4y agoIf they change the case on their username on the other hand, the Go ecosystem explodes: https://github.com/sirupsen/logrus/issues/570#issuecomment-313933276 https://github.com/sirupsen/logrus/issues/570#issuecomment-3...
- MrStonedOne 4y ago[dead]
- xena 4y agoThey should probably use GOPRIVATE[1] instead. GOPRIVATE doesn't disable the module cache globally, it just disables it for individual domains or paths on domains. This is mainly used with private repo dependencies on GitHub. [1]: https://goproxy.io/docs/GOPRIVATE-env.html https://goproxy.io/docs/GOPRIVATE-env.html
- ddevault 4y agoThanks for the tip, will update the post.
- pantojax45 4y agoNot a good look for Google here.
- indrora 4y ago[flagged]
- galleywest200 4y ago> It should not be necessary to fetch the same git module up to 2,000 times per day. Holy cow Google! Wouldn't it behoove us to check if any changes occurred before downloading an entire repo?
- pdonis 4y agoI don't understand why they have to do a fresh clone every time.
- winterqt 4y agoOr, as mentioned in the post, why they don't do a shallow clone if they have to fetch it every time for whatever reason. Seems like a weird decision either way.
- mmis1000 4y agoYep, a shallow clone is enough to get the latest version. And you can even filter the tree to make the download size even smaller given you only want the hash but not the contents (if the git server supports this feature) A checkout with this can literally clone nothing but hash git clone --depth=1 --filter=tree:0 --no-checkout https://xxxx/repo.git cd repo git log
- WorldMaker 4y agoPeople using Go modules should be using git tags, right? They should have at least one hash already that should be infinitely cacheable, the tag commit. Of course, I have seen alleged examples of Go modules using tags like branches and force pushing them regularly, but that kind of horror sends shivers down my back, at least, and I don't understand why you'd build an ecosystem supporting that sort of nonsense and which needs to be this paranoid and do full repository clones just for caching tag contents. If anything: lock it down more by requiring tag signatures and throwing errors if a signed tag ever changes. So much of what I read about the Go module ecosystem sounds to me like they want supply chain failures. I don't understand the Go ecosystem.
- teeray 4y ago> and implies a trust relationship with Google to return authentic packages. The entire point of the sumdb (go.sum), is to prevent the need for such a relationship. If Google (or any proxy you use) tries to return questionable packages, it will be detected by that system.
- ignaloidas 4y agoIf the proxy changes a new version of a package, when you update it, there's no way to detect it since it fetches through the cache anyways, so a poisoned sum will be added to sumdb, and anyone who isn't fetching their packages through Google's proxy will get told that whatever they're using is trying to trick them.
- morelisp 4y ago> anyone who isn't fetching their packages through Google's proxy will get told that whatever they're using is trying to trick them. No, the error message you get is neutral about which side might be wrong - it says "verifying module: checksum mismatch" and "This download does NOT match the one reported by the checksum server." (I've seen it a lot because it also appears when module authors rebase, which a small but surprisingly high number do...)
- lanstin 4y agoWow, that is shocking. There is never a reason to rebase a public git repo, except maybe credentials leak in the past.
- adityasaky 4y agoEven then, you want to revoke those credentials rather than try to wipe it from history, no?
- lanstin 4y agoThat is what I think but security people want both.
- leetrout 4y agoI feel like Drew has been in a pissing match with the Go team for a while so this outcome doesnt surprise me. I love Go but not Google's stewardship of it. The tracking proxy, Russ' takeover / squash of the package management work, the weird silence / stonewalling on other community issues... Drew has a valid complaint. I hate to hear he was banned from the issue tracker but that sounds about right. As a sibling said - GOPRIVATE is probably a good solution without throwing the baby out with the bathwater.
- wwarner 4y agoHave to agree in as much banning people with legitimate issues is unhelpful. I was also taken aback when Peter Bourgon, a great programmer and contributor to the go ecosystem was banned from all go channels.
- Beltalowda 4y ago> I was also taken aback when Peter Bourgon, a great programmer and contributor to the go ecosystem was banned from all go channels. Bourgon was frequently helpful and great, but also frequently rude, condensing, dismissive, and generally just unpleasant. I've seen this countless of times first-hand on Slack, Reddit, and Lobsters. I specifically stopped interacting with him long before he was banned. Whether he's a great programmer/contributor not isn't really important here. I kind of hate how it's brought up here because I think he's not a bad bloke at all and I know the entire situation caused great personal hurt to him. But that doesn't change that he was the kind of "brilliant jerk" that would chase people out of the community with his behaviour, and that he was unreceptive to criticism of it (often getting pretty defensive/aggressive). No one liked how all of this turned out, but it did make the Go community a better place. Being helpful yesterday doesn't cancel out being a jerk today. Same with Drew: he posted legitimate helpful issues. And he also ranted about how people were all a bunch of morons. I don't blame anyone for getting tired of that.
- Semaphor 4y ago> And he also ranted about how people were all a bunch of morons. I don't blame anyone for getting tired of that. Do you have a source?
- xenophonf 4y agoThis: > I was banned from the Go issue tracker without explanation, and was unable to continue discussing the problem with Google. is completely asinine. But it's also par for the course when it comes to interacting with Google. When is anyone going to hold them to account for their terrible customer service and community interaction?
- Thaxll 4y agoIt's one side of the story. I don't think he was banned for no reasons.
- yxhuvud 4y agoPerhaps, but those reasons should still be communicated.
- ComputerGuru 4y agoKnowing him, it probably wasn’t nothing. But that also does Dr matter - he runs a hosting service that sees a fair amount of traffic and the onus is on Google as stewards of the golang community and the brilliant minds behind invisibly proxing requests through Google’s servers to continue to figure this issue out. They have his email address.
- ddevault 4y agoI would share their side as well, but I never heard it. This is a violation of their own code of conduct, which requires them to notify the affected person, explain why, and offer the opportunity to mediate the situation. This is not the first time I was banned from the Go community without notice or explanation, and the first time turned out to be frivolous -- the ban was overturned months later with an admission that it was never justified in the first place. Community management in Go strikes me as very insular and unprofessional. Regardless, I don't really want to re-litigate it here. The main issue is that Google has been DoS'ing SourceHut's servers for two years, and I think we can all agree that there is no conduct violation for which DoS'ing your servers is a valid recourse.
- jchw 4y agoI find it annoying that blocking the Go module proxy (on the server side) doesn't cause a graceful fallback. Am I unreasonable in thinking that it should? Doesn't the sum file prevent you from getting a maliciously modified copy?
- MrStonedOne 4y ago[dead]
- ysleepy 4y agoAre google's requests done on-demand, or part of some scheduled refresh? I am wondering, because even if google's traffic is unreasonable, it might still be less than without the proxy. CI configurations are notoriously inefficient with dependency fetching, so I would not be surprised if the actual client traffic is massive and might overwhelm sourcehut if all migrate to direct fetches.
- ComputerGuru 4y agoPer the issue, both.
- lanstin 4y agoI recently moved to docker build pipeline for a project, and it’s redownloading all deps on each source file change, unlike the efficient on disk incremental compilation, because of how docker layer caching works, so my usage skyrocketed (and my build times went from seconds to minutes).
- lanstin 4y agoSince writing this, I have seen the trick - copy just the go.sum and go.mid files into an earlier layer, then get the deps, then copy all the other source code.
- ComputerGuru 4y agoFrom the GitHub issue, by a Googler: > For boring technical reasons, it would be a fair bit of extra work for us to read robots.txt […] This is coming from one of the biggest, richest, most well-staffed companies on the planet. It’s too much work for them to read a robots.txt file like the rest of the world (and plenty of one-man teams) do before hammering a server with terabytes of requests. If this is too much for them then no wonder they won’t implement smarter logic like differential data downloads or traffic synchronization among peer nodes.
- psanford 4y agoWhy did sourcehut not take the offer to be added to the refresh exclusion list like the other two small hosting providers did? It seems like that would have resolved this issue last year.
- fluidcruft 4y agoI wondered that too, but then I wondered if that's what sourcehut have actually done. I didn't notice any details about how go module mirror will be blocked. Wouldn't the effect on sourcehut users be identical?
- psanford 4y agoNo. The Go team offered to add sourcehat to a list that would stop background refreshes. It would still allow fetches initiated by end users. The change sourcehat is making is to break end users unless they set some environment variables. I've not seen any explanation about why the solution offered by the Go team was unacceptable. Its weird that that is completely left out of the blog post here.
- Tomte 4y agoI think Google should DDoS noone. Not everyone until they opt out.
- 4y ago
- comex 4y agoAccording to some comments in the linked GitHub issue, including [1] from last May, Drew could have simply asked to be excluded from automatic refresh traffic from the mirror. If I understand correctly, that would still leave traffic from the mirror when it’s acting as a direct proxy for someone’s request, but that is traffic that would be going to sr.ht regardless. For some reason he did not do this and instead chose an option that causes breakage. [1] https://github.com/golang/go/issues/44577#issuecomment-1137822116 https://github.com/golang/go/issues/44577#issuecomment-11378...
- deleted 4y ago[deleted]
- FiloSottile 4y agoI can confirm asking to be excluded from refreshes (which AFAIK is still a standing offer, but I obviously can't speak for the mirror team because I am not at Google anymore) would stop all automated traffic from the mirror, and that sr.ht could send a simple HTTP GET to proxy.golang.org for new tags if it wished not to wait for users to request new versions.
- throwaway_62022 4y agoI think it is problematic that we are using github issues as "support forum" for asking a git host provider to be excluded from the refresh list. This should not have come to that. Whatever happened to "reasonable defaults", so as a random person hosting a single Go module doesn't get DOSed - https://github.com/golang/go/issues/44577#issuecomment-860877028 https://github.com/golang/go/issues/44577#issuecomment-86087... ?
- FiloSottile 4y agoEveryone can make their own assessment of what is a reasonable default and what counts as a DoS (and they are welcome to opt-out of any traffic), but note that 4GB per day is 0.3704 Mbps.
- nemith 4y ago
- deleted 4y ago[deleted]
- zozbot234 4y agoSr.ht should set up their own graceful rate-limiting anyway, or they're going to have problems with badly coded CI setups as the service becomes more popular.
- necron99 4y agoWouldn't this just make the failures more mysterious and harder to track down for users? Failing sometimes for no reason that's obvious or apparent to end users is worse than always failing, IMO. E.g.: source-based packages on distributions where users may not be Go programmers, or even non-programmers, will compile and install Go software where some nested library dependency is on sr.ht. These packages will now fail and, sadly, this is going to cause widespread disruption. I think it'd be worse if those failures only happened occasionally, and not reliably repeatably.
- deleted 4y ago[deleted]
- cratermoon 4y ago> The situation remained so for over a year. In that time, I was banned from the Go issue tracker without explanation, and was unable to continue discussing the problem with Google Something tells me there's another side to this story that we aren't hearing.
- josephcsible 4y agoWhy? Google has quite a reputation for wrongfully banning people from things. This doesn't seem unusual for them at all.
- cratermoon 4y ago> Google has quite a reputation for wrongfully banning people from things. Extraordinary claims require extraordinary evidence.
- wnoise 4y agoExtraordinary claims do. Ordinary claims only require ordinary evidence.
- jessaustin 4y agoThere's nothing "extraordinary" about that claim. All online service providers do this, Google especially, and everyone on the internet knows that.
- josephcsible 4y agoHere's some evidence: https://news.ycombinator.com/item?id=3803568 https://news.ycombinator.com/item?id=3803568 https://news.ycombinator.com/item?id=15066518 https://news.ycombinator.com/item?id=15066518 https://news.ycombinator.com/item?id=19124324 https://news.ycombinator.com/item?id=19124324 https://news.ycombinator.com/item?id=20826618 https://news.ycombinator.com/item?id=20826618 https://news.ycombinator.com/item?id=20841586 https://news.ycombinator.com/item?id=20841586 https://news.ycombinator.com/item?id=21247759 https://news.ycombinator.com/item?id=21247759 https://news.ycombinator.com/item?id=24791357 https://news.ycombinator.com/item?id=24791357 https://news.ycombinator.com/item?id=24965432 https://news.ycombinator.com/item?id=24965432 https://news.ycombinator.com/item?id=26061935 https://news.ycombinator.com/item?id=26061935 https://news.ycombinator.com/item?id=26956077 https://news.ycombinator.com/item?id=26956077 https://news.ycombinator.com/item?id=26965146 https://news.ycombinator.com/item?id=26965146 https://news.ycombinator.com/item?id=28621412 https://news.ycombinator.com/item?id=28621412 https://news.ycombinator.com/item?id=30855065 https://news.ycombinator.com/item?id=30855065
- SuperSandro2000 4y agoSo either you disable the proxy completely for your site or get overwhelmed by traffic from a not to well written service, similar to a small DDOS attack, which is run by google and they are not planning to fix this? Did I get anything wrong here?
- tssva 4y agoThere is a refresh exclusion list which you can request your site to be added to. The proxy will continue to process requests for modules from your site but will not perform the automatic refresh which caused issues for Sourcehut. The Go team extended an offer to add Sourcehut to the list if a request to do so was made. The request never came and instead Sourcehut blocked the proxy.
- 0cf8612b2e1e 4y agoDoes every site need to personally request they not be DoS attacked?
- cbrozefsky 4y ago"Opt out of me DoSing" you is not legit behavior -- especially when the victim has raised it with you multiple times, suggested fixes, and you have then blocked them from communicating in your issue tracker. That's some really entitled thinking on the part of the Go team at Google, and it's sad to see people stanning for them.
- sidewndr46 4y agoThey should have them add a file to their web service at a path of "/i_want_to_live.txt" to indicate not to DoS the server.
- jdiff 4y agoIf an amount of traffic that nobody else even notices brings you to your knees, you're doing something wrong. Go could be far more efficient but pointing at this and calling it a DDoS is silly.
- deleted 4y ago[deleted]
- simonw 4y ago> Perform a shallow git clone rather than a full git clone; or, ideally, store the last seen commit hash for each reference and only fetch if it has been updated. I'd be interested to understand why that solution hasn't been implemented yet.
- deleted 4y ago[deleted]
- paulcarroty 4y agohttps://git.sr.ht/robots.txt https://git.sr.ht/robots.txt > YandexBot is a dickhead, too aggressive > User-agent: Yandex > Disallow: / Hahaha.
- n8ta 4y ago# Used for Alexa, I guess, who cares User-agent: Amazonbot Disallow: /
- deleted 4y ago[deleted]
- rakoo 4y ago> or, ideally, store the last seen commit hash for each reference and only fetch if it has been updated. Isn't that exactly what "git fetch" is doing already ?
- WhyNotHugo 4y agoYes, but they do a complete clone on each run.
- necron99 4y agoWhether Drew's right or wrong, this is disappointing. At the very least, it's going to make recommending Sourcehut as an alternative to Github harder to justify. This is really unfortunate, especially as sr.ht isn't free (any longer). As far as I know, it's the only remaining managed source hosting/CI option for Mercurial users. What a depressing way to start out 2023.
- MaxBarraclough 4y ago> especially as sr.ht isn't free (any longer) This isn't a bad thing, it was always meant to be financially supported by its users. The alternatives are ads or for Drew and co to run it at a loss indefinitely.
- counttheforks 4y agoI'll just rehost my git repos on a git host who doesn't mind actually doing their job as a git host by letting git clients clone git repos.
- Arnavion 4y agoWhat does your ideal git host do when one of its (non-paying) clients uses bots to clones repos so hard that its (paying) human clients are unable to clone their repos?
- counttheforks 4y agoImprove infrastructure. Apparently they are trivially DDOSable by anyone renting a few servers and running git clone in a loop? That's a problem they should solve. And no, blocking by user agent is not adequate protection.
- Arnavion 4y agoInfrastructure costs. So you'd be willing to pay more for that?
- counttheforks 4y agoI already pay for GitHub, am fine with that and they manage. So yes.
- lmas 4y agoYou're not paying for the traffic load though, Microsoft does. I think it's unfair to compare a forge run by a giant and one run by a tiny company, or for that matter, self hosting users.
- 0cf8612b2e1e 4y agoSome napkin math: - peak queries are 2500 requests per hour - some repos are 4gb in size - let’s round and say 2000 requests at 1gb = 2000 gb/hour = 48000 gb/day - AWS bandwidth at $0.02 / gb = $960 / day = $28,800 / month So, one of the richest companies in the world is charging you nearly $30k monthly because they cannot be bothered to be polite. Would you be ok with that situation?
- adenozine 4y agoWhat's the value proposition of not using Github? Github is such an incredibly useful project. I actually go out of my way to avoid projects on Gitlab and co, just because I don't want to have to worry that it's going to just disappear one day because they thought they could out-build Microsoft.
- vladharbuz 4y agoYour argument seems to be based on the idea that Microsoft would do a better job than its competition technically. The fact that you're happy to rely on them also seems to imply you think they're unlikely to act abusively. However, you're writing your comment in a thread about a similar megacorporation, Google, who is acting abusively, because their engineers are saying parsing a robots.txt file would be too difficult.
- adenozine 4y agoI actually wasn't making an argument but posing a question. Your reply seems defensive and emotional. I'm not sure what you mean by "act abusively" since the core of the product is storing and viewing source code. I mean, I could verify checksums if I needed to, and such a thing is common in workplaces I frequent. What I asked before, and I will reiterate for an additional attempt to your comprehension, is how can Sourcehut store and view source code BETTER than Github can?
- rsc 4y agoThe Go team has been making progress toward a complete fix to this problem. Go 1.19 added "go mod download -reuse", which lets it be told about the previous download result including the Git commit refs involved and their hashes. If the relevant parts of the server's advertised ref list is unchanged since the previous download, then the refresh will do nothing more than the ref list, which is very cheap. The proxy.golang.org service has not yet been updated to use -reuse, but it is on our list of planned work for this year. On the one hand Sourcehut claims this is a big problem for them, but on the other hand Sourcehut also has told us they don't want us to put in a special case to disable background refreshes (see the comment thread elsewhere on this page [1]). The offer to disable background refreshes until a more complete fix can be deployed still stands, both to Sourcehut and to anyone else who is bothered by the current load. Feel free to post an issue at https://go.dev/issue/new https://go.dev/issue/new or email me at rsc@golang.org if you would like to opt your server out of background refreshes. [1] https://news.ycombinator.com/item?id=34311621 https://news.ycombinator.com/item?id=34311621
- cstejerean 4y agoI suspect they have a problem with this DDoS by default unless you ask to opt out behavior. Why is anyone getting hit with these expensive background refreshes until you have a chance to do it right? Why is it still not done right 2 years after this was first reported? Maybe it should be an opt-in list where the big providers (such as github) can be hit by an army of bots and everyone else is safe by default.
- dpkirchner 4y agoAnd why isn't this a priority? These folks are offering a service that benefits the Go ecosystem. It smells like Go is on its way out.
- jdiff 4y agoThis smells wildly overdramatic. They've been working on solutions big and small since it was reported, it's just that the big solutions take time, this was communicated to Drew.
- n4jm4 4y agoVendoring in go mod should be mandatory.
- kwhitefoot 4y ago> "it would be a fair bit of extra work for us to read robots.txt", Can someone explain what this means? After all reading a small text file is plainly easy to do so its meaning must be something other than obvious.
- yamtaddle 4y agoAll I can figure is they're trying really, really hard to keep this very parallel on their side and also avoiding having to coordinate between nodes. It can't possibly be the reading of the robots.txt that's hard, so I think that statement has more to do with applying those policies to all the nodes—they must regard coordination between nodes to ensure the system as a whole isn't exceeding e.g. request rate maximums, to be "a fair bit of extra work". Judging just from the linked post, the issue on which this was discussed, and this thread, it's feeling a lot like this proxy was some kind of proof-of-concept that escaped its cage and got elevated to production.
- daenney 4y agoI can see this happening on my own git hosting too. I've started moving my Go code off GitHub and the module mirror shows up every 25min for each repo it's aware of doing a full clone. Thankfully the few modules I've moved are very small ones with very little history. This won't come anywhere near my egress allocation for that box. But the whole thing is frankly a little rude.
- lmas 4y agoYeah I wanted to do the same thing and self host my repos but that will have to wait now until this issue is fixed, it looks like... Could you tell me more about what tool you used to host the repos? And I assume you noticed the traffic in your web logs?
- davidbanham 4y agoThe proposed resolutions from Google have been along the lines of changes to the proxy application code. The resistance has been that this is hard because of the way responsibility is divided between the thin layer that provides the proxy web service and the ‘go’ command itself, which actually fetches the packages. Would it be simple to solve this with an additional layer of the same proxy? Currently, end users request a package from proxy.golang.org as per the default value of the GO_MOD_PROXY env var. Google runs many of these to handle the traffic, let’s say 1000. They all maintain a mirror of all packages that have been recently requested (note that I expect there’s more nuance here around shared lists of required packages, etc, but the structure should hold true) The result is that every one of the 1000 proxy instances requests the source data from git.sr.ht every day. Google could set GO_MOD_PROXY on the existing instances to internalproxy.golang.org. They could then run 100, or maybe 10 of these internal instances. This would drop the traffic to hit.sr.ht by one or two orders of magnitude. I suspect it would require minimal if any change to application code. This might be accomplished entirely within the remit of a sysadmin (SRE?). Any holes in my reasoning?