15 ms·
Google has been DDoSing Sourcehut for over a year
- nurettin 4y agoWhy not just stop using Go? Why not Nim? Crystal? why not even Haxe? Why not stop using things built by and built for EvilCorp (TM)?
- kS6n6GN8If 4y agoAthens, a self-hosted free Go proxy implementation, implements rate-limiting for GitHub (with a pretty horrific Github-specific behavior). This means GitHub does implement rate-limiting to cope with aggressive Go proxies. https://github.com/gomods/athens/blob/723c06bd8c13cc7bd238e650a559258ff7e23142/pkg/module/go_get_fetcher.go#L145-L148 https://github.com/gomods/athens/blob/723c06bd8c13cc7bd238e6... Food for thoughts.
- grawlinson 4y agoThis is still occurring? I remember the original incident and just assumed that Google had sorted it out with Drew/Sourcehut. This is absolutely shocking behaviour, and I’m mortified at the precedent that it sets.
- rsstack 4y agoThey didn't sort it out, they just banned him from commenting. So kind of problem solved for Google?
- deleted 4y ago[deleted]
- ocdtrekkie 4y agoThe fact that a programming language calls home to by Google by default should make it a non-starter for most sane developers. The fact that it calls home so it can DDoS other sites is low-key hilarious. And you'd think Google would know how to like... operate an efficient CDN, perhaps? Like, if this was managed by a competent company, you'd think this service would be akin to putting Cloudflare in front of your servers: It should minimize the heck out of your traffic because Google is able to cache it and serve it to the masses more efficiently. But it's Google, so it Googles. Being blocked from the issue tracker for not being okay with being DDoS'd is peak Google. I am sure you were accused of a Code of Conduct violation, because declaring a CoC violation is much, much cheaper than fixing their infrastructure.
- sascha_sl 4y agoAre you going to get upset at node for calling home to Microsoft (npm owned by github owned by microsoft) when using the supplied package management too?
- ocdtrekkie 4y agoI mean, I strongly just feel the npm ecosystem is a dumpster fire of sadness. Though there is a fundamental difference in that if you use NPM, you are choosing to download packages from NPM. But if you specify a bunch of repos you want to download Go packages from, and you get them from Google anyways, that's pretty uncomfortable. Especially if Google is in turn also DDoSing the repos in return.
- pretext-1 4y agoNo I did not "choose to download packages from NPM". Choosing means there is an alternative. And besides that the registry part is invisible in your package file (hardcoded default like GOPROXY). If you look at it, it's pretty similar. The part before the first slash is invisible: npmjs.com/org/package proxy.golang.org/org.com/package The difference with Go is that you can actually change the first part if you want, or even disable it.
- bsder 4y agoYes. I find all of the modern "always connected, always call home" behaviors quite annoying in my programming tools. But, to be fair, Node was a dumpster fire long before being bought by Microsoft.
- deleted 4y ago[deleted]
- ezfe 4y agoHow could node not do that? NPM hosts all the packages so of course it does.
- dmm 4y ago> The Go team holds that this service is not a crawler, and thus they do not obey robots.txt This seems wrong. I guess I always assumed that robots.txt applied to non-humans.
- jaywalk 4y agoI'm sure the reasoning would be something along the lines of these requests being made in response to user activity of some sort.
- kevincox 4y agoI think they are right not to obey robots.txt in this case. If I tell Go to download a module it shouldn't follow robots.txt because I am a human and I requested it. This is similar to if you had private iCal URLs on your server the right think would be to deny them in robots.txt (in case a crawler found a leaked link) but a service that is monitoring a specific iCal calendar should still fetch it. Basically robots.txt is more about what should be crawled than how. In this problem it appears that the traffic is desired in general, but it is being done far too often. robots.txt does have primative rate limiting configs but that seems to be a minor part of the file. Of course like anything there is nuance and there is definitely some middle ground between crawlers and humans.
- forty 4y agoPrecisely, here the service is fetching stuff regularly even if no one is asking anything, if I understand correctly.
- willcipriano 4y agoIt apparently pulled a repo only used by the author 500 times in a day.
- AshamedCaptain 4y agoEven the main Google search crawler does not respect Crawl-Delay from robots.txt (as of today), so it's a moot point anyway. I suspect it's just a cultural thing inside Google.
- bryan_w 4y agoWait... Why not black hole them? Make it their problem and they'll eventually do the correct thing.
- toast0 4y agoIf it were me, and I wasn't willing to just block the traffic, I might just set a 128 kbps limit on it and call it a day[1]. Eventually, the other side will figure out that their fetchers are all backed up and work out how to do their job without burning so much bandwidth. [1] Yeah, that can be a bit of a pain to setup depending on the server settings, but some people have to pay for bandwidth and server resources, so it's probably worthwhile.
- WhyNotHugo 4y agoKeeping the rate-limited connection alive eats up resources. In this case, the organisation executing the DDOS attack has virtually limitless resources.
- mpol 4y agoI guess this is the professional response ;) I would just blacklist the IP addressess, just to see what would happen.
- toast0 4y agoYeah, but that's not an option according to the article. > I can’t blackhole their IP addresses, because that would make all Go modules hosted on git.sr.ht stop working for default Go configurations (i.e. without GOPROXY=direct)
- CryptoBanker 4y agoThat’s more of a “won’t” than a “can’t”
- threatofrain 4y ago> I was banned from the Go issue tracker for mysterious reasons, so I cannot continue to nag them for a fix.¹ I can’t blackhole their IP addresses, because that would make all Go modules hosted on git.sr.ht stop working for default Go configurations (i.e. without GOPROXY=direct). I tried to advocate for Linux distros to patch out GOPROXY by default, citing privacy reasons, but I was unsuccessful. I have no further recourse but to tolerate having our little-fish service DoS’d by a 1.38 trillion dollar company. But I will say that if I was in their position, and my service was mistakenly sending an excessive amount of traffic to someone else, I would make it my first priority to fix it. But I suppose no one will get promoted for prioritizing that at Google. > [1]: In violation of Go’s own Code of Conduct, by the way, which requires that participants are notified moderator actions against them and given the opportunity to appeal. I happen to be well versed in Go’s CoC given that I was banned once before without notice — a ban which was later overturned on the grounds that the moderator was wrong in the first place. Great community, guys.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- FiloSottile 4y ago> In the meantime, if you would prefer, we can turn off all refresh traffic for your domain while we continue to improve this on our end. That would mean that the only traffic you would receive from us would be the result of a request directly from a user. This may impact the freshness of your domain's data which users receive from our servers, since we need to have some caching on our end to prevent too frequent fetches. https://github.com/golang/go/issues/44577#issuecomment-856928441 https://github.com/golang/go/issues/44577#issuecomment-85692... > "EFAIL" is an alarmist puff piece written by morons to slander PGP and inflate their egos. [...] https://github.com/golang/go/issues/30141#issuecomment-464278183 https://github.com/golang/go/issues/30141#issuecomment-46427... Disclosure: I was on the Go team at Google until earlier this month. Dealing with DeVault's bad faith arguments is one of the few things I won't miss of that job.
- yowlingcat 4y agoProbably the most egregious part is the footnote: ``` I was banned from the Go issue tracker for mysterious reasons[1], so I cannot continue to nag them for a fix. [1]In violation of Go’s own Code of Conduct, by the way, which requires that participants are notified moderator actions against them and given the opportunity to appeal. I happen to be well versed in Go’s CoC given that I was banned once before without notice — a ban which was later overturned on the grounds that the moderator was wrong in the first place. Great community, guys. ↩ ``` What happened to "don't be evil" I wonder? (And I know, it's that Google has now become a big grown up corporation)
- joshuamorton 4y agoThere is a very simple way to get them to stop sending the .5 qps that is described as a ddos. The linked bug show another user successfully saying "please opt me out", and Google building the feature to do that in a week. Drew has for some reason chosen not to ask for an opt out, even though it appears trivial and would probably be fixed by the weekend if he asked for it.
- Rantenki 4y agoCalling it .5 QPS to downplay the severity is willfully ignoring the complaint in the article. It's not just a query, it's a full Git clone of the entire repo with it's history. That's a huge difference.
- tptacek 4y agoThat's what a "Q" means in this context. It's a git hosting service.
- CaliforniaKarl 4y agoTrue, though I don't think a "Q" from a client doing a fetch involves as much work as a "Q" from a client doing a full clone. It seems like this Git hosting service's workload-per-Q has increased due to the Go proxy change, which (IMO) is a cause for concern.
- mig39 4y agoI think he's been banned from the issue tracker.
- joshuamorton 4y agoPossibly, but he was actively participating in https://github.com/golang/go/issues/44577 https://github.com/golang/go/issues/44577 up until the week it was fixed. If that's the root of the issue though, most of the article ("no one will get promoted for prioritizing that at Google.", "the go team has not prioritized it", etc.) is wrong. They may not have addressed the issue in the precise way he wanted, but I also think it's rather unreasonable to expect someone to not only address your issue, but address it in the precise way you suggest. That's rarely globally optimal and feels rather entitled.
- dgb23 4y agoSo this has been going on for a year, Drew DeVault has actively tried and explored several sane options and channels. That's a quite a bit of patience and goodwill before making a public blog post. There are basic mechanisms in place both on the git side and on the protocol side that can prevent this kind of wasteful, digital harassment.
- deleted 4y ago[deleted]
- zaphar 4y agoDrew has built himself a reputation of complaining loudly about stuff that most of the general public of users don't care about and generally unnecessarily increasing the burden for Open Source maintainers. In this particular case I think he has a valid point but his history is probably getting in the way of successfully getting people to see it. The lesson here, is that if you make everyone dislike dealing with you then you'll find it very hard to get them to work with you when you need them to.
- Ventito 4y agoHave you not tried to get in concat with anyone else before? "mysterious reasons" Did you try go get in contact with them directly and clarify it? Did you talk to a moderator? Besides that, you are hosting code right? How is 70gb of traffic a DDos? I have to say, traffic numbers like you posted, would not concern me at all. I'm very curious why your underlying git usage can't do better caching.
- crooked-v 4y ago> Did you try go get in contact with them directly and clarify it? Did you talk to a moderator? As noted in the article, he did, and then Google banned him without warning.
- tptacek 4y agoThat's not what the article says. It says he doesn't know why he was banned; it doesn't establish causality.
- uoaei 4y agoThe comment above does not claim causality, only a suggestive ordering of events.
- Ventito 4y agoI was not able to find anything in this regard. I only saw a quite pro active ticket thread
- garaetjjte 4y agoI don't get it. Without the proxy, wouldn't all the (potentially larger) traffic just hit the server directly anyway?
- 8organicbits 4y agoMy read is that there's a refresh job that's doing hundreds of full git clones even for packages that don't have users. Package with approx. zero users: https://github.com/golang/go/issues/44577#issuecomment-851079949 https://github.com/golang/go/issues/44577#issuecomment-85107... Full clones: https://github.com/golang/go/issues/44577#issuecomment-789242343 https://github.com/golang/go/issues/44577#issuecomment-78924... They disabled refreshes (for someone else) to fix the issue: https://github.com/golang/go/issues/44577#issuecomment-862719746 https://github.com/golang/go/issues/44577#issuecomment-86271... So maybe there is some caching happening between the user and the mirror, but the mirror is hammering the origin with much more load than should be needed.
- captainmuon 4y agoIt sucks having to work around something like this, but maybe the following would work: only allow the first checkout from a given Go node, blackhole later accesses. If the repository is modified or a certain time elapsed, reset and allow a download again. Also, if you want to escalate, I wonder if there is a way to create a fake git repository that expands to a huge amount of data when cloning, but uses minimal bandwidth on the server side. Set up a bunch of those on some other host, and use them from go, and wait until google notices. Something like this: https://github.com/ilikenwf/git-zlib-bomb https://github.com/ilikenwf/git-zlib-bomb
- verdverm 4y agoMalicious behavior would come back to byte the author and his users & business
- SSLy 4y agoYes, I also hope EU breaks Google apart.
- captainmuon 4y agoNot neccesarily. I was thinking of this story: https://news.ycombinator.com/item?id=26074139 https://news.ycombinator.com/item?id=26074139 > At the height of the browser wars I once woke up to Microsoft hotlinking a small button for downloading our software from the MSN homepage. I tried to reach someone there for hours but nobody cared enough to do something about it. The image was small (no more than a few K), but the millions of requests that page got were enough to totally kill our server. > Finally, I replaced the image on there with a 'Netscape Now' button. Within 15 minutes the matter was resolved. And there was another case where some Indian app was using excessive traffic from Wikimedia, which was solved similarly. Sometimes you have to do something to get the attention of the right people. (And I could imagine if you do it right and are not too malicious, you might actually get a bug bounty on top.)
- minus7 4y agoThe problem is that a) you want it to work properly for users of Go modules hosted on your Git forge and b) you don't want to effectively waste time on working around a problem that the GoProxy engineers could just solve with a more intelligent (but maybe more complex) design.
- 8organicbits 4y agoFrom https://github.com/golang/go/issues/44577#issuecomment-851079949 https://github.com/golang/go/issues/44577#issuecomment-85107... > Yesterday, GoModuleMirror downloaded 4 gigabytes of data from my server requesting a single module over 500 times (log attached). As far as I know, I am the only person in the world using this Go module. From https://github.com/golang/go/issues/44577#issuecomment-789242343 https://github.com/golang/go/issues/44577#issuecomment-78924... > yes we make a fresh clone every time I like golang as a developer, but this is a terrible implementation. I'm somewhat tempted to say that blocking the Google IP addresses is the correct answer in that it will force some sort of wider action (linux repos setting `GOPROXY=direct`, Google fixing their code, or unfortunately, golang modules moving off sourcehut).
- theamk 4y agoFrom https://github.com/golang/go/issues/44577#issuecomment-1137818914 https://github.com/golang/go/issues/44577#issuecomment-11378... > Anyone who's receiving too much traffic from proxy.golang.org can request that they be excluded from the refresh traffic, as we did for git.lubar.me. Nobody asked for sr.ht be added to the exclusion set, so as far as it's concerned nothing has changed.
- broknbottle 4y agosounds totally scalable especially coming from a company that is notorious for having zero support. /s
- deleted 4y ago[deleted]
- waych 4y agoI'm curious if the Go crawler would correctly back-off with "429 Too Many Requests" responses. Either way it could save the bandwidth.
- PufPufPuf 4y agoOf course not, the crawler doesn't even read robots.txt because it's "too difficult to implement"
- VWWHFSfQ 4y agoNo offense, but you built a git hosting website. and now you don't like the (pre-existing) ecosystem of things that use it? Maybe just block go packages altogether if it's that big of a deal
- yjftsjthsd-h 4y agoThere's a difference between making something available and being okay with wasteful and excessive usage.
- trhway 4y agoone can definitely see what people are promoted for at Google :) > the Go Module Mirror runs some crawlers that periodically clone Git repositories with Go modules in them to check for updates. >The service is distributed across many nodes which all crawl modules independently of one another, resulting in very redundant git traffic. basically slapping together a very inefficient alpha, and obviously people aren't promoted for fixing such glaring inefficiencies to make it even into a half reasonable beta. And that is just hilarious, like people in Google never heard of CDN, HEAD, git fetch, etc. - of course they know it, and it is really just an arrogance of an 800lb gorilla toward "small-fish" - "https://github.com/golang/go/issues/44577#issuecomment-856928441 https://github.com/golang/go/issues/44577#issuecomment-85692... - with a passive-aggressive blackmail of a cherry on top : >In the meantime, if you would prefer, we can turn off all refresh traffic for your domain while we continue to improve this on our end. That would mean that the only traffic you would receive from us would be the result of a request directly from a user. This may impact the freshness of your domain's data which users receive from our servers
- 8organicbits 4y agoI'm wondering how much load this is sending to github. Github has a ton of golang packages, including many non-popular ones that wouldn't otherwise get much traffic. A refresh job running full clones many times a day must be burning up bandwidth and compute over there as well. I suppose it's a drop in the bucket for Github's usage, but it's got to be a huge number.
- exikyut 4y agoI wouldn't be surprised if half the value of the Microsoft acquisition was to get GitHub's infra closer to the super-scale pipes and peering that Azure/Office/Update/etc leverage. I would expect they probably run tens to hundreds of PBs per month.
- tete 4y agoOr they have custom code for bigger services.
- midislack 4y agoI’m surprised. Most DDOS’s come from Microsoft these days. Azure Cloud is the #1 go to place for botnets to attack the Internet. I suspect it’s allowed on an official level, it’s too flagrant to be done without their knowledge or consent.
- tptacek 4y agoIt's not DDOS. That's "rhetorical flair" from the author.
- spacexsucks 4y ago
- sreevisakh 4y agoThis a consistent trend on threads that are critical of Google and Apple.
- tptacek 4y agoYou cannot write comments like this on HN.
- spacexsucks 4y agoStop this go madness.
- mro_name 4y agothat's what big organisations do, it' their DNA to flood and exhaust smaller ones. They just don't care about anybody not their scale. Ignorant egoists.
- kcartlidge 4y agoRegardless of whether or not Drew is abrasive (I've never dealt with him so have no opinion), this is on Google. If a single service (Go's module crawler) is doing this to multiple third parties (and it is, by design) then requiring Drew to implement a workaround is the same thing as relying on every (relevant to the issue) third party to do the same thing. Which is bad design. This should be fixed at source or switched off. As for the workaround itself it's true that going on a crawler exclusion list will not stop SourceHut being usable, only make it a little stale, but that doesn't matter. The problem is not at Drew's end. And once again for this workaround to be an effective solution every (relevant to the issue) third party needs to do the same thing. Which is bad design. As a general rule, with excessive polling the system doing the polling is the one at fault. And as a general rule if one system impacts many fix at the originator. If Google devs don't know this, then why on earth are they getting the salaries they do?
- tptacek 4y agoCan you help me understand the problem with the workaround, which is precisely to have Google's proxy not excessively poll DeVault's service? It really seems like DeVault's real argument here isn't about the impact of this on his service, but that he doesn't like the design of the proxy. He has a lot of standing to complain about impacts on his service, but essentially no standing to complain about designs he finds sub-par.
- WhyNotHugo 4y agoThe proxy _is_ pretty nasty design. Basically every time `golang` fetches modules, it uses Google's proxy out of the box. This lets Google collect very granular data on who uses which modules, and also makes it opaque to webmasters who's using their services (it's all hidden behind google). It's honestly pretty nasty how Google has hidden this kind of "analytics" within a pretty dark grey area. At the same time, the bot that handles the proxy seems to crawl everything, very often, and excessively, effectively DDOS'ing anyone hosting go modules.
- kcartlidge 4y agoI can't really speak for Drew's problem with the workaround, only my own from a design point of view - he may not even agree with my own issue at all for all I know, so I don't want to put words in his mouth. From my own perspective, however, the issue is about impact and responsibility. The excess traffic is their impact as it is caused by their design and is entirely of their choosing. Thus it becomes their responsibility. The workaround they offer (as with so much in tech) means their stuff becomes opt-out and not opt-in. By which I mean their own decisions/designs cause an impact where they require everyone else suffering from it to specifically and individually opt-out of being excessively polled. This is an outrageous requirement they can only impose due to the power imbalance.
- lizardactivist 4y agoAre there any ethical, reliable, and privacy-respecting implementations of Go? The reference implementation at go.dev is clearly a piece of malware with a built-in compiler.
- derekzhouzhen 4y agoWith rubygems.org or hex.pm, the package maintainer must explicitly publish a package to the registry. The registry never poll the original source, so the whole issue would not exist in the first place. I found Go's package registry very strange and limited: * What if I am using a source control program other than git? Mercury? * What if I have files in my git repo that I don't want to show to the world, like tests?
- akagusu 4y agoWhat I find shocking is not Google doing this or even how Go team didn't care about the issue and apparently banished him, but how much people here think this is an acceptable behavior and how much people take their time to defend Google and to justify this behavior blaming not Google but Drew because "it is hard to deal with him".
- lbschenkel 4y agoI'm not a Go user so I don't really know how to Go proxy works, but if webmasters had a way to opt out via robots.txt or specific status codes (429) and then the proxy at Google sends a response to the client meaning "try it yourself" (i.e., try again in direct mode) then it could have been an adequate solution for this scenario.