6 ms·
Gentoo bugzilla closed due AI bot scraper overload
- c-fe 2mo agogoing on a bit of a tangent here - some discussion there seems to be imply that AI bots mostly use IPv4s, which makes sense to me given that they are probably bots hosted by some cloud. Whereas IPv6 may be more organic traffic from e.g. mobile users (not in this case probably)... which made me think if at some point IPv6 may at some point win against IPv4 just because its more organic traffic (i.e. not from big tech cloud), leading to pages blocking IPv4? Just speculation on my side
- thayne 2mo agoMy residential ISP doesn't support IPv6 at all. If a site is ipv6 only, I can't reach it unless I use a VPN.
- neuroticnews25 2mo agoThis is weird, every VPS I've rented in the last 8 years had IPv6, and some didn't even had IPv4.
- TingPing 2mo agoIf a problematic bot is on any mainstream VPS it will just get shut down.
- Symbiote 2mo agoHow would I shut down a bot on Digital Ocean making over a million requests a day?
- coldbrewed 2mo agoSend an email to the Digitalocean abuse email with the time ranges and origin IPs.
- neuroticnews25 2mo agoI did this with OVH, never got any response.
- progval 2mo agoabuse@ contacts never send responses (though some send acknowledgements) because they don't want to give any details about their abuse process.
- Retr0id 2mo agoThe hardest-to-mitigate bot traffic tends to come from residential proxies, and most residential connections are still v4-only. However, it is also easy to get large numbers of v6 addresses cheaply.
- mrweasel 2mo agoWhile we're dealing with the same issue at work, I sometimes still wonder exactly who these scrapers are. OpenAI, Google, Anthropic and others are normally fairly well behaved (Minus Anthropic attempting to hide behind a browser-for-hire company). Mostly you can get IP range and user-agents for the large players, while it problems mostly stem from bots pretending to be Chrome. Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation. I also don't recall ever seeing Grok IP ranges or a specific Grok UA, but that doesn't mean that they're hiding, perhaps they're just not interested.
- marginalia_nu 2mo agoMost of the nonsense comes through botnets/residential proxies, so it's very hard to say.
- shevy-java 2mo agoWell, some websites claimed China is behind it, which could make sense (I would not know either way). At the same time, though, I kind of doubt your carte blanche here for all those companies. Why would you think none of them are responsible for the AI slop spam? > Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation Ok. So you also don't know. Well, I don't know either, but I don't make a speculation by claiming x, y, and z companies to be exempt. In my book they are all responsible.
- coldbrewed 2mo agoWe see constant abuse from the Tencent ASN/associated ACE ASN, and I've memorized the china169 backbone asn as AS4837 because of thr nonstop crawlers splattered across their network ranges. It's not possible to ID the operator running the crawlers running from these networks but there's a clear signal of the origin of some of these entities.
- rcxdude 2mo agoWell, those companies have bots that identify themselves, and you can see what they're doing. Google especially have decades of experience of designing scrapers and seem to be able to the job of scraping the whole internet without causing problems in that time. So, it's still possible they are also running a bunch of poorly-coded scraper bots through residential proxies, but it seems a little non-obvious why they would be doing this. (It is worth pointing out that most of this traffic seems to be dumb: it's stuff like getting lost in generated link forests of some web apps or repeatly re-querying the same endpoint on a super-high frequency. This isn't exactly going to give a good return on investment for AI training data, especially since AIUI the main race for LLM performance now is in good quality training data)
- ComputerPerson 2mo agoThere are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable. We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time. Most scrapers are relatively honest in some way shape or form.
- sporp 2mo agoHow did you implement this? My site's getting hammered, any tips would be appreciated
- ComputerPerson 2mo agoWhat kind of uptime/expectations/etc. are you working with? Scraper "attacks" don't take down our robot-specific server very often; it's safe for us to take heavy-handed approaches that sometimes redirect users there. 99% (made-up high number) of the time, the misdirected users don't realize anything is amiss. Start by analyzing your traffic, specifically user agents. Look for "robot" or even "bot" in the user agent and load balance those to a robot-specific server. This can all be done within Cloudflare. The only code is the user agent condition. Note: I'm very open to input here if anyone reading notices that we're shooting ourselves in the feet. Based on our analysis, the remaining traffic is a good picture of our human users. We have loads of other conditions, mostly balancing specific IP ranges for entities when we know exactly who they are, but this is a good start.
- subnetcap 2mo ago[flagged]
- configdesk 2mo ago[flagged]
- broker_desk 2mo ago[flagged]
- 2mo ago
- shevy-java 2mo agoAI skynet is winning. It is stealing time from humans, thus forcing down their activity, as can be seen here. A headline would be great if those AI companies would close down. I hold them all responsible for this.
- lilbigdoot 2mo agoPlot twist: the AI wins not through terminators but a kafka nightmare
- calvinmorrison 2mo agoThe year is 2026, somehow peoples basic web apps are not able to keep up with scrapers. Scrapers are not new. My side load is like .01 even with 10x the traffic of last year. It's called "serving static content", "caching" and many other things that are not new concepts. Then you've got good old cloudflare which is free to use
- sethops1 2mo agoAre you volunteering to setup and host that extra complexity for the Gentoo project?
- thayne 2mo agoA bugtracker is inherently dynamic.
- stryan 2mo ago1. You shouldn't need to be a professional sysadmin to be able to deploy a basic semi-dynamic website 2. Considering the spike of complaints over the past two years and the fact that they range from small time site operators to larger orgs (Linux Kernel, GNOME, Duke University, etc), do you think that maybe you're missing part of the problem space?
- calvinmorrison 2mo agoAll these experts here in advanced computing but people don't know how to cache with webpages?
- littlecranky67 2mo agoIt is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.
- kangalioo 2mo ago0.05 cents even. The beautiful thing is, minuscule amounts of micropayments are already enough to fix the incentives
- comboy 2mo agoNot sure it works, isn't it already like that if you have the cursed javascript-only pages? It's basically proof of work to access their content (and if you are scrapping en masse you likely also need some LLM getting involved)
- port11 2mo agoI suppose the cost of hardware, network usage, and electricity is still under 0.05? I mean, we know the unit economics for LLMs don't make a ton of sense, except if enterprise customers really are what keeps the lights on.
- tarpitt 2mo agoWith micropayments, the server owner makes 1 million requests times 0.05 cents = ~$500 The status quo with those js PoW pages doesn't really benefit the server owner at all, it's wasted energy. I mean proof of work is always wasted energy, but I figure it's better to kill two birds with one stone. CoinHive was one example of this. (I think this is a correct link? https://github.com/cazala/coin-hive https://github.com/cazala/coin-hive). Although I think ideally you would want to have some sort of browser plugin or app that runs on bare metal instead of a proof-of-work in the browser, because RandomX is designed such that it's slow when implemented in JS (https://github.com/tevador/RandomX/blob/master/doc/design.md https://github.com/tevador/RandomX/blob/master/doc/design.md)
- draw_down 2mo ago[dead]
- ece 2mo agoEvery website that wants to, can just charge for dumps. Anubis or even Cloudflare can handle the rest I think.
- jiffygist 2mo agoSo where are bugs reported now?
- nubinetwork 2mo agoThat's the thing, they aren't.
- edoceo 2mo agoGentoo still has an active IRC and mailing-list. Back to basics? Then scrapers just see the static archives from, MARC I guess.
- nubinetwork 2mo agoGentoo has pretty much always had a Bugzilla, even back in the early 00s... there never was an alternative plan. The biggest thing about what's happened today though, is how a "janitor" on the infrastructure team can shut down the entire distros ability to work on bugs. The decision to shut down should have been left to someone else like robbat. Mgorny has consistently caused issues for Gentoo, and many people have left the community after having to deal with him, but it seems comrel/devrel can't/won't remove him for whatever reason. Edit: it looks like bgo is up now. Hopefully it stays that way.
- jpcom 2mo agoWhat are they scraping the gentoo bugzilla for? I'm confused. Unless you're actively using Gentoo why would this be a resource? Very confusing. Also you'd think we'd have LLM BitTorrent by now, where if they want to scrape something we get a DHT hash for the content and share it with one another, rather than melt servers with the millionth request of the day.
- BoingBoomTschak 2mo agoLots of build failure detailed investigations and gcc/kernel expertise in debugging misbehaving or outright ICEs. That's it, I guess?
- xena 2mo agoThey're scraping everything. It doesn't matter what. It doesn't matter if it makes sense. They just scrape it all.
- FranOntanaya 2mo ago"There Will Be Blood" but it's 2026 and Plainview's child is AI.
- KennyBlanken 2mo agoThey're mining the written output of technical talent discussing troubleshooting, bug finding, patches, inter-dependencies, etc. It's an AI trainer's wet dream. Also: where exactly are AI companies incentivized to be anything but shitty 'neighbors'? Spam, DDoS attacks and other network abuse used to cause your hosting company or provider to call you and tell you to knock it off or you'll get disconnected, if your provider was reputable. If your provider wasn't reputable, it was likely a matter of time before they would get a nasty call from their upstream provider. Now it just gets you a thank-you from the sales team for all the bandwidth you bought. Meanwhile, do any of the cloud providers have any incentive to do anything about this? Hell no. They're making money off you having to ramp up extra or bigger instances. They're making money off the bandwidth. They're making money off the people doing the crawling, too. They're incentivized to do the exact opposite of effectively help you with your AI bot problem.
- pmdr 2mo agoThere's a reddit thread from 9 years ago with people complaining the site blocking crawlers and thus not being indexed by search engines. So I'm not sure what they consider "overload." At this point they should dump the db on thepiratebay.
- progval 2mo agoDumping the database won't stop the scrappers because humans running the scrappers are not even aware Gentoo exists. They're just scrapping every website their bots find.
- Gander5739 2mo agoAnd people complain about cloudflare/anubis/etc. Unfortunately, it's looking like this is the alternative.
- ltbarcly3 2mo agoCorrected title: "Gentoo bugzilla closed because the guy running it, who says it is 'unusable anyway', saw a lot of traffic from different IP addresses with no clear pattern and accuses AI"
- ltbarcly3 2mo agoThis got modded down to oblivion, but let me just copy and paste the first line of the tweet: I've taken #Gentoo Bugzilla down, because it was unusable anyway. No point in feeding the #LLM scrapers that are using thousands of different IPv4 addresses, with no obvious patterns I can see. Now you might try to interpret what he said as the LLM scrapers made it unusable, but apparently it was usable to the scrapers as he turned it off to stop feeding them. If he can actually attribute the traffic to LLM scrapes, then he can use that same method to identify their requests and block them or shape (slow) their traffic. He then says not to tell him how to fix it because he doesn't care.
- maelito 2mo agoIndexers should pay for each request. Maybe Taler could help ?
- capitainenemo 2mo agoHedgewars had to lock down its bugzilla and repo too, for the exact same reason. So far the really simple method used (basic auth with the creds posted on the main site) seems to have resisted the scrapers. Presumably if they are too stupid to scrape correctly they are also too stupid to incorporate a trivial auth.
- eYrKEC2 2mo agoI don't really understand why there is resistance to in-browser Anubis-style gating using crypto mining for public sites like this. You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal. You want to bang on my servers with the fury of a thousand madmen, then I'll scale up the servers and you can pay the marginal cost of your access. Or micropayments, of course, but hard to get normal users to sign up for micropayments. Micropayments could of course be the way to bypass the crypto mining gate.
- rufasterisco 2mo agoInstead of punishing a criminal, you make everybody pay. This is not only unfair to legitimate users having to pay. It also needs to explain what actually happens when the Ddos succeeds (I know, you are talking about scrapers, but what’s the difference really?). Does that mean that the attacker just gets to shrug it off. Don’t take my statements as facts, I just want to outline a few reason I could come up with that show a purely technical solution might not be enough. Akin to “just use Cloudflare, it’s free”. Yes it works but why conceding defeat and say “oh from now on you depend on a business to publish a server”?
- xena 2mo agoIf we could identify who is doing this, I bet that those people would be publicly burned at the stake by now.
- rufasterisco 2mo agoI don’t like people being burnt at stakes and what I see when I google “residential proxy” does not feel like we cannot identify. More like we are not even trying. I mean, if residential proxies are tools used to DDOS websites, why is Google advertising them? And if you have Google ads for them, can we really say “ah, if we only knew who these guys are?”.
- gruez 2mo ago>I don't really understand why there is resistance to in-browser Anubis-style gating using crypto mining for public sites like this. But in the case of anubis it's not even used for crypto. It's just wasted. >You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal. No, the problem is time wasted. I don't care about the electricity cost. Spending 10s to solve a challenge on a 5W SoC translates to 0.0005 cents (yes, cents, not dollars). Meanwhile if I click on a link and it doesn't load in 5s, I'm seriously questioning the value of your blog or whatever, and will probably just close the tab.
- nirbendavid 2mo ago[flagged]
- rvz 2mo agoJust put the website behind an x402 gate [0] (yes it uses stablecoins) requiring payment if a bot wants to access the content which acts like a Captcha and detect them if they are on direct data center IP or are using residential proxies. Stablecoins have a great use-case and you get paid for bots to access the site with the humans living in peace without the site getting botted. Job done. [0] https://stevekrouse.com/x402 https://stevekrouse.com/x402
- xena 2mo agoPlease write a function that returns true when the user is using a residential proxy and false when it is a normal user on a residential IP address.
- FranOntanaya 2mo agoAfter a decade and half we had to restrict our public side and reorganize our old TED content because scrappers were really hungry for assorted captioned videos. If you have anything of value for training you get eaten alive if you stick out, it's like wearing short pants in the summer tundra. It's going to change a lot the internet we knew, unless somehow we managed to agree on a common high quality dataset repository.
- rufasterisco 2mo agoEnough with playing around the issue. The way out of this mess is not to protect with tech that works but with principles and laws. This is not a tech problem. This is about what should or should not be legal. Nor it’s a question of having time to implement solution X or Y. If someone attacks you yes, you should have better security but you also need to have legal recourse, or it will never stop. Ddos is already illegal. I am not a lawyer so don’t ask me for exact resources, which vary by country anyway, but stop treating scrapers as an inescapable force of nature.
- Certhas 2mo agoI agree in principle. The tech community naturally has a tendency to reach for tech solutions to societal problems. But without jurisdiction, laws are meaningless. If you create a law that says you have to honour robots.txt files, what do you do if an IP from another country fails to do so?
- rufasterisco 2mo agoYou check the laws of your country. Likely they say that DDOS are criminal offences, and you can get 10 or 20 years of jail time, already. Then you report a crime and send the logs. And maybe report to some politician that is tough on crime. And you are not alone. I am sure you follow through, Pls check also my other comments. Laws without enforcement are meaningless, without jurisdiction they are less effective. Maybe yes, some poor guy will find a letter that says he needs to pay 50 USD or EUR to some faraway country because the puzzle app in his tv has weird ToS. Or whatever, a cease and desist letter? A “you are part of a hacking circle please uninstall this app” letter, signed by a judge? I don’t advocate for people getting thrown in jail, I just think that the whole “residential proxies” issue might be a charade that would not last longer if citizens and businesses would claim rights instead of play victim. I also really dislike grasping for a tech solution because I don’t want to start to mine crypto to read an article or to depend on some “free” cloud service to host an open bug tracker. It just feels so weird: why is it that no-one is doing it? I feel like I an one “comment away” from someone explaining to me how naive I am, and yet it’s never coming.
- 2mo ago
- vova_hn2 2mo agoAre we assuming that this is the companies themselves scraping data from training or is this "agents" acting on behalf of users? Nowadays every major chat UI (ChatGPT, Claude etc) has a "tool" that allows LLM to load web pages, so it must generate some traffic.
- xena 2mo agoNo, most of the load comes from armies of residential IP addresses that look like Google Chrome on the wire. The major chat UIs properly identify themselves. These waves of attacks do not.
- arjie 2mo agoI like lesswrong's approach to this. You get 4xx if you scrape that site but greaterwrong.com serves some kind of static site that is scrape friendly. Seems fine to me. Though, checking, I see that it requires a login but is still up. I misunderstood what 'closed' meant. This seems fine.
- clearbacklog 2mo ago[flagged]
- subnetcap 2mo ago[flagged]
- charcircuit 2mo agoI want AI to progress to a point where it can fix these static sites who fall over due to how inefficient they are.
- configdesk 2mo ago[flagged]
- stefantalpalaru 2mo ago[dead]
- s09dfhks 2mo ago(Not saying that it is the only solution / the right solution) Would fronting it with Anubis have helped at all here? (please do not ramble about how requiring javascript is a violation of your human rights)
- userbinator 2mo agoManufactured consent. This is not AI, this is just DDoS'ing for extortion and driving towards the closed proprietary "can only be accessed by a sanctioned UA" Internet that they're after. Why else do you think identity verification and attempting to turn the Internet into a walled garden became such a big thing in the recent years?
- simmerup 2mo agoYou don’t think it has anything to do with hostile states using fake profiles on social media to drum up discontent in the democratic world?