4 ms·
The Chinese AI scrapers/bots are killing quite a bit of the regular web now. YisouSpider absolutely pummeled my open source project's hosting for weeks. Like al
by JohnTHaller 8mo ago
The Chinese AI scrapers/bots are killing quite a bit of the regular web now. YisouSpider absolutely pummeled my open source project's hosting for weeks. Like all Chinese AI scrapers, it ignores robots.txt. So forget about it respecting a Crawl-delay. If you block the user agent, it would calm down for a bit, then it would just come back again using a generic browser user agent from the same IP addresses. It does this across 10s of thousands of IPs.
- kevin_thibedeau 8mo agoStart blocking /16s.
- mono442 8mo agoJust block the whole China, India and similar countries.
- AlexeyBelov 8mo agoYou'd need to block the US as well, as most such traffic comes from there. Which is not really reasonable.
- olivia-banks 8mo agoI had no idea Yisou was still around, or did somebody buy them? I'm not as up to date on Chinese tech as I should be.