3 ms·
There’s been multiple articles on the front page of HN about how there’s a ton of AI crawlers that are really bad citizens - ignoring robots.txt, ignoring cache
by egypturnash 2y ago
There’s been multiple articles on the front page of HN about how there’s a ton of AI crawlers that are really bad citizens - ignoring robots.txt, ignoring cache, re-scanning pages multiple times a day. The commons is already on fire and it’s not because of the actions of any of the “locals”.
- TeMPOraL 2y agoHow is that different than non-AI crawlers doing the same for the past decade or so? Tons of businesses engage in site crawling and scrapping, and many of them are bad citizens. My issue isn't with blocking bad-behaving bots - it's with singling out LLMs (both training and use), or worse, assuming the problem is being associated with AI and not bad bot behavior.
- pona-a 2y agoVolume and incentives. Before this LLM craze, the biggest crawlers were search engines. They had a motivation to not bring down their targets, because who needs an index full of dead links. With LLM crawlers, all you need is text, and if the site is forced to shut down because of you, that's just less data for your competitors.
- throw10920 2y agoAlso, nobody else steals your stuff like AI does. Doesn't take much thought to figure out the difference.
- egypturnash 2y agohttps://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali... https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ https://thelibre.news/foss-infrastructure-is-under-attack-by...