6 ms·
How is that different than non-AI crawlers doing the same for the past decade or so? Tons of businesses engage in site crawling and scrapping, and many of them
by TeMPOraL 2y ago
How is that different than non-AI crawlers doing the same for the past decade or so? Tons of businesses engage in site crawling and scrapping, and many of them are bad citizens.
My issue isn't with blocking bad-behaving bots - it's with singling out LLMs (both training and use), or worse, assuming the problem is being associated with AI and not bad bot behavior.
- pona-a 2y agoVolume and incentives. Before this LLM craze, the biggest crawlers were search engines. They had a motivation to not bring down their targets, because who needs an index full of dead links. With LLM crawlers, all you need is text, and if the site is forced to shut down because of you, that's just less data for your competitors.
- throw10920 2y agoAlso, nobody else steals your stuff like AI does. Doesn't take much thought to figure out the difference.
- egypturnash 2y agohttps://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali... https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ https://thelibre.news/foss-infrastructure-is-under-attack-by...