4 ms·

I guess my products fall into a niche that doesn’t seem to attract AI crawlers. I’ve seen only a few and they haven’t been too aggressive. I mean they ignore ty

I guess my products fall into a niche that doesn’t seem to attract AI crawlers. I’ve seen only a few and they haven’t been too aggressive. I mean they ignore typical crawl rate limits defined in robots.txt but account for maybe only 1-2% of my overall traffic.

esseph 1y ago
That's crazy! I've been watching the scraping go up exponentially for months now. https://www.eff.org/deeplinks/2025/06/keeping-web-under-weight-ai-crawlers https://www.eff.org/deeplinks/2025/06/keeping-web-under-weig...