3 ms·

I don't think this traffic is actually coming from crawlers for training. Think about it, why would a training scraper need to hit the same page hundreds of ti

I don't think this traffic is actually coming from crawlers for training. Think about it, why would a training scraper need to hit the same page hundreds of times a day? They only need to download it once. I think this is LLMs doing web searches at runtime in response to user queries. There's no caching at this level, so similar queries by many different users could lead the LLM to request the same page many times.

Anamon 6mo ago
Another possibility is that their crawling code is just that bad. A lot of these GenAI companies are dogfooding in their software development, and the quality can be seen whenever some of their code is released or leaked. It seems very plausible to me that their crawlers could simply barely functional, buggy messes. They have unlimited venture capital to burn, so it doesn't really matter if they scrape a site 100,000 times a day when once would have been enough.