2 ms·
The linked page claims that most identified crawlers are related to scraping for training data of LLMs, which seems likely.
by lblume 1y ago
The linked page claims that most identified crawlers are related to scraping for training data of LLMs, which seems likely.