3 ms·
Do really think LLM vendors that download 80TB+ of data over torrents are going to be labeling their crawler agents correctly and running them out of known data
by mtkd 2y ago
Do really think LLM vendors that download 80TB+ of data over torrents are going to be labeling their crawler agents correctly and running them out of known datacenters?
- teh_infallible 2y agoApparently they use smart appliances to scrape websites from residential accounts.
- Arnt 2y agoThe ones I noticed in my logfiles behave impeccably: retrieve robots.txt every week or so and act on it. (I noticed Claude, OpenAI and a couple of others whose names were less familiar to me.)