4 ms·
While it’s true that chatbots fetch information from websites in response to requests, the load from those requests is tiny compared to the volume of requests i
by snowwrestler 1y ago
While it’s true that chatbots fetch information from websites in response to requests, the load from those requests is tiny compared to the volume of requests indexing content to build training corpuses.
The reason is that user requests are similar to other web traffic because they reflect user interest. So those requests will mostly hit content that is already popular, and therefore well-cached.
Corpus-building crawlers do not reflect current user interest and try to hit every URL available. As a result these hit URLs that are mostly uncached. That is a much heavier load.
- shikon7 1y agoBut surely there aren't thousands of new corpuses built every minute.
- bgwalter 1y agoWhy would the Register point out Meta and OpenAI as the worst offenders? I'm sure they do not continuously build new corpuses every day. It is probably the search function, as mentioned in the top comments.
- snowwrestler 1y agoIt says in the first sentence of the article that it is 80% bots (crawlers) and only 20% fetchers. Of course they are crawling every day to improve their training data. The goal is LLMs that know everything, but “everything” changes on a daily basis. Meta and OpenAI are simply the largest after Google, but Google has had ~20 more years to learn how to politely operate crawlers at full-Internet scale.