4 ms·
It says in the first sentence of the article that it is 80% bots (crawlers) and only 20% fetchers. Of course they are crawling every day to improve their train
by snowwrestler 1y ago
It says in the first sentence of the article that it is 80% bots (crawlers) and only 20% fetchers.
Of course they are crawling every day to improve their training data. The goal is LLMs that know everything, but “everything” changes on a daily basis.
Meta and OpenAI are simply the largest after Google, but Google has had ~20 more years to learn how to politely operate crawlers at full-Internet scale.