3 ms·
Why would the Register point out Meta and OpenAI as the worst offenders? I'm sure they do not continuously build new corpuses every day. It is probably the sear
by bgwalter 1y ago
Why would the Register point out Meta and OpenAI as the worst offenders? I'm sure they do not continuously build new corpuses every day. It is probably the search function, as mentioned in the top comments.
- snowwrestler 1y agoIt says in the first sentence of the article that it is 80% bots (crawlers) and only 20% fetchers. Of course they are crawling every day to improve their training data. The goal is LLMs that know everything, but “everything” changes on a daily basis. Meta and OpenAI are simply the largest after Google, but Google has had ~20 more years to learn how to politely operate crawlers at full-Internet scale.