3 ms·

Why would anyone scrape wikipedia, isn’t there a full archive available via torrent?

• lxgr 2y ago
  These are presumably LLMs with access to web search answering user queries, not crawlers scraping data for training. It's probably both cheaper and more accurate (for fast-changing content) to them to just hit Wikipedia servers every time than to special case search results pointing there and keeping a local copy.
  • recursivecaveat 2y ago
    They talk about how a lot of the problem is all these bot requests hitting cold pages and especially multimedia that are not cached because they are rarely accessed. So I don't think it is likely that this activity is in direct response to user queries. I don't know why using an LLM would make people any more interested in very boring and obscure geographic and historical trivia than they would be via search engines.
  • ricardobeat 2y ago
    > Automated bots seeking AI model training data for LLMs Second sentence in the article.
    • lxgr 2y ago
      Yes, but I'm not convinced that that's actually (the majority of) what this is. How many large models are realistically being trained on all of Wikimedia, compared to the number of people using AI search, possibly fetching sites very inefficiently (e.g. using a headless browser spun up from scratch, i.e. with a cold cache, every single time).