4 ms·

Newb question. If the big boys are crawling, scraping, stealing the whole of the entire internet to train their models — how are these OS alternatives aiming to

Newb question. If the big boys are crawling, scraping, stealing the whole of the entire internet to train their models — how are these OS alternatives aiming to outperform them? Is it as simple as... ALSO crawl, scrape, steal the whole of the entire internet?

plufz 2y ago
I am no expert. But wasn’t part of deepseeks success to use synthetic data, atleast for code and math?
sebzim4500 2y ago
Yes, but in this case the RL stage doesn't actually need a huge amount of data. Most of the data is ingested in the pre-training phase, which in this case was done by Alibaba. There was a recent paper [1] that showed that for one particular model it was possible to get very good reasoning performance by finetuning on under a thousand examples of reasoning. [1] https://arxiv.org/abs/2502.03387 https://arxiv.org/abs/2502.03387
smallmancontrov 2y ago
No, the secret is to take outputs from previous models and put them in a clothes dryer with starch to give them the gently worn appearance of organically collected data.
yieldcrv 2y ago
No, fine tuning doesn't require that and models are using synthetic data sets now, for like 2 years as in, the huge models did the crawling and then the next models had conversations with the huge models oversimplified but basically the scraping ship has sailed, artists/writers will never be satisfied, and other companies bragging about using whatever aspirational goal post of licensed material are just doing it for brownie points
- mekaron7 2y ago
  Does this mean the model of LLM = search engine is also coming to an end? Ignoring things like functions and google search capabilities, current chatgpt models have quite a lot of niche knowledge, I assume from crawling the entire internet.
  - astrange 2y ago
    Creative writing is also harmed by limiting the pretraining data. eg if I ask DeepSeek to write a story it tends to write the bullet points outline of a story. And some other models always give characters the same names.
  - kingo55 2y ago
    I speculate we'll just see classes of models developed. Ones with lots of knowledge (even domain specific) and models with reasoning capabilities.
  - yieldcrv 2y ago
    no, companies are in a quagmire of having to update them every quarter for now, they're not doing that but just releasing new models that also have updated information there is one group thats scraping the latest information and including it in their latest models, another group thats only doing synthetic data sets from the prior group's models, and every enthusiast everywhere making fine tuned LLMs with synthetic data sets and whatever niche they want
  - regularfry 2y ago
    We already know how to build search engines. The far better model is to give one to an LLM as a tool to use. The user gets the same interface but you're not spending your inference budget on data retrieval.
deleted 2y ago
[deleted]
simonw 2y ago
"Stealing the whole of the entire internet" isn't a particularly accurate mental model these days. That's more or less what people were doing back in 2023 - crawling everything and dumping as much data in as possible. It's not a great strategy to build a best-in-class model though, as a lot of the internet is junk. The SolidGoldMagikarp/davidjl bug is the kind of thing that happens if you crawl all of https://www.reddit.com/r/counting/ https://www.reddit.com/r/counting/ for example: https://simonwillison.net/2023/Jun/8/gpt-tokenizers/#glitch-tokens https://simonwillison.net/2023/Jun/8/gpt-tokenizers/#glitch-... These days model training labs are more selective about what they train on. Most of the game of training a great model comes down to selectively training your data. They still use a lot of unlicensed data but it's a bit more sophisticated than just dumping in everything they can find.
- nicce 2y ago
  You can just ”steal” all the books and paywalled research papers. That is a lot already.
  - simonw 2y ago
    Hence the recent story/scandal about Meta torrenting LibGen: https://arstechnica.com/tech-policy/2025/02/meta-torrented-over-81-7tb-of-pirated-books-to-train-ai-authors-say/ https://arstechnica.com/tech-policy/2025/02/meta-torrented-o...
- woctordho 2y ago
  Crawling the whole internet still helps solve problems in the long tail of the distribution. The purpose of big data is garbage in, signal out
Aiguru31415666 2y ago
[dead]
littlestymaar 2y ago
You don't need to crawl and filter lots of internet data when you can disteal large models.
janalsncm 2y ago
The answer is that crawling the whole internet is only for training a base model which is expensive and compute-intensive. R1 didn’t train a base model, they performed additional steps on top of a previously-trained base model (V3). These guys are doing something similar.

plufz 2y ago

I am no expert. But wasn’t part of deepseeks success to use synthetic data, atleast for code and math?

sebzim4500 2y ago

Yes, but in this case the RL stage doesn't actually need a huge amount of data. Most of the data is ingested in the pre-training phase, which in this case was done by Alibaba. There was a recent paper [1] that showed that for one particular model it was possible to get very good reasoning performance by finetuning on under a thousand examples of reasoning. [1] https://arxiv.org/abs/2502.03387 https://arxiv.org/abs/2502.03387

smallmancontrov 2y ago

No, the secret is to take outputs from previous models and put them in a clothes dryer with starch to give them the gently worn appearance of organically collected data.

yieldcrv 2y ago

No, fine tuning doesn't require that and models are using synthetic data sets now, for like 2 years as in, the huge models did the crawling and then the next models had conversations with the huge models oversimplified but basically the scraping ship has sailed, artists/writers will never be satisfied, and other companies bragging about using whatever aspirational goal post of licensed material are just doing it for brownie points

mekaron7 2y ago
Does this mean the model of LLM = search engine is also coming to an end? Ignoring things like functions and google search capabilities, current chatgpt models have quite a lot of niche knowledge, I assume from crawling the entire internet.
- astrange 2y ago
  Creative writing is also harmed by limiting the pretraining data. eg if I ask DeepSeek to write a story it tends to write the bullet points outline of a story. And some other models always give characters the same names.
- kingo55 2y ago
  I speculate we'll just see classes of models developed. Ones with lots of knowledge (even domain specific) and models with reasoning capabilities.
- yieldcrv 2y ago
  no, companies are in a quagmire of having to update them every quarter for now, they're not doing that but just releasing new models that also have updated information there is one group thats scraping the latest information and including it in their latest models, another group thats only doing synthetic data sets from the prior group's models, and every enthusiast everywhere making fine tuned LLMs with synthetic data sets and whatever niche they want
- regularfry 2y ago
  We already know how to build search engines. The far better model is to give one to an LLM as a tool to use. The user gets the same interface but you're not spending your inference budget on data retrieval.

deleted 2y ago

[deleted]

simonw 2y ago

"Stealing the whole of the entire internet" isn't a particularly accurate mental model these days. That's more or less what people were doing back in 2023 - crawling everything and dumping as much data in as possible. It's not a great strategy to build a best-in-class model though, as a lot of the internet is junk. The SolidGoldMagikarp/davidjl bug is the kind of thing that happens if you crawl all of https://www.reddit.com/r/counting/ https://www.reddit.com/r/counting/ for example: https://simonwillison.net/2023/Jun/8/gpt-tokenizers/#glitch-tokens https://simonwillison.net/2023/Jun/8/gpt-tokenizers/#glitch-... These days model training labs are more selective about what they train on. Most of the game of training a great model comes down to selectively training your data. They still use a lot of unlicensed data but it's a bit more sophisticated than just dumping in everything they can find.

nicce 2y ago
You can just ”steal” all the books and paywalled research papers. That is a lot already.
- simonw 2y ago
  Hence the recent story/scandal about Meta torrenting LibGen: https://arstechnica.com/tech-policy/2025/02/meta-torrented-over-81-7tb-of-pirated-books-to-train-ai-authors-say/ https://arstechnica.com/tech-policy/2025/02/meta-torrented-o...
woctordho 2y ago
Crawling the whole internet still helps solve problems in the long tail of the distribution. The purpose of big data is garbage in, signal out

Aiguru31415666 2y ago

[dead]

littlestymaar 2y ago

You don't need to crawl and filter lots of internet data when you can disteal large models.

janalsncm 2y ago

The answer is that crawling the whole internet is only for training a base model which is expensive and compute-intensive. R1 didn’t train a base model, they performed additional steps on top of a previously-trained base model (V3). These guys are doing something similar.