3 ms·

I encourage everyone with even a slight interest in the subject to download a random sample of Common Crawl (the chunks are ~100MB) and see for yourself what is

by andai 1y ago

I encourage everyone with even a slight interest in the subject to download a random sample of Common Crawl (the chunks are ~100MB) and see for yourself what is being used for training data. https://data.commoncrawl.org/crawl-data/CC-MAIN-2025-38/segments/1757047532641.17/wet/CC-MAIN-20250905112101-20250905142101-00000.warc.wet.gz https://data.commoncrawl.org/crawl-data/CC-MAIN-2025-38/segm... I spotted here a large number of things that it would be unwise to repeat here. But I assume the data cleaning process removes such content before pretraining? ;) Although I have to wonder. I played with some of the base/text Llama models, and got very disturbing output from them. So there's not that much cleaning going on.

deleted 1y ago
[deleted]
throwaway314155 1y ago
> But I assume the data cleaning process removes such content before pretraining? ;) I didn't check what you're referring to but yes, the major providers likely have state of the art classifiers for censoring and filtering such content. And when that doesn't work, they can RLHF the behavior from occurring. You're trying to make some claim about garbage in/garbage out, but if there's even a tiny moat - it's in the filtering of these datasets and the purchasing of licenses to use other larger sources of data that (unlike Common Crawl) _aren't_ freely available for competition and open source movements to use.
- jedimastert 1y ago
  > purchasing of licenses to use other larger sources of data https://www.npr.org/2025/09/05/g-s1-87367/anthropic-authors-settlement-pirated-chatbot-training-material https://www.npr.org/2025/09/05/g-s1-87367/anthropic-authors-...
dist-epoch 1y ago
Karpathy made a point recently that the random Common Crawl sample is complete junk, and that something like an WSJ article is extremely rare in it, and it's a miracle the models can learn anything at all.
- andai 1y ago
  >Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. >The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all. You'd think it's random articles but it's not, it's weird data dumps, ad spam and SEO, terabytes of stock ticker updates, etc. And then there are diamonds mixed in there, the challenge is pick them out. https://x.com/karpathy/status/1797313173449764933 https://x.com/karpathy/status/1797313173449764933 Context: FineWeb-Edu, which used Llama 70B to [train a classifier to] filter FineWeb for quality, rejecting >90% of pages. https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1 https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb...
  - WA 1y ago
    Don‘t forget the terabytes of torrented ebooks. https://www.tomshardware.com/tech-industry/artificial-intelligence/meta-staff-torrented-nearly-82tb-of-pirated-books-for-ai-training-court-records-reveal-copyright-violations https://www.tomshardware.com/tech-industry/artificial-intell... https://www.classaction.org/news/1.5b-anthropic-settlement-ends-authors-class-action-lawsuit-over-alleged-ai-copyright-infringement https://www.classaction.org/news/1.5b-anthropic-settlement-e...
    - nativeit 1y ago
      I suspect this is the "secret sauce" to making a coherent model.
- jojobas 1y ago
  From the current WSJ front page: Paul Ingrassia's 'Nazi Streak' Musk Tosses Barbs at NASA Chie After SpaceX Criticism Travis Kelce Teams Up With Investor for Activist Campaign at Six Flags A Small North Carolina College Becomes a Magnet for Wealthy Students Cracker Barrel CEO Explains Short-Lived Logo Change If that's the benchmark for high quality training material we're in trouble.

by andai 1y ago

deleted 1y ago

[deleted]

throwaway314155 1y ago

> But I assume the data cleaning process removes such content before pretraining? ;) I didn't check what you're referring to but yes, the major providers likely have state of the art classifiers for censoring and filtering such content. And when that doesn't work, they can RLHF the behavior from occurring. You're trying to make some claim about garbage in/garbage out, but if there's even a tiny moat - it's in the filtering of these datasets and the purchasing of licenses to use other larger sources of data that (unlike Common Crawl) _aren't_ freely available for competition and open source movements to use.

jedimastert 1y ago
> purchasing of licenses to use other larger sources of data https://www.npr.org/2025/09/05/g-s1-87367/anthropic-authors-settlement-pirated-chatbot-training-material https://www.npr.org/2025/09/05/g-s1-87367/anthropic-authors-...

dist-epoch 1y ago

Karpathy made a point recently that the random Common Crawl sample is complete junk, and that something like an WSJ article is extremely rare in it, and it's a miracle the models can learn anything at all.

andai 1y ago
>Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. >The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all. You'd think it's random articles but it's not, it's weird data dumps, ad spam and SEO, terabytes of stock ticker updates, etc. And then there are diamonds mixed in there, the challenge is pick them out. https://x.com/karpathy/status/1797313173449764933 https://x.com/karpathy/status/1797313173449764933 Context: FineWeb-Edu, which used Llama 70B to [train a classifier to] filter FineWeb for quality, rejecting >90% of pages. https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1 https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb...
- WA 1y ago
  Don‘t forget the terabytes of torrented ebooks. https://www.tomshardware.com/tech-industry/artificial-intelligence/meta-staff-torrented-nearly-82tb-of-pirated-books-for-ai-training-court-records-reveal-copyright-violations https://www.tomshardware.com/tech-industry/artificial-intell... https://www.classaction.org/news/1.5b-anthropic-settlement-ends-authors-class-action-lawsuit-over-alleged-ai-copyright-infringement https://www.classaction.org/news/1.5b-anthropic-settlement-e...
  - nativeit 1y ago
    I suspect this is the "secret sauce" to making a coherent model.
jojobas 1y ago
From the current WSJ front page: Paul Ingrassia's 'Nazi Streak' Musk Tosses Barbs at NASA Chie After SpaceX Criticism Travis Kelce Teams Up With Investor for Activist Campaign at Six Flags A Small North Carolina College Becomes a Magnet for Wealthy Students Cracker Barrel CEO Explains Short-Lived Logo Change If that's the benchmark for high quality training material we're in trouble.