4 ms·

Karpathy made a point recently that the random Common Crawl sample is complete junk, and that something like an WSJ article is extremely rare in it, and it's a

Karpathy made a point recently that the random Common Crawl sample is complete junk, and that something like an WSJ article is extremely rare in it, and it's a miracle the models can learn anything at all.

andai 1y ago
>Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. >The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all. You'd think it's random articles but it's not, it's weird data dumps, ad spam and SEO, terabytes of stock ticker updates, etc. And then there are diamonds mixed in there, the challenge is pick them out. https://x.com/karpathy/status/1797313173449764933 https://x.com/karpathy/status/1797313173449764933 Context: FineWeb-Edu, which used Llama 70B to [train a classifier to] filter FineWeb for quality, rejecting >90% of pages. https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1 https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb...
- WA 1y ago
  Don‘t forget the terabytes of torrented ebooks. https://www.tomshardware.com/tech-industry/artificial-intelligence/meta-staff-torrented-nearly-82tb-of-pirated-books-for-ai-training-court-records-reveal-copyright-violations https://www.tomshardware.com/tech-industry/artificial-intell... https://www.classaction.org/news/1.5b-anthropic-settlement-ends-authors-class-action-lawsuit-over-alleged-ai-copyright-infringement https://www.classaction.org/news/1.5b-anthropic-settlement-e...
  - nativeit 1y ago
    I suspect this is the "secret sauce" to making a coherent model.
jojobas 1y ago
From the current WSJ front page: Paul Ingrassia's 'Nazi Streak' Musk Tosses Barbs at NASA Chie After SpaceX Criticism Travis Kelce Teams Up With Investor for Activist Campaign at Six Flags A Small North Carolina College Becomes a Magnet for Wealthy Students Cracker Barrel CEO Explains Short-Lived Logo Change If that's the benchmark for high quality training material we're in trouble.
- stocksinsmocks 1y ago
  There is very, very little written work that will stand the test of time. Maybe the real bitter lesson is that training data quality is inversely proportional to scale and the technical capabilities exist but can never be realized
- anigbrowl 1y ago
  In general I find WSJ articles very well written. It's not their fault if much of today's news is about clowns.
  - dclowd9901 1y ago
    Their editorial department is an embarrassment imo. Sycophancy for conservatism thinly veiled as intellectualism.
    - anigbrowl 1y ago
      I also hate their editorial department, I'm just saying that the news articles are well written in a technical sense rather than because I like their editorial positions or choice of subject mattter.

andai 1y ago

>Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. >The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all. You'd think it's random articles but it's not, it's weird data dumps, ad spam and SEO, terabytes of stock ticker updates, etc. And then there are diamonds mixed in there, the challenge is pick them out. https://x.com/karpathy/status/1797313173449764933 https://x.com/karpathy/status/1797313173449764933 Context: FineWeb-Edu, which used Llama 70B to [train a classifier to] filter FineWeb for quality, rejecting >90% of pages. https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1 https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb...

WA 1y ago
Don‘t forget the terabytes of torrented ebooks. https://www.tomshardware.com/tech-industry/artificial-intelligence/meta-staff-torrented-nearly-82tb-of-pirated-books-for-ai-training-court-records-reveal-copyright-violations https://www.tomshardware.com/tech-industry/artificial-intell... https://www.classaction.org/news/1.5b-anthropic-settlement-ends-authors-class-action-lawsuit-over-alleged-ai-copyright-infringement https://www.classaction.org/news/1.5b-anthropic-settlement-e...
- nativeit 1y ago
  I suspect this is the "secret sauce" to making a coherent model.

jojobas 1y ago

From the current WSJ front page: Paul Ingrassia's 'Nazi Streak' Musk Tosses Barbs at NASA Chie After SpaceX Criticism Travis Kelce Teams Up With Investor for Activist Campaign at Six Flags A Small North Carolina College Becomes a Magnet for Wealthy Students Cracker Barrel CEO Explains Short-Lived Logo Change If that's the benchmark for high quality training material we're in trouble.

stocksinsmocks 1y ago
There is very, very little written work that will stand the test of time. Maybe the real bitter lesson is that training data quality is inversely proportional to scale and the technical capabilities exist but can never be realized
anigbrowl 1y ago
In general I find WSJ articles very well written. It's not their fault if much of today's news is about clowns.
- dclowd9901 1y ago
  Their editorial department is an embarrassment imo. Sycophancy for conservatism thinly veiled as intellectualism.
  - anigbrowl 1y ago
    I also hate their editorial department, I'm just saying that the news articles are well written in a technical sense rather than because I like their editorial positions or choice of subject mattter.