36 ms·
"Stealing the whole of the entire internet" isn't a particularly accurate mental model these days. That's more or less what people were doing back in 2023 - cr
by simonw 2y ago
"Stealing the whole of the entire internet" isn't a particularly accurate mental model these days.
That's more or less what people were doing back in 2023 - crawling everything and dumping as much data in as possible.
It's not a great strategy to build a best-in-class model though, as a lot of the internet is junk. The SolidGoldMagikarp/davidjl bug is the kind of thing that happens if you crawl all of https://www.reddit.com/r/counting/ https://www.reddit.com/r/counting/ for example: https://simonwillison.net/2023/Jun/8/gpt-tokenizers/#glitch-tokens https://simonwillison.net/2023/Jun/8/gpt-tokenizers/#glitch-...
These days model training labs are more selective about what they train on. Most of the game of training a great model comes down to selectively training your data. They still use a lot of unlicensed data but it's a bit more sophisticated than just dumping in everything they can find.
- nicce 2y agoYou can just ”steal” all the books and paywalled research papers. That is a lot already.
- simonw 2y agoHence the recent story/scandal about Meta torrenting LibGen: https://arstechnica.com/tech-policy/2025/02/meta-torrented-over-81-7tb-of-pirated-books-to-train-ai-authors-say/ https://arstechnica.com/tech-policy/2025/02/meta-torrented-o...
- woctordho 2y agoCrawling the whole internet still helps solve problems in the long tail of the distribution. The purpose of big data is garbage in, signal out