4 ms·
Why is it no one ever talks about the one thing no one can get their hands on except the big labs ? I'm talking about the training set. Sure there are some op
by ur-whale 6mo ago
Why is it no one ever talks about the one thing no one can get their hands on except the big labs ?
I'm talking about the training set.
Sure there are some open sets out there.
But my guess is they are nowhere near what OpenAI, Google and Anthropic are actually using.
Happy to be proven wrong.
- CamperBob2 6mo agoI think OpenAI and Anthropic just downloaded the same torrents from Anna's Archive that anyone else can. But it's only OK when they do it. The rest of us get nastygrams from law offices. Anthropic actually had to cough up some bucks, for that matter. At that point, a lot depends on the quality of the preprocessing applied to the raw text dumps. It is reportedly not that trivial to go from DumpOfSketchyRussianPirateSite.zip to a data set suitable for ingestion during pretraining. A few bad chunks of data can apparently do more harm than one would expect. AFAIK Google scans almost everything in print as part of the Google Books initiative, so they may have been able to skip the torrenting step.
- ur-whale 6mo ago> I think OpenAI and Anthropic just downloaded the same torrents from Anna's Archive I think you're underestimating how much chat conversation data they've gathered at this point, and how much of it is part of the training set. None of that is available to anyone who wants to train a frontier model. And when it comes to Google ... the hoard of data they're sitting on goes back to what? 1998? They've basically got a digital record of what happened since the birth of the internet.