3 ms·
Ideally a LLM would have every text in existence in its training set, right? I doubt we’re anywhere near that now.
by sosodev 2y ago
Ideally a LLM would have every text in existence in its training set, right? I doubt we’re anywhere near that now.
- simonw 2y agoNo, I don't think that would help. Andrej certainly doesn't seem to think so.
- Chris_Newton 2y agoIdeally a LLM would have every text in existence in its training set, right? Only if every text in existence provides good source material to factor into generating the response. Given that half of all texts are even worse than average, this seems unlikely, and Karpathy’s argument seems very reasonable to me. The final paragraph from the original tweet, after the quote in the GP comment, mentions another interesting aspect. Even if we do have sufficient expert-level source material in a particular field to train a useful generative AI model and that in itself produces more desirable responses than training on a larger but more average-quality data set, is there still potentially useful information that could be extracted from the larger training set as well? How can we classify which aspects of a larger training set are desirable to keep while filtering out noise that competes with higher-quality source material? It feels like the progress of generative AI over the next few years might be defined more by these kinds of questions than just trying to build ever larger models using ever larger sets of training data.