3 ms·

We really need to get away from the notion that the way LLMs are and will always be trained is by feeding them ‘the entire internet’ That’s not how existing LL

We really need to get away from the notion that the way LLMs are and will always be trained is by feeding them ‘the entire internet’ That’s not how existing LLMs have been trained, and the goal of their training wasn’t to have them memorize what was on the internet. The goal was to give them a large corpus of human writing, for which datasets like common crawl were a good start point. Grabbing random content and shoving it into your training set is a cheap way to increase the dataset size but as increasingly the pool of internet content gets polluted with non-human-originated (LLM-generated) writing, and stuff like that that is intended to screw with LLM training, the validity of ‘just grab as much internet content as you like’ as a way to get training data looks increasingly less valuable. LLM training data doesn't have to be gathered googlebot crawler style, with the assumption that if it’s online it should be in the dataset.

theptip 3y ago
Don’t forget, the only reason we are having an AI revolution is that it turns out there is lots of crystallized intelligence in the internet corpus, providing a contiguous loss slope to descend all the way to GPT-4 level intelligence. It did not have to be that way, the would could have been such that there is no brute force path to intelligence without investing evolutionary timescales in your search. It’s not clear to me how you’d get enough training data for Chinchilla-optimal LLMs without doing something like crawl the internet. Perhaps we can get distillation, textbooks, and other synthetic data to be good enough that we don’t need to crawl the web every time, but we are not there yet. All the frontier models require so much text that these massive piles are required. There is already a lot of filtering going on, Karpathy’s point is this sort of thing is hard to detect.

theptip 3y ago

Don’t forget, the only reason we are having an AI revolution is that it turns out there is lots of crystallized intelligence in the internet corpus, providing a contiguous loss slope to descend all the way to GPT-4 level intelligence. It did not have to be that way, the would could have been such that there is no brute force path to intelligence without investing evolutionary timescales in your search. It’s not clear to me how you’d get enough training data for Chinchilla-optimal LLMs without doing something like crawl the internet. Perhaps we can get distillation, textbooks, and other synthetic data to be good enough that we don’t need to crawl the web every time, but we are not there yet. All the frontier models require so much text that these massive piles are required. There is already a lot of filtering going on, Karpathy’s point is this sort of thing is hard to detect.