Don’t forget, the only reason we are having an AI revolution is that it turns out there is lots of crystallized intelligence in the internet corpus, providing a

Don’t forget, the only reason we are having an AI revolution is that it turns out there is lots of crystallized intelligence in the internet corpus, providing a contiguous loss slope to descend all the way to GPT-4 level intelligence. It did not have to be that way, the would could have been such that there is no brute force path to intelligence without investing evolutionary timescales in your search. It’s not clear to me how you’d get enough training data for Chinchilla-optimal LLMs without doing something like crawl the internet. Perhaps we can get distillation, textbooks, and other synthetic data to be good enough that we don’t need to crawl the web every time, but we are not there yet. All the frontier models require so much text that these massive piles are required. There is already a lot of filtering going on, Karpathy’s point is this sort of thing is hard to detect.