its not really memorizing the internet. its passing the internet through the network. imagine you are in class for an hour, and the professor was boring - speak

its not really memorizing the internet. its passing the internet through the network. imagine you are in class for an hour, and the professor was boring - speaking in a monotone voice. did you really memorize everything they said? Now imagine the professor is extremely engaging, maybe you learned more, but you still haven't memorized what everything they said. so neural network architechers are going to evolve and get more complicated or even just more efficient at learning. a standard NN was supposed to be able to memorize even time series data, but a memory component was added and LSTMs did a lot better at that task. so for now, we can keep adding model size and topology to these NN and still get huge value add. gpt4 is a proof if that. lastly, its not just about throwing data at it. openai spent a lot of time giving the model feedback.