The way I think of the phenomenon, neural networks interpolate. Facts form only points in this space, while the network weights represents the patterns of the p

The way I think of the phenomenon, neural networks interpolate. Facts form only points in this space, while the network weights represents the patterns of the points by "curves". They are so good at it, though, that it almost seem like they remember each point. Their ability to generalize, though, comes from representing the learning as these "curves". Sometimes they produce generalizations that are poor or false. Even when they are false, though, they may be good. In fact, Stephen Wolfram is claiming that when GPT-4 "halucinates" a function name for the Wolfram Language, the team often agrees with GPT-4 that precisely that call SHOULD be part of the language, so they go on and implement it AFTER it came up as a halucination. One might (and I would) argue that this behavior is precisely what most human creativity (in technical domains) is about.