>LLMs use a neural network to interpolate "interpolate" in what vector space, pray tell? What does "interpolate" even mean, when I prompt it "write me a story

>LLMs use a neural network to interpolate "interpolate" in what vector space, pray tell? What does "interpolate" even mean, when I prompt it "write me a story about a sentient banana in the style of Hemingway and oh make it a commentary on class consciousness"? You can't assemble such a thing by cutting and pasting pieces of other text. That kind of "interpolation" has to happen at the semantic level - ipso facto, there is a semantic level. Not to mention that no, they don't predict the next token in the training set. Give any LLM the first paragraph of any Wikipedia article - almost certainly in the training set, and uniquely so - and it won't predict the next word correctly, a lot of the time. But it will predict a word that is grammatically correct, stylistically apropos, and most likely factually correct. So what's it really doing, hm? LLMs aren't even large enough to contain their training data - not even remotely close. It can't "stitch together things it saw" because it doesn't remember them. It only remembers the ideas used to construct them. The learned abstraction is the entire point of the exercise. LLMs would be useless if they were overfit the way you say they are.