3 ms·
They are emergent in the sense that there is nothing in the pre-training dataset that would show the LLM by example how to, for example, compare and contrast an
by selfhoster11 3y ago
They are emergent in the sense that there is nothing in the pre-training dataset that would show the LLM by example how to, for example, compare and contrast any given pairing of fruit, technologies, or fictional settings, while thinking with the mindset of a doctor that hates both options, and on top of that make sure that this ends up formatted as a stream-of-consciousness. It can learn all these aspects from the source data individually in isolation, but there's no way there are examples that show how to combine it all (awareness of world information + knowledge of how to use it) into a single answer. That's probably a very clumsy example - others online have supplied more rigorous ones that I recommend checking out.
Strictly speaking, it might be "stochastic parroting". But really, if you want to be a great and supremely effective stochastic parrot, you have to learn an internal representation of certain things so that you can predict them. And there are hints that this is exactly what a sufficiently-large large language model is doing.