4 ms·
Except the paper in the OPs comment literally talks about how these LLM's are building actual models of the real world. Literally. The root parent post links a
by kilgnad 4y ago
Except the paper in the OPs comment literally talks about how these LLM's are building actual models of the real world. Literally. The root parent post links a research paper that categorically falsifies your statement.
No need to be worried about knocking chatGPT. I have no pride invested in the thing. But I do think that people who view it solely as statistical word generators are biased.
- theGnuMe 4y agoYou can imagine that the embedding for the token “boat” includes wood, steel, water and so forth in some set of probabilistic paths as learnt by autorgressive training since the words appear together in the past N tokens. So they are directly in frame. A question is how to connect out of frame elements and are overlapping tokens sufficient to do “that”. Specifically is the token “that” sufficiently trained to reveal what it refers to? I think this depends on the fine tuning q/a task which adds in a layer of alignment rather than being an emergent property of the LLM in general. Still alignment tasks are autoregressive (I think)… they could be masked or masked part of speech potentially.. but if autoregressive then I suspect you’re looking at regularities in positioning to identify things.