3 ms·
> At this point there is no framework that suggests GPT understands the underlying data. It can’t assign meaning as a human would. Actually there are many indi
by naasking 4y ago
> At this point there is no framework that suggests GPT understands the underlying data. It can’t assign meaning as a human would.
Actually there are many indications that GPT understands the data, because its output mostly makes sense. The reason it can't assign meaning the way a human would is because a human can correlate words with other sensory data that GPT doesn't have access to. That's where GPT creates nonsense.
Think carefully about what "understanding" means in a mechanistic sense. It's a form of compression, and a few billion parameters encoding the contents of a large part of the internet seems like pretty good compression to me.
- ivanbakel 4y agoGPT doesn't display understanding of purely abstract systems, so I doubt it's an issue of lacking sensory information. It can't consistently do arithmetic, for example - and I think it would be presumptuous to insist that sensory information is a prerequisite for mathematics, even though that's how humans arrived at it.
- naasking 4y agoIt's not yet clear why it struggles with arithmetic. It could be data-related, could be model-related, although scaling both seems to improve the situation. In any case, GPT could still understand non-abstract things just fine. People with low IQ also struggle with abstract reasoning, and IQ tests place GPT-3 at around 83.