4 ms·
> To be able to predict accurately sentences that make sense, GPT-4 must have a internal way of representing concepts, such as "objects', "time", "family" and e
by westernpopular 3y ago
> To be able to predict accurately sentences that make sense, GPT-4 must have a internal way of representing concepts, such as "objects', "time", "family" and everything else under the sun.
[citation needed]
- lyu07282 3y agohttps://arxiv.org/abs/2210.13382 https://arxiv.org/abs/2210.13382 https://arxiv.org/abs/2310.02207 https://arxiv.org/abs/2310.02207 among others
- vidarh 3y agoWhile one might well argue over whether they "must have" this, it's clear that they do end up with internal models of the world. Even quite literally: https://twitter.com/wesg52/status/1709551516577902782 https://twitter.com/wesg52/status/1709551516577902782
- sidlls 3y agoThat's more or less exactly what these models do by design, otherwise the predictors/estimators for the next token in a sequence would fail spectacularly. These models aren't just plucking random tokens out of a bag and ordering them based on some brute force large-scale memory lookup or whatever. There is an internal representation of the tokens in a more meaningful sense. That sense, however, is limited to the statistical/mathematical framework upon which the models are built. It's a huge (in my opinion completely unjustified, wishful thinking exercise) leap to call it "reasoning."
- immibis 3y agoHow can any system use a word such as "time" in a way that makes sense without representing the word? It definitely has at least one representation: the binary ASCII code 01110100011010010110110101100101.