3 ms·

> isn’t ‘holding a representation of a fact or relation’ basically what we call ‘knowing’ something? No. Knowledge is fundamentally and inextricably tied to ou

> isn’t ‘holding a representation of a fact or relation’ basically what we call ‘knowing’ something? No. Knowledge is fundamentally and inextricably tied to our experience of reality. The billions of years of evolution that created our perceptual framework has created a bridge to understanding reality machines don’t have. I’d argue a chicken brain has a far better greater amount of true knowledge than any of these language models. That does not diminish the utility and sophistication of these language models. But they are not knowledge. They are a disconnected meta map of what we have written down. We calibrate these language models to knowledge because our language usually corresponds to things we know. If the internet was mostly populated with sentences like “a druid flew into houston, cats pool into mud vampires wealth umpire onion buy now saxophone”, a language model mapping it all would be useless.

jameshart 4y ago
You’re saying that a language model trained on ‘sensical’ language is qualitatively different from one trained on ‘nonsensical’ language use. I agree. What is your name for the quality that such a language model possesses, that makes it more ‘useful’? Why is ‘knowledge’ not a good word for that quality?
- didericis 4y ago
  This is a deep topic that I don’t feel fully qualified to speak precisely about, but the key point is that knowledge isn’t about mapping symbols to other symbols like what language models do, it’s about mapping symbols/mental representations to reality. I think knowledge is in fact a good word for ‘sensical’ language, but the point most people like me are trying to make when talking like this is that language models can’t determine what is ‘sensical’ language. The amazing thing about these newer models is that they seem to get remarkably close to properly understanding real world concepts just through symbol mapping on massive amounts of data, but that’s highly dependent on having being fed sensical data where that mapping of symbols to reality was already done. I’m trying to argue that knowledge only really exists in the data feeding step, when we map symbols to reality. What the language models do is create a map of symbols. But they mix things which are knowledge with things that are not, and cannot determine what about their output and their mapping is actually valid knowledge. I think most of why this is has to do with the nearly endless amount of embodied physical and historical context that goes into our own sense making. A surprisingly large degree seems to be embedded in language, but there’s a lot more that is not. EDIT: I rambled, sorry; direct answer to your question about what I’d call the property that makes these language models useful is “relatively high degree of correspondence with reality”. What I consider “knowledge” is directly obtained by something which can interact with reality/directly maps reality to some representation.
djexjms 4y ago
I think more caution is warranted here. That argument seems to rely on our experience of reality being an accurate representation of objective reality. There are some ideas that are surprisingly difficult to dismiss that essentially state that our perceptions of reality do not have this property, and that in fact that any organism that is the result of an evolutionary process is really only ever incentivized to evolve a perception system that presents them with what is essentially a user interface. Spacetime itself may just be a data structure generated in the mind as a simplification of some much more complex and informationally overwhelming reality.
- didericis 4y ago
  I’m quite sympathetic to that line of thinking. Your language is suggestive of an interview on Lex Fridman’s podcast/am guessing you’ve listened to it. Nietzche also basically recognized this over a century ago, though he put it in different terms/couldn’t articulate in as much mathematical detail as evolutionary game theorists allegedly can. That’s actually precisely why I think evolutionarily adapted embodied perceptual systems need to be a part of any true knowledge: reality is so incredibly complex that we don’t know what our perception is or is not doing. We just know it’s doing something reality based because we exist. Perceptual models don’t need to correspond with reality like actual embodied perception (and corresponding cognition) I don’t buy the argument that the correspondence with objective reality in evolved perceptual systems is zero as is claimed in that interview, simply because we exist. I think that conclusion is an artifact of precisely the same kind of over abstraction and non-correspondence that invalidates language models as knowledge systems. Our existence means our perception corresponds with at least some kind of compatibility with and ability to navigate through reality. A system which does not have that embodied evolutionary history with millennia of required correspondence with reality, simplified or not, does not have all of the essential embedded context and understanding of complexity that goes into our sense making. Science is about discovering more of that hidden context, and perhaps these language models can help identify patterns we haven’t seen yet that do in fact correspond with reality. But the model can’t test it’s own correspondence.

jameshart 4y ago

You’re saying that a language model trained on ‘sensical’ language is qualitatively different from one trained on ‘nonsensical’ language use. I agree. What is your name for the quality that such a language model possesses, that makes it more ‘useful’? Why is ‘knowledge’ not a good word for that quality?

didericis 4y ago
This is a deep topic that I don’t feel fully qualified to speak precisely about, but the key point is that knowledge isn’t about mapping symbols to other symbols like what language models do, it’s about mapping symbols/mental representations to reality. I think knowledge is in fact a good word for ‘sensical’ language, but the point most people like me are trying to make when talking like this is that language models can’t determine what is ‘sensical’ language. The amazing thing about these newer models is that they seem to get remarkably close to properly understanding real world concepts just through symbol mapping on massive amounts of data, but that’s highly dependent on having being fed sensical data where that mapping of symbols to reality was already done. I’m trying to argue that knowledge only really exists in the data feeding step, when we map symbols to reality. What the language models do is create a map of symbols. But they mix things which are knowledge with things that are not, and cannot determine what about their output and their mapping is actually valid knowledge. I think most of why this is has to do with the nearly endless amount of embodied physical and historical context that goes into our own sense making. A surprisingly large degree seems to be embedded in language, but there’s a lot more that is not. EDIT: I rambled, sorry; direct answer to your question about what I’d call the property that makes these language models useful is “relatively high degree of correspondence with reality”. What I consider “knowledge” is directly obtained by something which can interact with reality/directly maps reality to some representation.

djexjms 4y ago

I think more caution is warranted here. That argument seems to rely on our experience of reality being an accurate representation of objective reality. There are some ideas that are surprisingly difficult to dismiss that essentially state that our perceptions of reality do not have this property, and that in fact that any organism that is the result of an evolutionary process is really only ever incentivized to evolve a perception system that presents them with what is essentially a user interface. Spacetime itself may just be a data structure generated in the mind as a simplification of some much more complex and informationally overwhelming reality.

didericis 4y ago
I’m quite sympathetic to that line of thinking. Your language is suggestive of an interview on Lex Fridman’s podcast/am guessing you’ve listened to it. Nietzche also basically recognized this over a century ago, though he put it in different terms/couldn’t articulate in as much mathematical detail as evolutionary game theorists allegedly can. That’s actually precisely why I think evolutionarily adapted embodied perceptual systems need to be a part of any true knowledge: reality is so incredibly complex that we don’t know what our perception is or is not doing. We just know it’s doing something reality based because we exist. Perceptual models don’t need to correspond with reality like actual embodied perception (and corresponding cognition) I don’t buy the argument that the correspondence with objective reality in evolved perceptual systems is zero as is claimed in that interview, simply because we exist. I think that conclusion is an artifact of precisely the same kind of over abstraction and non-correspondence that invalidates language models as knowledge systems. Our existence means our perception corresponds with at least some kind of compatibility with and ability to navigate through reality. A system which does not have that embodied evolutionary history with millennia of required correspondence with reality, simplified or not, does not have all of the essential embedded context and understanding of complexity that goes into our sense making. Science is about discovering more of that hidden context, and perhaps these language models can help identify patterns we haven’t seen yet that do in fact correspond with reality. But the model can’t test it’s own correspondence.