4 ms·

"Knowledge" is a bit of a shorthand term in a case like this, yes, but not in itself much more wrong than going looking for "knowledge" in a book for instance (

by etiam 4y ago

"Knowledge" is a bit of a shorthand term in a case like this, yes, but not in itself much more wrong than going looking for "knowledge" in a book for instance (though arguably the ML context is more prone to misunderstandings about actually knowing anything, due to the public image of "AI"). The models still capture statistical regularities in the "language" (overwhelmingly that is text corpora) they were optimized for tasks against, and to the extent what went in during optimization really represents facts and relations in the world, the model probably holds representations related to facts and relations in the world. It probably also holds representations of spurious facts and relations in the world. The optimization on language is not explicitly designed for capturing semantic information and, as far as I know, there's typically no mechanisms built in to measure or report anything about the veracity of the claims. Which makes relying on anything coming out of them a shaky business at best for now of course.

jameshart 4y ago
Forget about whether the ‘latent knowledge’ in the LLM is ‘correct’ or not, this kind of research is at least predicated on the idea that the LLM holds certain things as true, and consistently holds their opposite as false. The point of this kind of work is to query the LLM and figure out what it ‘thinks is true’ - which is then presumably useful for deciding how much to rely on its output. I just find it amusing that people in the ML field are always so adamant that ‘it doesn’t know’ or ‘it’s not thinking’ or ‘it doesn’t understand’, but I don’t see much engagement with what is actually meant by ‘know’, ‘think’ or ‘understand’ in the first place. Like, you just said that a language model ‘probably… holds representations of spurious facts and relations’. But… isn’t ‘holding a representation of a fact or relation’ basically what we call ‘knowing’ something?
- naasking 4y ago
  > I just find it amusing that people in the ML field are always so adamant that ‘it doesn’t know’ or ‘it’s not thinking’ or ‘it doesn’t understand’, but I don’t see much engagement with what is actually meant by ‘know’, ‘think’ or ‘understand’ in the first place. Exactly. This is why these experts should shut up about claiming these AI models don't know things, or aren't sentient. We don't have any mechanistic explanations for what knowledge is or how it works, or what sentience is or how it works, so all of these claims are just bullshit. At best we have some hand-wavy intuitions about what some properties of knowledge or sentience have (Gettier problem, etc), but that's nowhere near enough to make the definitive claims I've seen.
- didericis 4y ago
  > isn’t ‘holding a representation of a fact or relation’ basically what we call ‘knowing’ something? No. Knowledge is fundamentally and inextricably tied to our experience of reality. The billions of years of evolution that created our perceptual framework has created a bridge to understanding reality machines don’t have. I’d argue a chicken brain has a far better greater amount of true knowledge than any of these language models. That does not diminish the utility and sophistication of these language models. But they are not knowledge. They are a disconnected meta map of what we have written down. We calibrate these language models to knowledge because our language usually corresponds to things we know. If the internet was mostly populated with sentences like “a druid flew into houston, cats pool into mud vampires wealth umpire onion buy now saxophone”, a language model mapping it all would be useless.
  - jameshart 4y ago
    You’re saying that a language model trained on ‘sensical’ language is qualitatively different from one trained on ‘nonsensical’ language use. I agree. What is your name for the quality that such a language model possesses, that makes it more ‘useful’? Why is ‘knowledge’ not a good word for that quality?
    - didericis 4y ago
      This is a deep topic that I don’t feel fully qualified to speak precisely about, but the key point is that knowledge isn’t about mapping symbols to other symbols like what language models do, it’s about mapping symbols/mental representations to reality. I think knowledge is in fact a good word for ‘sensical’ language, but the point most people like me are trying to make when talking like this is that language models can’t determine what is ‘sensical’ language. The amazing thing about these newer models is that they seem to get remarkably close to properly understanding real world concepts just through symbol mapping on massive amounts of data, but that’s highly dependent on having being fed sensical data where that mapping of symbols to reality was already done. I’m trying to argue that knowledge only really exists in the data feeding step, when we map symbols to reality. What the language models do is create a map of symbols. But they mix things which are knowledge with things that are not, and cannot determine what about their output and their mapping is actually valid knowledge. I think most of why this is has to do with the nearly endless amount of embodied physical and historical context that goes into our own sense making. A surprisingly large degree seems to be embedded in language, but there’s a lot more that is not. EDIT: I rambled, sorry; direct answer to your question about what I’d call the property that makes these language models useful is “relatively high degree of correspondence with reality”. What I consider “knowledge” is directly obtained by something which can interact with reality/directly maps reality to some representation.

by etiam 4y ago

jameshart 4y ago

Forget about whether the ‘latent knowledge’ in the LLM is ‘correct’ or not, this kind of research is at least predicated on the idea that the LLM holds certain things as true, and consistently holds their opposite as false. The point of this kind of work is to query the LLM and figure out what it ‘thinks is true’ - which is then presumably useful for deciding how much to rely on its output. I just find it amusing that people in the ML field are always so adamant that ‘it doesn’t know’ or ‘it’s not thinking’ or ‘it doesn’t understand’, but I don’t see much engagement with what is actually meant by ‘know’, ‘think’ or ‘understand’ in the first place. Like, you just said that a language model ‘probably… holds representations of spurious facts and relations’. But… isn’t ‘holding a representation of a fact or relation’ basically what we call ‘knowing’ something?

naasking 4y ago
> I just find it amusing that people in the ML field are always so adamant that ‘it doesn’t know’ or ‘it’s not thinking’ or ‘it doesn’t understand’, but I don’t see much engagement with what is actually meant by ‘know’, ‘think’ or ‘understand’ in the first place. Exactly. This is why these experts should shut up about claiming these AI models don't know things, or aren't sentient. We don't have any mechanistic explanations for what knowledge is or how it works, or what sentience is or how it works, so all of these claims are just bullshit. At best we have some hand-wavy intuitions about what some properties of knowledge or sentience have (Gettier problem, etc), but that's nowhere near enough to make the definitive claims I've seen.
didericis 4y ago
> isn’t ‘holding a representation of a fact or relation’ basically what we call ‘knowing’ something? No. Knowledge is fundamentally and inextricably tied to our experience of reality. The billions of years of evolution that created our perceptual framework has created a bridge to understanding reality machines don’t have. I’d argue a chicken brain has a far better greater amount of true knowledge than any of these language models. That does not diminish the utility and sophistication of these language models. But they are not knowledge. They are a disconnected meta map of what we have written down. We calibrate these language models to knowledge because our language usually corresponds to things we know. If the internet was mostly populated with sentences like “a druid flew into houston, cats pool into mud vampires wealth umpire onion buy now saxophone”, a language model mapping it all would be useless.
- jameshart 4y ago
  You’re saying that a language model trained on ‘sensical’ language is qualitatively different from one trained on ‘nonsensical’ language use. I agree. What is your name for the quality that such a language model possesses, that makes it more ‘useful’? Why is ‘knowledge’ not a good word for that quality?
  - didericis 4y ago
    This is a deep topic that I don’t feel fully qualified to speak precisely about, but the key point is that knowledge isn’t about mapping symbols to other symbols like what language models do, it’s about mapping symbols/mental representations to reality. I think knowledge is in fact a good word for ‘sensical’ language, but the point most people like me are trying to make when talking like this is that language models can’t determine what is ‘sensical’ language. The amazing thing about these newer models is that they seem to get remarkably close to properly understanding real world concepts just through symbol mapping on massive amounts of data, but that’s highly dependent on having being fed sensical data where that mapping of symbols to reality was already done. I’m trying to argue that knowledge only really exists in the data feeding step, when we map symbols to reality. What the language models do is create a map of symbols. But they mix things which are knowledge with things that are not, and cannot determine what about their output and their mapping is actually valid knowledge. I think most of why this is has to do with the nearly endless amount of embodied physical and historical context that goes into our own sense making. A surprisingly large degree seems to be embedded in language, but there’s a lot more that is not. EDIT: I rambled, sorry; direct answer to your question about what I’d call the property that makes these language models useful is “relatively high degree of correspondence with reality”. What I consider “knowledge” is directly obtained by something which can interact with reality/directly maps reality to some representation.