3 ms·

> Many attempts at making them refuse to answer what they don't know caused them to refuse to answer things they did in fact know. Are we sure they know these

> Many attempts at making them refuse to answer what they don't know caused them to refuse to answer things they did in fact know. Are we sure they know these things as opposed to being able to consistently guess correctly? With LLMs I'm not sure we even have a clear definition of what it means for it to "know" something.

redox99 1y ago
Yes. You could ask for factual information like "Tallest building in X place" and first it would answer it did not know. After pressuring it, it would answer with the correct building and height. But also things where guessing was desirable. For example with a riddle it would tell you it did not know or there wasn't enough information. After pressuring it to answer anyway it would correctly solve the riddle. The official llama 2 finetune was pretty bad with this stuff.
- Volundr 1y ago
  > After pressuring it, it would answer with the correct building and height. And if you bully it enough on something nonsensical it'll give you a wrong answer. You press it, and it takes a guess even though you told it not to, and gets it right, then you go "see it knew!". There's no database hanging out in ChatGPT/Claude/Gemini's weights with a list of cities and the tallest buildings. There's a whole bunch of opaque stats derived from the content it's been trained on that means that most of the time it'll come up with the same guess. But there's no difference in process between that highly consistent response to you asking the tallest building in New York and the one where it hallucinates a Python method that doesn't exist, or suggests glue to keep the cheese on your pizza. It's all the same process to the LLM.
ajross 1y ago
> Are we sure they know these things as opposed to being able to consistently guess correctly? What is the practical difference you're imagining between "consistently correct guess" and "knowledge"? LLMs aren't databases. We have databases. LLMs are probabilistic inference engines. All they do is guess, essentially. The discussion here is about how to get the guess to "check itself" with a firmer idea of "truth". And it turns out that's hard because it requires that the guessing engine know that something needs to be checked in the first place.
- mynameisvlad 1y ago
  Simple, and even simpler from your own example. Knowledge has an objective correctness. We know that there is a "right" and "wrong" answer and we know what a "right" answer is. "Consistently correct guesses", based on the name itself, is not reliable enough to actually be trusted. There's absolutely no guarantee that the next "consistently correct guess" is knowledge or a hallucination.
  - ajross 1y ago
    This is a circular semantic argument. You're saying knowledge is knowledge because it's correct, where guessing is guessing because it's a guess. But "is it correct?" is precisely the question you're asking the poor LLM to answer in the first place. It's not helpful to just demand a computation device work the way you want, you need to actually make it work. Also, too, there are whole subfields of philosophy that make your statement here kinda laughably naive. Suffice it to say that, no, knowledge as rigorously understood does not have "an objective correctness".
    - mynameisvlad 1y ago
      I mean, it clearly does based on your comments showing a need for a correctness check to disambiguate between made up "hallucinations" and actual "knowledge" (together, a "consistently correct guess"). The fact that you are humanizing an LLM is honestly just plain weird. It does not have feelings. It doesn't care that it has to answer "is it correct?" and saying poor LLM is just trying to tug on heartstrings to make your point.
      - ajross 1y ago
        FWIW "asking the poor <system> to do <requirement>" is an extremely common idiom. It's used as a metaphor for an inappropriate or unachievable design requirement. Nothing to do with LLMs. I work on microcontrollers for a living.
    - Volundr 1y ago
      > You're saying knowledge is knowledge because it's correct, where guessing is guessing because it's a guess. Knowledge is knowledge because the knower knows it to be correct. I know I'm typing this into my phone, because it's right here in my hand. I'm guessing you typed your reply into some electronic device. I'm guessing this is true for all your comments. Am I 100% accurate? You'll have to answer that for me. I don't know it to be true, it's a highly informed guess. Being wrong sometimes is not what makes a guess a guess. It's the different between pulling something from your memory banks, be they biological or mechanical, vs inferring it from some combination of your knowledge (what's in those memory banks), statistics, intuition, and whatever other fairy dust you sprinkle on.

redox99 1y ago

Yes. You could ask for factual information like "Tallest building in X place" and first it would answer it did not know. After pressuring it, it would answer with the correct building and height. But also things where guessing was desirable. For example with a riddle it would tell you it did not know or there wasn't enough information. After pressuring it to answer anyway it would correctly solve the riddle. The official llama 2 finetune was pretty bad with this stuff.

Volundr 1y ago
> After pressuring it, it would answer with the correct building and height. And if you bully it enough on something nonsensical it'll give you a wrong answer. You press it, and it takes a guess even though you told it not to, and gets it right, then you go "see it knew!". There's no database hanging out in ChatGPT/Claude/Gemini's weights with a list of cities and the tallest buildings. There's a whole bunch of opaque stats derived from the content it's been trained on that means that most of the time it'll come up with the same guess. But there's no difference in process between that highly consistent response to you asking the tallest building in New York and the one where it hallucinates a Python method that doesn't exist, or suggests glue to keep the cheese on your pizza. It's all the same process to the LLM.

ajross 1y ago

> Are we sure they know these things as opposed to being able to consistently guess correctly? What is the practical difference you're imagining between "consistently correct guess" and "knowledge"? LLMs aren't databases. We have databases. LLMs are probabilistic inference engines. All they do is guess, essentially. The discussion here is about how to get the guess to "check itself" with a firmer idea of "truth". And it turns out that's hard because it requires that the guessing engine know that something needs to be checked in the first place.

mynameisvlad 1y ago
Simple, and even simpler from your own example. Knowledge has an objective correctness. We know that there is a "right" and "wrong" answer and we know what a "right" answer is. "Consistently correct guesses", based on the name itself, is not reliable enough to actually be trusted. There's absolutely no guarantee that the next "consistently correct guess" is knowledge or a hallucination.
- ajross 1y ago
  This is a circular semantic argument. You're saying knowledge is knowledge because it's correct, where guessing is guessing because it's a guess. But "is it correct?" is precisely the question you're asking the poor LLM to answer in the first place. It's not helpful to just demand a computation device work the way you want, you need to actually make it work. Also, too, there are whole subfields of philosophy that make your statement here kinda laughably naive. Suffice it to say that, no, knowledge as rigorously understood does not have "an objective correctness".
  - mynameisvlad 1y ago
    I mean, it clearly does based on your comments showing a need for a correctness check to disambiguate between made up "hallucinations" and actual "knowledge" (together, a "consistently correct guess"). The fact that you are humanizing an LLM is honestly just plain weird. It does not have feelings. It doesn't care that it has to answer "is it correct?" and saying poor LLM is just trying to tug on heartstrings to make your point.
    - ajross 1y ago
      FWIW "asking the poor <system> to do <requirement>" is an extremely common idiom. It's used as a metaphor for an inappropriate or unachievable design requirement. Nothing to do with LLMs. I work on microcontrollers for a living.
  - Volundr 1y ago
    > You're saying knowledge is knowledge because it's correct, where guessing is guessing because it's a guess. Knowledge is knowledge because the knower knows it to be correct. I know I'm typing this into my phone, because it's right here in my hand. I'm guessing you typed your reply into some electronic device. I'm guessing this is true for all your comments. Am I 100% accurate? You'll have to answer that for me. I don't know it to be true, it's a highly informed guess. Being wrong sometimes is not what makes a guess a guess. It's the different between pulling something from your memory banks, be they biological or mechanical, vs inferring it from some combination of your knowledge (what's in those memory banks), statistics, intuition, and whatever other fairy dust you sprinkle on.