4 ms·

> LLMs so far seem to be entirely unverifiable. I don't understand this complaint. Are they any less verifiable than a human?

by ta8645 2y ago

threeseed 2y ago
I can ask a human to explain the steps they took to answer a question. I can ask a human a question 100 times and I don't get back 100 different answers. None of those applies to an LLM.
- ta8645 2y ago
  You can ask an LLM to explain itself.. it will give you a logical stepwise progression from your question to its answer. It will often contain a mistake, but the same is true for a human. And if your LLM is giving you 100 different answers, then it has been configured to do so. Because instead, it could be configured to never vary at all. It could be 100% reproducible if so desired.
  - soloist11 2y ago
    [dead]
  - bluefirebrand 2y ago
    > it will give you a logical stepwise progression from your question to its answer. No, it will generate a new hallucination that might be a logical stepwise progression from the question you asked to the answer you gave, but it is not due to any actual internal reasoning being done by the LLM
    - ta8645 2y ago
      So what? You have no way to know for sure if the human you ask the same question, does either. The question that started this thread was related to verifiability. And i still think it is a spurious complaint, given that we have exactly the same limitations when dealing with any human agent.
      - wizzwizz4 2y ago
        The problem of other minds is no reason to throw everything out the window. Humans are capable of being conscious of their reasoning processes; token-at-a-time predictive text models wired up as chatbots aren't capable of it. Your choice is between a possibly-mistaken, possibly-lying human, and a 100%-definitely incapable computer program. You don't know either "for sure", but you don't know that the external world exists "for sure" either. It's an insight-free observation, and shouldn't be the focus of anyone's decision-making.
        ta8645 2y ago
        You've made some interesting points, which are debatable, for sure. But you've failed to address the question being asked about "verifiability".
        vanviegen 2y ago
        When you ask an LLM to carefully reason step by step before arriving at its answer, that seems pretty much the same as conscious reasoning to me. Of course, when asked to justify a gut reaction after the fact, it will just come up with something that sounds plausible (and may or may not be true). Just like humans do.
      - bluefirebrand 2y ago
        > And i still think it is a spurious complaint, given that we have exactly the same limitations when dealing with any human agent We're not talking about an LLM that is trying to do the job of a human, here We're talking about an LLM that is trying to give authoritative answers to any question typed into the Google search bar It's already well past the scale that humans could handle Talking about human shortcomings when discussing LLMs is a red herring at best, or some kind of deliberate goalpost shifting at worst
        ta8645 2y ago
        Nothing of the sort. I'm trying to understand why anyone cares about formal verifiability in this context, since it's not something we rely on when asking humans to answer questions for us. We evaluate any answer we get without such mathematical proofs, and instead simply judge the answer we're given on its fit and usefulness. Anyone who doubts the usefulness of even these nascent LLMs is fooling themselves. The proof is in the pudding, they already do a great job, even with all their obvious limitations.
    - vidarh 2y ago
      We have no clear evidence the same isn't true for humans, and some that it might be. See experiments with split brain patients, that have shown the brain halves will readily explain how they made decisions they provably never made.
      - bluefirebrand 2y ago
        I think we have thousands of years of evidence that the same isn't true for humans The fact that one human brain is composed in two half brains that each seem to be able to function fairly independently when separated doesn't seem like it changes that much
        vidarh 2y ago
        It changes that we can prove it. In as much as you can do experiments where you "hide" actions from one half, mess with it (e.g. totally change "choices" supposedly made by the brain) and ask the other half to explain why "it" made the choice, and it will do so, unaware that the choice was made by the experimenters. It won't go "sorry, but I don't know" or similar.
  - Vetch 2y ago
    > It will often contain a mistake...but the same is true for a human. If this were true textbooks could not work. Given a question, we don't consult random humans but experts of their field. If I have a question on algorithms, I might check a text by Knuth, I wouldn't randomly ask on the street. > It could be 100% reproducible if so desired. Reproducible does not mean better. For harder questions, it's often best to generate multiple answers at a higher temperature than to greedily pick the highest probability tokens.
- kenjackson 2y ago
  Ask a human what the meaning of life is and how it impacts their day to day interactions. I know I can tell you an answer but I couldn’t tell you steps about how I got it. And if you asked it to me twice I’d definitely give different answers unless you told me to give the same answer. In part I’d give a different answer because if someone asks me the same question twice I assume the first answer wasn’t sufficient.
  - threeseed 2y ago
    No one is taking about existential questions about meaning of life. We are talking about basic things like whether or not to eat rocks or put glue in recipes. We can answer those questions with a chain of logic and repeatability.
    - kenjackson 2y ago
      And those specific questions get repeatable answers on ChatGPT for me. Here are two answers I got which seem as close as you’d expect any human to give: “No, people should not eat rocks. Rocks are not digestible and can cause serious harm to the digestive system, including blockages and damage to internal organs. Eating rocks can lead to severe health problems and should be avoided.” “No, people should not eat rocks. Rocks are not digestible and can cause serious harm to the digestive system, including blockages, abrasions, and potential poisoning from harmful minerals or substances. It's important to consume only food items that are safe and meant for human consumption.”
- astrange 2y ago
  The only reason you can't verify a server side LLM is you can't see the model. It is possible to look at its activations if you have the model.
  - tux1968 2y ago
    Do the activations tell you anything more than what the LLM delivers in plain text? Other than for trivial bugs in the LLM code, I don't think so.
    - astrange 2y ago
      Yes, "making up an answer" will look different from "quoting pretrained knowledge" because eg the model might've decided you were asking a creative writing question.
      - duskwuff 2y ago
        Can you cite a source for this, or are you speculating? My understanding was the opposite -- that the activity of a confabulating LLM is indistinguishable from one giving factually accurate responses. https://arxiv.org/abs/2401.11817 https://arxiv.org/abs/2401.11817
        astrange 2y ago
        Some things like: https://arxiv.org/abs/2310.18168 https://arxiv.org/abs/2310.18168 https://arxiv.org/abs/2310.06824 https://arxiv.org/abs/2310.06824 There are various reasons an LLM might have incorrect "beliefs" - the input text was false, training doesn't try to preserve true beliefs, quantization certainly doesn't. So it can't be perfectly addressed, but some things leading to it seem like they can be found. > https://arxiv.org/abs/2401.11817 https://arxiv.org/abs/2401.11817 This seems like it's true since LLMs are a finite size, but in Google's case it has a "truth oracle" (the websites it's quoting)… the problem is it's a bad oracle.
      - lukev 2y ago
        This is confidently stated and incorrect.
- mewpmewp2 2y ago
  Isn't it actually known that every time a human brain recalls a piece of memory the memory gets slightly changed? If the answer has any length at all, I imagine the answer can vary every single time the person answers, unless they prepared for it, memorized it word by word.
  - vidarh 2y ago
    It's also known that the brain is prone to outright constructing demonstrably fictional rationalisations of decisions it's never made. Any notion that we're reliable narrators of our own thoughts and actions is fiction.
    - mewpmewp2 2y ago
      Right, that makes sense, our brain will do a quick black box judgment (some may call it system 1), and then rational process only works to justify that or explain the black box, assuming that black box is always correct (depending on the person and how much they trust their black box or system 1). So system 2 is "hallucinating" the best justification for system 1. And usually system 2 will do it only when it's required to justify it for anyone else.
- vidarh 2y ago
  And for most cases that human explanation is likely with a disturbing frequency a complete fabrication after the fact. See experiments on split brain patients. With respect to repeatability, yes, LLMs are currently frozen in time. That is not an inherent limitation, but it is one that is practical for a lot uses and a problem for some.

> LLMs so far seem to be entirely unverifiable. I don't understand this complaint. Are they any less verifiable than a human?

by ta8645 2y ago

> LLMs so far seem to be entirely unverifiable. I don't understand this complaint. Are they any less verifiable than a human?

threeseed 2y ago

I can ask a human to explain the steps they took to answer a question. I can ask a human a question 100 times and I don't get back 100 different answers. None of those applies to an LLM.

ta8645 2y ago
You can ask an LLM to explain itself.. it will give you a logical stepwise progression from your question to its answer. It will often contain a mistake, but the same is true for a human. And if your LLM is giving you 100 different answers, then it has been configured to do so. Because instead, it could be configured to never vary at all. It could be 100% reproducible if so desired.
- soloist11 2y ago
  [dead]
- bluefirebrand 2y ago
  > it will give you a logical stepwise progression from your question to its answer. No, it will generate a new hallucination that might be a logical stepwise progression from the question you asked to the answer you gave, but it is not due to any actual internal reasoning being done by the LLM
  - ta8645 2y ago
    So what? You have no way to know for sure if the human you ask the same question, does either. The question that started this thread was related to verifiability. And i still think it is a spurious complaint, given that we have exactly the same limitations when dealing with any human agent.
    - wizzwizz4 2y ago
      The problem of other minds is no reason to throw everything out the window. Humans are capable of being conscious of their reasoning processes; token-at-a-time predictive text models wired up as chatbots aren't capable of it. Your choice is between a possibly-mistaken, possibly-lying human, and a 100%-definitely incapable computer program. You don't know either "for sure", but you don't know that the external world exists "for sure" either. It's an insight-free observation, and shouldn't be the focus of anyone's decision-making.
      - ta8645 2y ago
        You've made some interesting points, which are debatable, for sure. But you've failed to address the question being asked about "verifiability".
      - vanviegen 2y ago
        When you ask an LLM to carefully reason step by step before arriving at its answer, that seems pretty much the same as conscious reasoning to me. Of course, when asked to justify a gut reaction after the fact, it will just come up with something that sounds plausible (and may or may not be true). Just like humans do.
    - bluefirebrand 2y ago
      > And i still think it is a spurious complaint, given that we have exactly the same limitations when dealing with any human agent We're not talking about an LLM that is trying to do the job of a human, here We're talking about an LLM that is trying to give authoritative answers to any question typed into the Google search bar It's already well past the scale that humans could handle Talking about human shortcomings when discussing LLMs is a red herring at best, or some kind of deliberate goalpost shifting at worst
      - ta8645 2y ago
        Nothing of the sort. I'm trying to understand why anyone cares about formal verifiability in this context, since it's not something we rely on when asking humans to answer questions for us. We evaluate any answer we get without such mathematical proofs, and instead simply judge the answer we're given on its fit and usefulness. Anyone who doubts the usefulness of even these nascent LLMs is fooling themselves. The proof is in the pudding, they already do a great job, even with all their obvious limitations.
  - vidarh 2y ago
    We have no clear evidence the same isn't true for humans, and some that it might be. See experiments with split brain patients, that have shown the brain halves will readily explain how they made decisions they provably never made.
    - bluefirebrand 2y ago
      I think we have thousands of years of evidence that the same isn't true for humans The fact that one human brain is composed in two half brains that each seem to be able to function fairly independently when separated doesn't seem like it changes that much
      - vidarh 2y ago
        It changes that we can prove it. In as much as you can do experiments where you "hide" actions from one half, mess with it (e.g. totally change "choices" supposedly made by the brain) and ask the other half to explain why "it" made the choice, and it will do so, unaware that the choice was made by the experimenters. It won't go "sorry, but I don't know" or similar.
- Vetch 2y ago
  > It will often contain a mistake...but the same is true for a human. If this were true textbooks could not work. Given a question, we don't consult random humans but experts of their field. If I have a question on algorithms, I might check a text by Knuth, I wouldn't randomly ask on the street. > It could be 100% reproducible if so desired. Reproducible does not mean better. For harder questions, it's often best to generate multiple answers at a higher temperature than to greedily pick the highest probability tokens.
kenjackson 2y ago
Ask a human what the meaning of life is and how it impacts their day to day interactions. I know I can tell you an answer but I couldn’t tell you steps about how I got it. And if you asked it to me twice I’d definitely give different answers unless you told me to give the same answer. In part I’d give a different answer because if someone asks me the same question twice I assume the first answer wasn’t sufficient.
- threeseed 2y ago
  No one is taking about existential questions about meaning of life. We are talking about basic things like whether or not to eat rocks or put glue in recipes. We can answer those questions with a chain of logic and repeatability.
  - kenjackson 2y ago
    And those specific questions get repeatable answers on ChatGPT for me. Here are two answers I got which seem as close as you’d expect any human to give: “No, people should not eat rocks. Rocks are not digestible and can cause serious harm to the digestive system, including blockages and damage to internal organs. Eating rocks can lead to severe health problems and should be avoided.” “No, people should not eat rocks. Rocks are not digestible and can cause serious harm to the digestive system, including blockages, abrasions, and potential poisoning from harmful minerals or substances. It's important to consume only food items that are safe and meant for human consumption.”
astrange 2y ago
The only reason you can't verify a server side LLM is you can't see the model. It is possible to look at its activations if you have the model.
- tux1968 2y ago
  Do the activations tell you anything more than what the LLM delivers in plain text? Other than for trivial bugs in the LLM code, I don't think so.
  - astrange 2y ago
    Yes, "making up an answer" will look different from "quoting pretrained knowledge" because eg the model might've decided you were asking a creative writing question.
    - duskwuff 2y ago
      Can you cite a source for this, or are you speculating? My understanding was the opposite -- that the activity of a confabulating LLM is indistinguishable from one giving factually accurate responses. https://arxiv.org/abs/2401.11817 https://arxiv.org/abs/2401.11817
      - astrange 2y ago
        Some things like: https://arxiv.org/abs/2310.18168 https://arxiv.org/abs/2310.18168 https://arxiv.org/abs/2310.06824 https://arxiv.org/abs/2310.06824 There are various reasons an LLM might have incorrect "beliefs" - the input text was false, training doesn't try to preserve true beliefs, quantization certainly doesn't. So it can't be perfectly addressed, but some things leading to it seem like they can be found. > https://arxiv.org/abs/2401.11817 https://arxiv.org/abs/2401.11817 This seems like it's true since LLMs are a finite size, but in Google's case it has a "truth oracle" (the websites it's quoting)… the problem is it's a bad oracle.
    - lukev 2y ago
      This is confidently stated and incorrect.
mewpmewp2 2y ago
Isn't it actually known that every time a human brain recalls a piece of memory the memory gets slightly changed? If the answer has any length at all, I imagine the answer can vary every single time the person answers, unless they prepared for it, memorized it word by word.
- vidarh 2y ago
  It's also known that the brain is prone to outright constructing demonstrably fictional rationalisations of decisions it's never made. Any notion that we're reliable narrators of our own thoughts and actions is fiction.
  - mewpmewp2 2y ago
    Right, that makes sense, our brain will do a quick black box judgment (some may call it system 1), and then rational process only works to justify that or explain the black box, assuming that black box is always correct (depending on the person and how much they trust their black box or system 1). So system 2 is "hallucinating" the best justification for system 1. And usually system 2 will do it only when it's required to justify it for anyone else.
vidarh 2y ago
And for most cases that human explanation is likely with a disturbing frequency a complete fabrication after the fact. See experiments on split brain patients. With respect to repeatability, yes, LLMs are currently frozen in time. That is not an inherent limitation, but it is one that is practical for a lot uses and a problem for some.