4 ms·

The "failure modes" in humans do not show we lack the capacity. Eg., do you have capacity to reason about physics? Well if you're extremely drunk, less so. But

The "failure modes" in humans do not show we lack the capacity. Eg., do you have capacity to reason about physics? Well if you're extremely drunk, less so. But not if I permute the name of the object. > I've found more often than not, simply changing names of variables Yes, lol --- why do you think that is? Because in the digitised dataset of "everything ever written" those names correspond to places in that dataset that can be sampled from by the LLM. Showing Hyp1 to be the case. P(Hyp1| ChangeNameMakesDifference) >>>>>> P(Hyp2|ChangeNameMakesDifference) To such a degree that the latter is vanishingly close to zero.

famouswaffles 3y ago
>The "failure modes" in humans do not show we lack the capacity. Then they don't in LLMs too >Yes, lol --- why do you think that is? Being able to solve a changed common puzzle but also with different names than it would ever see in training is not an indication of a lack of ability lol. and changing names isn't the only way to get it out of memory, just the easiest/most straightforward. You can converse it out of there too but that doesn't work as often.
- mjburgess 3y ago
  > Then they don't in LLMs too LLMs don't get drunk . If a child answers questions from a book of answers then they'll appear to understand the domain insofar as those questions appear. They do not. They will fail to answer questions under, eg., permutations of words (say, a question asks about "norepinephrine" but the book only contains "noradrenaline" etc.). Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain. But these are not the kinds of failures experienced with those who have some capacity, eg., for counter-factual reasoning about their environment's physics. In those people it is environmental illusion and cognitive impairment -- not trivial permutations of phrasing which lead to catastrophic loss of apparent understanding. Cognitive impairment = reasoning machine is broken Environmental illusion = data is ambigious and actions cannto resolve it These "failure modes" are expected if you actually have the relevant capacity.
  - famouswaffles 3y ago
    >Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain. alright let me humor you for a bit. Lets start with some solid examples of GPT-4 failing this "trivial linguistic permutation" then ?
    - mjburgess 3y ago
      see, just one reference in the paper: https://arxiv.org/pdf/2302.08399.pdf https://arxiv.org/pdf/2302.08399.pdf
      - famouswaffles 3y ago
        they can answer those https://medium.com/@nathanbos/prompting-better-theory-of-mind-performance-from-large-language-models-b43cc6bbbd3 https://medium.com/@nathanbos/prompting-better-theory-of-min...
        mjburgess 3y ago
        Yes, by changing the words The whole point is that irrelevant word permutation should not "turn on" or "turn off" this capacity. That you can "prompt engineer" your way to the answer shows that the prompt engineer knows the answer and can "use the right search terms" to find it.
        famouswaffles 3y ago
        "But the bag is transparent" is not "irrelevant word permutation" and neither is the additive question that spurs the correct resolution. And it certainly isn't random. a human that isn't paying attention could fail the question too which is kind of the point i'm making. There's no way a model that can't model protein structures does this - https://www.researchgate.net/publication/367453911_Large_language_models_generate_functional_protein_sequences_across_diverse_families https://www.researchgate.net/publication/367453911_Large_lan...
lgessler 3y ago
This is a false dichotomy. It's not the case that models are truly capable of reasoning if and only if they are insensitive to irrelevant perturbations to input. In other words, the mere fact that sensitivity to names sometimes causes significant degradations in model performance doesn't mean that we've observed models are incapable of anything we might call "reasoning"—leaving aside the matter of how we'd define that.
- mjburgess 3y ago
  I didnt say "if and only if" -- this is a conceptual analysis condition which applies only under deductive analysis. I am using science, ie., abduction, to compare a class of hypotheses. P(CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases) is much much much lower than, P(-CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases)
  - lgessler 3y ago
    Idk, that isn't the sense I got from "It is absolutely trivial to show Hyp2 is false", but sure, I agree with you that this evidence certainly ought to tip the scales one way and not the other.
  - achrono 3y ago
    This might be a naive question, but here me out. Do we really know what the difference is between statistics and the capacity to think? Is "true understanding" rather a continuum of sophistication from a simple adder to Albert Einstein? My point here isn't "if it quacks like a duck...", but more so that while we are talking about intelligent apparatus we should be comparing apples to apples, and not say "this is a mere engine and that is a living brain".
    - deleted 3y ago
      [deleted]
lostmsu 3y ago
> less so. But not if I permute the name of the object. You need to realize that you wrote it on a forum where the most known joke is "there are two hard things in programming". That would immediately show you how this assumption is exactly false.

famouswaffles 3y ago

>The "failure modes" in humans do not show we lack the capacity. Then they don't in LLMs too >Yes, lol --- why do you think that is? Being able to solve a changed common puzzle but also with different names than it would ever see in training is not an indication of a lack of ability lol. and changing names isn't the only way to get it out of memory, just the easiest/most straightforward. You can converse it out of there too but that doesn't work as often.

mjburgess 3y ago
> Then they don't in LLMs too LLMs don't get drunk . If a child answers questions from a book of answers then they'll appear to understand the domain insofar as those questions appear. They do not. They will fail to answer questions under, eg., permutations of words (say, a question asks about "norepinephrine" but the book only contains "noradrenaline" etc.). Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain. But these are not the kinds of failures experienced with those who have some capacity, eg., for counter-factual reasoning about their environment's physics. In those people it is environmental illusion and cognitive impairment -- not trivial permutations of phrasing which lead to catastrophic loss of apparent understanding. Cognitive impairment = reasoning machine is broken Environmental illusion = data is ambigious and actions cannto resolve it These "failure modes" are expected if you actually have the relevant capacity.
- famouswaffles 3y ago
  >Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain. alright let me humor you for a bit. Lets start with some solid examples of GPT-4 failing this "trivial linguistic permutation" then ?
  - mjburgess 3y ago
    see, just one reference in the paper: https://arxiv.org/pdf/2302.08399.pdf https://arxiv.org/pdf/2302.08399.pdf
    - famouswaffles 3y ago
      they can answer those https://medium.com/@nathanbos/prompting-better-theory-of-mind-performance-from-large-language-models-b43cc6bbbd3 https://medium.com/@nathanbos/prompting-better-theory-of-min...
      - mjburgess 3y ago
        Yes, by changing the words The whole point is that irrelevant word permutation should not "turn on" or "turn off" this capacity. That you can "prompt engineer" your way to the answer shows that the prompt engineer knows the answer and can "use the right search terms" to find it.
        famouswaffles 3y ago
        "But the bag is transparent" is not "irrelevant word permutation" and neither is the additive question that spurs the correct resolution. And it certainly isn't random. a human that isn't paying attention could fail the question too which is kind of the point i'm making. There's no way a model that can't model protein structures does this - https://www.researchgate.net/publication/367453911_Large_language_models_generate_functional_protein_sequences_across_diverse_families https://www.researchgate.net/publication/367453911_Large_lan...

lgessler 3y ago

This is a false dichotomy. It's not the case that models are truly capable of reasoning if and only if they are insensitive to irrelevant perturbations to input. In other words, the mere fact that sensitivity to names sometimes causes significant degradations in model performance doesn't mean that we've observed models are incapable of anything we might call "reasoning"—leaving aside the matter of how we'd define that.

mjburgess 3y ago
I didnt say "if and only if" -- this is a conceptual analysis condition which applies only under deductive analysis. I am using science, ie., abduction, to compare a class of hypotheses. P(CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases) is much much much lower than, P(-CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases)
- lgessler 3y ago
  Idk, that isn't the sense I got from "It is absolutely trivial to show Hyp2 is false", but sure, I agree with you that this evidence certainly ought to tip the scales one way and not the other.
- achrono 3y ago
  This might be a naive question, but here me out. Do we really know what the difference is between statistics and the capacity to think? Is "true understanding" rather a continuum of sophistication from a simple adder to Albert Einstein? My point here isn't "if it quacks like a duck...", but more so that while we are talking about intelligent apparatus we should be comparing apples to apples, and not say "this is a mere engine and that is a living brain".
  - deleted 3y ago
    [deleted]

lostmsu 3y ago

> less so. But not if I permute the name of the object. You need to realize that you wrote it on a forum where the most known joke is "there are two hard things in programming". That would immediately show you how this assumption is exactly false.