4 ms·

>It is absolutely trivial to show Hyp2 is false No it's not > Current LLMs can produce impressive results on a set of linguistic inputs and then fail complete

>It is absolutely trivial to show Hyp2 is false No it's not > Current LLMs can produce impressive results on a set of linguistic inputs and then fail completely on others that make trivial alterations to the same underlying domain. >Indeed: because there're no relevant prior cases to sample from in that case. That's not what that tells us. Humans have weird failure modes that look absurd outside the context of evolutionary biology (some still look absurd) and that don't speak to any lack or presence of intelligence or complex thought. Not sure why it's so hard to grasp that LLMs are bound to have odd failure modes regardless of the above. and trivial here is relative. In my experience, "trivial" often turns out to be trivial in the way a person may not pay close attention to and be similarly tricked. For instance, GPT-4 might solve a classic puzzle correctly then fail the same puzzle subtlety changed. I've found more often than not, simply changing names of variables in the puzzle to something completely different can get it to solve the changed puzzle. It takes memory shortcuts but can be pulled out of that. LLMs have failure modes that look like human failure modes too.

mjburgess 3y ago
The "failure modes" in humans do not show we lack the capacity. Eg., do you have capacity to reason about physics? Well if you're extremely drunk, less so. But not if I permute the name of the object. > I've found more often than not, simply changing names of variables Yes, lol --- why do you think that is? Because in the digitised dataset of "everything ever written" those names correspond to places in that dataset that can be sampled from by the LLM. Showing Hyp1 to be the case. P(Hyp1| ChangeNameMakesDifference) >>>>>> P(Hyp2|ChangeNameMakesDifference) To such a degree that the latter is vanishingly close to zero.
- famouswaffles 3y ago
  >The "failure modes" in humans do not show we lack the capacity. Then they don't in LLMs too >Yes, lol --- why do you think that is? Being able to solve a changed common puzzle but also with different names than it would ever see in training is not an indication of a lack of ability lol. and changing names isn't the only way to get it out of memory, just the easiest/most straightforward. You can converse it out of there too but that doesn't work as often.
  - mjburgess 3y ago
    > Then they don't in LLMs too LLMs don't get drunk . If a child answers questions from a book of answers then they'll appear to understand the domain insofar as those questions appear. They do not. They will fail to answer questions under, eg., permutations of words (say, a question asks about "norepinephrine" but the book only contains "noradrenaline" etc.). Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain. But these are not the kinds of failures experienced with those who have some capacity, eg., for counter-factual reasoning about their environment's physics. In those people it is environmental illusion and cognitive impairment -- not trivial permutations of phrasing which lead to catastrophic loss of apparent understanding. Cognitive impairment = reasoning machine is broken Environmental illusion = data is ambigious and actions cannto resolve it These "failure modes" are expected if you actually have the relevant capacity.
    - famouswaffles 3y ago
      >Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain. alright let me humor you for a bit. Lets start with some solid examples of GPT-4 failing this "trivial linguistic permutation" then ?
      - mjburgess 3y ago
        see, just one reference in the paper: https://arxiv.org/pdf/2302.08399.pdf https://arxiv.org/pdf/2302.08399.pdf
        famouswaffles 3y ago
        they can answer those https://medium.com/@nathanbos/prompting-better-theory-of-mind-performance-from-large-language-models-b43cc6bbbd3 https://medium.com/@nathanbos/prompting-better-theory-of-min...
        3y ago
- lgessler 3y ago
  This is a false dichotomy. It's not the case that models are truly capable of reasoning if and only if they are insensitive to irrelevant perturbations to input. In other words, the mere fact that sensitivity to names sometimes causes significant degradations in model performance doesn't mean that we've observed models are incapable of anything we might call "reasoning"—leaving aside the matter of how we'd define that.
  - mjburgess 3y ago
    I didnt say "if and only if" -- this is a conceptual analysis condition which applies only under deductive analysis. I am using science, ie., abduction, to compare a class of hypotheses. P(CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases) is much much much lower than, P(-CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases)
    - lgessler 3y ago
      Idk, that isn't the sense I got from "It is absolutely trivial to show Hyp2 is false", but sure, I agree with you that this evidence certainly ought to tip the scales one way and not the other.
    - achrono 3y ago
      This might be a naive question, but here me out. Do we really know what the difference is between statistics and the capacity to think? Is "true understanding" rather a continuum of sophistication from a simple adder to Albert Einstein? My point here isn't "if it quacks like a duck...", but more so that while we are talking about intelligent apparatus we should be comparing apples to apples, and not say "this is a mere engine and that is a living brain".
      - deleted 3y ago
        [deleted]
- lostmsu 3y ago
  > less so. But not if I permute the name of the object. You need to realize that you wrote it on a forum where the most known joke is "there are two hard things in programming". That would immediately show you how this assumption is exactly false.
sgt101 3y ago
was with you until: > look absurd outside the context of evolutionary biology for humans, everything (everthing) is within the context of evolutionary biology! > LLMs have failure modes that look like human failure modes too. Yes - because LLM's are trained on 2020 Reddit.
- famouswaffles 3y ago
  >for humans, everything (everthing) is within the context of evolutionary biology! Sure but if some alien species were observing us, some of our actions would look downright odd. Evolutionary biology doesn't necessarily hold the same reference frame for other species, even on earth. Octopi are weird to us. Not so much to other Octopi. >Yes - because LLM's are trained on 2020 Reddit. I wasn't making any comment on why this was the case. Simply that it was. There'll be failure models LLMs adopt from training data, but there's also bound to be failure modes LLMs adopt from the training scheme itself.

mjburgess 3y ago

The "failure modes" in humans do not show we lack the capacity. Eg., do you have capacity to reason about physics? Well if you're extremely drunk, less so. But not if I permute the name of the object. > I've found more often than not, simply changing names of variables Yes, lol --- why do you think that is? Because in the digitised dataset of "everything ever written" those names correspond to places in that dataset that can be sampled from by the LLM. Showing Hyp1 to be the case. P(Hyp1| ChangeNameMakesDifference) >>>>>> P(Hyp2|ChangeNameMakesDifference) To such a degree that the latter is vanishingly close to zero.

famouswaffles 3y ago
>The "failure modes" in humans do not show we lack the capacity. Then they don't in LLMs too >Yes, lol --- why do you think that is? Being able to solve a changed common puzzle but also with different names than it would ever see in training is not an indication of a lack of ability lol. and changing names isn't the only way to get it out of memory, just the easiest/most straightforward. You can converse it out of there too but that doesn't work as often.
- mjburgess 3y ago
  > Then they don't in LLMs too LLMs don't get drunk . If a child answers questions from a book of answers then they'll appear to understand the domain insofar as those questions appear. They do not. They will fail to answer questions under, eg., permutations of words (say, a question asks about "norepinephrine" but the book only contains "noradrenaline" etc.). Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain. But these are not the kinds of failures experienced with those who have some capacity, eg., for counter-factual reasoning about their environment's physics. In those people it is environmental illusion and cognitive impairment -- not trivial permutations of phrasing which lead to catastrophic loss of apparent understanding. Cognitive impairment = reasoning machine is broken Environmental illusion = data is ambigious and actions cannto resolve it These "failure modes" are expected if you actually have the relevant capacity.
  - famouswaffles 3y ago
    >Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain. alright let me humor you for a bit. Lets start with some solid examples of GPT-4 failing this "trivial linguistic permutation" then ?
    - mjburgess 3y ago
      see, just one reference in the paper: https://arxiv.org/pdf/2302.08399.pdf https://arxiv.org/pdf/2302.08399.pdf
      - famouswaffles 3y ago
        they can answer those https://medium.com/@nathanbos/prompting-better-theory-of-mind-performance-from-large-language-models-b43cc6bbbd3 https://medium.com/@nathanbos/prompting-better-theory-of-min...
        3y ago
lgessler 3y ago
This is a false dichotomy. It's not the case that models are truly capable of reasoning if and only if they are insensitive to irrelevant perturbations to input. In other words, the mere fact that sensitivity to names sometimes causes significant degradations in model performance doesn't mean that we've observed models are incapable of anything we might call "reasoning"—leaving aside the matter of how we'd define that.
- mjburgess 3y ago
  I didnt say "if and only if" -- this is a conceptual analysis condition which applies only under deductive analysis. I am using science, ie., abduction, to compare a class of hypotheses. P(CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases) is much much much lower than, P(-CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases)
  - lgessler 3y ago
    Idk, that isn't the sense I got from "It is absolutely trivial to show Hyp2 is false", but sure, I agree with you that this evidence certainly ought to tip the scales one way and not the other.
  - achrono 3y ago
    This might be a naive question, but here me out. Do we really know what the difference is between statistics and the capacity to think? Is "true understanding" rather a continuum of sophistication from a simple adder to Albert Einstein? My point here isn't "if it quacks like a duck...", but more so that while we are talking about intelligent apparatus we should be comparing apples to apples, and not say "this is a mere engine and that is a living brain".
    - deleted 3y ago
      [deleted]
lostmsu 3y ago
> less so. But not if I permute the name of the object. You need to realize that you wrote it on a forum where the most known joke is "there are two hard things in programming". That would immediately show you how this assumption is exactly false.

sgt101 3y ago

was with you until: > look absurd outside the context of evolutionary biology for humans, everything (everthing) is within the context of evolutionary biology! > LLMs have failure modes that look like human failure modes too. Yes - because LLM's are trained on 2020 Reddit.

famouswaffles 3y ago
>for humans, everything (everthing) is within the context of evolutionary biology! Sure but if some alien species were observing us, some of our actions would look downright odd. Evolutionary biology doesn't necessarily hold the same reference frame for other species, even on earth. Octopi are weird to us. Not so much to other Octopi. >Yes - because LLM's are trained on 2020 Reddit. I wasn't making any comment on why this was the case. Simply that it was. There'll be failure models LLMs adopt from training data, but there's also bound to be failure modes LLMs adopt from the training scheme itself.