6 ms·

GP meant that LLMs trained to learned "A is a B" won't necessarily be able to answer "give an example of B". Eg given the fact "Foo is the capital city of Bar",

GP meant that LLMs trained to learned "A is a B" won't necessarily be able to answer "give an example of B". Eg given the fact "Foo is the capital city of Bar", it might be able to complete "Foo is the capital city of _" but not "The capital city of Bar is _". (Which isn't necessarily a evidence against truth models for out-of-distribution facts, it could just be a matter of indexing.)

xcv123 3y ago
Yes it can only learn to solve the problems that it was trained to solve. Not surprising really. Tautological.
- ngneer 3y ago
  Right, but that ignores the key piece, which is whether such systems can infer the equivalence relation denoted by "is". A is A. A is B implies B is A. A is B and B is C implies A is C. When a system sees that A is B, yet cannot infer that B is A, it exhibits an asymmetry, which is interesting. Whether this asymmetry exists in the underlying language is unclear.
  - xcv123 3y ago
    > When a system sees that A is B, yet cannot infer that B is A So the issue is that they cannot infer that general rule due to a fundamental limitation of the transformer LLM architecture, not just a training data issue? I skimmed the paper and it seems to be the case.
  - dr_kiszonka 3y ago
    > A is B implies B is A. I get the overall idea, but this statement isn't always true, right?
    - xcv123 3y ago
      Correct. Not in natural language. These are natural language systems, not logic systems. "The sky is blue" does not imply "blue is the sky".
      - TeMPOraL 3y ago
        There's an asymmetry here, too. "The sky is ${color}", with no extra context, has one obvious answer for us living here, today. Whereas for "Blue is the color of ${thing}", with no extra answer, there's an insane amount of equally sensible substitutions for ${thing}. Without extra content, the model has no reason to privilege "sky" over any other equally valid answer.
bondarchuk 3y ago
>"Eg given the fact "Foo is the capital city of Bar", it might be able to complete "Foo is the capital city of _" but not "The capital city of Bar is _"." Is that really so? It would seem to be well within what I thought they were capable of based on all the other things they can do correctly.
- lucubratory 3y ago
  The reversal curse itself is real. An example I can remember from the paper is "Who is Tom Cruise's mother? [Tom Cruise's mother's name]" paired with "Who is [Tom Cruise's mother's name]'s son? [incorrect answer or "Can't answer that"]". It is an interesting side effect of the way they think and a significant annoyance in getting them to work as we want them to. That said, I think people make too much of it as an "LLMs can't reason" point, when I don't think that's accurate. What it says is that LLMs instant recall is not logically bidirectional, but this is something that humans do as well. Humans take longer to respond to (and are less accurate answering) "Who is Tom Cruise's mother?" than "Who is [her name]'s son?". At least for me, when I get questions that are the "wrong way around", I have to literally run through it logically in my head, generally along the lines of "(What does that name remind me of, is she a spy? Or her son is a spy? Is she a fictional character? Wait I think this is is a celebrity thing, which spy celebrity has her as his mum? Oh yeah, Tom Cruise.) [Out loud:] Tom Cruise." Also, some people misunderstand the actual deficiency, and think that the LLM can't answer the question at all, rather than just zero shot. The LLM can answer the question if it has the information in context, it can reason "If A=B, then B=A" just fine. It just can't do the less popular halves of AB equivalencies zero shot.
  - alphablended 3y ago
    > An example I can remember from the paper is "Who is Tom Cruise's mother? [Tom Cruise's mother's name]" paired with "Who is [Tom Cruise's mother's name]'s son? [incorrect answer or "Can't answer that"]". The paper is, apparently, still under review. In the mean time, may I suggest you to verify that example by yourself?
    - 3y ago

xcv123 3y ago

Yes it can only learn to solve the problems that it was trained to solve. Not surprising really. Tautological.

ngneer 3y ago
Right, but that ignores the key piece, which is whether such systems can infer the equivalence relation denoted by "is". A is A. A is B implies B is A. A is B and B is C implies A is C. When a system sees that A is B, yet cannot infer that B is A, it exhibits an asymmetry, which is interesting. Whether this asymmetry exists in the underlying language is unclear.
- xcv123 3y ago
  > When a system sees that A is B, yet cannot infer that B is A So the issue is that they cannot infer that general rule due to a fundamental limitation of the transformer LLM architecture, not just a training data issue? I skimmed the paper and it seems to be the case.
- dr_kiszonka 3y ago
  > A is B implies B is A. I get the overall idea, but this statement isn't always true, right?
  - xcv123 3y ago
    Correct. Not in natural language. These are natural language systems, not logic systems. "The sky is blue" does not imply "blue is the sky".
    - TeMPOraL 3y ago
      There's an asymmetry here, too. "The sky is ${color}", with no extra context, has one obvious answer for us living here, today. Whereas for "Blue is the color of ${thing}", with no extra answer, there's an insane amount of equally sensible substitutions for ${thing}. Without extra content, the model has no reason to privilege "sky" over any other equally valid answer.

bondarchuk 3y ago

>"Eg given the fact "Foo is the capital city of Bar", it might be able to complete "Foo is the capital city of _" but not "The capital city of Bar is _"." Is that really so? It would seem to be well within what I thought they were capable of based on all the other things they can do correctly.

lucubratory 3y ago
The reversal curse itself is real. An example I can remember from the paper is "Who is Tom Cruise's mother? [Tom Cruise's mother's name]" paired with "Who is [Tom Cruise's mother's name]'s son? [incorrect answer or "Can't answer that"]". It is an interesting side effect of the way they think and a significant annoyance in getting them to work as we want them to. That said, I think people make too much of it as an "LLMs can't reason" point, when I don't think that's accurate. What it says is that LLMs instant recall is not logically bidirectional, but this is something that humans do as well. Humans take longer to respond to (and are less accurate answering) "Who is Tom Cruise's mother?" than "Who is [her name]'s son?". At least for me, when I get questions that are the "wrong way around", I have to literally run through it logically in my head, generally along the lines of "(What does that name remind me of, is she a spy? Or her son is a spy? Is she a fictional character? Wait I think this is is a celebrity thing, which spy celebrity has her as his mum? Oh yeah, Tom Cruise.) [Out loud:] Tom Cruise." Also, some people misunderstand the actual deficiency, and think that the LLM can't answer the question at all, rather than just zero shot. The LLM can answer the question if it has the information in context, it can reason "If A=B, then B=A" just fine. It just can't do the less popular halves of AB equivalencies zero shot.
- alphablended 3y ago
  > An example I can remember from the paper is "Who is Tom Cruise's mother? [Tom Cruise's mother's name]" paired with "Who is [Tom Cruise's mother's name]'s son? [incorrect answer or "Can't answer that"]". The paper is, apparently, still under review. In the mean time, may I suggest you to verify that example by yourself?
  - 3y ago