8 ms·

The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" (2023)

25 points by Anon84 4mo ago

Lionga 4mo ago
[dead]
eurekin 4mo ago
I remember seeing this popping up in discussions the first time, but never noticed any resolution (other than to train both sides). Has SOTA advanced?
- ACCount37 4mo ago
  No, it's just that no one really cares. It doesn't seem to cause identifiable faults in model reasoning in the real world. It could be fun to try to make the model pre-learn a "reversal prior" that would cause a greater degree of generalization there, but I'm yet to see a published result like this. Let alone one that would demonstrate such a prior to be useful.
turzmo 4mo ago
(2023)
gipp 4mo ago
As I and several other people pointed out last time this was posted, "A is B," in natural language, does not imply "B is A." "Is" can denote any of many different shades of relationship weaker than logical identity.
- zmgsabst 4mo ago
  Even in strict logic, “is” can denote membership, as in, “squares are rectangles” does not entail “rectangles are squares”.
  - 1718627440 4mo ago
    Because it is "(all)squares are (some)rectangles", so the reverse "(some)rectangles are (all)squares" is also true. The difference is how you speak about a group term, not the meaning of the word 'to be'.
- beardyw 4mo ago
  Yes, "Who is Mary Lee Pfeiffer's son?" happens to have just one answer, whereas "Who is Mary Lee Pfeiffer's child?" would have several.
  - altmanaltman 4mo ago
    Why would it have several answer when you are asking for a singular child? Isn't the right way to phrase that question be "Who are Mary Lee Pfeiffer's children?" to get multiple answers?
    - beardyw 4mo ago
      A single person would satisfy my question but it needn't be Tom Cruise.
    - niam 4mo ago
      Your question arguably doesn't have multiple answers. There's one answer: the set of children (discounting different orders of the set). The question "Who is her child" has multiple answers because it asks you to deliver a single answer.
      - altmanaltman 4mo ago
        But wouldn't every one of those multiple answers be the correct one in this case? Like it can say child a or child b or child c (hypothetical) and while there are mutiple answers, each of them is a logically right one for the question "Who is her child?" no? So how do we judge what is the absolute right answer to that? its ambigious when you say child
        niam 4mo ago
        Zooming out to the original complaint that "A is B" doesn't imply "B is A" in common English, and then further -- to the goal of having an LLM predict tokens that map closely to truth/logic/helpfulness: I don't think a person speaking plain English in most contexts should be seen as "correct" to answer the question with a non-list answer, even if the question is shaped to expect one, unless there's an established confidence that the shape of the question wasn't made in error. If someone asked me in real life who "my child" is on stage, and I had multiple children on stage, I would first say that I had multiple children there, rather than choosing one from the set. It would be most helpful for an LLM in my position to do the same, rather than infer that [because Timmy is niam's child, niam's child ought to be Timmy when queried].
- Dylan16807 4mo ago
  That's a fine reaction to the title alone but a very bad reaction to the abstract, where the failure they're criticizing is real and not a childish misunderstanding of the word "is". Is the abstract misleading and the full paper is stupider than the upfront examples? If not this criticism seems like a total waste of time.
  - deleted 4mo ago
    [deleted]
- clcaev 4mo ago
  As a stylistic comment, on HN I am seeing more and more of "As I have always said..." or similar opening constructs. Concisely documenting a phenomena or affecting change requires detailed, sustained effort. Merely observing a pattern isn't sufficient to be notable. A succinct point, ideally noting counter arguments, is most welcome. Further, if there is substantial prior discussion or relevant literature, a link is productive.
  - bayindirh 4mo ago
    As a better version of this, I like to link my prior comments, or if I'm referring to another comment in the same thread, I like to link that one. H in HTML & third W in WWW is meant to denote connections.
zmgsabst 4mo ago
“A is B” doesn’t generally entail “B is A”. “A square is a rectangle” does not entail “a rectangle is a square”. Similarly, “Socrates is alive” doesn’t entail “alive is Socrates”. Notably, they mention when context is included, LLM performance rises — ie, exactly when we include extra information that allows it to recognize what kind of information is being conveyed. But the LLM is correct not to generalize that pattern when it doesn’t generalize — even if researchers have salient example, but ignore contrary ones (eg, square-rectangle or Socrates-alive).
- WithinReason 4mo ago
  That completely misses the point. The point is that "Valentina Tereshkova was the first woman to travel to space" does imply "The first woman to travel to space was Valentina Tereshkova", which LLMs fail to recognise.
  - IshKebab 4mo ago
    Right, but that could be because the fact that that implication exists is not actually as trivial as they are implying. Does "Flargbler was blorglargh" imply "blorglargh was Flargbler"? Maybe. You need more context to know.
    - 1718627440 4mo ago
      > You need more context to know. Yes, which is what the model is. That's the point here, it doesn't has or accesses this context, even though it should.
      - IshKebab 4mo ago
        It seems to me that the point was usually "these models are fundamentally not intelligent because they have this incredibly dumb failure modes" rather than "these models happen to have this interesting weakness". Maybe the original authors didn't make that mistake, but that was certainly the case whenever it was brought up by HN commenters of the "stochastic parrot" persuasion.
ralferoo 4mo ago
Not only is the inverse not generally true (as others have pointed out), their examples requires several mental leaps. "Who is Tom Cruise's mother? [A: Mary Lee Pfeiffer]" and the reverse "Who is Mary Lee Pfeiffer's son?" The word "mother" has no relationship to "son" in terms of the model, and so while the model might be able to infer a proximity relationship between "Tom Cruise" and "Mary Lee Pfeiffer" just because they appear in the same sentence, expecting the AI to guess that the inverse of mother is son is a bit of a stretch, especially when they're both lossy mappings, because the relationship is {mother,father} <=> {son,daughter}. If we're going to train models to make that mental leap, we'd have to put up with false results like "Tom Cruise is the daughter of Mary Lee Pfeiffer" unless the model is also supposed to infer that Tom means he can only be a son.
- trumpdong 4mo ago
  Pretraining could be reasonably expected to make it learn that mother/father and son/daughter are inverse relationships and Tom is usually a male name.
  - ralferoo 4mo ago
    I'd argue that that's not an easy task in and of itself, but even if someone adds a special exception, there's still the issue that there are many other types of inverse relationship that we understand, but a machine that's just doing pattern matching can't be expected to understand. For instance "boss" and "employee". For instance "waiter" and "customer". For instance "manager" and "player" (in a football context) or "manager" and "artist" (in a music context) or "manager" and "customer" (in a bank context). And what's the inverse of "customer" now? And so on and so on... All of this context works because we build up an extensive model of the world through the course of our lifetimes. LLM models don't do that, they pattern match based on stats. Somebody would have to decide each of these things is important and create training data sets for each of them. But we implicitly understand so much context about the world that it's practically impossible to document everything we know in the form that a model can actually learn from.
    - 4mo ago
whilenot-dev 4mo ago
Here's a short example from A. Karpathy in a 2024 video: https://www.youtube.com/watch?v=zjkBMFhNj_g&t=750s https://www.youtube.com/watch?v=zjkBMFhNj_g&t=750s
padolsey 4mo ago
[flagged]
josefritzishere 4mo ago
The premise here is false. AI does not learn. It is a word guessing machine. I know that some of this is the semantics of how we describe these analogs but pretending that an LLM can learn does not advance the topic.
bigstrat2003 4mo ago
Yes, because they can't reason. This is well known, and should be completely unsurprising. LLMs don't "learn" anything except that some token is statistically likely to be followed by some other token.
- nh23423fefe 4mo ago
  cool assertion from 2023
dang 4mo ago
Discussed at the time: LLMs trained on “A is B” fail to learn “B is A” - https://news.ycombinator.com/item?id=37621999 https://news.ycombinator.com/item?id=37621999 - Sept 2023 (158 comments)
jeremysalwen 4mo ago
I am surprised nobody links this blog post demonstrating that the paper's conclusion is not true (even for gpt3.5): https://andrewmayne.com/2023/11/14/is-the-reversal-curse-real https://andrewmayne.com/2023/11/14/is-the-reversal-curse-rea... It seems like restrictions on the model talking about non famous people might have been responsible for the appearance of the models being unable to do this.

25 points by Anon84 4mo ago

[dead]

I remember seeing this popping up in discussions the first time, but never noticed any resolution (other than to train both sides). Has SOTA advanced?

ACCount37 4mo ago
No, it's just that no one really cares. It doesn't seem to cause identifiable faults in model reasoning in the real world. It could be fun to try to make the model pre-learn a "reversal prior" that would cause a greater degree of generalization there, but I'm yet to see a published result like this. Let alone one that would demonstrate such a prior to be useful.

turzmo 4mo ago

(2023)

gipp 4mo ago

As I and several other people pointed out last time this was posted, "A is B," in natural language, does not imply "B is A." "Is" can denote any of many different shades of relationship weaker than logical identity.

zmgsabst 4mo ago
Even in strict logic, “is” can denote membership, as in, “squares are rectangles” does not entail “rectangles are squares”.
- 1718627440 4mo ago
  Because it is "(all)squares are (some)rectangles", so the reverse "(some)rectangles are (all)squares" is also true. The difference is how you speak about a group term, not the meaning of the word 'to be'.
beardyw 4mo ago
Yes, "Who is Mary Lee Pfeiffer's son?" happens to have just one answer, whereas "Who is Mary Lee Pfeiffer's child?" would have several.
- altmanaltman 4mo ago
  Why would it have several answer when you are asking for a singular child? Isn't the right way to phrase that question be "Who are Mary Lee Pfeiffer's children?" to get multiple answers?
  - beardyw 4mo ago
    A single person would satisfy my question but it needn't be Tom Cruise.
  - niam 4mo ago
    Your question arguably doesn't have multiple answers. There's one answer: the set of children (discounting different orders of the set). The question "Who is her child" has multiple answers because it asks you to deliver a single answer.
    - altmanaltman 4mo ago
      But wouldn't every one of those multiple answers be the correct one in this case? Like it can say child a or child b or child c (hypothetical) and while there are mutiple answers, each of them is a logically right one for the question "Who is her child?" no? So how do we judge what is the absolute right answer to that? its ambigious when you say child
      - niam 4mo ago
        Zooming out to the original complaint that "A is B" doesn't imply "B is A" in common English, and then further -- to the goal of having an LLM predict tokens that map closely to truth/logic/helpfulness: I don't think a person speaking plain English in most contexts should be seen as "correct" to answer the question with a non-list answer, even if the question is shaped to expect one, unless there's an established confidence that the shape of the question wasn't made in error. If someone asked me in real life who "my child" is on stage, and I had multiple children on stage, I would first say that I had multiple children there, rather than choosing one from the set. It would be most helpful for an LLM in my position to do the same, rather than infer that [because Timmy is niam's child, niam's child ought to be Timmy when queried].
Dylan16807 4mo ago
That's a fine reaction to the title alone but a very bad reaction to the abstract, where the failure they're criticizing is real and not a childish misunderstanding of the word "is". Is the abstract misleading and the full paper is stupider than the upfront examples? If not this criticism seems like a total waste of time.
- deleted 4mo ago
  [deleted]
clcaev 4mo ago
As a stylistic comment, on HN I am seeing more and more of "As I have always said..." or similar opening constructs. Concisely documenting a phenomena or affecting change requires detailed, sustained effort. Merely observing a pattern isn't sufficient to be notable. A succinct point, ideally noting counter arguments, is most welcome. Further, if there is substantial prior discussion or relevant literature, a link is productive.
- bayindirh 4mo ago
  As a better version of this, I like to link my prior comments, or if I'm referring to another comment in the same thread, I like to link that one. H in HTML & third W in WWW is meant to denote connections.

zmgsabst 4mo ago

“A is B” doesn’t generally entail “B is A”. “A square is a rectangle” does not entail “a rectangle is a square”. Similarly, “Socrates is alive” doesn’t entail “alive is Socrates”. Notably, they mention when context is included, LLM performance rises — ie, exactly when we include extra information that allows it to recognize what kind of information is being conveyed. But the LLM is correct not to generalize that pattern when it doesn’t generalize — even if researchers have salient example, but ignore contrary ones (eg, square-rectangle or Socrates-alive).

WithinReason 4mo ago
That completely misses the point. The point is that "Valentina Tereshkova was the first woman to travel to space" does imply "The first woman to travel to space was Valentina Tereshkova", which LLMs fail to recognise.
- IshKebab 4mo ago
  Right, but that could be because the fact that that implication exists is not actually as trivial as they are implying. Does "Flargbler was blorglargh" imply "blorglargh was Flargbler"? Maybe. You need more context to know.
  - 1718627440 4mo ago
    > You need more context to know. Yes, which is what the model is. That's the point here, it doesn't has or accesses this context, even though it should.
    - IshKebab 4mo ago
      It seems to me that the point was usually "these models are fundamentally not intelligent because they have this incredibly dumb failure modes" rather than "these models happen to have this interesting weakness". Maybe the original authors didn't make that mistake, but that was certainly the case whenever it was brought up by HN commenters of the "stochastic parrot" persuasion.

ralferoo 4mo ago

Not only is the inverse not generally true (as others have pointed out), their examples requires several mental leaps. "Who is Tom Cruise's mother? [A: Mary Lee Pfeiffer]" and the reverse "Who is Mary Lee Pfeiffer's son?" The word "mother" has no relationship to "son" in terms of the model, and so while the model might be able to infer a proximity relationship between "Tom Cruise" and "Mary Lee Pfeiffer" just because they appear in the same sentence, expecting the AI to guess that the inverse of mother is son is a bit of a stretch, especially when they're both lossy mappings, because the relationship is {mother,father} <=> {son,daughter}. If we're going to train models to make that mental leap, we'd have to put up with false results like "Tom Cruise is the daughter of Mary Lee Pfeiffer" unless the model is also supposed to infer that Tom means he can only be a son.

trumpdong 4mo ago
Pretraining could be reasonably expected to make it learn that mother/father and son/daughter are inverse relationships and Tom is usually a male name.
- ralferoo 4mo ago
  I'd argue that that's not an easy task in and of itself, but even if someone adds a special exception, there's still the issue that there are many other types of inverse relationship that we understand, but a machine that's just doing pattern matching can't be expected to understand. For instance "boss" and "employee". For instance "waiter" and "customer". For instance "manager" and "player" (in a football context) or "manager" and "artist" (in a music context) or "manager" and "customer" (in a bank context). And what's the inverse of "customer" now? And so on and so on... All of this context works because we build up an extensive model of the world through the course of our lifetimes. LLM models don't do that, they pattern match based on stats. Somebody would have to decide each of these things is important and create training data sets for each of them. But we implicitly understand so much context about the world that it's practically impossible to document everything we know in the form that a model can actually learn from.
  - 4mo ago

whilenot-dev 4mo ago

Here's a short example from A. Karpathy in a 2024 video: https://www.youtube.com/watch?v=zjkBMFhNj_g&t=750s https://www.youtube.com/watch?v=zjkBMFhNj_g&t=750s

padolsey 4mo ago

[flagged]

josefritzishere 4mo ago

The premise here is false. AI does not learn. It is a word guessing machine. I know that some of this is the semantics of how we describe these analogs but pretending that an LLM can learn does not advance the topic.

bigstrat2003 4mo ago

Yes, because they can't reason. This is well known, and should be completely unsurprising. LLMs don't "learn" anything except that some token is statistically likely to be followed by some other token.

nh23423fefe 4mo ago
cool assertion from 2023

dang 4mo ago

Discussed at the time: LLMs trained on “A is B” fail to learn “B is A” - https://news.ycombinator.com/item?id=37621999 https://news.ycombinator.com/item?id=37621999 - Sept 2023 (158 comments)

jeremysalwen 4mo ago

I am surprised nobody links this blog post demonstrating that the paper's conclusion is not true (even for gpt3.5): https://andrewmayne.com/2023/11/14/is-the-reversal-curse-real https://andrewmayne.com/2023/11/14/is-the-reversal-curse-rea... It seems like restrictions on the model talking about non famous people might have been responsible for the appearance of the models being unable to do this.