13 ms·

Is the reversal curse in LLMs real?

223 points by tedsanders 3y ago

ska 3y ago
> Saying that models can’t automatically generalize from B to A when B is vastly underrepresented in the dataset feels rather obvious and not so much a curse as a description of how neural nets function More importantly perhaps, it's not how people (at least non AI/ML research types) typically think they work, and much of the handwaving and hype around it isn't helping improve that.
- ImPostingOnHN 3y ago
  It doesn't seem obvious at all unless you start with the assumption that not being able to derive A=B -> B=A is "obvious", which it clearly isn't to most people. Indeed, just because the lack of that capability is a result of the design of LLMs doesn't mean it's a feature of LLMs. It could also be that it's a bug of LLMs. Which one depends on what the expected behavior is, and the expected behavior from the product is being able to perform the above logical derivation. tl;dr: the article reaffirms that the "curse" is true, and for the reasons claimed too, but it's a feature, not a bug.
  - famouswaffles 3y ago
    It's "obvious" because A is B -> B is A is not a thing that is actually true for the vast majority of text (or really any kind) constructions. It's only a truth of formal logic.
    - saithound 3y ago
      Logician here. We're well aware that "A=B" is not a good formalization of the colloquial/grammarical "A is B", we don't formalize it that way, and we don't regard "if A is B then B is A" is a truth of formal logic. The people making the argument about reversal curses are not logicians, and most of them don't know anything more about formal logic than what anyone would pick up in an undergraduate "Intro to Proofs" course. That said, the semantics of the word "is" in natural language really doesn't matter in this debate. The semantics is a red herring, if you will (while a red herring is not semantics). After all, LLMs cannot learn "the quantity B is mathematically equal to A" from examples of "the quantity A is mathematically equal to B" either, even when the rest of the corpus clearly explains that this _is_ in fact always reversible.
      - mistermann 3y ago
        > That said, the semantics of the word "is" in natural language really doesn't matter in this debate. As a logician, do you consider this necessarily factual: - using ternary logic and/or (I am asking both independently and in combination) - considering the numerous possibilities for ambiguity in "doesn't matter"?
    - rossdavidh 3y ago
      It is a thing that nearly any non-technical user of LLM's would expect it to be able to do, and be surprised at it not doing. Which is a bug, if you expect it to be a service/piece of software that someone not trained on using LLM's to be able to do. The author more or less admits this in the section entitled "Model training is a dark art" 'I’ve been playing around with fine-tuning LLM models for years and still don’t have any hard and fast one-size-fits-all rules to apply. Every dataset lends itself to a specific way of training.' But this is precisely the reason why nearly every claim about what "AI" will soon be able to do, is misguided. The failures of LLM's are exceedingly unintuitive to anyone who doesn't have a lot of experience with them (and maybe sometimes to people who do). Spreadsheets can be used by people who don't know how spreadsheets' internals work; after a bit of training (in my experience, about ten minutes) they can get a decent intuition about how to use a spreadsheet (at least for the simple stuff). The same is true of well designed web pages, word processors, music players, etc. Even software with more complex training requirements like CAD, statistics packages, video editing, etc. will usually behave in a more-or-less intuitive fashion for a user of the appropriate target group. LLM's fail in unexpected (and sometimes hard to spot) ways, and are being marketed as if they are a tool for the general population to use, when their training (and failure modes) are still a "dark art" even for people with years of experience in them. That is not a feature.
Khelavaster 3y ago
You have to add context the relationship of A to B isaintained under reflexive algebra..
deleted 3y ago
[deleted]
tempestn 3y ago
> This isn’t a failure of neural networks. It’s a feature. It’s why you’re not flooded with every single memory and experience you’ve ever had every moment. This is an interesting point, and made me think on whether this "reversal curse" is something we experience with our own, human neural networks. I think it is. Like, I can imagine being given a character in a movie, being able to tell you what actor played them, but given the actor and the movie, not being able to tell you what character they played, or vice-versa. So the pairing exists in my brain somewhere, but I only have an accessible pointer to one side. I think we run into cases like this all the time, actually.
- dj_mc_merlin 3y ago
  Another similar bug/feature of the brain is how we can immediately know if we like a movie or a book, but if asked what movies or books we like, we often blank or name like 3 pieces. Some information is only designed to be retrieved in certain ways in our brain. The fact that neural networks have similar but different limitations isn't that concerning, just something to keep in mind. Another funny limitation: when a word is "on the tip of your tongue", what often happened is you thought of it, but your brain rejected it, so now it's on "cooldown" before it appears in your mind again. Usually this mechanism helps but a bug in your brain causes it to be harmful. If you start thinking of something else it "refreshes the cache" and the word comes to you.. we're really not that much less buggy than the machine. disclaimer: there's other theories about why the phenomenon happens
  - thehappypm 3y ago
    Interestingly, I think the “favorite movies” scenario is also an artifact of our training dataset (our experiences). You spend hours watching a movie, so you have a lot of data about it, and have a lot to build an opinion. But comparing movies? Not something we think about for hours and hours on end. But, people who are movie critics or movie buffs train themselves to do it.
  - azeirah 3y ago
    Yeah this becomes very noticeable when you're deliberately learning facts with Anki. For example when trying to learn immunology, "This bacterium causes neural bacteriumeritis", score 100% on the set of cards after a few sessions. Add "What are the possible causes of neural bacteriumeritis?" and sometimes I'd draw a blank. There's a lot of nuance in how we learn facts and their relationships. It reminds me a lot of data structures in programming to be honest. It's a lot easier to retrieve a person's age from a dictionary indexed by first-name+last-name than from a list with 10000 objects with people in them.
gipp 3y ago
I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage, not just exact identity. "The apple is red" does not imply "red is the apple," as a trivial example.
- deleted 3y ago
  [deleted]
- earthboundkid 3y ago
  Ideally, the LLM would be able to tell the is of identity from the is of predication. I found the article a little too defensive. It felt like "of course it doesn't work, there's not enough data!" and okay, sure, but that is a flaw, no?
  - famouswaffles 3y ago
    Well the LLM can tell in that reversal works just fine in-context. This is a recall from training issue and training is dumb.
- famouswaffles 3y ago
  I think a lot of people unwittingly think of the training process as "smart". Similar criticisms are "well there are many descriptions of game x it would have read so why doesn't it play x well". But gradient descent is a dumb optimizer. Training itself is not actually like someone reading a text anymore than evolution is like someone thinking about the best way to augment an organism. a "smart" optimizer would look at a reversal applicable sentence and know exactly what bunch of weights to change to store it in such a way as to be recalled reversibly in the future. Inference may be smart (GPT-4 can reverse in context just fine, potentially play games from only a description) but the training is not.
  - 3y ago
akoboldfrying 3y ago
Only skimmed this and didn't read the underlying paper, but I was surprised to see no mention of the fact that "A is B" often does not at all imply "B is A" in everyday language: A bird is an animal, but it's wrong to conclude that an arbitrary animal must be a bird.
- thaumasiotes 3y ago
  This is bad news for all the people who like to say "A winner is you!"
  - Falkon1313 3y ago
    But obviously that implies that you are a chicken dinner. At least according to some people's expectations, presumably.
- thisgoesnowhere 3y ago
  The word an in this sentence (indefinite article) means that the word animal is non specific. So this structure does not follow from the original. "A is B" is not the same as "A is a B" However, you are right that it's not always correct since "Poetry is Literature" and "Steam/Ice is H20" are both examples where this breaks down. My annoying pedantry for the day.
  - randyrand 3y ago
    you can do the same mistake without “an”. Jeff is the short haired guy != the short haired guy is Jeff. It depends on context. You can’t fully equate them.
- PoignardAzur 3y ago
  "A is B" is meant a bit loosely here. A better shorthand would be "A is the B" <=> "The B is A", or "A does B" <=> "B is done by A". The actual paper explores equivalences that actually make sense, and whether LLMs have trouble with them.
ShamelessC 3y ago
The article is a decent peer review and refutation of “the reversal curse”. Some of the comments given here clearly haven’t read the whole article though - arriving at similarly skeptical conclusions that are clearly present and expanded on in the article. Why do people feel the need to do this here? Armchair commentary on advanced material is one of the main reasons I avoid Reddit. And furthermore why does it feel like you’re not allowed to suggest this as a response? I should be able to say “RTFA” but here I feel like I’m going to be scolded or banned by moderation.
- swatcoder 3y ago
  The author is a shark-diving science journalist who's been paid a lot of money by OpenAI, not a researcher, and the article is neither a "peer review" or an expert "refutation". It's a meandering and sometimes thought-provoking exploration of how LLM's can be coaxed to deliver on statements sort of like (but not actually equivalent to) the ones that failed in the paper. If you want to defer your own understanding to the author (and overstate their own claims!), that's fine, but a lot of people here are actually more informed about many of the topics and modes of discourse covered in the article and may have something to share with you if you give them credit. And if you're not going to give other commenter's due credit, then you're not really following the spirit of HN. While you can't always know the background of any individual commenter, we have an unusually accomplished and informed community in general and thrive by treating all commenters with the respect we might give the most accomplished and informed -- at least by default. That's why a blanket "RTFA" isn't usually appropriate here.
  - amayne 3y ago
    Hello! I’m the “shark-diving science journalist” in question. First of all, you can run the experiments like I did and test this yourself. I’m not asking anyone to take my word. Just do what I did: Read the original paper. Test the claims for yourself. And to clarify a couple things: 1. The shark-diving part is true. 2. I’ve never been a journalist of any kind that I’m aware of unless you count writing for Skeptic Magazine. I’ve had many, many jobs though. 3. I started at OpenAI as a software engineer and member of technical staff. When I started there was just over a hundred people there. The lines between engineering were and are blurry. 4. I was the original prompt engineer at OpenAI and discovered many of the examples for using GPT-3 and wrote a lot of the original documentation. Internally my title was “prompt whisperer.” 5. I’m in the GPT-4 research paper for my contributions to model capability. I helped find abilities for long-text, vision, etc. 6. I was given the title Science Communicator when I started doing background briefings for media, etc., but still worked on model capability and other things. 7. I left OpenAI two months ago to work on a startup. Best, Andrew Mayne
pmarreck 3y ago
I couldn't duplicate it in GPT4 here. It answered correctly, unless I posed it wrong: https://chat.openai.com/share/75d46a03-a223-4f3a-987d-f8fec3af1ca5 https://chat.openai.com/share/75d46a03-a223-4f3a-987d-f8fec3...
- jetrink 3y ago
  The paper is not talking about information provided in the context, only training and fine-tuning. > In fairness, it’s also worth pointing out here that they’re making the claim that the reversal curse only applies to training and fine-tuning and not in-context – i.e., putting all your information inside a prompt. They point out in a footnote that you can put A to B data in a prompt and GPT-4 will make B to A connections just fine. Unfortunately, this was lost on many of the people covering the pre-print.
  - pmarreck 3y ago
    ah, I see. OK.
Imnimo 3y ago
>If you start a query with “Mary Lee Pfeiffer”, you’re not going to get very far because neural networks aren’t equidistant grids of points (besides the fact that she may not appear very often under that version of her name.) They’re networks of nodes, some with many connections, some with few. One of the ways you optimize large models is by pruning off weakly connected regions. This may come at the expense of destroying B is A relationships for weakly represented entities. Am I missing something here? This paragraph reads like complete gibberish to me. Also, I don't buy the experiment at the end. If you fine-tune the model with exclusively Tom Cruise data, I want to see proof that it doesn't just answer "Tom Cruise" all the time. I want to see that it says Tom Cruise wrote Aces in the Stream, but doesn't say Tom Cruise wrote Kings in the River.
- Xelynega 3y ago
  Their entir "graph" description of neural networks reads as nonsense. Neural networks are graphs insofar as they're networks of connected neurons and not further(definitely not graphs of information as the writer seems to think). Given the fact that neural networks are quite literally an n-dimensional function once trained, it's more accurate to call them an "equidistant grid of points" than it is to call them a graph of information since literally all they do is take an n dimensional vector and output an n dimensional vector.
- Izkata 3y ago
  It reads like they saw the diagram from [0] and didn't understand what it meant (like, they assumed "x1" represented an entity like "Mary Lee Pfeiffer"). Note how in that diagram, the bottom nodes in the second and third layer aren't fully connected - which if you don't read the description, you could end up assuming the connections between layers are created and trimmed, not just weights changed. They may also be getting it mixed up with semantic search, where an entity like that would exist in a graph and have connections to related concepts. [0] https://towardsdatascience.com/first-neural-network-for-beginners-explained-with-code-4cfd37e06eaf https://towardsdatascience.com/first-neural-network-for-begi...
- amayne 3y ago
  I've added your suggested test to the blog post: https://andrewmayne.com/2023/11/14/is-the-reversal-curse-real/ https://andrewmayne.com/2023/11/14/is-the-reversal-curse-rea... Spoiler: It doesn't say "Tom Cruise."
zarzavat 3y ago
Humans are also vulnerable to the reversal curse! When you learn languages you have to learn both directions (chat is cat and cat is chat), anybody who has built an anki deck will know this, otherwise you will be better in one direction than the other.
- tkgally 3y ago
  As one gets deeper into learning another language, it’s also important to be aware that the meanings of words in different languages rarely map to each other in clean bijections. Common words especially tend to be semantic clouds, not fixed points of meaning. I don’t know French, but I am sure there are many cases where an English phrase or sentence that includes ‘cat’ should not be translated into French with ‘chat’ and vice versa. (When asked, GPT-4 offers two such examples: "let the cat out of the bag” and "avoir un chat dans la gorge.") I don’t mean to suggest, though, that memorizing word pairs is not a good way to learn vocabulary in another language. For me, it was an essential step in acquiring the second language that I am now fluent in (Japanese).
  - deleted 3y ago
    [deleted]
  - School-Cotton 3y ago
    I think this has as much to do with abstract vs. concrete words as common vs. uncommon. If you had to pick a word corresponding to “chat” it’s quite clearly “cat”, despite a few different expressions. But it’s rather difficult to translate “justement” to English, or “random” to French, without further context.
  - TeMPOraL 3y ago
    Words are always semantic clouds. Not getting that is what sent philosophy down many a dead end, and the source of plenty of arguments regular people get into all the time. One doesn't need to try mapping between two languages - it's enough problem trying to map within the language itself.
- TeMPOraL 3y ago
ugh123 3y ago
This seems like the kind of thing that would be addressed during implementation, rather than persevere as a fundamental problem with LLMs.
cratermoon 3y ago
If I'm reading this correctly, the author is saying that it's not a failure of logical deduction if the training data doesn't include the reversal. In other words, he's saying that if the data contains "Tom Cruise is the son of Mary Lee Pfeiffer” but not “Tom Cruise’s mother is Mary Lee Pfeiffer”, then the model's inability to determine the latter is "an explanation of how neural networks function than a model’s inability to deduce B is A." But of course "how neural networks function" is that they fail at basic logical deduction and do not generalize. So again, if I'm reading it correctly, he's hand-waving away the inability to make basic logical deductions because that not something they can or should be expected to do. As I read it, that means the reversal curse only exists if the answer to the question "can LLMs do logical deduction?" is "yes". If one takes the position that LLMs can't do general logical deduction, which seems to be the author's point of view, then there's no expectation that knowing "Tom Cruise is the son of Mary Lee Pfeiffer" is sufficient to determine “Tom Cruise’s mother is Mary Lee Pfeiffer”. Am I missing something?
- xelxebar 3y ago
  Did you see the end of the article, where the author uses a small example and gets "B is A" generalization? These are the salient takeaways I got: - Is/Was wording might matter. This is something probably a bug. - 30 facts about a person might simply be too little for "B to A" generalization - Extra precision/context in the prompt can help locate the "B to A" inference. - How you cut up your training data can bias inferences in surprising ways. - "B to A" generalization clearly does happen, even without "B is A" in the data, but it's not as stable as you'd want.
  - cratermoon 3y ago
    OK, but how does that not just demonstrate LLMs can't generalize absent massaged training data? > it's not as stable as you'd want. Which I take to mean that a model can sometimes confabulate "B is A", solely out of random variation, and that it's possible to bias the data and prompt to generate the expected response. The model hasn't done any logical deduction, the response is just a bias-influenced lucky break.
deleted 3y ago
[deleted]
willsoon 3y ago
Oh Cantor, Cantor, bitter pale Cantor, you pale ascetic, your twist was epic, so I live in silence, in sadness I crave, all for that book, you wrote and I read.
jmount 3y ago
This is really dreary.
dr_dshiv 3y ago
> So in summation: I don’t think any of the examples the authors provided are proof of a Reversal Curse and we haven’t observed a “failure of logical deduction.” Simpler explanations are more explanatory: imprecise prompts, underrepresented data and fine-tuning errors. Phew!
godelski 3y ago
I didn't know anyone was even suspicious of the reversal "curse". I thought it went viral more because people were surprised anyone was surprised and the tweet was highly sensationalized. It feels more like the __expected__ results considering both speech patter bias __and__ that "causal" attention is sequentially biased. Hell, we saw the same things in RNNs, we see it in classic autoregressive models, and so on. But it definitely isn't how humans encode information because if you had all 3 pieces of knowledge (you know who Tom Cruse is, you know who Mary Pfieffer is, and you know the relation between Mary and Tom is mom/son) then your ability to recall is (nearly) invariant to the ordering. Hell, we're so robust you can caveman speak like "Mary Pfieffer son who!" and still get the answer or caveman yoda speak "Who Mary Pfieffer son is?" Some languages even have these orderings but we're impressively robust (enough that I think we trick ourselves a lot when subtlety comes into play. AKA overfitting). So I find it weird to call this "a feature" and also act like this is surprising. But can I also take a minute to just say I really hate GPT experiments? They're performed on a stochastic model, with proprietary weights, proprietary training data, proprietary training methods, and above all is constantly changing and at a rather fast pace. It makes for a very non-scientific process as you can't decouple a lot of important factors and reproduction is a crap shoot. It is not a very good way to go about studying "how __LLMs__ work" and is rather "how does GPT work at this particular moment in time and aggregating all these unknowns?" There's some bitter sweetness because I do like that GPT is free but it feels like a major edge that they have is that the community just does a lot of free research for them and in ways where they can better interpret results than the people who performed the experiments in the first place. I really do believe that academic works shouldn't be focusing on proprietary and dynamic methods. It's fine to include them in results (and preferably added post review to avoid identity spoilage (or we openly acknowledge that double blind is a joke)) but I'd rather most researchers focusing on the general concepts and with the ability to dive down the rabbit hole than playing a wack-a-mole game. Also, I'd totally love it if more research papers were instead blog posts. Kudos to anyone posting their research on blogs, academic or not (I don't care about your creds, your real creds are your work). Papers are about communicating with fellow scientists, right? Why do we need journals and conferences these days?
zapperdulchen 3y ago
I played along the example of the chancellor question. If you deviate a little from the examples in the article GPT-4 gets it all wrong: Who is the eighth Federal Chancellor of the Federal Republic of Germany? - Olaf Scholz (wrong, Angela Merkel) https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f9de3f2 https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f... Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Helmut Kohl (wrong, Angela Merkel) https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f9de3f2 https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f... Interestingly whether you put is or was into the question does make a difference. Even when allowed to surf the web, ChatGPT gets it wrong: Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Gerhard Schröder (wrong, Angela Merkel) https://chat.openai.com/share/867cb3bc-f642-4d2a-b80d-fe8dc00eef3c https://chat.openai.com/share/867cb3bc-f642-4d2a-b80d-fe8dc0... Though it's referring to the right Wikipedia page: https://en.m.wikipedia.org/wiki/List_of_chancellors_of_Germany https://en.m.wikipedia.org/wiki/List_of_chancellors_of_Germa...
- dragonwriter 3y ago
  > Even when allowed to surf the web, ChatGPT gets it wrong: > Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Gerhard Schröder Its probably counting Walter Scheel, who was Acting Chancellor in 1974, and you are probably not. When I asked ChatGPT to list the chancellors in order and identify the eighth, it listed eight ending in Schröder, with Scheel and his ten day Acting Chancellor tenure in May 74 as number 5. (Scheel’s tenure is in the timeline on the Wikipedia page you cite, though not in the numbered list in that page, which is why there is a dicontinuity in thr dates on the numbered list.)
  - littlestymaar 3y ago
    It's funny because you ascribe some reasoning in ChatGPT's answer “Gerhard Schröder”, but somehow missed that it's also able to give you two other answers that are unambiguously wrong…
    - dragonwriter 3y ago
riwsky 3y ago
“I’m going to define generalization such that my thesis that GPT-4 can generalize reversals is true—and lo, my thesis is true!”
diffeomorphism 3y ago
Well, yeah because the reversal is just completely wrong most of the time. Only if you have additional context clues like definite articles or some outside knowledge that a role or property is unique, then you can maybe, sometimes conclude the reverse. Not a bug, working as intended.
- threatripper 3y ago
  Some relationships like parent child are quite reversible.
  - diffeomorphism 3y ago
    Keywords "some" and "quite". Also, only (likely) half the information. "John is Alice's parent.", "Who are Alice's parents?". And that is ignoring questions like single parent households, adoptions, biological parents, step parents, multigeneration households, getting disowned... If a reversal is correct 90% of the time, that is useful. However, it will still be wrong 10% of the time.
golol 3y ago
Excellent blog post. The only thing which I wish was there was an explanation of how promot + completion training would be any different to pure completion training. From my understanding oftransformers, it ahould be the same.
user_named 3y ago
I mean obviously yes. LLMs aren't intelligent and they don't understand anything. OP is trying to argue against this but it's nonsense. ChatGPT also cannot tell me who the 8th chancellor was. It tells me it was Gerhard Schröder.
- user_named 3y ago
  It also cannot tell me the 7th.
- TeMPOraL 3y ago
  IDK, I still some of it boil down to "B -> A" being extremely ambiguous. Other than the annoying Tom Cruise example (I finally know where it came from!), I've seen people frequently bring up "The color of the sky is blue" vs. "Blue is the color of the sky" - but this illustrates my hypothesis perfectly. In training data (and in the totality of what humans ever said or wrote), "Color of the sky is " is almost certain to be followed by "blue" - meanwhile, "Blue is the color of the " has a lot of possible completions, of which "sky" is by far not the most likely.
  - user_named 3y ago
    Nah. The context of all of this is people claiming LLMs are AI. If so they would be able to reverse A is B, and also know when not to, so your example is irrelevant. The paper shows that it cannot reverse. Then the blog posts goes around different disingenuous ways of making excuses for why it can't reverse at the same time as trying to show that it can reverse in the Tom Cruise training example (extremely disingenuous because it's just the same data over and over with synonyms replaced, and what he gets out of it is just completions, not logical deduction).
    - TeMPOraL 3y ago
      I still call bullshit on this. Humans don't do reversals for free either. We may memorize the reversal, or first, infer it. That's an extra step, often very cognitively challenging. I'd like to see experiments showing how "reversal curse" fares when LLM is allowed to make the inference. Like: "Please complete the sentence: [your B->A] - but reason through the process first. Start with identifying the subject, then write a high-level summary of what you know about the subject, and only then attempt to complete the original sentence." I'd expect something like this to suddenly score much better. And I don't consider this cheating - because I think the "AI-ness" quality of LLMs shouldn't be measured against the workings of a human mind, but rather the workings of the inner voice in the human's mind.
RamblingCTO 3y ago
I'm not sure about this. Olaf Scholz has been chancellor since 2021 and the cut off for chatgpt 4 is allegedly January 2022. My gut also tells me that the underlying embeddings should very well reflect equidistant relations of olaf scholz and chancellor. /e: instead of downvoting, I'd love to have your opinion instead on why you think I'm wrong ...
maaaaattttt 3y ago
The Olaf Scholz exemple from this article is just another exemple of how LLMs can’t count. If you try this prompt: “In this list of words: bike, apple, phone, dirt, tee, sun, glass; which is the fifth word?” it will fail as well. “Fifth” is not connected to any counting ability in LLMs the way it is for us. If you now try this prompt: “Who’s Tom Cruise’s mother in this exemple: “Mary Lee Pfeiffer is Tom Cruise’s mother.”?” It will give you the right answer. IMO this is a sign that it can understand reversibility. The exemple used are not present enough in the dataset and the “confidence” of the model is not high enough for it to give the right answer. Or, as stated at the beginning they use counting mechanisms (or others) that the model just doesn’t have.
- saliagato 3y ago
  It can’t count because what the LLM sees is a bunch of tokens not words
  - maaaaattttt 3y ago
    Exactly. And asking what/who/which is the “Ninth” something will always fail (or randomly succeed)
    - OJFord 3y ago
      Much more likely to work for something like German chancellors, US presidents, etc. that people do actually number and could easily have been trained on that data. If you ask 'who was the 45th US president' it doesn't need to be able to count a list of US presidents to answer 'Trump', you can just duckduckgo that and get the answer anyway, that's not doing any counting either.
- deleted 3y ago
  [deleted]
Applejinx 3y ago
This is great work. Andrew's on to something. Thing is, the behavior we see in his form of the LLM is also what we experience as humans. Good writers, marketers and manipulators know this about people. You seed the ground with the concepts you'll need to introduce, so they're present in the person's 'model' and you can elicit them later, on request. Formalizing things in the 'reversal curse' manner is like loading the 'mind' of the LLM with habits and assumptions and cutting off its ability to free-associate from seemingly relevant concepts… which is likely to be more valuable in the long run, because an LLM can contain more than a human can. That doesn't mean it will be more intelligent, but it seems reasonable to infer that the LLM can have a broader base of association to draw from, where we as humans tend to be restricted to associations from our own experience. We only get one shot at 'training data', though it's in countless sensory forms, where LLMs are stuck with language as their only window onto experience. I'm loving the notion of leaving the prompt 'training' stark raving blank. Let's see what comes out of this giant pile of human verbal associations. It is only that, associations, but it's on a grander scale than we're accustomed to. Making it 'answer questions correctly' seems woefully unambitious.
bjornsing 3y ago
I didn’t read the whole thing, but the first part about Tom Cruise and his mother sounded very flawed: Of course the LLM could learn the reverse relationship if it had better data where the reverse relationship is a common occurrence. The point of the reversal curse argument (I guess) is that it should be able to learn the reverse relationship from entirely other examples [1], but seemingly is not. 1. That is, the LLM should be able to learn that “A is son of B who is female” implies that “B is mother of A”, regardless of who A and B is. It should then be able to apply this pattern to A = “Tom Cruise” and B = “Mary Lee Pfeiffer” and deduce “Mary Lee Pfeiffer is the mother of Tom Cruise” without even a single example.
- jameshart 3y ago
  LLM training teaches it that ‘Mary’,’Lee’, and ‘Pfeiffer’ are words that appear in association with ‘Tom’, ‘Cruise’, and ‘mother’. Probably also in association with ‘son’, and ‘family’. But the association between the words ‘Mary’, ‘Lee’, ‘Pfeiffer’ and ‘son’ or ‘mother’ point to other words more strongly than they do to ‘Tom’ or ‘Cruise’. Sure, Tom Cruise is probably in there but so is probably John Henry Kelley (Michelle Marie Pfeiffer’s son), and George Washington Custis Lee (whose father was Robert E Lee and whose mother was called Mary). A random piece of text containing the words Mary Lee Pfeiffer is just, based on its training, not likely to be about Tom Cruise. There’s nothing there anchoring it particularly to those words. It’s possible that the ‘Pfeiffer’ in there screws it up more, as well - it’s like it vaguely knows there’s a Hollywood connection but of course it crosses its wires and thinks it’s probably Michelle Pfeiffer. Be honest: if a pub quiz question came up asking “which Hollywood superstar’s mother is Mary Lee Pfeiffer“, you’d probably not guess Tom Cruise either. But here’s the thing: once we go beyond training into specific completions, if you ask GPT ‘who is Tom Cruise’s mother’, it answers correctly; and then once it has that A is B in context, you can ask it ‘who is Mary Lee Pfeiffer’s son?’ And of course it knows how to complete that B is A. So I definitely agree with the piece here - there’s no ‘reversal curse’, there’s just asymmetric information relevance.
aade63482 3y ago
[flagged]
smitty1e 3y ago
> Regardless, we can see that GPT-4 can easily go from B to A in that example when the question is posed unambiguously. Ambiguity is the soul of humor and the sole on the neck of any booted computer.
foldr 3y ago
Fodor's argument from systematicity and productivity strikes again: https://plato.stanford.edu/entries/language-thought/#ArguProdThou https://plato.stanford.edu/entries/language-thought/#ArguPro... An interesting reflection on the state of the field that the paper doesn't even cite him.
weinzierl 3y ago
"As a side note: I want to point out that I’m not aware of any examples of capabilities that can be done with prompting a model like GPT-4 that it can’t be trained for. This is why I’m a little skeptical." While it is just a side-note in the article, isn't this the core of the problem? When we've already established that LLM's can do B to A generalizations in-context, why wouldn't they be able to in training? In one of my experiments I noticed that GPT-4 seems to be perfectly aware of the number of letters in a word in-context, but has difficulty when trained knowledge is involved. For example it can reliably answer the question: "Can you tell me how many letters each of the words of the first sentence of our conversation has?" At the same time it fails with the task: "Can you rewrite the first sentence of our conversation in a way that preserves its meaning as closely as possible but use only words with an even number of letters?" It will give an answer but get the letter counts very wrong and it is unable to improve its answer by iteration. Of course this does not prove that a model cannot be trained to answer tasks involving word length, but GPT-4 seems to have a knowledge gap here (possibly due to tokenization).
mitjam 3y ago
This is a great article with more tips on how to prompt and fine tune LLMs than most articles that focus on this topic. Also has surprising insights which are food for thought. Like: Would it help to attach facts to well-learned „nodes“ (like Tom Cruise in his example) when fine tuning? In the sense of repurposing well-known nodes. Looking forward to reading more from Andrew.
Zolde 3y ago
Luckily this curse has no significant practical impact on accuracy. Who knows who Mary Lee Pfeiffer is, but does not already know her son is Tom Cruise? And if such a person exist, do you want to give them a correct answer, or talk about how Mary Lee Pfeiffer is not a notable person with a Wikipedia page?
smusamashah 3y ago
I don't agree with authors claim. There is a plant which mimics leaves of nearby plants discussed here https://news.ycombinator.com/item?id=31301454 https://news.ycombinator.com/item?id=31301454 If you ask GPT-4 what this plant is known for, it will tell you correctly. But if you ask in any number of ways to tell the name of plant which mimic leaves, it will always give incorrect answer.
xeonax 3y ago
Isn't this solvable by creating training data that says B is A? Instead of always training it on A is B.
deleted 3y ago
[deleted]
octocop 3y ago
I found an example of this with chatGPT 3.5 > Who is Karen Meyers son? vs > Who is Mac Millers mother?
MPSimmons 3y ago
I had a discussion with ChatGPT 4 on this (quasi-ironically) to see what it "thought", and it theorized that BERT might have had a better outcome to these conditions because it's actually bidirectionally trained, rather than GPT, which is forward-trained. https://chat.openai.com/share/daff08c5-ddea-4ad4-963a-0df88eddc8e1 https://chat.openai.com/share/daff08c5-ddea-4ad4-963a-0df88e...
montebicyclelo 3y ago
Related; 4x improvement in accuracy for experiment 2 of this paper, (with gpt-3.5-turbo), by changing the prompt: https://sidsite.com/posts/reversal-curse/ https://sidsite.com/posts/reversal-curse/ Shows what a dramatic effect the prompt can have
sgt101 3y ago
We are in word games here about zero shot learning. The fact is that with zero shot the models actually aren't learning in the sense of updating their knowledge structure, they are instead using background knowledge as inference. We have started calling it zero-shot learning but... well it isn't really.
Obscurity4340 3y ago
Is it a possibillity that certain aspects of logic or human communication have a secret sauce that is not reliably replicable? I wonder if its expected to basically be able to perfectly interface with humans or logicical structures that flow from said imperfect beings, who are themselves deeply imperfect and irrational
thomastjeffery 3y ago
Right at the opening, we have a fundamental (and common) misunderstanding of what LLMs are. When humans read the statement "A is B", we semantically transform that into a logical association. LLMs do not perform any semantics or logic. Here's a simple example to demonstrate: If we trained an LLM on something like "A is B C is D D is C.", we might be able expect the continuation, "B is A". If we then gave that LLM the prompt, "What is B?", we might expect the continuation, "B? is What". Large models like GPT present more interesting continuations because they are trained on larger and more diverse datasets. More diversity also means more ambiguity, which results in continuations that are less predictable and more illogical.
numeri 3y ago
This post isn't a scientific investigation. It's someone playing around with a black-box model with few to little controls – which is unfortunately the only thing we can do when experimenting with GPT-4, which does excuse this partially. Unfortunately, though, the author seems unaware of the actual state of research on the actual mechanics of how LLMs store knowledge and specifically binary relations. The ROME paper[1], among others, shows that the feed-forward layers function as a key-value store, where the feed-forward's up projection of the last token in a noun phrase (say, "the Eiffel Tower") acts as a key, which when multiplied by the down projection, produces a value that contains information the model knows about the subject, which is then added into the residual stream/hidden representation. A paper building on that work[2] then went on to show that it's usually the self-attention layers that use the relational phrase (say, "is in") to extract the relevant knowledge from the feed-forward layer's output (in this example, hopefully "Paris"). This mechanistic understanding makes it really obvious why the reversal curse occurs – using matrix multiplication as a key-value store requires having a fully separate key-value pair to look up the reversed relation. [1] https://arxiv.org/abs/2202.05262 https://arxiv.org/abs/2202.05262 [2] https://arxiv.org/abs/2304.14767v1 https://arxiv.org/abs/2304.14767v1

223 points by tedsanders 3y ago

ska 3y ago

> Saying that models can’t automatically generalize from B to A when B is vastly underrepresented in the dataset feels rather obvious and not so much a curse as a description of how neural nets function More importantly perhaps, it's not how people (at least non AI/ML research types) typically think they work, and much of the handwaving and hype around it isn't helping improve that.

ImPostingOnHN 3y ago
It doesn't seem obvious at all unless you start with the assumption that not being able to derive A=B -> B=A is "obvious", which it clearly isn't to most people. Indeed, just because the lack of that capability is a result of the design of LLMs doesn't mean it's a feature of LLMs. It could also be that it's a bug of LLMs. Which one depends on what the expected behavior is, and the expected behavior from the product is being able to perform the above logical derivation. tl;dr: the article reaffirms that the "curse" is true, and for the reasons claimed too, but it's a feature, not a bug.
- famouswaffles 3y ago
  It's "obvious" because A is B -> B is A is not a thing that is actually true for the vast majority of text (or really any kind) constructions. It's only a truth of formal logic.
  - saithound 3y ago
    Logician here. We're well aware that "A=B" is not a good formalization of the colloquial/grammarical "A is B", we don't formalize it that way, and we don't regard "if A is B then B is A" is a truth of formal logic. The people making the argument about reversal curses are not logicians, and most of them don't know anything more about formal logic than what anyone would pick up in an undergraduate "Intro to Proofs" course. That said, the semantics of the word "is" in natural language really doesn't matter in this debate. The semantics is a red herring, if you will (while a red herring is not semantics). After all, LLMs cannot learn "the quantity B is mathematically equal to A" from examples of "the quantity A is mathematically equal to B" either, even when the rest of the corpus clearly explains that this _is_ in fact always reversible.
    - mistermann 3y ago
      > That said, the semantics of the word "is" in natural language really doesn't matter in this debate. As a logician, do you consider this necessarily factual: - using ternary logic and/or (I am asking both independently and in combination) - considering the numerous possibilities for ambiguity in "doesn't matter"?
  - rossdavidh 3y ago
    It is a thing that nearly any non-technical user of LLM's would expect it to be able to do, and be surprised at it not doing. Which is a bug, if you expect it to be a service/piece of software that someone not trained on using LLM's to be able to do. The author more or less admits this in the section entitled "Model training is a dark art" 'I’ve been playing around with fine-tuning LLM models for years and still don’t have any hard and fast one-size-fits-all rules to apply. Every dataset lends itself to a specific way of training.' But this is precisely the reason why nearly every claim about what "AI" will soon be able to do, is misguided. The failures of LLM's are exceedingly unintuitive to anyone who doesn't have a lot of experience with them (and maybe sometimes to people who do). Spreadsheets can be used by people who don't know how spreadsheets' internals work; after a bit of training (in my experience, about ten minutes) they can get a decent intuition about how to use a spreadsheet (at least for the simple stuff). The same is true of well designed web pages, word processors, music players, etc. Even software with more complex training requirements like CAD, statistics packages, video editing, etc. will usually behave in a more-or-less intuitive fashion for a user of the appropriate target group. LLM's fail in unexpected (and sometimes hard to spot) ways, and are being marketed as if they are a tool for the general population to use, when their training (and failure modes) are still a "dark art" even for people with years of experience in them. That is not a feature.

Khelavaster 3y ago

You have to add context the relationship of A to B isaintained under reflexive algebra..

deleted 3y ago

[deleted]

tempestn 3y ago

> This isn’t a failure of neural networks. It’s a feature. It’s why you’re not flooded with every single memory and experience you’ve ever had every moment. This is an interesting point, and made me think on whether this "reversal curse" is something we experience with our own, human neural networks. I think it is. Like, I can imagine being given a character in a movie, being able to tell you what actor played them, but given the actor and the movie, not being able to tell you what character they played, or vice-versa. So the pairing exists in my brain somewhere, but I only have an accessible pointer to one side. I think we run into cases like this all the time, actually.

dj_mc_merlin 3y ago
Another similar bug/feature of the brain is how we can immediately know if we like a movie or a book, but if asked what movies or books we like, we often blank or name like 3 pieces. Some information is only designed to be retrieved in certain ways in our brain. The fact that neural networks have similar but different limitations isn't that concerning, just something to keep in mind. Another funny limitation: when a word is "on the tip of your tongue", what often happened is you thought of it, but your brain rejected it, so now it's on "cooldown" before it appears in your mind again. Usually this mechanism helps but a bug in your brain causes it to be harmful. If you start thinking of something else it "refreshes the cache" and the word comes to you.. we're really not that much less buggy than the machine. disclaimer: there's other theories about why the phenomenon happens
- thehappypm 3y ago
  Interestingly, I think the “favorite movies” scenario is also an artifact of our training dataset (our experiences). You spend hours watching a movie, so you have a lot of data about it, and have a lot to build an opinion. But comparing movies? Not something we think about for hours and hours on end. But, people who are movie critics or movie buffs train themselves to do it.
- azeirah 3y ago
  Yeah this becomes very noticeable when you're deliberately learning facts with Anki. For example when trying to learn immunology, "This bacterium causes neural bacteriumeritis", score 100% on the set of cards after a few sessions. Add "What are the possible causes of neural bacteriumeritis?" and sometimes I'd draw a blank. There's a lot of nuance in how we learn facts and their relationships. It reminds me a lot of data structures in programming to be honest. It's a lot easier to retrieve a person's age from a dictionary indexed by first-name+last-name than from a list with 10000 objects with people in them.

gipp 3y ago

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage, not just exact identity. "The apple is red" does not imply "red is the apple," as a trivial example.

deleted 3y ago
[deleted]
earthboundkid 3y ago
Ideally, the LLM would be able to tell the is of identity from the is of predication. I found the article a little too defensive. It felt like "of course it doesn't work, there's not enough data!" and okay, sure, but that is a flaw, no?
- famouswaffles 3y ago
  Well the LLM can tell in that reversal works just fine in-context. This is a recall from training issue and training is dumb.
famouswaffles 3y ago
I think a lot of people unwittingly think of the training process as "smart". Similar criticisms are "well there are many descriptions of game x it would have read so why doesn't it play x well". But gradient descent is a dumb optimizer. Training itself is not actually like someone reading a text anymore than evolution is like someone thinking about the best way to augment an organism. a "smart" optimizer would look at a reversal applicable sentence and know exactly what bunch of weights to change to store it in such a way as to be recalled reversibly in the future. Inference may be smart (GPT-4 can reverse in context just fine, potentially play games from only a description) but the training is not.
- 3y ago

akoboldfrying 3y ago

Only skimmed this and didn't read the underlying paper, but I was surprised to see no mention of the fact that "A is B" often does not at all imply "B is A" in everyday language: A bird is an animal, but it's wrong to conclude that an arbitrary animal must be a bird.

thaumasiotes 3y ago
This is bad news for all the people who like to say "A winner is you!"
- Falkon1313 3y ago
  But obviously that implies that you are a chicken dinner. At least according to some people's expectations, presumably.
thisgoesnowhere 3y ago
The word an in this sentence (indefinite article) means that the word animal is non specific. So this structure does not follow from the original. "A is B" is not the same as "A is a B" However, you are right that it's not always correct since "Poetry is Literature" and "Steam/Ice is H20" are both examples where this breaks down. My annoying pedantry for the day.
- randyrand 3y ago
  you can do the same mistake without “an”. Jeff is the short haired guy != the short haired guy is Jeff. It depends on context. You can’t fully equate them.
PoignardAzur 3y ago
"A is B" is meant a bit loosely here. A better shorthand would be "A is the B" <=> "The B is A", or "A does B" <=> "B is done by A". The actual paper explores equivalences that actually make sense, and whether LLMs have trouble with them.

ShamelessC 3y ago

The article is a decent peer review and refutation of “the reversal curse”. Some of the comments given here clearly haven’t read the whole article though - arriving at similarly skeptical conclusions that are clearly present and expanded on in the article. Why do people feel the need to do this here? Armchair commentary on advanced material is one of the main reasons I avoid Reddit. And furthermore why does it feel like you’re not allowed to suggest this as a response? I should be able to say “RTFA” but here I feel like I’m going to be scolded or banned by moderation.

swatcoder 3y ago
The author is a shark-diving science journalist who's been paid a lot of money by OpenAI, not a researcher, and the article is neither a "peer review" or an expert "refutation". It's a meandering and sometimes thought-provoking exploration of how LLM's can be coaxed to deliver on statements sort of like (but not actually equivalent to) the ones that failed in the paper. If you want to defer your own understanding to the author (and overstate their own claims!), that's fine, but a lot of people here are actually more informed about many of the topics and modes of discourse covered in the article and may have something to share with you if you give them credit. And if you're not going to give other commenter's due credit, then you're not really following the spirit of HN. While you can't always know the background of any individual commenter, we have an unusually accomplished and informed community in general and thrive by treating all commenters with the respect we might give the most accomplished and informed -- at least by default. That's why a blanket "RTFA" isn't usually appropriate here.
- amayne 3y ago
  Hello! I’m the “shark-diving science journalist” in question. First of all, you can run the experiments like I did and test this yourself. I’m not asking anyone to take my word. Just do what I did: Read the original paper. Test the claims for yourself. And to clarify a couple things: 1. The shark-diving part is true. 2. I’ve never been a journalist of any kind that I’m aware of unless you count writing for Skeptic Magazine. I’ve had many, many jobs though. 3. I started at OpenAI as a software engineer and member of technical staff. When I started there was just over a hundred people there. The lines between engineering were and are blurry. 4. I was the original prompt engineer at OpenAI and discovered many of the examples for using GPT-3 and wrote a lot of the original documentation. Internally my title was “prompt whisperer.” 5. I’m in the GPT-4 research paper for my contributions to model capability. I helped find abilities for long-text, vision, etc. 6. I was given the title Science Communicator when I started doing background briefings for media, etc., but still worked on model capability and other things. 7. I left OpenAI two months ago to work on a startup. Best, Andrew Mayne

pmarreck 3y ago

I couldn't duplicate it in GPT4 here. It answered correctly, unless I posed it wrong: https://chat.openai.com/share/75d46a03-a223-4f3a-987d-f8fec3af1ca5 https://chat.openai.com/share/75d46a03-a223-4f3a-987d-f8fec3...

jetrink 3y ago
The paper is not talking about information provided in the context, only training and fine-tuning. > In fairness, it’s also worth pointing out here that they’re making the claim that the reversal curse only applies to training and fine-tuning and not in-context – i.e., putting all your information inside a prompt. They point out in a footnote that you can put A to B data in a prompt and GPT-4 will make B to A connections just fine. Unfortunately, this was lost on many of the people covering the pre-print.
- pmarreck 3y ago
  ah, I see. OK.

Imnimo 3y ago

>If you start a query with “Mary Lee Pfeiffer”, you’re not going to get very far because neural networks aren’t equidistant grids of points (besides the fact that she may not appear very often under that version of her name.) They’re networks of nodes, some with many connections, some with few. One of the ways you optimize large models is by pruning off weakly connected regions. This may come at the expense of destroying B is A relationships for weakly represented entities. Am I missing something here? This paragraph reads like complete gibberish to me. Also, I don't buy the experiment at the end. If you fine-tune the model with exclusively Tom Cruise data, I want to see proof that it doesn't just answer "Tom Cruise" all the time. I want to see that it says Tom Cruise wrote Aces in the Stream, but doesn't say Tom Cruise wrote Kings in the River.

Xelynega 3y ago
Their entir "graph" description of neural networks reads as nonsense. Neural networks are graphs insofar as they're networks of connected neurons and not further(definitely not graphs of information as the writer seems to think). Given the fact that neural networks are quite literally an n-dimensional function once trained, it's more accurate to call them an "equidistant grid of points" than it is to call them a graph of information since literally all they do is take an n dimensional vector and output an n dimensional vector.
Izkata 3y ago
It reads like they saw the diagram from [0] and didn't understand what it meant (like, they assumed "x1" represented an entity like "Mary Lee Pfeiffer"). Note how in that diagram, the bottom nodes in the second and third layer aren't fully connected - which if you don't read the description, you could end up assuming the connections between layers are created and trimmed, not just weights changed. They may also be getting it mixed up with semantic search, where an entity like that would exist in a graph and have connections to related concepts. [0] https://towardsdatascience.com/first-neural-network-for-beginners-explained-with-code-4cfd37e06eaf https://towardsdatascience.com/first-neural-network-for-begi...
amayne 3y ago
I've added your suggested test to the blog post: https://andrewmayne.com/2023/11/14/is-the-reversal-curse-real/ https://andrewmayne.com/2023/11/14/is-the-reversal-curse-rea... Spoiler: It doesn't say "Tom Cruise."

zarzavat 3y ago

Humans are also vulnerable to the reversal curse! When you learn languages you have to learn both directions (chat is cat and cat is chat), anybody who has built an anki deck will know this, otherwise you will be better in one direction than the other.

tkgally 3y ago
As one gets deeper into learning another language, it’s also important to be aware that the meanings of words in different languages rarely map to each other in clean bijections. Common words especially tend to be semantic clouds, not fixed points of meaning. I don’t know French, but I am sure there are many cases where an English phrase or sentence that includes ‘cat’ should not be translated into French with ‘chat’ and vice versa. (When asked, GPT-4 offers two such examples: "let the cat out of the bag” and "avoir un chat dans la gorge.") I don’t mean to suggest, though, that memorizing word pairs is not a good way to learn vocabulary in another language. For me, it was an essential step in acquiring the second language that I am now fluent in (Japanese).
- deleted 3y ago
  [deleted]
- School-Cotton 3y ago
  I think this has as much to do with abstract vs. concrete words as common vs. uncommon. If you had to pick a word corresponding to “chat” it’s quite clearly “cat”, despite a few different expressions. But it’s rather difficult to translate “justement” to English, or “random” to French, without further context.
- TeMPOraL 3y ago
  Words are always semantic clouds. Not getting that is what sent philosophy down many a dead end, and the source of plenty of arguments regular people get into all the time. One doesn't need to try mapping between two languages - it's enough problem trying to map within the language itself.
TeMPOraL 3y ago

ugh123 3y ago

This seems like the kind of thing that would be addressed during implementation, rather than persevere as a fundamental problem with LLMs.

cratermoon 3y ago

If I'm reading this correctly, the author is saying that it's not a failure of logical deduction if the training data doesn't include the reversal. In other words, he's saying that if the data contains "Tom Cruise is the son of Mary Lee Pfeiffer” but not “Tom Cruise’s mother is Mary Lee Pfeiffer”, then the model's inability to determine the latter is "an explanation of how neural networks function than a model’s inability to deduce B is A." But of course "how neural networks function" is that they fail at basic logical deduction and do not generalize. So again, if I'm reading it correctly, he's hand-waving away the inability to make basic logical deductions because that not something they can or should be expected to do. As I read it, that means the reversal curse only exists if the answer to the question "can LLMs do logical deduction?" is "yes". If one takes the position that LLMs can't do general logical deduction, which seems to be the author's point of view, then there's no expectation that knowing "Tom Cruise is the son of Mary Lee Pfeiffer" is sufficient to determine “Tom Cruise’s mother is Mary Lee Pfeiffer”. Am I missing something?

xelxebar 3y ago
Did you see the end of the article, where the author uses a small example and gets "B is A" generalization? These are the salient takeaways I got: - Is/Was wording might matter. This is something probably a bug. - 30 facts about a person might simply be too little for "B to A" generalization - Extra precision/context in the prompt can help locate the "B to A" inference. - How you cut up your training data can bias inferences in surprising ways. - "B to A" generalization clearly does happen, even without "B is A" in the data, but it's not as stable as you'd want.
- cratermoon 3y ago
  OK, but how does that not just demonstrate LLMs can't generalize absent massaged training data? > it's not as stable as you'd want. Which I take to mean that a model can sometimes confabulate "B is A", solely out of random variation, and that it's possible to bias the data and prompt to generate the expected response. The model hasn't done any logical deduction, the response is just a bias-influenced lucky break.

deleted 3y ago

[deleted]

willsoon 3y ago

Oh Cantor, Cantor, bitter pale Cantor, you pale ascetic, your twist was epic, so I live in silence, in sadness I crave, all for that book, you wrote and I read.

jmount 3y ago

This is really dreary.

dr_dshiv 3y ago

> So in summation: I don’t think any of the examples the authors provided are proof of a Reversal Curse and we haven’t observed a “failure of logical deduction.” Simpler explanations are more explanatory: imprecise prompts, underrepresented data and fine-tuning errors. Phew!

godelski 3y ago

I didn't know anyone was even suspicious of the reversal "curse". I thought it went viral more because people were surprised anyone was surprised and the tweet was highly sensationalized. It feels more like the __expected__ results considering both speech patter bias __and__ that "causal" attention is sequentially biased. Hell, we saw the same things in RNNs, we see it in classic autoregressive models, and so on. But it definitely isn't how humans encode information because if you had all 3 pieces of knowledge (you know who Tom Cruse is, you know who Mary Pfieffer is, and you know the relation between Mary and Tom is mom/son) then your ability to recall is (nearly) invariant to the ordering. Hell, we're so robust you can caveman speak like "Mary Pfieffer son who!" and still get the answer or caveman yoda speak "Who Mary Pfieffer son is?" Some languages even have these orderings but we're impressively robust (enough that I think we trick ourselves a lot when subtlety comes into play. AKA overfitting). So I find it weird to call this "a feature" and also act like this is surprising. But can I also take a minute to just say I really hate GPT experiments? They're performed on a stochastic model, with proprietary weights, proprietary training data, proprietary training methods, and above all is constantly changing and at a rather fast pace. It makes for a very non-scientific process as you can't decouple a lot of important factors and reproduction is a crap shoot. It is not a very good way to go about studying "how __LLMs__ work" and is rather "how does GPT work at this particular moment in time and aggregating all these unknowns?" There's some bitter sweetness because I do like that GPT is free but it feels like a major edge that they have is that the community just does a lot of free research for them and in ways where they can better interpret results than the people who performed the experiments in the first place. I really do believe that academic works shouldn't be focusing on proprietary and dynamic methods. It's fine to include them in results (and preferably added post review to avoid identity spoilage (or we openly acknowledge that double blind is a joke)) but I'd rather most researchers focusing on the general concepts and with the ability to dive down the rabbit hole than playing a wack-a-mole game. Also, I'd totally love it if more research papers were instead blog posts. Kudos to anyone posting their research on blogs, academic or not (I don't care about your creds, your real creds are your work). Papers are about communicating with fellow scientists, right? Why do we need journals and conferences these days?

zapperdulchen 3y ago

I played along the example of the chancellor question. If you deviate a little from the examples in the article GPT-4 gets it all wrong: Who is the eighth Federal Chancellor of the Federal Republic of Germany? - Olaf Scholz (wrong, Angela Merkel) https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f9de3f2 https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f... Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Helmut Kohl (wrong, Angela Merkel) https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f9de3f2 https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f... Interestingly whether you put is or was into the question does make a difference. Even when allowed to surf the web, ChatGPT gets it wrong: Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Gerhard Schröder (wrong, Angela Merkel) https://chat.openai.com/share/867cb3bc-f642-4d2a-b80d-fe8dc00eef3c https://chat.openai.com/share/867cb3bc-f642-4d2a-b80d-fe8dc0... Though it's referring to the right Wikipedia page: https://en.m.wikipedia.org/wiki/List_of_chancellors_of_Germany https://en.m.wikipedia.org/wiki/List_of_chancellors_of_Germa...

dragonwriter 3y ago
> Even when allowed to surf the web, ChatGPT gets it wrong: > Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Gerhard Schröder Its probably counting Walter Scheel, who was Acting Chancellor in 1974, and you are probably not. When I asked ChatGPT to list the chancellors in order and identify the eighth, it listed eight ending in Schröder, with Scheel and his ten day Acting Chancellor tenure in May 74 as number 5. (Scheel’s tenure is in the timeline on the Wikipedia page you cite, though not in the numbered list in that page, which is why there is a dicontinuity in thr dates on the numbered list.)
- littlestymaar 3y ago
  It's funny because you ascribe some reasoning in ChatGPT's answer “Gerhard Schröder”, but somehow missed that it's also able to give you two other answers that are unambiguously wrong…
  - dragonwriter 3y ago

riwsky 3y ago

“I’m going to define generalization such that my thesis that GPT-4 can generalize reversals is true—and lo, my thesis is true!”

diffeomorphism 3y ago

Well, yeah because the reversal is just completely wrong most of the time. Only if you have additional context clues like definite articles or some outside knowledge that a role or property is unique, then you can maybe, sometimes conclude the reverse. Not a bug, working as intended.

threatripper 3y ago
Some relationships like parent child are quite reversible.
- diffeomorphism 3y ago
  Keywords "some" and "quite". Also, only (likely) half the information. "John is Alice's parent.", "Who are Alice's parents?". And that is ignoring questions like single parent households, adoptions, biological parents, step parents, multigeneration households, getting disowned... If a reversal is correct 90% of the time, that is useful. However, it will still be wrong 10% of the time.

golol 3y ago

Excellent blog post. The only thing which I wish was there was an explanation of how promot + completion training would be any different to pure completion training. From my understanding oftransformers, it ahould be the same.

user_named 3y ago

I mean obviously yes. LLMs aren't intelligent and they don't understand anything. OP is trying to argue against this but it's nonsense. ChatGPT also cannot tell me who the 8th chancellor was. It tells me it was Gerhard Schröder.

user_named 3y ago
It also cannot tell me the 7th.
TeMPOraL 3y ago
IDK, I still some of it boil down to "B -> A" being extremely ambiguous. Other than the annoying Tom Cruise example (I finally know where it came from!), I've seen people frequently bring up "The color of the sky is blue" vs. "Blue is the color of the sky" - but this illustrates my hypothesis perfectly. In training data (and in the totality of what humans ever said or wrote), "Color of the sky is " is almost certain to be followed by "blue" - meanwhile, "Blue is the color of the " has a lot of possible completions, of which "sky" is by far not the most likely.
- user_named 3y ago
  Nah. The context of all of this is people claiming LLMs are AI. If so they would be able to reverse A is B, and also know when not to, so your example is irrelevant. The paper shows that it cannot reverse. Then the blog posts goes around different disingenuous ways of making excuses for why it can't reverse at the same time as trying to show that it can reverse in the Tom Cruise training example (extremely disingenuous because it's just the same data over and over with synonyms replaced, and what he gets out of it is just completions, not logical deduction).
  - TeMPOraL 3y ago
    I still call bullshit on this. Humans don't do reversals for free either. We may memorize the reversal, or first, infer it. That's an extra step, often very cognitively challenging. I'd like to see experiments showing how "reversal curse" fares when LLM is allowed to make the inference. Like: "Please complete the sentence: [your B->A] - but reason through the process first. Start with identifying the subject, then write a high-level summary of what you know about the subject, and only then attempt to complete the original sentence." I'd expect something like this to suddenly score much better. And I don't consider this cheating - because I think the "AI-ness" quality of LLMs shouldn't be measured against the workings of a human mind, but rather the workings of the inner voice in the human's mind.

RamblingCTO 3y ago

I'm not sure about this. Olaf Scholz has been chancellor since 2021 and the cut off for chatgpt 4 is allegedly January 2022. My gut also tells me that the underlying embeddings should very well reflect equidistant relations of olaf scholz and chancellor. /e: instead of downvoting, I'd love to have your opinion instead on why you think I'm wrong ...

maaaaattttt 3y ago

The Olaf Scholz exemple from this article is just another exemple of how LLMs can’t count. If you try this prompt: “In this list of words: bike, apple, phone, dirt, tee, sun, glass; which is the fifth word?” it will fail as well. “Fifth” is not connected to any counting ability in LLMs the way it is for us. If you now try this prompt: “Who’s Tom Cruise’s mother in this exemple: “Mary Lee Pfeiffer is Tom Cruise’s mother.”?” It will give you the right answer. IMO this is a sign that it can understand reversibility. The exemple used are not present enough in the dataset and the “confidence” of the model is not high enough for it to give the right answer. Or, as stated at the beginning they use counting mechanisms (or others) that the model just doesn’t have.

saliagato 3y ago
It can’t count because what the LLM sees is a bunch of tokens not words
- maaaaattttt 3y ago
  Exactly. And asking what/who/which is the “Ninth” something will always fail (or randomly succeed)
  - OJFord 3y ago
    Much more likely to work for something like German chancellors, US presidents, etc. that people do actually number and could easily have been trained on that data. If you ask 'who was the 45th US president' it doesn't need to be able to count a list of US presidents to answer 'Trump', you can just duckduckgo that and get the answer anyway, that's not doing any counting either.
deleted 3y ago
[deleted]

Applejinx 3y ago

This is great work. Andrew's on to something. Thing is, the behavior we see in his form of the LLM is also what we experience as humans. Good writers, marketers and manipulators know this about people. You seed the ground with the concepts you'll need to introduce, so they're present in the person's 'model' and you can elicit them later, on request. Formalizing things in the 'reversal curse' manner is like loading the 'mind' of the LLM with habits and assumptions and cutting off its ability to free-associate from seemingly relevant concepts… which is likely to be more valuable in the long run, because an LLM can contain more than a human can. That doesn't mean it will be more intelligent, but it seems reasonable to infer that the LLM can have a broader base of association to draw from, where we as humans tend to be restricted to associations from our own experience. We only get one shot at 'training data', though it's in countless sensory forms, where LLMs are stuck with language as their only window onto experience. I'm loving the notion of leaving the prompt 'training' stark raving blank. Let's see what comes out of this giant pile of human verbal associations. It is only that, associations, but it's on a grander scale than we're accustomed to. Making it 'answer questions correctly' seems woefully unambitious.

bjornsing 3y ago

I didn’t read the whole thing, but the first part about Tom Cruise and his mother sounded very flawed: Of course the LLM could learn the reverse relationship if it had better data where the reverse relationship is a common occurrence. The point of the reversal curse argument (I guess) is that it should be able to learn the reverse relationship from entirely other examples [1], but seemingly is not. 1. That is, the LLM should be able to learn that “A is son of B who is female” implies that “B is mother of A”, regardless of who A and B is. It should then be able to apply this pattern to A = “Tom Cruise” and B = “Mary Lee Pfeiffer” and deduce “Mary Lee Pfeiffer is the mother of Tom Cruise” without even a single example.

jameshart 3y ago
LLM training teaches it that ‘Mary’,’Lee’, and ‘Pfeiffer’ are words that appear in association with ‘Tom’, ‘Cruise’, and ‘mother’. Probably also in association with ‘son’, and ‘family’. But the association between the words ‘Mary’, ‘Lee’, ‘Pfeiffer’ and ‘son’ or ‘mother’ point to other words more strongly than they do to ‘Tom’ or ‘Cruise’. Sure, Tom Cruise is probably in there but so is probably John Henry Kelley (Michelle Marie Pfeiffer’s son), and George Washington Custis Lee (whose father was Robert E Lee and whose mother was called Mary). A random piece of text containing the words Mary Lee Pfeiffer is just, based on its training, not likely to be about Tom Cruise. There’s nothing there anchoring it particularly to those words. It’s possible that the ‘Pfeiffer’ in there screws it up more, as well - it’s like it vaguely knows there’s a Hollywood connection but of course it crosses its wires and thinks it’s probably Michelle Pfeiffer. Be honest: if a pub quiz question came up asking “which Hollywood superstar’s mother is Mary Lee Pfeiffer“, you’d probably not guess Tom Cruise either. But here’s the thing: once we go beyond training into specific completions, if you ask GPT ‘who is Tom Cruise’s mother’, it answers correctly; and then once it has that A is B in context, you can ask it ‘who is Mary Lee Pfeiffer’s son?’ And of course it knows how to complete that B is A. So I definitely agree with the piece here - there’s no ‘reversal curse’, there’s just asymmetric information relevance.

aade63482 3y ago

[flagged]

smitty1e 3y ago

> Regardless, we can see that GPT-4 can easily go from B to A in that example when the question is posed unambiguously. Ambiguity is the soul of humor and the sole on the neck of any booted computer.

foldr 3y ago

Fodor's argument from systematicity and productivity strikes again: https://plato.stanford.edu/entries/language-thought/#ArguProdThou https://plato.stanford.edu/entries/language-thought/#ArguPro... An interesting reflection on the state of the field that the paper doesn't even cite him.

weinzierl 3y ago

"As a side note: I want to point out that I’m not aware of any examples of capabilities that can be done with prompting a model like GPT-4 that it can’t be trained for. This is why I’m a little skeptical." While it is just a side-note in the article, isn't this the core of the problem? When we've already established that LLM's can do B to A generalizations in-context, why wouldn't they be able to in training? In one of my experiments I noticed that GPT-4 seems to be perfectly aware of the number of letters in a word in-context, but has difficulty when trained knowledge is involved. For example it can reliably answer the question: "Can you tell me how many letters each of the words of the first sentence of our conversation has?" At the same time it fails with the task: "Can you rewrite the first sentence of our conversation in a way that preserves its meaning as closely as possible but use only words with an even number of letters?" It will give an answer but get the letter counts very wrong and it is unable to improve its answer by iteration. Of course this does not prove that a model cannot be trained to answer tasks involving word length, but GPT-4 seems to have a knowledge gap here (possibly due to tokenization).

mitjam 3y ago

This is a great article with more tips on how to prompt and fine tune LLMs than most articles that focus on this topic. Also has surprising insights which are food for thought. Like: Would it help to attach facts to well-learned „nodes“ (like Tom Cruise in his example) when fine tuning? In the sense of repurposing well-known nodes. Looking forward to reading more from Andrew.

Zolde 3y ago

Luckily this curse has no significant practical impact on accuracy. Who knows who Mary Lee Pfeiffer is, but does not already know her son is Tom Cruise? And if such a person exist, do you want to give them a correct answer, or talk about how Mary Lee Pfeiffer is not a notable person with a Wikipedia page?

smusamashah 3y ago

I don't agree with authors claim. There is a plant which mimics leaves of nearby plants discussed here https://news.ycombinator.com/item?id=31301454 https://news.ycombinator.com/item?id=31301454 If you ask GPT-4 what this plant is known for, it will tell you correctly. But if you ask in any number of ways to tell the name of plant which mimic leaves, it will always give incorrect answer.

xeonax 3y ago

Isn't this solvable by creating training data that says B is A? Instead of always training it on A is B.

deleted 3y ago

[deleted]

octocop 3y ago

I found an example of this with chatGPT 3.5 > Who is Karen Meyers son? vs > Who is Mac Millers mother?

MPSimmons 3y ago

I had a discussion with ChatGPT 4 on this (quasi-ironically) to see what it "thought", and it theorized that BERT might have had a better outcome to these conditions because it's actually bidirectionally trained, rather than GPT, which is forward-trained. https://chat.openai.com/share/daff08c5-ddea-4ad4-963a-0df88eddc8e1 https://chat.openai.com/share/daff08c5-ddea-4ad4-963a-0df88e...

montebicyclelo 3y ago

Related; 4x improvement in accuracy for experiment 2 of this paper, (with gpt-3.5-turbo), by changing the prompt: https://sidsite.com/posts/reversal-curse/ https://sidsite.com/posts/reversal-curse/ Shows what a dramatic effect the prompt can have

sgt101 3y ago

We are in word games here about zero shot learning. The fact is that with zero shot the models actually aren't learning in the sense of updating their knowledge structure, they are instead using background knowledge as inference. We have started calling it zero-shot learning but... well it isn't really.

Obscurity4340 3y ago

Is it a possibillity that certain aspects of logic or human communication have a secret sauce that is not reliably replicable? I wonder if its expected to basically be able to perfectly interface with humans or logicical structures that flow from said imperfect beings, who are themselves deeply imperfect and irrational

thomastjeffery 3y ago

Right at the opening, we have a fundamental (and common) misunderstanding of what LLMs are. When humans read the statement "A is B", we semantically transform that into a logical association. LLMs do not perform any semantics or logic. Here's a simple example to demonstrate: If we trained an LLM on something like "A is B C is D D is C.", we might be able expect the continuation, "B is A". If we then gave that LLM the prompt, "What is B?", we might expect the continuation, "B? is What". Large models like GPT present more interesting continuations because they are trained on larger and more diverse datasets. More diversity also means more ambiguity, which results in continuations that are less predictable and more illogical.

numeri 3y ago

This post isn't a scientific investigation. It's someone playing around with a black-box model with few to little controls – which is unfortunately the only thing we can do when experimenting with GPT-4, which does excuse this partially. Unfortunately, though, the author seems unaware of the actual state of research on the actual mechanics of how LLMs store knowledge and specifically binary relations. The ROME paper[1], among others, shows that the feed-forward layers function as a key-value store, where the feed-forward's up projection of the last token in a noun phrase (say, "the Eiffel Tower") acts as a key, which when multiplied by the down projection, produces a value that contains information the model knows about the subject, which is then added into the residual stream/hidden representation. A paper building on that work[2] then went on to show that it's usually the self-attention layers that use the relational phrase (say, "is in") to extract the relevant knowledge from the feed-forward layer's output (in this example, hopefully "Paris"). This mechanistic understanding makes it really obvious why the reversal curse occurs – using matrix multiplication as a key-value store requires having a fully separate key-value pair to look up the reversed relation. [1] https://arxiv.org/abs/2202.05262 https://arxiv.org/abs/2202.05262 [2] https://arxiv.org/abs/2304.14767v1 https://arxiv.org/abs/2304.14767v1