10 ms·

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless

by gipp 3y ago

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage, not just exact identity. "The apple is red" does not imply "red is the apple," as a trivial example.

deleted 3y ago
[deleted]
earthboundkid 3y ago
Ideally, the LLM would be able to tell the is of identity from the is of predication. I found the article a little too defensive. It felt like "of course it doesn't work, there's not enough data!" and okay, sure, but that is a flaw, no?
- famouswaffles 3y ago
  Well the LLM can tell in that reversal works just fine in-context. This is a recall from training issue and training is dumb.
famouswaffles 3y ago
I think a lot of people unwittingly think of the training process as "smart". Similar criticisms are "well there are many descriptions of game x it would have read so why doesn't it play x well". But gradient descent is a dumb optimizer. Training itself is not actually like someone reading a text anymore than evolution is like someone thinking about the best way to augment an organism. a "smart" optimizer would look at a reversal applicable sentence and know exactly what bunch of weights to change to store it in such a way as to be recalled reversibly in the future. Inference may be smart (GPT-4 can reverse in context just fine, potentially play games from only a description) but the training is not.
- LoganDark 3y ago
  A prior (now-deleted) comment of yours put this in such a great way, and I'm sad that it got deleted: > You could potentially describe how a game is played to GPT-4 without examples and get it playing it correctly but passing that same description into the training process of the model just gets you a model that can describe your game correctly.
- tysam_and 3y ago
  I'm not really sure if I understand the intuition here, this seems rather disconnected from what I understand to be the math of optimization. It seems like you're referring to an associative Hebbian/Hopfield-like lookup, which the current 'dumb' optimizers already do. Better yet, the learning rates are normalized by the diagonal of the empirical Fisher so that the learning w.r.t. to some estimated expected information is more constant, for said associative lookup operation. Additionally, the training loop (which you call 'dumb') is just...a teacher-forced version of inference, which you call 'smart'? It's better to simply minimize the log-likelihood in a scalable way. Hand-engineered solutions rarely survive compared to strong-scaling ones.
  - famouswaffles 3y ago
    smart is in quotes here for a reason. It's 'dumb' in relation to many people's expectations. >Additionally, the training loop (which you call 'dumb') is just...a teacher-forced version of inference, which you call 'smart'? What powers In context learning is not very well understood but it doesn't appear to be or really work exactly like just a non teacher-forced version of training. There are qualitative differences. The same models have no problem with this 'curse' when the information is provided in context for example. >It's better to simply minimize the log-likelihood in a scalable way. Hand-engineered solutions rarely survive compared to strong-scaling ones. I never said anything about it being bad.
    - tysam_and 3y ago
      > What powers In context learning is not very well understood but it doesn't appear to be or really work exactly like just a non teacher-forced version of training. I mean, yes. One distills information from a training set into a compressed representation, and the other generates a compressed representation that yields (more or less) fixed state space attractors. It's just inducing a bias over the state space of the network, nothing incredibly special, though I'd consider the initial stage of training to be the most important, as it is responsible for all of the ingest of all of the embedded information the network will be using during autoregressive inference (especially w.r.t. the context of doing so during longer generations). So the notion of 'in context learning' is one I find to be a bit of an illusion, of course, as no actual learning is being done, just the induction of biases, which appears to give rise to a transiently-'better trained' network. You could see this as a bit straightforward perhaps, but I feel it needs to be said.
majormajor 3y ago
I don't think this is the right explanation, or really that relevant, the suggestions from 'og_kalu and in the article sound more accurate to me. It seems like understanding when "is" is reversible is pretty core to the capabilities of the model, but that's different than having a lot of facts memorized. For instance, a model should be able to answer "who is the star of Mission Impossible" with "Tom Cruise" based on context or internalized-training of "Tom Cruise is the star of Mission Impossible" while not answering "Tom Cruise" to a much more general question like "who is covered in water" even if it had internalized a bit from a review like "there's a scene in Mission Impossible where Tom Cruise is covered in water..." unless it had context pointing it at that specific case.
- tysam_and 3y ago
  It's a bit of mathematical bikeshedding, hardcoding reversability would cause far more problems than it would help. Best to simply scale log-likelihood-based training, next-token-based training trivially contains a requirement for learning all of the subproblems that predict said, next token, and hardcoding something to get warm human fuzzies would be creating a biased estimator (and move us back towards the 90s a bit). Models already very constantly do context-dependent token utilization, it's an autoregressive feature based on the entire stream of incoming tokens. Humans have a bias to focus on the 'last token used', this is not what language models look at.
  - thfuran 3y ago
    >it's an autoregressive feature based on the entire stream of incoming tokens. Humans have a bias to focus on the 'last token used', this is not what language models look at. But human language is created for and by humans. Is not then operating on language in a categorically different manner an incorrect usage/understanding of language?
    - tysam_and 3y ago
      No. For more information, please see https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf https://people.math.harvard.edu/~ctm/home/text/others/shanno...
p1necone 3y ago
I don't think it's as simple as that - "the apple is red" is "single thing belongs to category", whereas "Olaf Scholz was the ninth Chancellor of Germany" is "single thing is single thing" - the latter is reversible, the former is not. I would expect a good language model to be able to parse both sentences correctly.
- willsoon 3y ago
  You are right. You are thinking correctly. But "the apple is red" might not mean that this particular apple belongs ---to put it in your wording--- to the category red, but that the category of things we call apple also belongs to the category of things that are red. And generally speaking, I think that is the meaning.
  - School-Cotton 3y ago
    > but that the category of things we call apple also belongs to the category of things that are red No, in English this would be “apples are red”, not “the apple is red”.
    - bloak 3y ago
      I don't disagree though perhaps it's worth mentioning that sentences such as "the great spotted woodpecker is a medium-sized woodpecker" are used in English with the meaning "great spotted woodpeckers are medium-sized woodpeckers" so it seems to me that "apples are red" is grammatically a possible meaning of "the apple is red" even if it would be stylistically and pragmatically so weird that nobody would ever do that except perhaps as some kind of joke.
      - smegsicle 3y ago
        grammatically you'd capitalize Great Spotted Woodpecker, and similarly if speaking of all apples as a category, Apple should form a proper noun
      - karatinversion 3y ago
foobarqux 3y ago
But that's the problem isn't it? In some cases they are equivalent and in some cases they are not and a next-word-predictor needs to have "explicit" training data (i.e. it is not doing "reasoning") whereas a human can infer. This isn't surprising if you think about how the machine actually works rather than treating it like a sacred magic box. The default assumption for why there is any "success" for in-context learning should be that it's just picking a nearby token that "fits" not a process of logical deduction. edit: The obvious band-aid-fix is to feed reversed sentences into the training data without telling anyone, after which LLM boosters will proclaim that LLMs "learned" to reverse logical implications.
- famouswaffles 3y ago
  LLMs can already reverse logical implications. Literally this is not a problem in context. The model can infere all this just fine. Even the original paper makes this clear. This is a recall from training problem, not a logical inference one.
  - foobarqux 3y ago
    I don't understand what you are saying. LLMs can't generally reverse implications, that's what the paper says. This in turn suggests that what you are observing in-context is some other statistical heuristic, not a "logical inference capability". I don't know in what way this is some trivial "recall" problem as you suggest. Does the reversed implication in question need to be in the training data explicitly or not? If it does I don't understand how you can claim a logical inference capability.
    - famouswaffles 3y ago
      >LLMs can't generally reverse implications, that's what the paper says No that's not what the paper says. It says training won't immediately store this information in a way as to make it reversible. Let's get one thing straight. This is a common problem for human learning as well. Anyone who has used Anki for language learning will tell you that if you just train on target language word on the front and native language word on the back, you will fail the reverse unless you specifically train for that. This is specifically a problem of recall. Not a problem of making the logical inference. If you ask the human language learner immediately he has learnt the new word for the reverse direction, he will obviously tell you the correct word. But later he may not recall the reverse even if he remembers the original. Again this is a problem of recall rather than the ability to make logical inferences. In the same vein, if you give the LLM the original direction in context and immediately ask the reverse, it will correctly tell you. It can make the logical inference.
deleted 3y ago
[deleted]
notahacker 3y ago
the examples are about relations in natural language rather than formal logic though. Mary Lee Pfeiffer being Tom Cruise's mother definitely does imply that Tom Cruise is a valid answer to questions about who Mary Lee Pfeiffer's son is (it doesn't necessarily imply she doesn't have other sons or there isn't another lady of that name who is childless, but that isn't what's tripping the model up). And there is absolutely nothing ambiguous about the failures of the fine-tuned model where the author gave it specific phrases exclusively associated with a fake name in the fine-tuning training set and when prompted for "who is $specific_phrase", supplied different names exclusively associated with completely different phrases (unlike the author, I don't see [serendipitously or otherwise] picking name words from the right training set as "B kinda-has-something-to-do-with A generalization", not when it's lost so much information that the combination of name words which exclusively appears in sentences with that phrase isn't treated as a more probable response. Never mind emergent understanding of syntax, it isn't even making obvious inferences from proximity here) If GPT4 rarely makes those errors it's clearly not an insurmountable problem at this level of training, but it does imply a lot more difficulty fine-tuning models to reliably retrieve correct information from specific text.
tysam_and 3y ago
Bit of a nitpick, "red is the apple" is both a valid sentence and one that also conveys the original relationship held by the opposite phrasing, so, in this case, "the apple is red" does indeed imply "red is the apple", and accurately so.
- emme 3y ago
  A proper example is probably "an apple is red" and "red is an apple".
  - tikimcfee 3y ago
    What you’re missing that is “red is an apple” is also possibly saying in a metaphorical sense that red is like an apple to someone - delicious, a treat, perhaps otherwise representative. In that way, the encoding of “is”’ is exactly correct - it’s an ordered pair of glyphs that imply a weak form of assignment or description. : apologies, replied to the wrong post, meant to push this up one.
    - cryptoz 3y ago
      The correct sentence would start with a capital letter: “Red is an apple.” This is also completely valid as in a cartoon character of an apple named Red. The subject being the start of the sentence compounds the uncertainty in meaning.
      - tysam_and 3y ago
        Yes, this is because there's no assignment happening up front w.r.t. the word 'red'. However, you'll notice the same kind of ambiguity for the word 'apple' in the reversed sentence, as the word 'The' can imply quite a few things following it. The entropy gotta get slung around somehow.
- eru 3y ago
  Perhaps have a look at this famous syllogism instead: (1) Mortal was Socrates. (2) All humans are mortal. (3) Therefor all humans are Socrates.
anon291 3y ago
> skeptic side of the LLM spectrum Can you explain what you are skeptical of? There is ample evidence that LLMs are what they say they are: a series of transformer (usually) functions with learned weights that can successfully be used to generate human like text in many domains. Do you disbelieve this? Or are you skeptical of something else.
johnfn 3y ago
You're thinking of LLMs as basically pattern matching "a is b", but that's not really how they work. In fact, the incredible thing that LLMs can do is that they can understand (some) colloquial language and shades of meaning. LLMs can grasp all sorts of strange textual nuance; they can't grasp the cases "Foo is Bar's mother" strictly implies "Bar is Foo's son"?
- famouswaffles 3y ago
  >they can't grasp the cases "Foo is Bar's mother" strictly implies "Bar is Foo's son"? Correction. The Model can. Training cannot.
  - TeMPOraL 3y ago
    I'm starting to wonder, why should it in the first place? Very late stages, or fine-tuning, I'd perhaps understand. But early on? Everything is possible, and "Foo is Bar's mother" doesn't imply much over "these tokens come together, sometimes". It's not that humans aren't suffering from this too. It's possible to spot it when you're learning, or even easier, when you're teaching someone a new thing. A person who learned that "A is B" does not automatically learn that "B is A"; they need to first process it, perhaps run the reversion explicitly in their head. I'd say that checking if the student can infer "B is A" having learned "A is B" is a good way to tell whether they're starting to comprehend the material, vs. just memorizing it.
ryukoposting 3y ago
The (purported) point of an LLM is its ability to represent language. The inability to represent such a critical nuance would represent a fundamental failure in that regard.
Grimblewald 3y ago
While red is the apple is not a correct logical statment using conventional understanding of each term, it is still an understandable english sentence, a structure which likely pops up a fair bit. Even if just for scripts of yoda lines. It just goes to show truly how fucky language is and how amazing it is that something even remotely comprehensible comes out of LLMs
foldr 3y ago
This is explained very clearly in the paper. They are not looking at all sentences of the form "A is B": >While it’s useful to relate the Reversal Curse to logical deduction, it’s a simplification of the full picture. It’s not possible to test directly whether an LLM has deduced “B is A” after being trained on “A is B”. LLMs are trained to predict what humans would write and not what is true (Lin et al., 2022). So even if an LLM had inferred “B is A”, it might not “tell us” when prompted. Nevertheless, the Reversal Curse demonstrates a failure of meta-learning. Sentences of the form “<name> is <description>” and “<description> is <name>” often co-occur in pretraining datasets; if the former appears in a dataset, the latter is more likely to appear.4 This is because humans often vary the order of elements in a sentence or paragraph.5 Thus, a good meta-learner would increase the probability of an instance of “<description> is <name>” after being trained on “<name> is <description>” . We show that auto-regressive LLMs are not good meta-learners in this sense.
bob1029 3y ago
This is approximately where I wound up. I had an initial knee-jerk reaction after reading the paper, but your kind of reasoning brought me back down to earth. I don't know that I even want these things to generalize. I get it in academic terms, but from a CTO perspective how much this matters to the business? We already know our policies and procedures and aren't exactly excited about the idea of something getting curious with flipping things around. I don't want it to generalize and start making weird assumptions like "some men are tall". I want it to say "I don't know", or provide a statistical signal indicating the same.
- user_named 3y ago
  Then you don't want an LLM you want something deterministic
  - bob1029 3y ago
    There is no opportunity for nuance?
    - sgt101 3y ago
      There's an opportunity to build a completely different technology. LLM's have a specific architecture that makes them good at approximate retrieval. They have a specific learning process that makes them able to do this over a vast training set. Doing things like reasoning about facts is possible using either symbolic or specific sub symbolic structures, but combining these with an LLM is a bit tricky. It's something that could be done for a demo in a 6hr/day/week project (depends on the demo) but to do it for real in an application... or as a robust bit of science... well... 6 people for 6 mths/years/?
deleted 3y ago
[deleted]
croes 3y ago
But humans know when "is" means equal.
- mistermann 3y ago
  It only takes two humans getting it correct once for your statement to be technically true though, demonstrating how easily seemingly innocuous and straightforward language can be misleading. The opposite statement, humans don't know when "is" means equal is also a true statement, perhaps even more true.
  - croes 3y ago
    But you wouldn't think of the latter group of humans as the next big thing for many important tasks. I doubt MS would put their service in lot of their products.
    - mistermann 3y ago
      > But you wouldn't think of the latter group of humans as the next big thing for many important tasks. I consider them the target audience for upgrades, I think the ROI could be massive.
bastawhiz 3y ago
> "A is B" does not imply "B is A" in the first place Not to Bill Clinton this, but I think you are considering this for the wrong interpretation of "is". In this case, "RMS is the founder of the FSF" is the sort of statement implied here. Nobody else was or ever will be the founder of the FSF. So yes, for what the author is describing, "B is A" should always hold. B, in this case, isn't a category, it's an exact identity. The author doesn't explicitly state this, but it's clear this is what they're talking about.
- mistermann 3y ago
  You are using your mind to determine what the proper usage is in this specific concrete instance, LLM's do not have your mind to perform that step. LLM's have to work at higher levels of abstraction, and there the multiple meanings of "is" (which is also very commonly used to mean it is my opinion that X "is" Y) makes sorting things out very difficult. This problem exists for both LLM's and humans, I think it may be fundamental to reality itself, as it currently is at least. I wonder if they changed the training data to consistently use "equals" where possible (even if it sounds weird to the reader) would make a difference.
  - marcosdumay 3y ago
    LLMs are supposed to derive a huge amount of those rules from their training sets. That's the one thing they are good at. They should be pretty good at differentiating the reflexive from the non-reflexive usage of "is" from the context pretty fine. The fact that the current ones don't is surprising.
  - bastawhiz 3y ago
    > there the multiple meanings of "is" (which is also very commonly used to mean it is my opinion that X "is" Y) makes sorting things out very difficult. I'm not going to hype the technology more than it needs, but it seems to do a pretty damn good job of just this. And to the author's point, it does get this right. And in fact, I'd suspect many of the cases where the LLM gets this wrong are caused by lots of examples in the training data which were written at different times, where someone else was chancellor. Or where the LLM gets numbering wrong (which is a well established problem).

by gipp 3y ago

deleted 3y ago

[deleted]

earthboundkid 3y ago

Ideally, the LLM would be able to tell the is of identity from the is of predication. I found the article a little too defensive. It felt like "of course it doesn't work, there's not enough data!" and okay, sure, but that is a flaw, no?

famouswaffles 3y ago
Well the LLM can tell in that reversal works just fine in-context. This is a recall from training issue and training is dumb.

famouswaffles 3y ago

I think a lot of people unwittingly think of the training process as "smart". Similar criticisms are "well there are many descriptions of game x it would have read so why doesn't it play x well". But gradient descent is a dumb optimizer. Training itself is not actually like someone reading a text anymore than evolution is like someone thinking about the best way to augment an organism. a "smart" optimizer would look at a reversal applicable sentence and know exactly what bunch of weights to change to store it in such a way as to be recalled reversibly in the future. Inference may be smart (GPT-4 can reverse in context just fine, potentially play games from only a description) but the training is not.

LoganDark 3y ago
A prior (now-deleted) comment of yours put this in such a great way, and I'm sad that it got deleted: > You could potentially describe how a game is played to GPT-4 without examples and get it playing it correctly but passing that same description into the training process of the model just gets you a model that can describe your game correctly.
tysam_and 3y ago
I'm not really sure if I understand the intuition here, this seems rather disconnected from what I understand to be the math of optimization. It seems like you're referring to an associative Hebbian/Hopfield-like lookup, which the current 'dumb' optimizers already do. Better yet, the learning rates are normalized by the diagonal of the empirical Fisher so that the learning w.r.t. to some estimated expected information is more constant, for said associative lookup operation. Additionally, the training loop (which you call 'dumb') is just...a teacher-forced version of inference, which you call 'smart'? It's better to simply minimize the log-likelihood in a scalable way. Hand-engineered solutions rarely survive compared to strong-scaling ones.
- famouswaffles 3y ago
  smart is in quotes here for a reason. It's 'dumb' in relation to many people's expectations. >Additionally, the training loop (which you call 'dumb') is just...a teacher-forced version of inference, which you call 'smart'? What powers In context learning is not very well understood but it doesn't appear to be or really work exactly like just a non teacher-forced version of training. There are qualitative differences. The same models have no problem with this 'curse' when the information is provided in context for example. >It's better to simply minimize the log-likelihood in a scalable way. Hand-engineered solutions rarely survive compared to strong-scaling ones. I never said anything about it being bad.
  - tysam_and 3y ago
    > What powers In context learning is not very well understood but it doesn't appear to be or really work exactly like just a non teacher-forced version of training. I mean, yes. One distills information from a training set into a compressed representation, and the other generates a compressed representation that yields (more or less) fixed state space attractors. It's just inducing a bias over the state space of the network, nothing incredibly special, though I'd consider the initial stage of training to be the most important, as it is responsible for all of the ingest of all of the embedded information the network will be using during autoregressive inference (especially w.r.t. the context of doing so during longer generations). So the notion of 'in context learning' is one I find to be a bit of an illusion, of course, as no actual learning is being done, just the induction of biases, which appears to give rise to a transiently-'better trained' network. You could see this as a bit straightforward perhaps, but I feel it needs to be said.

majormajor 3y ago

I don't think this is the right explanation, or really that relevant, the suggestions from 'og_kalu and in the article sound more accurate to me. It seems like understanding when "is" is reversible is pretty core to the capabilities of the model, but that's different than having a lot of facts memorized. For instance, a model should be able to answer "who is the star of Mission Impossible" with "Tom Cruise" based on context or internalized-training of "Tom Cruise is the star of Mission Impossible" while not answering "Tom Cruise" to a much more general question like "who is covered in water" even if it had internalized a bit from a review like "there's a scene in Mission Impossible where Tom Cruise is covered in water..." unless it had context pointing it at that specific case.

tysam_and 3y ago
It's a bit of mathematical bikeshedding, hardcoding reversability would cause far more problems than it would help. Best to simply scale log-likelihood-based training, next-token-based training trivially contains a requirement for learning all of the subproblems that predict said, next token, and hardcoding something to get warm human fuzzies would be creating a biased estimator (and move us back towards the 90s a bit). Models already very constantly do context-dependent token utilization, it's an autoregressive feature based on the entire stream of incoming tokens. Humans have a bias to focus on the 'last token used', this is not what language models look at.
- thfuran 3y ago
  >it's an autoregressive feature based on the entire stream of incoming tokens. Humans have a bias to focus on the 'last token used', this is not what language models look at. But human language is created for and by humans. Is not then operating on language in a categorically different manner an incorrect usage/understanding of language?
  - tysam_and 3y ago
    No. For more information, please see https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf https://people.math.harvard.edu/~ctm/home/text/others/shanno...

p1necone 3y ago

I don't think it's as simple as that - "the apple is red" is "single thing belongs to category", whereas "Olaf Scholz was the ninth Chancellor of Germany" is "single thing is single thing" - the latter is reversible, the former is not. I would expect a good language model to be able to parse both sentences correctly.

willsoon 3y ago
You are right. You are thinking correctly. But "the apple is red" might not mean that this particular apple belongs ---to put it in your wording--- to the category red, but that the category of things we call apple also belongs to the category of things that are red. And generally speaking, I think that is the meaning.
- School-Cotton 3y ago
  > but that the category of things we call apple also belongs to the category of things that are red No, in English this would be “apples are red”, not “the apple is red”.
  - bloak 3y ago
    I don't disagree though perhaps it's worth mentioning that sentences such as "the great spotted woodpecker is a medium-sized woodpecker" are used in English with the meaning "great spotted woodpeckers are medium-sized woodpeckers" so it seems to me that "apples are red" is grammatically a possible meaning of "the apple is red" even if it would be stylistically and pragmatically so weird that nobody would ever do that except perhaps as some kind of joke.
    - smegsicle 3y ago
      grammatically you'd capitalize Great Spotted Woodpecker, and similarly if speaking of all apples as a category, Apple should form a proper noun
    - karatinversion 3y ago

foobarqux 3y ago

But that's the problem isn't it? In some cases they are equivalent and in some cases they are not and a next-word-predictor needs to have "explicit" training data (i.e. it is not doing "reasoning") whereas a human can infer. This isn't surprising if you think about how the machine actually works rather than treating it like a sacred magic box. The default assumption for why there is any "success" for in-context learning should be that it's just picking a nearby token that "fits" not a process of logical deduction. edit: The obvious band-aid-fix is to feed reversed sentences into the training data without telling anyone, after which LLM boosters will proclaim that LLMs "learned" to reverse logical implications.

famouswaffles 3y ago
LLMs can already reverse logical implications. Literally this is not a problem in context. The model can infere all this just fine. Even the original paper makes this clear. This is a recall from training problem, not a logical inference one.
- foobarqux 3y ago
  I don't understand what you are saying. LLMs can't generally reverse implications, that's what the paper says. This in turn suggests that what you are observing in-context is some other statistical heuristic, not a "logical inference capability". I don't know in what way this is some trivial "recall" problem as you suggest. Does the reversed implication in question need to be in the training data explicitly or not? If it does I don't understand how you can claim a logical inference capability.
  - famouswaffles 3y ago
    >LLMs can't generally reverse implications, that's what the paper says No that's not what the paper says. It says training won't immediately store this information in a way as to make it reversible. Let's get one thing straight. This is a common problem for human learning as well. Anyone who has used Anki for language learning will tell you that if you just train on target language word on the front and native language word on the back, you will fail the reverse unless you specifically train for that. This is specifically a problem of recall. Not a problem of making the logical inference. If you ask the human language learner immediately he has learnt the new word for the reverse direction, he will obviously tell you the correct word. But later he may not recall the reverse even if he remembers the original. Again this is a problem of recall rather than the ability to make logical inferences. In the same vein, if you give the LLM the original direction in context and immediately ask the reverse, it will correctly tell you. It can make the logical inference.

deleted 3y ago

[deleted]

notahacker 3y ago

the examples are about relations in natural language rather than formal logic though. Mary Lee Pfeiffer being Tom Cruise's mother definitely does imply that Tom Cruise is a valid answer to questions about who Mary Lee Pfeiffer's son is (it doesn't necessarily imply she doesn't have other sons or there isn't another lady of that name who is childless, but that isn't what's tripping the model up). And there is absolutely nothing ambiguous about the failures of the fine-tuned model where the author gave it specific phrases exclusively associated with a fake name in the fine-tuning training set and when prompted for "who is $specific_phrase", supplied different names exclusively associated with completely different phrases (unlike the author, I don't see [serendipitously or otherwise] picking name words from the right training set as "B kinda-has-something-to-do-with A generalization", not when it's lost so much information that the combination of name words which exclusively appears in sentences with that phrase isn't treated as a more probable response. Never mind emergent understanding of syntax, it isn't even making obvious inferences from proximity here) If GPT4 rarely makes those errors it's clearly not an insurmountable problem at this level of training, but it does imply a lot more difficulty fine-tuning models to reliably retrieve correct information from specific text.

tysam_and 3y ago

Bit of a nitpick, "red is the apple" is both a valid sentence and one that also conveys the original relationship held by the opposite phrasing, so, in this case, "the apple is red" does indeed imply "red is the apple", and accurately so.

emme 3y ago
A proper example is probably "an apple is red" and "red is an apple".
- tikimcfee 3y ago
  What you’re missing that is “red is an apple” is also possibly saying in a metaphorical sense that red is like an apple to someone - delicious, a treat, perhaps otherwise representative. In that way, the encoding of “is”’ is exactly correct - it’s an ordered pair of glyphs that imply a weak form of assignment or description. : apologies, replied to the wrong post, meant to push this up one.
  - cryptoz 3y ago
    The correct sentence would start with a capital letter: “Red is an apple.” This is also completely valid as in a cartoon character of an apple named Red. The subject being the start of the sentence compounds the uncertainty in meaning.
    - tysam_and 3y ago
      Yes, this is because there's no assignment happening up front w.r.t. the word 'red'. However, you'll notice the same kind of ambiguity for the word 'apple' in the reversed sentence, as the word 'The' can imply quite a few things following it. The entropy gotta get slung around somehow.
eru 3y ago
Perhaps have a look at this famous syllogism instead: (1) Mortal was Socrates. (2) All humans are mortal. (3) Therefor all humans are Socrates.

anon291 3y ago

> skeptic side of the LLM spectrum Can you explain what you are skeptical of? There is ample evidence that LLMs are what they say they are: a series of transformer (usually) functions with learned weights that can successfully be used to generate human like text in many domains. Do you disbelieve this? Or are you skeptical of something else.

johnfn 3y ago

You're thinking of LLMs as basically pattern matching "a is b", but that's not really how they work. In fact, the incredible thing that LLMs can do is that they can understand (some) colloquial language and shades of meaning. LLMs can grasp all sorts of strange textual nuance; they can't grasp the cases "Foo is Bar's mother" strictly implies "Bar is Foo's son"?

famouswaffles 3y ago
>they can't grasp the cases "Foo is Bar's mother" strictly implies "Bar is Foo's son"? Correction. The Model can. Training cannot.
- TeMPOraL 3y ago
  I'm starting to wonder, why should it in the first place? Very late stages, or fine-tuning, I'd perhaps understand. But early on? Everything is possible, and "Foo is Bar's mother" doesn't imply much over "these tokens come together, sometimes". It's not that humans aren't suffering from this too. It's possible to spot it when you're learning, or even easier, when you're teaching someone a new thing. A person who learned that "A is B" does not automatically learn that "B is A"; they need to first process it, perhaps run the reversion explicitly in their head. I'd say that checking if the student can infer "B is A" having learned "A is B" is a good way to tell whether they're starting to comprehend the material, vs. just memorizing it.

ryukoposting 3y ago

The (purported) point of an LLM is its ability to represent language. The inability to represent such a critical nuance would represent a fundamental failure in that regard.

Grimblewald 3y ago

While red is the apple is not a correct logical statment using conventional understanding of each term, it is still an understandable english sentence, a structure which likely pops up a fair bit. Even if just for scripts of yoda lines. It just goes to show truly how fucky language is and how amazing it is that something even remotely comprehensible comes out of LLMs

foldr 3y ago

This is explained very clearly in the paper. They are not looking at all sentences of the form "A is B": >While it’s useful to relate the Reversal Curse to logical deduction, it’s a simplification of the full picture. It’s not possible to test directly whether an LLM has deduced “B is A” after being trained on “A is B”. LLMs are trained to predict what humans would write and not what is true (Lin et al., 2022). So even if an LLM had inferred “B is A”, it might not “tell us” when prompted. Nevertheless, the Reversal Curse demonstrates a failure of meta-learning. Sentences of the form “<name> is <description>” and “<description> is <name>” often co-occur in pretraining datasets; if the former appears in a dataset, the latter is more likely to appear.4 This is because humans often vary the order of elements in a sentence or paragraph.5 Thus, a good meta-learner would increase the probability of an instance of “<description> is <name>” after being trained on “<name> is <description>” . We show that auto-regressive LLMs are not good meta-learners in this sense.

bob1029 3y ago

This is approximately where I wound up. I had an initial knee-jerk reaction after reading the paper, but your kind of reasoning brought me back down to earth. I don't know that I even want these things to generalize. I get it in academic terms, but from a CTO perspective how much this matters to the business? We already know our policies and procedures and aren't exactly excited about the idea of something getting curious with flipping things around. I don't want it to generalize and start making weird assumptions like "some men are tall". I want it to say "I don't know", or provide a statistical signal indicating the same.

user_named 3y ago
Then you don't want an LLM you want something deterministic
- bob1029 3y ago
  There is no opportunity for nuance?
  - sgt101 3y ago
    There's an opportunity to build a completely different technology. LLM's have a specific architecture that makes them good at approximate retrieval. They have a specific learning process that makes them able to do this over a vast training set. Doing things like reasoning about facts is possible using either symbolic or specific sub symbolic structures, but combining these with an LLM is a bit tricky. It's something that could be done for a demo in a 6hr/day/week project (depends on the demo) but to do it for real in an application... or as a robust bit of science... well... 6 people for 6 mths/years/?

deleted 3y ago

[deleted]

croes 3y ago

But humans know when "is" means equal.

mistermann 3y ago
It only takes two humans getting it correct once for your statement to be technically true though, demonstrating how easily seemingly innocuous and straightforward language can be misleading. The opposite statement, humans don't know when "is" means equal is also a true statement, perhaps even more true.
- croes 3y ago
  But you wouldn't think of the latter group of humans as the next big thing for many important tasks. I doubt MS would put their service in lot of their products.
  - mistermann 3y ago
    > But you wouldn't think of the latter group of humans as the next big thing for many important tasks. I consider them the target audience for upgrades, I think the ROI could be massive.

bastawhiz 3y ago

> "A is B" does not imply "B is A" in the first place Not to Bill Clinton this, but I think you are considering this for the wrong interpretation of "is". In this case, "RMS is the founder of the FSF" is the sort of statement implied here. Nobody else was or ever will be the founder of the FSF. So yes, for what the author is describing, "B is A" should always hold. B, in this case, isn't a category, it's an exact identity. The author doesn't explicitly state this, but it's clear this is what they're talking about.

mistermann 3y ago
You are using your mind to determine what the proper usage is in this specific concrete instance, LLM's do not have your mind to perform that step. LLM's have to work at higher levels of abstraction, and there the multiple meanings of "is" (which is also very commonly used to mean it is my opinion that X "is" Y) makes sorting things out very difficult. This problem exists for both LLM's and humans, I think it may be fundamental to reality itself, as it currently is at least. I wonder if they changed the training data to consistently use "equals" where possible (even if it sounds weird to the reader) would make a difference.
- marcosdumay 3y ago
  LLMs are supposed to derive a huge amount of those rules from their training sets. That's the one thing they are good at. They should be pretty good at differentiating the reflexive from the non-reflexive usage of "is" from the context pretty fine. The fact that the current ones don't is surprising.
- bastawhiz 3y ago
  > there the multiple meanings of "is" (which is also very commonly used to mean it is my opinion that X "is" Y) makes sorting things out very difficult. I'm not going to hype the technology more than it needs, but it seems to do a pretty damn good job of just this. And to the author's point, it does get this right. And in fact, I'd suspect many of the cases where the LLM gets this wrong are caused by lots of examples in the training data which were written at different times, where someone else was chancellor. Or where the LLM gets numbering wrong (which is a well established problem).