3 ms·
“A is B” doesn’t generally entail “B is A”. “A square is a rectangle” does not entail “a rectangle is a square”. Similarly, “Socrates is alive” doesn’t entail
by zmgsabst 4mo ago
“A is B” doesn’t generally entail “B is A”.
“A square is a rectangle” does not entail “a rectangle is a square”.
Similarly, “Socrates is alive” doesn’t entail “alive is Socrates”.
Notably, they mention when context is included, LLM performance rises — ie, exactly when we include extra information that allows it to recognize what kind of information is being conveyed.
But the LLM is correct not to generalize that pattern when it doesn’t generalize — even if researchers have salient example, but ignore contrary ones (eg, square-rectangle or Socrates-alive).
- WithinReason 4mo agoThat completely misses the point. The point is that "Valentina Tereshkova was the first woman to travel to space" does imply "The first woman to travel to space was Valentina Tereshkova", which LLMs fail to recognise.
- IshKebab 4mo agoRight, but that could be because the fact that that implication exists is not actually as trivial as they are implying. Does "Flargbler was blorglargh" imply "blorglargh was Flargbler"? Maybe. You need more context to know.
- 1718627440 4mo ago> You need more context to know. Yes, which is what the model is. That's the point here, it doesn't has or accesses this context, even though it should.
- IshKebab 4mo agoIt seems to me that the point was usually "these models are fundamentally not intelligent because they have this incredibly dumb failure modes" rather than "these models happen to have this interesting weakness". Maybe the original authors didn't make that mistake, but that was certainly the case whenever it was brought up by HN commenters of the "stochastic parrot" persuasion.
- Dibby053 4mo ago"A is the B" does entail "the B is A", because "the" establishes an identity/bijection.