3 ms·

I don't think this is the right explanation, or really that relevant, the suggestions from 'og_kalu and in the article sound more accurate to me. It seems like

I don't think this is the right explanation, or really that relevant, the suggestions from 'og_kalu and in the article sound more accurate to me. It seems like understanding when "is" is reversible is pretty core to the capabilities of the model, but that's different than having a lot of facts memorized. For instance, a model should be able to answer "who is the star of Mission Impossible" with "Tom Cruise" based on context or internalized-training of "Tom Cruise is the star of Mission Impossible" while not answering "Tom Cruise" to a much more general question like "who is covered in water" even if it had internalized a bit from a review like "there's a scene in Mission Impossible where Tom Cruise is covered in water..." unless it had context pointing it at that specific case.

tysam_and 3y ago
It's a bit of mathematical bikeshedding, hardcoding reversability would cause far more problems than it would help. Best to simply scale log-likelihood-based training, next-token-based training trivially contains a requirement for learning all of the subproblems that predict said, next token, and hardcoding something to get warm human fuzzies would be creating a biased estimator (and move us back towards the 90s a bit). Models already very constantly do context-dependent token utilization, it's an autoregressive feature based on the entire stream of incoming tokens. Humans have a bias to focus on the 'last token used', this is not what language models look at.
- thfuran 3y ago
  >it's an autoregressive feature based on the entire stream of incoming tokens. Humans have a bias to focus on the 'last token used', this is not what language models look at. But human language is created for and by humans. Is not then operating on language in a categorically different manner an incorrect usage/understanding of language?
  - tysam_and 3y ago
    No. For more information, please see https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf https://people.math.harvard.edu/~ctm/home/text/others/shanno...
    - thfuran 3y ago
      I'm not sure why you think that's a rebuttal.
      - tysam_and 3y ago
        It is because it is the foundation for the mathematics underpinning your question about communications systems, encodings, and their various structures. It also explains why the choice of the log likelihood is a better fit for estimating a token-based model than some other hand-crafted heuristic. The version with the Weaver introduction is quite good as well, there are other versions of similar papers covering the topic from different angles, I find it to be well-worth the read.
godelski 3y ago
I mean transformers are definitely sequentially biased. There's also human speech bias. But I think it's pretty clear that humans are generally invariant to this kind of prompting as well (given that they have the knowledge. More in a different comment). My surprisal is far higher that the reversal "curse" is considered controversial or even surprising than it was when that sensational tweet dropped. It feels pretty well studied that sequential algorithms are sequentially biased.
- aade63482 3y ago
  [flagged]
- TeMPOraL 3y ago
  > But I think it's pretty clear that humans are generally invariant to this kind of prompting as well (given that they have the knowledge. More in a different comment). But are they? Unless I'm misunderstanding what you mean here, I'd say the opposite is clearly the case - learning "A is B" doesn't automatically mean learning the reversal; you either have to learn it from external source, or infer and memorize it - both involve an extra cognitive effort. This maps to LLMs as training, or "in-context learning" and then adding the inference to the training set.
  - godelski 3y ago
    I think this would be reasonably testable actually (at least in a simplified case). Teach someone two relations that aren't meaningful naturally. Say Apples -> black holes and octopi -> industrial fan. You give them a few minutes to learn them. Then give some other unrelated tasks and say "if black holes -> apples, then what does industrial fan imply?" I feel pretty confident that humans would be able to do decently well on this task. Obviously got to account for people simply forgetting the forward relationship though. In the non-simplified case I'd actually say that we have some natural experiments, but that might spoil this as well because you can in fact say that this is training for such invariance (which I'm unsure is problematic to my claim but pushback is definitely welcome). If we're learning history we'd often learn information such as "On December 7th 1941 Japan attacked Pearl Harbor and this was a major event that got America directly involved in WW2." But then on the test we'd see questions such as "What important event happened December 7th 1941?", "What was the date of the Pearl Harbor attack?", "What major event led to the US becoming a direct player in the war and when did this event occur?" and so on. Granted, because we do such testing I'd expect GPT to be able to reasonably answer many of these same questions but I'm very much under the impression that humans are far more robust/generalized and would quickly learn these reversals. And to stress, conditioned on knowing the forward direction to a strong degree. If you're fuzzy about the forward direction I'm not surprised if the reversal is even more fuzzy. Definitely biased. But more importantly, I do not think we have the same strong algorithmic bias in our encoding methods that transformers have. That's my main claim, just to clarify. But I'm conjecturing a lot here. I think these are reasonable and we have some strong evidence for this. Maybe this is more situational than I'm presuming and I'm definitely working off of biased sampling. So discussion helps refine the ideas.

tysam_and 3y ago

It's a bit of mathematical bikeshedding, hardcoding reversability would cause far more problems than it would help. Best to simply scale log-likelihood-based training, next-token-based training trivially contains a requirement for learning all of the subproblems that predict said, next token, and hardcoding something to get warm human fuzzies would be creating a biased estimator (and move us back towards the 90s a bit). Models already very constantly do context-dependent token utilization, it's an autoregressive feature based on the entire stream of incoming tokens. Humans have a bias to focus on the 'last token used', this is not what language models look at.

thfuran 3y ago
>it's an autoregressive feature based on the entire stream of incoming tokens. Humans have a bias to focus on the 'last token used', this is not what language models look at. But human language is created for and by humans. Is not then operating on language in a categorically different manner an incorrect usage/understanding of language?
- tysam_and 3y ago
  No. For more information, please see https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf https://people.math.harvard.edu/~ctm/home/text/others/shanno...
  - thfuran 3y ago
    I'm not sure why you think that's a rebuttal.
    - tysam_and 3y ago
      It is because it is the foundation for the mathematics underpinning your question about communications systems, encodings, and their various structures. It also explains why the choice of the log likelihood is a better fit for estimating a token-based model than some other hand-crafted heuristic. The version with the Weaver introduction is quite good as well, there are other versions of similar papers covering the topic from different angles, I find it to be well-worth the read.

godelski 3y ago

I mean transformers are definitely sequentially biased. There's also human speech bias. But I think it's pretty clear that humans are generally invariant to this kind of prompting as well (given that they have the knowledge. More in a different comment). My surprisal is far higher that the reversal "curse" is considered controversial or even surprising than it was when that sensational tweet dropped. It feels pretty well studied that sequential algorithms are sequentially biased.

aade63482 3y ago
[flagged]
TeMPOraL 3y ago
> But I think it's pretty clear that humans are generally invariant to this kind of prompting as well (given that they have the knowledge. More in a different comment). But are they? Unless I'm misunderstanding what you mean here, I'd say the opposite is clearly the case - learning "A is B" doesn't automatically mean learning the reversal; you either have to learn it from external source, or infer and memorize it - both involve an extra cognitive effort. This maps to LLMs as training, or "in-context learning" and then adding the inference to the training set.
- godelski 3y ago
  I think this would be reasonably testable actually (at least in a simplified case). Teach someone two relations that aren't meaningful naturally. Say Apples -> black holes and octopi -> industrial fan. You give them a few minutes to learn them. Then give some other unrelated tasks and say "if black holes -> apples, then what does industrial fan imply?" I feel pretty confident that humans would be able to do decently well on this task. Obviously got to account for people simply forgetting the forward relationship though. In the non-simplified case I'd actually say that we have some natural experiments, but that might spoil this as well because you can in fact say that this is training for such invariance (which I'm unsure is problematic to my claim but pushback is definitely welcome). If we're learning history we'd often learn information such as "On December 7th 1941 Japan attacked Pearl Harbor and this was a major event that got America directly involved in WW2." But then on the test we'd see questions such as "What important event happened December 7th 1941?", "What was the date of the Pearl Harbor attack?", "What major event led to the US becoming a direct player in the war and when did this event occur?" and so on. Granted, because we do such testing I'd expect GPT to be able to reasonably answer many of these same questions but I'm very much under the impression that humans are far more robust/generalized and would quickly learn these reversals. And to stress, conditioned on knowing the forward direction to a strong degree. If you're fuzzy about the forward direction I'm not surprised if the reversal is even more fuzzy. Definitely biased. But more importantly, I do not think we have the same strong algorithmic bias in our encoding methods that transformers have. That's my main claim, just to clarify. But I'm conjecturing a lot here. I think these are reasonable and we have some strong evidence for this. Maybe this is more situational than I'm presuming and I'm definitely working off of biased sampling. So discussion helps refine the ideas.