3 ms·

GPT-3 proves that it is possible. It can reason about and answer novel questions just by looking at huge amounts of text.

rvense 5y ago
GPT-3 does not understand language like you and I do with reference to the world. It has no senses, so it is not possible for it to do so. It is also to my knowledge not able to translate anything, so even if we were able to make it "learn" Harrappan writing from these seals, it would not be able to generate English text from them, because the information required to do so does not exist. To reply to your other comment, rabbit and rocks would occur in different contexts, but you have no way of knowing which is which. You can get numbers, probabilities and mutual information scores and vectors and what have you, but if you can't ground them in anything, you have no way of getting to meaning, and therefore no way of translating.
- eru 5y ago
  > You can get numbers, probabilities and mutual information scores and vectors and what have you, but if you can't ground them in anything, you have no way of getting to meaning, and therefore no way of translating. But you have a grounding! You know it's a language used by humans.
- petters 5y ago
  > It is also to my knowledge not able to translate anything It translates very well actually. I've tried with English/French and English/German with very good results. I think it is quite well understood that deciphering a language without external information is possible? But GPT-3 was a recent example that I think proves it. There are probably tons of literature about this question. I think aliens would be able to understand lots of things about Earth just by listening in. > To reply to your other comment, rabbit and rocks would occur in different contexts, but you have no way of knowing which is which. You can get numbers, probabilities and mutual information scores and vectors and what have you, but if you can't ground them in anything, you have no way of getting to meaning, and therefore no way of translating. Well, GPT-3 translates very well, so this argument can not be correct. I don't see why it would be like that? It seems quite intuitive that it would be possible to extract meaning.
  - rvense 5y ago
    GPT-3 trained on only English can produce French text? Citation please! I've looked for one, but I can't find it. I can, however, find more than a little information suggesting that GPT-3 is trained on multilingual data. > I think it is quite well understood that deciphering a language without external information is possible? I don't think this is understood at all, and I would have to ask you for just one example of it. I would, in fact, say it is categorically impossible, if by "no external information" you mean that we have only text or sound. The reason that it is not possible to extract meaning is that meaning is a substance that is not present in the text. It is more than the relations between words, in that it is connected to the world.
    - rvense 5y ago
      I am looking more into the exact nature of the multilingual data used by GPT-3 and it's a little hard. The main paper mostly focuses on the algorithms which is less interesting to me. The key is, if you have aligned or parallel texts, then you can use structural similarities in the lexicon to bootstrap a mapping from one language to another without referring to substantive meaning (which I will maintain is how humans do translation), even if you don't have a dictionary as such. This is not surprising, but I would expect it to have severe limits.

rvense 5y ago

GPT-3 does not understand language like you and I do with reference to the world. It has no senses, so it is not possible for it to do so. It is also to my knowledge not able to translate anything, so even if we were able to make it "learn" Harrappan writing from these seals, it would not be able to generate English text from them, because the information required to do so does not exist. To reply to your other comment, rabbit and rocks would occur in different contexts, but you have no way of knowing which is which. You can get numbers, probabilities and mutual information scores and vectors and what have you, but if you can't ground them in anything, you have no way of getting to meaning, and therefore no way of translating.

eru 5y ago
> You can get numbers, probabilities and mutual information scores and vectors and what have you, but if you can't ground them in anything, you have no way of getting to meaning, and therefore no way of translating. But you have a grounding! You know it's a language used by humans.
petters 5y ago
> It is also to my knowledge not able to translate anything It translates very well actually. I've tried with English/French and English/German with very good results. I think it is quite well understood that deciphering a language without external information is possible? But GPT-3 was a recent example that I think proves it. There are probably tons of literature about this question. I think aliens would be able to understand lots of things about Earth just by listening in. > To reply to your other comment, rabbit and rocks would occur in different contexts, but you have no way of knowing which is which. You can get numbers, probabilities and mutual information scores and vectors and what have you, but if you can't ground them in anything, you have no way of getting to meaning, and therefore no way of translating. Well, GPT-3 translates very well, so this argument can not be correct. I don't see why it would be like that? It seems quite intuitive that it would be possible to extract meaning.
- rvense 5y ago
  GPT-3 trained on only English can produce French text? Citation please! I've looked for one, but I can't find it. I can, however, find more than a little information suggesting that GPT-3 is trained on multilingual data. > I think it is quite well understood that deciphering a language without external information is possible? I don't think this is understood at all, and I would have to ask you for just one example of it. I would, in fact, say it is categorically impossible, if by "no external information" you mean that we have only text or sound. The reason that it is not possible to extract meaning is that meaning is a substance that is not present in the text. It is more than the relations between words, in that it is connected to the world.
  - rvense 5y ago
    I am looking more into the exact nature of the multilingual data used by GPT-3 and it's a little hard. The main paper mostly focuses on the algorithms which is less interesting to me. The key is, if you have aligned or parallel texts, then you can use structural similarities in the lexicon to bootstrap a mapping from one language to another without referring to substantive meaning (which I will maintain is how humans do translation), even if you don't have a dictionary as such. This is not surprising, but I would expect it to have severe limits.