5 ms·

It seems like there's a hard limit on how well this approach can work, defined by the extent to which the distribution of words in texts uniquely identifies the

It seems like there's a hard limit on how well this approach can work, defined by the extent to which the distribution of words in texts uniquely identifies their meaning without reference to any external semantics. With monolingual corpora and no semantic grounding, all you can say is that certain words (or letters, morphemes, or whatever) covary in certain ways that are similar to the ways in which other words covary in the other language. It's super interesting that this is enough to identify some translation pairs across languages.

ragebol 9y ago
Very interesting. Can a language define all it's concepts with language alone? I.e Can a language map all it's concepts to words and other concepts defined by just words. Is there some sort of Godel's incompleteness theorem for language?
- canjobear 9y ago
  > Can a language map all it's concepts to words and other concepts defined by just words. I don't see why natural language would have this property. It is certainly possible to construct codes such that the distribution over codewords does not have this property.
  - ragebol 9y ago
    I also don't see why it would, but then that would of course be a limit to the idea in the paper. I'm amazed that what the paper shows even works.
    - letlambda 9y ago
      It seems extremely unlikely to me that it wouldn't. I conjecture that no such word exists in any widely used human language. This would be a word who's meaning is not contained in the span of meanings of all other words in the language. Can you think of one in english (or your native language)? You won't find it in any dictionary, since all those words have descriptions in terms of other words.
      - pechay 9y ago
        Gullible?
        letlambda 9y ago
        easily deceived
      - ragebol 9y ago
        The example I've been thinking about was 'pain', which, according to Wikipedia, is "an unpleasant sensory and emotional experience associated with actual or potential tissue damage." The native Dutch word "Gezelligheid" has no direct English translation and thus apparently has a wiki page in English [0]. Now I wonder what the relevance is of the distance between word vectors from different languages relating to nearly the same concept (e.g Gezelligheid to Cozyness) [0]: https://en.wikipedia.org/wiki/Gezelligheid https://en.wikipedia.org/wiki/Gezelligheid
- mike_n 9y ago
  actually, it's theoretically a hard problem -- https://aclanthology.info/pdf/J/J86/J86-4003.pdf https://aclanthology.info/pdf/J/J86/J86-4003.pdf I would have thought that they needed to 'ground' (https://en.wikipedia.org/wiki/Symbol_grounding_problem https://en.wikipedia.org/wiki/Symbol_grounding_problem) at least one shared reference word/concept so they could proceed to define everything relative to that and thereby communicate effectively and know how to map different languages to the same latent space. But I guess perhaps one can infer/approximate it statistically -- https://cis.upenn.edu/~ccb/publications/discriminative-bilingual-lexicon-induction.pdf https://cis.upenn.edu/~ccb/publications/discriminative-bilin...
OscarCunningham 9y ago
In particular I wonder if it can get "left" and "right" the correct way around.
- d--b 9y ago
  Excellent example!
- schoen 9y ago
  I'd imagine you can potentially get that by connections with strength, weakness, dexterity, etc.
  - harperlee 9y ago
    I'm not sure I understand you... But the parent's comment is quite interesting, how could the system end up with a correct left/right understanding or our world, instead of a mirror world? Without experiencing the outside world, it seems impossible!
    - pathsjs 9y ago
      The remark is that the two words, while apparently interchangeable, in actual use are not. One is often used with positive meaning related to dexterity and so on. Also, they have different political meanings. So they are not at all symmetric in actual use, much like every other word pair
      - OscarCunningham 9y ago
        Not all of those differences are cross-cultural.
        pathsjs 9y ago
        This is a good point. Still, for instance, the binomial left/right to indicate political position is common across many languages

ragebol 9y ago

Very interesting. Can a language define all it's concepts with language alone? I.e Can a language map all it's concepts to words and other concepts defined by just words. Is there some sort of Godel's incompleteness theorem for language?

canjobear 9y ago
> Can a language map all it's concepts to words and other concepts defined by just words. I don't see why natural language would have this property. It is certainly possible to construct codes such that the distribution over codewords does not have this property.
- ragebol 9y ago
  I also don't see why it would, but then that would of course be a limit to the idea in the paper. I'm amazed that what the paper shows even works.
  - letlambda 9y ago
    It seems extremely unlikely to me that it wouldn't. I conjecture that no such word exists in any widely used human language. This would be a word who's meaning is not contained in the span of meanings of all other words in the language. Can you think of one in english (or your native language)? You won't find it in any dictionary, since all those words have descriptions in terms of other words.
    - pechay 9y ago
      Gullible?
      - letlambda 9y ago
        easily deceived
    - ragebol 9y ago
      The example I've been thinking about was 'pain', which, according to Wikipedia, is "an unpleasant sensory and emotional experience associated with actual or potential tissue damage." The native Dutch word "Gezelligheid" has no direct English translation and thus apparently has a wiki page in English [0]. Now I wonder what the relevance is of the distance between word vectors from different languages relating to nearly the same concept (e.g Gezelligheid to Cozyness) [0]: https://en.wikipedia.org/wiki/Gezelligheid https://en.wikipedia.org/wiki/Gezelligheid
mike_n 9y ago
actually, it's theoretically a hard problem -- https://aclanthology.info/pdf/J/J86/J86-4003.pdf https://aclanthology.info/pdf/J/J86/J86-4003.pdf I would have thought that they needed to 'ground' (https://en.wikipedia.org/wiki/Symbol_grounding_problem https://en.wikipedia.org/wiki/Symbol_grounding_problem) at least one shared reference word/concept so they could proceed to define everything relative to that and thereby communicate effectively and know how to map different languages to the same latent space. But I guess perhaps one can infer/approximate it statistically -- https://cis.upenn.edu/~ccb/publications/discriminative-bilingual-lexicon-induction.pdf https://cis.upenn.edu/~ccb/publications/discriminative-bilin...

OscarCunningham 9y ago

In particular I wonder if it can get "left" and "right" the correct way around.

d--b 9y ago
Excellent example!
schoen 9y ago
I'd imagine you can potentially get that by connections with strength, weakness, dexterity, etc.
- harperlee 9y ago
  I'm not sure I understand you... But the parent's comment is quite interesting, how could the system end up with a correct left/right understanding or our world, instead of a mirror world? Without experiencing the outside world, it seems impossible!
  - pathsjs 9y ago
    The remark is that the two words, while apparently interchangeable, in actual use are not. One is often used with positive meaning related to dexterity and so on. Also, they have different political meanings. So they are not at all symmetric in actual use, much like every other word pair
    - OscarCunningham 9y ago
      Not all of those differences are cross-cultural.
      - pathsjs 9y ago
        This is a good point. Still, for instance, the binomial left/right to indicate political position is common across many languages