3 ms·

Figuring out how to process context is important for NLP, no question. But I think this is probably wrong on Wittgenstein. I'm pretty sure his entire point in

by akozak 8y ago

Figuring out how to process context is important for NLP, no question. But I think this is probably wrong on Wittgenstein. I'm pretty sure his entire point in the Philosophical Investigations was that "meaning" is exactly NOT probabilities of symbol co-occurrence, or just names of objects in the world. Symbols acquire meanings from their use by humans. Accounting for context in NLP via probabilities of occurrence might be useful in better reproducing language, but we should be careful not to say that this is the essence of meaning and language.

whatshisface 8y ago
>Accounting for context in NLP via probabilities of occurrence might be useful in better reproducing language, but we should be careful not to say that this is the essence of meaning and language. Yes, and the article actually includes evidence in favor of this and against its own conclusion. It mentions that vector "cat" is closer to vector "dog" than vector "dog" is to vector "dogs," which makes sense if you interpret it as a measure of appearance in sentences but no sense at all if you force it into the mold of "the meaning of words."
- akozak 8y ago
  Yes - but for me it was this paragraph: > And it’s now quite clear where the Wittgenstein’s theories jump in: context is crucial to learn the embeddings as it’s crucial in his theories to attach meaning. In the same way as two words have similar meanings they will have similar representations (small distance in the N-dimensional space) just because they often appear in similar contexts. So “cat” and “dog” will end up having close vectors because they often appear in the same contexts: it’s useful for the model to use for them similar embeddings because it’s the most convenient thing it can do to have better performances in predicting the two words given their contexts. I am actually fine to say that this approach is useful and convenient - and that we can fairly call measuring probabilities of co-occurrence measuring "context" in some sense. But "context" for Wittgenstein in his account of meaning was clearly not word or symbol occurrences. It was a much broader view of the way that language fits in with human intentions and behavior and the wide variety of uses for a word. I hate to quote Wikipedia, but from the PI article: "Wittgenstein argues that definitions emerge from what he termed "forms of life", roughly the culture and society in which they are used." https://en.wikipedia.org/wiki/Philosophical_Investigations#Meaning_and_definition https://en.wikipedia.org/wiki/Philosophical_Investigations#M...
  - kolbe 8y ago
    I'm in total agreement. It's especially confusing that the author would go through the bother of invoking Wittgenstein when it seems like he meant the exact opposite.
- adrianratnapala 8y ago
  Really? The words "dogs" refers to something like a set whereas "dog" (usually) refers to an individual. Thus they have very different meanings, even if those meanings have an obvious and simple relationship. On the other hand "cat" and "dog" refer to individuals that are similar in many ways (and interchangeable for some common purposes). That is, these two words have somewhat similar meanings.
- empath75 8y ago
  Two strings are more alike than a string and an array of strings.
  - goatlover 8y ago
    A dog and cats aren't data structures, they're a singular animal of a biological family and a multiple animals of another. Data structures belong to a different language game, which is programming jargon.
    - empath75 8y ago
      A representation of a dog or cat in your mind is a data structure of a kind. Embeddings are about the names, not the named. How about this? Is a drop of water more like a grain of sand or more like an ocean?
      - goatlover 8y ago
        > A representation of a dog or cat in your mind is a data structure of a kind. Embeddings are about the names, not the named. The later Wittgenstein's philosophy was about how meaning is language usage. As such, it was not interested in the names, which are arbitrary rules of the language game. In ordinary speech, talk of dog and cats is not about data structures or representations. That sort of talk takes place in a specialized language game like computer or cognitive science. As such, it's an abuse of language to reify the usage in one language game for all language games, as if a word had a universal meaning independent of the game.
- dwaltrip 8y ago
  It is not obvious to me which pair is more similar: (cat, dog) or (dog, dogs). I am leaning towards (cat, dog) being more similar. However, I think it could easily change depending on the context.

by akozak 8y ago

whatshisface 8y ago

>Accounting for context in NLP via probabilities of occurrence might be useful in better reproducing language, but we should be careful not to say that this is the essence of meaning and language. Yes, and the article actually includes evidence in favor of this and against its own conclusion. It mentions that vector "cat" is closer to vector "dog" than vector "dog" is to vector "dogs," which makes sense if you interpret it as a measure of appearance in sentences but no sense at all if you force it into the mold of "the meaning of words."

akozak 8y ago
Yes - but for me it was this paragraph: > And it’s now quite clear where the Wittgenstein’s theories jump in: context is crucial to learn the embeddings as it’s crucial in his theories to attach meaning. In the same way as two words have similar meanings they will have similar representations (small distance in the N-dimensional space) just because they often appear in similar contexts. So “cat” and “dog” will end up having close vectors because they often appear in the same contexts: it’s useful for the model to use for them similar embeddings because it’s the most convenient thing it can do to have better performances in predicting the two words given their contexts. I am actually fine to say that this approach is useful and convenient - and that we can fairly call measuring probabilities of co-occurrence measuring "context" in some sense. But "context" for Wittgenstein in his account of meaning was clearly not word or symbol occurrences. It was a much broader view of the way that language fits in with human intentions and behavior and the wide variety of uses for a word. I hate to quote Wikipedia, but from the PI article: "Wittgenstein argues that definitions emerge from what he termed "forms of life", roughly the culture and society in which they are used." https://en.wikipedia.org/wiki/Philosophical_Investigations#Meaning_and_definition https://en.wikipedia.org/wiki/Philosophical_Investigations#M...
- kolbe 8y ago
  I'm in total agreement. It's especially confusing that the author would go through the bother of invoking Wittgenstein when it seems like he meant the exact opposite.
adrianratnapala 8y ago
Really? The words "dogs" refers to something like a set whereas "dog" (usually) refers to an individual. Thus they have very different meanings, even if those meanings have an obvious and simple relationship. On the other hand "cat" and "dog" refer to individuals that are similar in many ways (and interchangeable for some common purposes). That is, these two words have somewhat similar meanings.
empath75 8y ago
Two strings are more alike than a string and an array of strings.
- goatlover 8y ago
  A dog and cats aren't data structures, they're a singular animal of a biological family and a multiple animals of another. Data structures belong to a different language game, which is programming jargon.
  - empath75 8y ago
    A representation of a dog or cat in your mind is a data structure of a kind. Embeddings are about the names, not the named. How about this? Is a drop of water more like a grain of sand or more like an ocean?
    - goatlover 8y ago
      > A representation of a dog or cat in your mind is a data structure of a kind. Embeddings are about the names, not the named. The later Wittgenstein's philosophy was about how meaning is language usage. As such, it was not interested in the names, which are arbitrary rules of the language game. In ordinary speech, talk of dog and cats is not about data structures or representations. That sort of talk takes place in a specialized language game like computer or cognitive science. As such, it's an abuse of language to reify the usage in one language game for all language games, as if a word had a universal meaning independent of the game.
dwaltrip 8y ago
It is not obvious to me which pair is more similar: (cat, dog) or (dog, dogs). I am leaning towards (cat, dog) being more similar. However, I think it could easily change depending on the context.