6 ms·

Locality is just how the embeddings are created, and at that only one way. “Ball” will also be closer to “round” and “net” will be closer to “string”. So this i

Locality is just how the embeddings are created, and at that only one way. “Ball” will also be closer to “round” and “net” will be closer to “string”. So this is the creation process, but you end up with something that sure does look a lot like meaning. You can do math with these vectors, like addition. king - man + woman = queen is the classic Word2Vec example. Based on you calling that phrase earlier “word salad”, I think you’re just not that familiar with word embeddings. https://en.m.wikipedia.org/wiki/Word_embedding https://en.m.wikipedia.org/wiki/Word_embedding

dvt 4y ago
> but you end up with something that sure does look a lot like meaning I suggest you read some Quine and Kripke; the argument that word embeddings encode meaning (particularly via locality) is an incredibly philosophically-naive position. People have been debating what language is for a very long time.
- travisjungroth 4y ago
  Wait, how are you holding this position and telling me what to read when you couldn’t understand what “high dimensional representation of meaning” meant a few hours ago? Maybe the thing that will land is to say that embeddings are an attempt at representing meaning. There’s lots of other stuff you could represent about words. You could represent popularity by giving each word a number between 0 and 1 representing what portion of the training text was that word. That would tell you if a text might be harder for someone new to the language to understand. You could represent the spelling of the word as a list of integers. You could represent the pronunciation difficulty with a score based on ambiguous and infrequent n-grams. Or, you could try to represent the concept that the word is pointing to so you can follow synonyms or categorize text by topic. This would be an attempt at representing the word’s meaning. I’ve been explaining as if you didn’t understand what this was, but if you disagree that’s another thing.
  - YeGoblynQueenne 4y ago
    >> Wait, how are you holding this position and telling me what to read when you couldn’t understand what “high dimensional representation of meaning” meant a few hours ago? From the conversation so far it's clear to me that the OP has at least some idea of how word embeddings work and they described as "word salad" the expression that related them to "meaning", i.e. they quibbled about the "... representation of meaning", not the "high dimensional representation... " part. I hope the OP can give a reference to the Quine and Kripke source they refer to. As far as I remember it, the point about arithmetic with word verctors (man, king, woman, queen and all that) is a motivational example for Word2Vec by Tomas Mikolov, which wikpedia tells me was published in 2013. I can't find the reference right now, but in 2014 I was taking a Master's in all that jazz and I remember one of my tutors having the paper on his desk, and commenting that the claim that word embeddings model meaning was "well, that's what he says". So there has been debate and disagreement about the ability of word embeddings to represent meaning for at least as long as Word2Vec has existed, and as far as I know the idea of word embeddings existed earlier than that (I was taught the concept unconnected to word2vec in my Master's). The bottom line is, just because somebody says their algorithm does a thing, doesn't mean that everyone has to accept it immediately, and without critical discussion. So I, for one, want to see that Quine and Kripke stuff the OP is referring to.
    - travisjungroth 4y ago
      > The bottom line is, just because somebody says their algorithm does a thing, doesn't mean that everyone has to accept it immediately, and without critical discussion. That critical discussion was 10 years ago. Embeddings are vector representations of meaning. They’re not perfect, but they can be used for enough meaning-dependent tasks that this statement shouldn’t be controversial. All this AI/ML stuff is so exhausting because the philosophers come out of the woodwork. They don’t prove anything or disprove anything, they just kinda whine. When I was a flight instructor and said the airplane wanted to float down the runway, no one tried to point out that airplanes don’t have desires. I can say a server knows another server is offline and that’s fine too. But say “reason”, “think”, or “meaning” with something AI related and it’s like all work has to stop until we stop using that precious word. I’ve tried to engage with that stuff less, but got tricked cause I thought this person just literally didn’t understand.
zwaps 4y ago
Embeddings can go beyond a contextual definition of meaning. You can use (transformer) LLMs to get a "word sense" aka its predicted probability distribution, which also induces a network of associations (not via context, but via whatever the LLM has learned as fitting words). This gives a structural definition of meaning the ye olde sense (v. Humboldt and so forth). This of course starts with the assumption that LLMs can encode meaning, albeit not in a simplistic linear fashion as Word2Vec or GloVe did https://arxiv.org/abs/2110.04151 https://arxiv.org/abs/2110.04151

dvt 4y ago

> but you end up with something that sure does look a lot like meaning I suggest you read some Quine and Kripke; the argument that word embeddings encode meaning (particularly via locality) is an incredibly philosophically-naive position. People have been debating what language is for a very long time.

travisjungroth 4y ago
Wait, how are you holding this position and telling me what to read when you couldn’t understand what “high dimensional representation of meaning” meant a few hours ago? Maybe the thing that will land is to say that embeddings are an attempt at representing meaning. There’s lots of other stuff you could represent about words. You could represent popularity by giving each word a number between 0 and 1 representing what portion of the training text was that word. That would tell you if a text might be harder for someone new to the language to understand. You could represent the spelling of the word as a list of integers. You could represent the pronunciation difficulty with a score based on ambiguous and infrequent n-grams. Or, you could try to represent the concept that the word is pointing to so you can follow synonyms or categorize text by topic. This would be an attempt at representing the word’s meaning. I’ve been explaining as if you didn’t understand what this was, but if you disagree that’s another thing.
- YeGoblynQueenne 4y ago
  >> Wait, how are you holding this position and telling me what to read when you couldn’t understand what “high dimensional representation of meaning” meant a few hours ago? From the conversation so far it's clear to me that the OP has at least some idea of how word embeddings work and they described as "word salad" the expression that related them to "meaning", i.e. they quibbled about the "... representation of meaning", not the "high dimensional representation... " part. I hope the OP can give a reference to the Quine and Kripke source they refer to. As far as I remember it, the point about arithmetic with word verctors (man, king, woman, queen and all that) is a motivational example for Word2Vec by Tomas Mikolov, which wikpedia tells me was published in 2013. I can't find the reference right now, but in 2014 I was taking a Master's in all that jazz and I remember one of my tutors having the paper on his desk, and commenting that the claim that word embeddings model meaning was "well, that's what he says". So there has been debate and disagreement about the ability of word embeddings to represent meaning for at least as long as Word2Vec has existed, and as far as I know the idea of word embeddings existed earlier than that (I was taught the concept unconnected to word2vec in my Master's). The bottom line is, just because somebody says their algorithm does a thing, doesn't mean that everyone has to accept it immediately, and without critical discussion. So I, for one, want to see that Quine and Kripke stuff the OP is referring to.
  - travisjungroth 4y ago
    > The bottom line is, just because somebody says their algorithm does a thing, doesn't mean that everyone has to accept it immediately, and without critical discussion. That critical discussion was 10 years ago. Embeddings are vector representations of meaning. They’re not perfect, but they can be used for enough meaning-dependent tasks that this statement shouldn’t be controversial. All this AI/ML stuff is so exhausting because the philosophers come out of the woodwork. They don’t prove anything or disprove anything, they just kinda whine. When I was a flight instructor and said the airplane wanted to float down the runway, no one tried to point out that airplanes don’t have desires. I can say a server knows another server is offline and that’s fine too. But say “reason”, “think”, or “meaning” with something AI related and it’s like all work has to stop until we stop using that precious word. I’ve tried to engage with that stuff less, but got tricked cause I thought this person just literally didn’t understand.

zwaps 4y ago

Embeddings can go beyond a contextual definition of meaning. You can use (transformer) LLMs to get a "word sense" aka its predicted probability distribution, which also induces a network of associations (not via context, but via whatever the LLM has learned as fitting words). This gives a structural definition of meaning the ye olde sense (v. Humboldt and so forth). This of course starts with the assumption that LLMs can encode meaning, albeit not in a simplistic linear fashion as Word2Vec or GloVe did https://arxiv.org/abs/2110.04151 https://arxiv.org/abs/2110.04151