4 ms·

> but you end up with something that sure does look a lot like meaning I suggest you read some Quine and Kripke; the argument that word embeddings encode meani

by dvt 4y ago

> but you end up with something that sure does look a lot like meaning I suggest you read some Quine and Kripke; the argument that word embeddings encode meaning (particularly via locality) is an incredibly philosophically-naive position. People have been debating what language is for a very long time.

travisjungroth 4y ago
Wait, how are you holding this position and telling me what to read when you couldn’t understand what “high dimensional representation of meaning” meant a few hours ago? Maybe the thing that will land is to say that embeddings are an attempt at representing meaning. There’s lots of other stuff you could represent about words. You could represent popularity by giving each word a number between 0 and 1 representing what portion of the training text was that word. That would tell you if a text might be harder for someone new to the language to understand. You could represent the spelling of the word as a list of integers. You could represent the pronunciation difficulty with a score based on ambiguous and infrequent n-grams. Or, you could try to represent the concept that the word is pointing to so you can follow synonyms or categorize text by topic. This would be an attempt at representing the word’s meaning. I’ve been explaining as if you didn’t understand what this was, but if you disagree that’s another thing.
- YeGoblynQueenne 4y ago
  >> Wait, how are you holding this position and telling me what to read when you couldn’t understand what “high dimensional representation of meaning” meant a few hours ago? From the conversation so far it's clear to me that the OP has at least some idea of how word embeddings work and they described as "word salad" the expression that related them to "meaning", i.e. they quibbled about the "... representation of meaning", not the "high dimensional representation... " part. I hope the OP can give a reference to the Quine and Kripke source they refer to. As far as I remember it, the point about arithmetic with word verctors (man, king, woman, queen and all that) is a motivational example for Word2Vec by Tomas Mikolov, which wikpedia tells me was published in 2013. I can't find the reference right now, but in 2014 I was taking a Master's in all that jazz and I remember one of my tutors having the paper on his desk, and commenting that the claim that word embeddings model meaning was "well, that's what he says". So there has been debate and disagreement about the ability of word embeddings to represent meaning for at least as long as Word2Vec has existed, and as far as I know the idea of word embeddings existed earlier than that (I was taught the concept unconnected to word2vec in my Master's). The bottom line is, just because somebody says their algorithm does a thing, doesn't mean that everyone has to accept it immediately, and without critical discussion. So I, for one, want to see that Quine and Kripke stuff the OP is referring to.
  - travisjungroth 4y ago
    > The bottom line is, just because somebody says their algorithm does a thing, doesn't mean that everyone has to accept it immediately, and without critical discussion. That critical discussion was 10 years ago. Embeddings are vector representations of meaning. They’re not perfect, but they can be used for enough meaning-dependent tasks that this statement shouldn’t be controversial. All this AI/ML stuff is so exhausting because the philosophers come out of the woodwork. They don’t prove anything or disprove anything, they just kinda whine. When I was a flight instructor and said the airplane wanted to float down the runway, no one tried to point out that airplanes don’t have desires. I can say a server knows another server is offline and that’s fine too. But say “reason”, “think”, or “meaning” with something AI related and it’s like all work has to stop until we stop using that precious word. I’ve tried to engage with that stuff less, but got tricked cause I thought this person just literally didn’t understand.
    - YeGoblynQueenne 4y ago
      I hope you're not calling me a "philosopher". I identify as a coder. And what I want to know is where is the code for "meaning". I don't care why it's the code for "meaning", but if you point me to the code for "word embeddings" and you say "that's the code for 'meaning", then I'm going to wonder whether, if I ask for the code for "quicksort" you'll point me to the code for "bubblesort". So, no, what we call things that do things is important, otherwise we don't know what things do what, and what things are being done. And lest we forget: However, in AI, our programs to a great degree are problems rather than solutions. If a researcher tries to write an "understanding" program, it isn't because he has thought of a better way of implementing this well-understood task, but because he thinks he can come closer to writing the implementation. If he calls the main loop of his program "UNDERSTAND', he i s (until proven innocent) merely begging the question. He may mislead a lot of people, most prominently himself, and enrage a lot of others. What he should do instead is refer to this main loop as "G0034", and see if he can convince himself or anyone else that G0034 implements some part of understanding. Or he could give i t a name that reveals its intrinsic properties, like NODE-NET- INTERSECTION-FINDER, it being the substance of his theory that finding intersections in networks of nodes constitutes understanding. If Quillian <1969> had called his program the "Teachable Language Node Net Intersection Finder", he would have saved us some reading. (Except for those of us fanatic about finding the part on teachability.) https://cs.fit.edu/~kgallagher/Schtick/Serious/McDermott.AI.MeetsNaturalStupidity.pdf https://cs.fit.edu/~kgallagher/Schtick/Serious/McDermott.AI.... Edit: And this is not right at all: >> That critical discussion was 10 years ago. In AI, there are lots of people who say lots of things. They keep saying them, even when people point out that they are just things that they say. They continue to say them even long after the other people get tired and give up trying to make sense of what is being said. That doesn't mean that the "discussion" is over, it's just that there is no real way to stop people saying things if that's what they really want to do.

by dvt 4y ago

travisjungroth 4y ago

Wait, how are you holding this position and telling me what to read when you couldn’t understand what “high dimensional representation of meaning” meant a few hours ago? Maybe the thing that will land is to say that embeddings are an attempt at representing meaning. There’s lots of other stuff you could represent about words. You could represent popularity by giving each word a number between 0 and 1 representing what portion of the training text was that word. That would tell you if a text might be harder for someone new to the language to understand. You could represent the spelling of the word as a list of integers. You could represent the pronunciation difficulty with a score based on ambiguous and infrequent n-grams. Or, you could try to represent the concept that the word is pointing to so you can follow synonyms or categorize text by topic. This would be an attempt at representing the word’s meaning. I’ve been explaining as if you didn’t understand what this was, but if you disagree that’s another thing.

YeGoblynQueenne 4y ago
>> Wait, how are you holding this position and telling me what to read when you couldn’t understand what “high dimensional representation of meaning” meant a few hours ago? From the conversation so far it's clear to me that the OP has at least some idea of how word embeddings work and they described as "word salad" the expression that related them to "meaning", i.e. they quibbled about the "... representation of meaning", not the "high dimensional representation... " part. I hope the OP can give a reference to the Quine and Kripke source they refer to. As far as I remember it, the point about arithmetic with word verctors (man, king, woman, queen and all that) is a motivational example for Word2Vec by Tomas Mikolov, which wikpedia tells me was published in 2013. I can't find the reference right now, but in 2014 I was taking a Master's in all that jazz and I remember one of my tutors having the paper on his desk, and commenting that the claim that word embeddings model meaning was "well, that's what he says". So there has been debate and disagreement about the ability of word embeddings to represent meaning for at least as long as Word2Vec has existed, and as far as I know the idea of word embeddings existed earlier than that (I was taught the concept unconnected to word2vec in my Master's). The bottom line is, just because somebody says their algorithm does a thing, doesn't mean that everyone has to accept it immediately, and without critical discussion. So I, for one, want to see that Quine and Kripke stuff the OP is referring to.
- travisjungroth 4y ago
  > The bottom line is, just because somebody says their algorithm does a thing, doesn't mean that everyone has to accept it immediately, and without critical discussion. That critical discussion was 10 years ago. Embeddings are vector representations of meaning. They’re not perfect, but they can be used for enough meaning-dependent tasks that this statement shouldn’t be controversial. All this AI/ML stuff is so exhausting because the philosophers come out of the woodwork. They don’t prove anything or disprove anything, they just kinda whine. When I was a flight instructor and said the airplane wanted to float down the runway, no one tried to point out that airplanes don’t have desires. I can say a server knows another server is offline and that’s fine too. But say “reason”, “think”, or “meaning” with something AI related and it’s like all work has to stop until we stop using that precious word. I’ve tried to engage with that stuff less, but got tricked cause I thought this person just literally didn’t understand.
  - YeGoblynQueenne 4y ago
    I hope you're not calling me a "philosopher". I identify as a coder. And what I want to know is where is the code for "meaning". I don't care why it's the code for "meaning", but if you point me to the code for "word embeddings" and you say "that's the code for 'meaning", then I'm going to wonder whether, if I ask for the code for "quicksort" you'll point me to the code for "bubblesort". So, no, what we call things that do things is important, otherwise we don't know what things do what, and what things are being done. And lest we forget: However, in AI, our programs to a great degree are problems rather than solutions. If a researcher tries to write an "understanding" program, it isn't because he has thought of a better way of implementing this well-understood task, but because he thinks he can come closer to writing the implementation. If he calls the main loop of his program "UNDERSTAND', he i s (until proven innocent) merely begging the question. He may mislead a lot of people, most prominently himself, and enrage a lot of others. What he should do instead is refer to this main loop as "G0034", and see if he can convince himself or anyone else that G0034 implements some part of understanding. Or he could give i t a name that reveals its intrinsic properties, like NODE-NET- INTERSECTION-FINDER, it being the substance of his theory that finding intersections in networks of nodes constitutes understanding. If Quillian <1969> had called his program the "Teachable Language Node Net Intersection Finder", he would have saved us some reading. (Except for those of us fanatic about finding the part on teachability.) https://cs.fit.edu/~kgallagher/Schtick/Serious/McDermott.AI.MeetsNaturalStupidity.pdf https://cs.fit.edu/~kgallagher/Schtick/Serious/McDermott.AI.... Edit: And this is not right at all: >> That critical discussion was 10 years ago. In AI, there are lots of people who say lots of things. They keep saying them, even when people point out that they are just things that they say. They continue to say them even long after the other people get tired and give up trying to make sense of what is being said. That doesn't mean that the "discussion" is over, it's just that there is no real way to stop people saying things if that's what they really want to do.