8 ms·

Deep or Shallow, NLP is breaking out

107 points by samiur1204 11y ago

n0us 11y ago
As research in NLP advances one of the things I am looking out for is how the field will be broken into separate problems. Are there some problems which are relatively low hanging fruit and others that we will be scratching our heads over 30 years from now? One thing that I think will be challenging is that language has observer depending meaning. The same statement might have a completely different meaning to someone with a different experience, or made in a different context, or stated by a different person. Games like Chess and Go have observer independent solutions. The winner is the same no matter who/what plays the game. Determining the meaning of a sentence is a problem where the real answer depends on observer dependent perspective and therefore will need a completely different way to measure success compared to more 'mathematical' tasks like Go. Trying to program a machine to account for this kind of personal experience that humans have, as well as for individual differences between people will be quite challenging I think. I also think that the most significant advances will come from cross cutting academic disciplines like Psychology, Linguistics, and Philosophy of Language.
- Radim 11y ago
  But this is relevant even in Go AI! As we saw with the recent AlphaGo matches, not understanding your opponent (always assuming you're playing against a clone of yourself, as AlphaGo does), can cause objectively suboptimal play. You just have no concept of a "trick play" -- in your world, everything you see, the opponent can see too! You could argue that with a little bit better theory of mind [1], AlphaGo could have executed a better strategy for its comeback. Instead of playing out obviously broken ladders, hoping the opponent will make a trivial mistake (ha), it was objectively strong enough to devise a cleverer plan, playing to the human's actual weaknesses. It didn't, and lost. So what you write is true, and only more true in the imperfect, fuzzy and intrinsically deceitful world of NLP. [1] https://en.wikipedia.org/wiki/Theory_of_mind https://en.wikipedia.org/wiki/Theory_of_mind
  - wodenokoto 11y ago
    Given that it won 4 out of 5 games, I think your argument is pretty weak. The only trick play you can do in a game of perfect information is exactly to hope that your opponent will make a trivial mistake in response to your move.
    - vonmoltke 11y ago
      Not at all. If the game is complex enough you may not have the ability to determine all permutations of your opponents strategy. As someone posted in another thread, the number of legal positions in Go is so high that, while calculating all possibilities is theoretically possible, actually doing so is effectively impossible. This requires you to limit your processing to the most likely scenarios, and a "trick play" becomes one that follows a highly unlikely strategy and banks on your opponent not realizing what you are actually doing.
      - Radim 11y ago
        Exactly. And it is the "highly unlikely strategy" bit that can be tailored to your specific opponent, opening a whole new can of research worms. Even in Go. For wodenokoto: you can read more complexity and the difference between "theoretical perfection" vs "practical execution" here [1] (especially the Scott Aaronson’s essay linked there). [1] http://rare-technologies.com/go_games_life/ http://rare-technologies.com/go_games_life/
- iofj 11y ago
  The answer for most of these problems where humans say "humans do X and X is difficult because ..." is either : 1) it's not complex at all. 2) same as 1, with "in 99.99...% of the cases" appended to it. Most of the problems in AI are a scaling problem. The reason you're not yet seeing androids taking over the economy are twofold: Portable energy. Our best technologies are not the equal of the human body when it comes to how much power can be stored. But they are constantly improving, even if we'll need another tenfold increase in energy density (less if we make mobile robots gasoline powered, which is impractical for other reasons, more if do the battery + electrical motors thing everyone wants). This means humans are cheaper and easier for a lot of tasks. Control. The human body (not including face and face-adjecent muscles) has about 300 actuators. That means controlling a human body means controlling 300 individual motors at the same time in a useful way (and every motor affects every other motor. Moving your hand forward means adjusting the power your little toe is applying to the ground to maintain balance). The state of the art is maybe 10-dimensional control (10 interdependent actuators), which can be increased to maybe 20 if the problem can be split into subproblems (e.g. cooperating robots, or parts of the robot that are attached, so imbalance cannot occur). In some ways every extra dimension adds an order of magnitude to the complexity, so it's not like we'll get there in 10 years. But in a century ... probably. The thing about these problems is that they are problems of degree. Just like today's AI algorithms were known in the 1960's. So why didn't we have live speech transcription in the 1960s ? We all know the answer : processing power was 50 orders of magnitude less than we needed. It was a problem of degree. We knew the problem, and had at least a good guess at what to do, but couldn't contemplate that actually acting on those ideas would make any serious progress due to the limitations of processing speeds at those times. If you wanted a 1960s computer to do a 50x50 matrix multiplication, you'd be waiting days. Doing millions of them was therefore considered useless. Even this is being gentle. A good case can be made that every component of these algorithms was known once differentiation was formalized, but nobody put them together, not because they didn't realize it could work, but because it was useless : doing things this way would have been incredibly inefficient compared to the then normal ways of doing things. E.g. finding formulas and constants by small adjustments on large chains of partial derivatives (ie. "Deep learning") is something that Isaac Newton knew how to do. It's just he would declare you totally mad for doing that. One might criticize that there were a few holes in the mathematical understanding of matrices in Newton's day, I wouldn't doubt that had he had a reason to really look into those problems, he would have fixed them. But he was more than 50 orders of magnitudes of : doing a single 50x50 matrix multiplication with any amount of resources then available wouldn't have finished before he died. Now machine learning tutorials tell you to run unrolled 30x30 LSTM expansions on audio data and unroll for 50 datapoints. And then doing that millions of times (tens of thousands of times on a few thousand samples). You can expect this to run on the computer you're reading this on in a matter of hours.
- wodenokoto 11y ago
  I have a hard time putting it to words, but I do feel that while computers are pretty good at statistically parsing language, humans have an extra sanity check that parses "meaning" (what ever that may satisfactionally be defined as). Maybe it means we need to hook up our NLP pipeline to deep-visual networks and knowledge-graphs in order to ask "Did the message of that sentence make sense, and if not, should check a similarly sounding sentence or a different parsing of the sentence in order to make it more cohesive"
  - bendykstra 11y ago
    The Wikipedia article on garden path sentences has a good description of that phenomenon. https://en.wikipedia.org/wiki/Garden_path_sentence https://en.wikipedia.org/wiki/Garden_path_sentence
    - wodenokoto 11y ago
      Thank you for the link. I do think we have NLP algorithms that can handle garden paths without understanding the sentence. A state of the art Japanese POS-tagger for example has a dictionary of all previously seen word/pos pairs and their transition probabilities. Sine there are no spaces in Japanese, the possible combinations of word/pos pairs you can lay out that matches the input sentence is quite big, and gives you a fairly large graph that you need to find the globally best path through. This means the algorithm does walk down the garden path, but at some point decides that there is a better path that will get you to the end of the sentence. If you go to http://www.atilika.org http://www.atilika.org, scroll down a bit, select the "viterbi" radio button, paste 私は日本人です into the text box and click tokenise, you can see an example of such a graph.
  - n0us 11y ago
    I agree that great advances can be made from statistically parsing language but it is another thing entirely to have a computer (for example) read all of Shakespeare's sonnets and write an essay about the influence they have had on other authors. Even if it is as a highschool level. I kind of see statistical parsing as another "parlor trick" type of AI. Impressive and hard to achieve but somehow separate from true intelligence. It's hard to formally describe this distinction which is why I think that advances will come from cross cutting academic departments and reaching out from the "throw more math and more processors at the problem approach" To achieve a true AI we will have to advance not just our understanding of algorithms, but our understanding of what it means to be intelligent which is a concept that I think most people lack any coherent definition of.
- YeGoblynQueenne 11y ago
  >> One thing that I think will be challenging is that language has observer depending meaning. The idea with word embeddings is that they provide a context for words, which informs their meaning. It goes back to Zellig Harris and the observation that "constituents of the same type can be replaced by each other" [1], or in other words: words that occur in the same context have the same meaning. [1] Zellig Harris. 1951. Methods in Structural Linguistics.
  - kajecounterhack 11y ago
    If anyone is interested in embeddings, they are really cool and here is a relevant paper: https://cs.stanford.edu/~quocle/paragraph_vector.pdf https://cs.stanford.edu/~quocle/paragraph_vector.pdf (This is to say there is such a thing as an unsupervised method for creating features that encode meaning, and it's being used today for various tasks :] )
  - zardo 11y ago
    The meaning to the reader is not encoded. Maybe something like book to book-review or book to cliff notes mappings could try to capture that.
- joe_the_user 11y ago
  Per force, there will be some aspects of language that will be easier to solve than others. At the same time, I think that there's been a tendency for people to overestimate the value of just a part of language understanding. To me, Google search with NLP is at best barely better than search with some kind of query language (and often worse imo). Those phone-based robobilling system now can mostly understand what you say but are still essentially only a notch above keypad menus. The thing with complex syntax-based meaning in human language is that really is only worth the trouble if one is setting up an ongoing relationship with an other. A very smart human with vast knowledge still couldn't give great answers to single-sentence questions from people he'd never seen before and who would never see him again. And ongoing language interactions actually tend to connect multiple aspect of human activity making divisions harder. So metaphorically we may have a few very small fruit and more or less one very large one.
- vonmoltke 11y ago
  > Determining the meaning of a sentence is a problem where the real answer depends on observer dependent perspective and therefore will need a completely different way to measure success compared to more 'mathematical' tasks like Go. Trying to program a machine to account for this kind of personal experience that humans have, as well as for individual differences between people will be quite challenging I think. I also think that the most significant advances will come from cross cutting academic disciplines like Psychology, Linguistics, and Philosophy of Language. I have worked off-and-on on a project to detect metaphors and identify the concrete meaning the metaphor is attempting to convey. Amongst many other things that project taught me that it is nearly impossible to get more than five linguists to agree on what a metaphor is in the first place.
xigency 11y ago
In terms of shallow natural language processing, some pretty simple observations can lead to a lot of bang for the buck. One project I completed was able to really excel at keyword matching simply by building a huge dictionary of words, in a literal sense: a dictionary of contextually relevant words to a phrase, generated by very large texts. I think baby steps are the key to getting further with NLP. For reference: http://nlp.stanford.edu/fsnlp/ http://nlp.stanford.edu/fsnlp/
- donpdonp 11y ago
  A quick idea of what the Stanford NLP parser is capable of can be gained by trying a few things in their online gateway. Its impressive. http://nlp.stanford.edu:8080/parser/ http://nlp.stanford.edu:8080/parser/
  - charlieegan3 11y ago
    I'd reccommend also checking out: http://corenlp.run/ http://corenlp.run/. It's a running version of the corenlp server. I'm making use of this, just have it running in a container and get parses via the API (which is good but a little odd to use).
meeper16 11y ago
More info on vectors https://www.kaggle.com/c/word2vec-nlp-tutorial/forums/t/12349/word2vec-is-based-on-an-approach-from-lawrence-berkeley-national-lab https://www.kaggle.com/c/word2vec-nlp-tutorial/forums/t/1234...
YeGoblynQueenne 11y ago
>> reasoning by analogy is the core kind of reasoning we do, and logic is just a sort of superficial thing on top of it that happens much later Well, for some that's certainly the case.
YeGoblynQueenne 11y ago
In a recent interview with Communications, Hinton said his own research on word vectors goes back to the mid-1980s, when he, David Rumelhart, and Ronald Williams published work in Nature that demonstrated family relationships as vectors. "The vectors were only six components long because computers were very small then, but it took a long time for it to catch on," Hinton said. Yeah, I know the work he's talking about. It's the one related to this dataset: https://archive.ics.uci.edu/ml/datasets/Kinship https://archive.ics.uci.edu/ml/datasets/Kinship From that page: Creator: Geoff Hinton Donor: J. Ross Quinlan Data Set Information: This relational database consists of 24 unique names in two families (they have equivalent structures). Hinton used one unique output unit for each person and was interested in predicting the following relations: wife, husband, mother, father, daughter, son, sister, brother, aunt, uncle, niece, and nephew. Hinton used 104 input-output vector pairs (from a space of 12x24=288 possible pairs). The prediction task is as follows: given a name and a relation, have the outputs be on for only those individuals (among the 24) that satisfy the relation. The outputs for all other individuals should be off. Hinton's results: Using 100 vectors as input and 4 for testing, his results on two passes yielded 7 correct responses out of 8. His network of 36 input units, 3 layers of hidden units, and 24 output units used 500 sweeps of the training set during training. Quinlan's results: Using FOIL, he repeated the experiment 20 times (rather than Hinton's 2 times). FOIL was correct 78 out of 80 times on the test cases. And yet, if you have a wee look at Hinton's publication on Rexa, there's 43 citations, while there's a single one on Quinlan's (from Muggleton, duh). So, you know, maybe it's not logic and reasoning that's the problem here, rather a certain tendency to drum up results of neural models even when they don't do any better than other techniques. But, really, it doesn't matter. Google has the airwaves (so to speak). No matter what happens anywhere else, in academia or business, their stuff is going to be publicised the most and that's what we all have to deal with.
- romaniv 11y ago
  > ut, really, it doesn't matter. Google has the airwaves (so to speak). No matter what happens anywhere else, in academia or business, their stuff is going to be publicised the most and that's what we all have to deal with. One thing that bothers me about ANN hype is that those models are very resource and data hungry. Which suits Google perfectly, but will keep that kind of AI out of people's hands/computers for decades to come.
YeGoblynQueenne 11y ago
>> "Most of our reasoning is by analogy; it's not logical reasoning." I keep reading this quote, showcased in the article, again and again, and I still cant believe that people are actually proposing this. They're basically advocating that we should abandon logic, stop trying to reason about things, and instead try stuff at random, and why? Because computers have been shown to do alright with that approach at tasks in which humans excel. It is clear as daylight that whatever humans do, computers don't - otherwise, you'd need to train your baby with the Brown Corpus before it could figure out you're its mama. We have managed to overcome the limitations of our primitive computational technology with some clever tricks and that's amazing. But to take that rightly celebrated fact and make it into an argument that we must now become ourselves as dumb as our computers, and also never try to make the poor things intelligent in the same way we are, that's ... well, it's dumb. That's what it is.
- robrenaud 11y ago
  Unfortunately, the article is down right now, so I don't have the context for that quote. But in "Thinking Fast and Slow", Daniel Kahneman makes that same point that most of our reasoning is kind of heuristic pattern matching. There are lots of experiments that show evidence of this behavior cited in his book. Certainly people can turn it off and logically reason about things when desired, but it's slow and taxing, and the heuristics people live by mostly work, so it's frequently unneeded.
- dwaltrip 11y ago
  A human child receives millions of "training" lessons every day for two years, and only then does it learn that you, indeed, are its "mother". This knowledge is composed from other, simpler and more basic facts, that it slowly learns along the way: 1) you are a reliable source of warmth and comfort 2) you make me feel safe when I'm anxious and scared 3) you feed me when I'm hungry 4) you always come back after going away 5) you make me laugh and feel happy These can be broken down into more basic facts, as the concepts here are still rather advanced (for a baby). The baby doesn't actually know what the it means to feel "safe", "fear", "happy", etc. It just has fuzzy experiential feelings that slowly become sharper and more contextualized as the baby is "trained" every second of every day by interacting with the world around it.
senthil_rajasek 11y ago
link to google cache http://webcache.googleusercontent.com/search?q=cache:145V9qmKz2gJ:cacm.acm.org/magazines/2016/3/198856-deep-or-shallow-nlp-is-breaking-out/fulltext+&cd=1&hl=en&ct=clnk&gl=us http://webcache.googleusercontent.com/search?q=cache:145V9qm...
nkurz 11y ago
Just as "Hello, World" may be the best-known general programming introductory example, Mikolov, who was then at Microsoft Research, also introduced what fast became a benchmark equation in natural language processing at the 2013 proceedings of the North American Association for Computational Linguistics, the kingman+woman=queen analogy, in which the computer solved the equation spontaneously. Since the ACM has professional editors, I was surprised that they would twice misrepresent the example "king – man + woman = queen" as "kingman+woman=queen" in the article. At least they spelled "Hello, World" right, even if they couldn't bring themselves to add the "!". It looks like the problem is that in the PDF version, the phrase happens to be hyphenated at the "minus sign" in both usages (http://delivery.acm.org/10.1145/2880000/2874915/p13-goth.pdf http://delivery.acm.org/10.1145/2880000/2874915/p13-goth.pdf) [1] although one might hope this is something an editor would have checked. [1] Looks like the ACM wants to you click on the PDF link yourself, from the "View As" bar in the text version.

107 points by samiur1204 11y ago

n0us 11y ago

As research in NLP advances one of the things I am looking out for is how the field will be broken into separate problems. Are there some problems which are relatively low hanging fruit and others that we will be scratching our heads over 30 years from now? One thing that I think will be challenging is that language has observer depending meaning. The same statement might have a completely different meaning to someone with a different experience, or made in a different context, or stated by a different person. Games like Chess and Go have observer independent solutions. The winner is the same no matter who/what plays the game. Determining the meaning of a sentence is a problem where the real answer depends on observer dependent perspective and therefore will need a completely different way to measure success compared to more 'mathematical' tasks like Go. Trying to program a machine to account for this kind of personal experience that humans have, as well as for individual differences between people will be quite challenging I think. I also think that the most significant advances will come from cross cutting academic disciplines like Psychology, Linguistics, and Philosophy of Language.

Radim 11y ago
But this is relevant even in Go AI! As we saw with the recent AlphaGo matches, not understanding your opponent (always assuming you're playing against a clone of yourself, as AlphaGo does), can cause objectively suboptimal play. You just have no concept of a "trick play" -- in your world, everything you see, the opponent can see too! You could argue that with a little bit better theory of mind [1], AlphaGo could have executed a better strategy for its comeback. Instead of playing out obviously broken ladders, hoping the opponent will make a trivial mistake (ha), it was objectively strong enough to devise a cleverer plan, playing to the human's actual weaknesses. It didn't, and lost. So what you write is true, and only more true in the imperfect, fuzzy and intrinsically deceitful world of NLP. [1] https://en.wikipedia.org/wiki/Theory_of_mind https://en.wikipedia.org/wiki/Theory_of_mind
- wodenokoto 11y ago
  Given that it won 4 out of 5 games, I think your argument is pretty weak. The only trick play you can do in a game of perfect information is exactly to hope that your opponent will make a trivial mistake in response to your move.
  - vonmoltke 11y ago
    Not at all. If the game is complex enough you may not have the ability to determine all permutations of your opponents strategy. As someone posted in another thread, the number of legal positions in Go is so high that, while calculating all possibilities is theoretically possible, actually doing so is effectively impossible. This requires you to limit your processing to the most likely scenarios, and a "trick play" becomes one that follows a highly unlikely strategy and banks on your opponent not realizing what you are actually doing.
    - Radim 11y ago
      Exactly. And it is the "highly unlikely strategy" bit that can be tailored to your specific opponent, opening a whole new can of research worms. Even in Go. For wodenokoto: you can read more complexity and the difference between "theoretical perfection" vs "practical execution" here [1] (especially the Scott Aaronson’s essay linked there). [1] http://rare-technologies.com/go_games_life/ http://rare-technologies.com/go_games_life/
iofj 11y ago
The answer for most of these problems where humans say "humans do X and X is difficult because ..." is either : 1) it's not complex at all. 2) same as 1, with "in 99.99...% of the cases" appended to it. Most of the problems in AI are a scaling problem. The reason you're not yet seeing androids taking over the economy are twofold: Portable energy. Our best technologies are not the equal of the human body when it comes to how much power can be stored. But they are constantly improving, even if we'll need another tenfold increase in energy density (less if we make mobile robots gasoline powered, which is impractical for other reasons, more if do the battery + electrical motors thing everyone wants). This means humans are cheaper and easier for a lot of tasks. Control. The human body (not including face and face-adjecent muscles) has about 300 actuators. That means controlling a human body means controlling 300 individual motors at the same time in a useful way (and every motor affects every other motor. Moving your hand forward means adjusting the power your little toe is applying to the ground to maintain balance). The state of the art is maybe 10-dimensional control (10 interdependent actuators), which can be increased to maybe 20 if the problem can be split into subproblems (e.g. cooperating robots, or parts of the robot that are attached, so imbalance cannot occur). In some ways every extra dimension adds an order of magnitude to the complexity, so it's not like we'll get there in 10 years. But in a century ... probably. The thing about these problems is that they are problems of degree. Just like today's AI algorithms were known in the 1960's. So why didn't we have live speech transcription in the 1960s ? We all know the answer : processing power was 50 orders of magnitude less than we needed. It was a problem of degree. We knew the problem, and had at least a good guess at what to do, but couldn't contemplate that actually acting on those ideas would make any serious progress due to the limitations of processing speeds at those times. If you wanted a 1960s computer to do a 50x50 matrix multiplication, you'd be waiting days. Doing millions of them was therefore considered useless. Even this is being gentle. A good case can be made that every component of these algorithms was known once differentiation was formalized, but nobody put them together, not because they didn't realize it could work, but because it was useless : doing things this way would have been incredibly inefficient compared to the then normal ways of doing things. E.g. finding formulas and constants by small adjustments on large chains of partial derivatives (ie. "Deep learning") is something that Isaac Newton knew how to do. It's just he would declare you totally mad for doing that. One might criticize that there were a few holes in the mathematical understanding of matrices in Newton's day, I wouldn't doubt that had he had a reason to really look into those problems, he would have fixed them. But he was more than 50 orders of magnitudes of : doing a single 50x50 matrix multiplication with any amount of resources then available wouldn't have finished before he died. Now machine learning tutorials tell you to run unrolled 30x30 LSTM expansions on audio data and unroll for 50 datapoints. And then doing that millions of times (tens of thousands of times on a few thousand samples). You can expect this to run on the computer you're reading this on in a matter of hours.
wodenokoto 11y ago
I have a hard time putting it to words, but I do feel that while computers are pretty good at statistically parsing language, humans have an extra sanity check that parses "meaning" (what ever that may satisfactionally be defined as). Maybe it means we need to hook up our NLP pipeline to deep-visual networks and knowledge-graphs in order to ask "Did the message of that sentence make sense, and if not, should check a similarly sounding sentence or a different parsing of the sentence in order to make it more cohesive"
- bendykstra 11y ago
  The Wikipedia article on garden path sentences has a good description of that phenomenon. https://en.wikipedia.org/wiki/Garden_path_sentence https://en.wikipedia.org/wiki/Garden_path_sentence
  - wodenokoto 11y ago
    Thank you for the link. I do think we have NLP algorithms that can handle garden paths without understanding the sentence. A state of the art Japanese POS-tagger for example has a dictionary of all previously seen word/pos pairs and their transition probabilities. Sine there are no spaces in Japanese, the possible combinations of word/pos pairs you can lay out that matches the input sentence is quite big, and gives you a fairly large graph that you need to find the globally best path through. This means the algorithm does walk down the garden path, but at some point decides that there is a better path that will get you to the end of the sentence. If you go to http://www.atilika.org http://www.atilika.org, scroll down a bit, select the "viterbi" radio button, paste 私は日本人です into the text box and click tokenise, you can see an example of such a graph.
- n0us 11y ago
  I agree that great advances can be made from statistically parsing language but it is another thing entirely to have a computer (for example) read all of Shakespeare's sonnets and write an essay about the influence they have had on other authors. Even if it is as a highschool level. I kind of see statistical parsing as another "parlor trick" type of AI. Impressive and hard to achieve but somehow separate from true intelligence. It's hard to formally describe this distinction which is why I think that advances will come from cross cutting academic departments and reaching out from the "throw more math and more processors at the problem approach" To achieve a true AI we will have to advance not just our understanding of algorithms, but our understanding of what it means to be intelligent which is a concept that I think most people lack any coherent definition of.
YeGoblynQueenne 11y ago
>> One thing that I think will be challenging is that language has observer depending meaning. The idea with word embeddings is that they provide a context for words, which informs their meaning. It goes back to Zellig Harris and the observation that "constituents of the same type can be replaced by each other" [1], or in other words: words that occur in the same context have the same meaning. [1] Zellig Harris. 1951. Methods in Structural Linguistics.
- kajecounterhack 11y ago
  If anyone is interested in embeddings, they are really cool and here is a relevant paper: https://cs.stanford.edu/~quocle/paragraph_vector.pdf https://cs.stanford.edu/~quocle/paragraph_vector.pdf (This is to say there is such a thing as an unsupervised method for creating features that encode meaning, and it's being used today for various tasks :] )
- zardo 11y ago
  The meaning to the reader is not encoded. Maybe something like book to book-review or book to cliff notes mappings could try to capture that.
joe_the_user 11y ago
Per force, there will be some aspects of language that will be easier to solve than others. At the same time, I think that there's been a tendency for people to overestimate the value of just a part of language understanding. To me, Google search with NLP is at best barely better than search with some kind of query language (and often worse imo). Those phone-based robobilling system now can mostly understand what you say but are still essentially only a notch above keypad menus. The thing with complex syntax-based meaning in human language is that really is only worth the trouble if one is setting up an ongoing relationship with an other. A very smart human with vast knowledge still couldn't give great answers to single-sentence questions from people he'd never seen before and who would never see him again. And ongoing language interactions actually tend to connect multiple aspect of human activity making divisions harder. So metaphorically we may have a few very small fruit and more or less one very large one.
vonmoltke 11y ago
> Determining the meaning of a sentence is a problem where the real answer depends on observer dependent perspective and therefore will need a completely different way to measure success compared to more 'mathematical' tasks like Go. Trying to program a machine to account for this kind of personal experience that humans have, as well as for individual differences between people will be quite challenging I think. I also think that the most significant advances will come from cross cutting academic disciplines like Psychology, Linguistics, and Philosophy of Language. I have worked off-and-on on a project to detect metaphors and identify the concrete meaning the metaphor is attempting to convey. Amongst many other things that project taught me that it is nearly impossible to get more than five linguists to agree on what a metaphor is in the first place.

xigency 11y ago

In terms of shallow natural language processing, some pretty simple observations can lead to a lot of bang for the buck. One project I completed was able to really excel at keyword matching simply by building a huge dictionary of words, in a literal sense: a dictionary of contextually relevant words to a phrase, generated by very large texts. I think baby steps are the key to getting further with NLP. For reference: http://nlp.stanford.edu/fsnlp/ http://nlp.stanford.edu/fsnlp/

donpdonp 11y ago
A quick idea of what the Stanford NLP parser is capable of can be gained by trying a few things in their online gateway. Its impressive. http://nlp.stanford.edu:8080/parser/ http://nlp.stanford.edu:8080/parser/
- charlieegan3 11y ago
  I'd reccommend also checking out: http://corenlp.run/ http://corenlp.run/. It's a running version of the corenlp server. I'm making use of this, just have it running in a container and get parses via the API (which is good but a little odd to use).

meeper16 11y ago

More info on vectors https://www.kaggle.com/c/word2vec-nlp-tutorial/forums/t/12349/word2vec-is-based-on-an-approach-from-lawrence-berkeley-national-lab https://www.kaggle.com/c/word2vec-nlp-tutorial/forums/t/1234...

YeGoblynQueenne 11y ago

>> reasoning by analogy is the core kind of reasoning we do, and logic is just a sort of superficial thing on top of it that happens much later Well, for some that's certainly the case.

YeGoblynQueenne 11y ago

In a recent interview with Communications, Hinton said his own research on word vectors goes back to the mid-1980s, when he, David Rumelhart, and Ronald Williams published work in Nature that demonstrated family relationships as vectors. "The vectors were only six components long because computers were very small then, but it took a long time for it to catch on," Hinton said. Yeah, I know the work he's talking about. It's the one related to this dataset: https://archive.ics.uci.edu/ml/datasets/Kinship https://archive.ics.uci.edu/ml/datasets/Kinship From that page: Creator: Geoff Hinton Donor: J. Ross Quinlan Data Set Information: This relational database consists of 24 unique names in two families (they have equivalent structures). Hinton used one unique output unit for each person and was interested in predicting the following relations: wife, husband, mother, father, daughter, son, sister, brother, aunt, uncle, niece, and nephew. Hinton used 104 input-output vector pairs (from a space of 12x24=288 possible pairs). The prediction task is as follows: given a name and a relation, have the outputs be on for only those individuals (among the 24) that satisfy the relation. The outputs for all other individuals should be off. Hinton's results: Using 100 vectors as input and 4 for testing, his results on two passes yielded 7 correct responses out of 8. His network of 36 input units, 3 layers of hidden units, and 24 output units used 500 sweeps of the training set during training. Quinlan's results: Using FOIL, he repeated the experiment 20 times (rather than Hinton's 2 times). FOIL was correct 78 out of 80 times on the test cases. And yet, if you have a wee look at Hinton's publication on Rexa, there's 43 citations, while there's a single one on Quinlan's (from Muggleton, duh). So, you know, maybe it's not logic and reasoning that's the problem here, rather a certain tendency to drum up results of neural models even when they don't do any better than other techniques. But, really, it doesn't matter. Google has the airwaves (so to speak). No matter what happens anywhere else, in academia or business, their stuff is going to be publicised the most and that's what we all have to deal with.

romaniv 11y ago
> ut, really, it doesn't matter. Google has the airwaves (so to speak). No matter what happens anywhere else, in academia or business, their stuff is going to be publicised the most and that's what we all have to deal with. One thing that bothers me about ANN hype is that those models are very resource and data hungry. Which suits Google perfectly, but will keep that kind of AI out of people's hands/computers for decades to come.

YeGoblynQueenne 11y ago

>> "Most of our reasoning is by analogy; it's not logical reasoning." I keep reading this quote, showcased in the article, again and again, and I still cant believe that people are actually proposing this. They're basically advocating that we should abandon logic, stop trying to reason about things, and instead try stuff at random, and why? Because computers have been shown to do alright with that approach at tasks in which humans excel. It is clear as daylight that whatever humans do, computers don't - otherwise, you'd need to train your baby with the Brown Corpus before it could figure out you're its mama. We have managed to overcome the limitations of our primitive computational technology with some clever tricks and that's amazing. But to take that rightly celebrated fact and make it into an argument that we must now become ourselves as dumb as our computers, and also never try to make the poor things intelligent in the same way we are, that's ... well, it's dumb. That's what it is.

robrenaud 11y ago
Unfortunately, the article is down right now, so I don't have the context for that quote. But in "Thinking Fast and Slow", Daniel Kahneman makes that same point that most of our reasoning is kind of heuristic pattern matching. There are lots of experiments that show evidence of this behavior cited in his book. Certainly people can turn it off and logically reason about things when desired, but it's slow and taxing, and the heuristics people live by mostly work, so it's frequently unneeded.
dwaltrip 11y ago
A human child receives millions of "training" lessons every day for two years, and only then does it learn that you, indeed, are its "mother". This knowledge is composed from other, simpler and more basic facts, that it slowly learns along the way: 1) you are a reliable source of warmth and comfort 2) you make me feel safe when I'm anxious and scared 3) you feed me when I'm hungry 4) you always come back after going away 5) you make me laugh and feel happy These can be broken down into more basic facts, as the concepts here are still rather advanced (for a baby). The baby doesn't actually know what the it means to feel "safe", "fear", "happy", etc. It just has fuzzy experiential feelings that slowly become sharper and more contextualized as the baby is "trained" every second of every day by interacting with the world around it.

senthil_rajasek 11y ago

link to google cache http://webcache.googleusercontent.com/search?q=cache:145V9qmKz2gJ:cacm.acm.org/magazines/2016/3/198856-deep-or-shallow-nlp-is-breaking-out/fulltext+&cd=1&hl=en&ct=clnk&gl=us http://webcache.googleusercontent.com/search?q=cache:145V9qm...

nkurz 11y ago

Just as "Hello, World" may be the best-known general programming introductory example, Mikolov, who was then at Microsoft Research, also introduced what fast became a benchmark equation in natural language processing at the 2013 proceedings of the North American Association for Computational Linguistics, the kingman+woman=queen analogy, in which the computer solved the equation spontaneously. Since the ACM has professional editors, I was surprised that they would twice misrepresent the example "king – man + woman = queen" as "kingman+woman=queen" in the article. At least they spelled "Hello, World" right, even if they couldn't bring themselves to add the "!". It looks like the problem is that in the PDF version, the phrase happens to be hyphenated at the "minus sign" in both usages (http://delivery.acm.org/10.1145/2880000/2874915/p13-goth.pdf http://delivery.acm.org/10.1145/2880000/2874915/p13-goth.pdf) [1] although one might hope this is something an editor would have checked. [1] Looks like the ACM wants to you click on the PDF link yourself, from the "View As" bar in the text version.