5 ms·

As research in NLP advances one of the things I am looking out for is how the field will be broken into separate problems. Are there some problems which are rel

by n0us 11y ago

As research in NLP advances one of the things I am looking out for is how the field will be broken into separate problems. Are there some problems which are relatively low hanging fruit and others that we will be scratching our heads over 30 years from now? One thing that I think will be challenging is that language has observer depending meaning. The same statement might have a completely different meaning to someone with a different experience, or made in a different context, or stated by a different person. Games like Chess and Go have observer independent solutions. The winner is the same no matter who/what plays the game. Determining the meaning of a sentence is a problem where the real answer depends on observer dependent perspective and therefore will need a completely different way to measure success compared to more 'mathematical' tasks like Go. Trying to program a machine to account for this kind of personal experience that humans have, as well as for individual differences between people will be quite challenging I think. I also think that the most significant advances will come from cross cutting academic disciplines like Psychology, Linguistics, and Philosophy of Language.

Radim 11y ago
But this is relevant even in Go AI! As we saw with the recent AlphaGo matches, not understanding your opponent (always assuming you're playing against a clone of yourself, as AlphaGo does), can cause objectively suboptimal play. You just have no concept of a "trick play" -- in your world, everything you see, the opponent can see too! You could argue that with a little bit better theory of mind [1], AlphaGo could have executed a better strategy for its comeback. Instead of playing out obviously broken ladders, hoping the opponent will make a trivial mistake (ha), it was objectively strong enough to devise a cleverer plan, playing to the human's actual weaknesses. It didn't, and lost. So what you write is true, and only more true in the imperfect, fuzzy and intrinsically deceitful world of NLP. [1] https://en.wikipedia.org/wiki/Theory_of_mind https://en.wikipedia.org/wiki/Theory_of_mind
- wodenokoto 11y ago
  Given that it won 4 out of 5 games, I think your argument is pretty weak. The only trick play you can do in a game of perfect information is exactly to hope that your opponent will make a trivial mistake in response to your move.
  - vonmoltke 11y ago
    Not at all. If the game is complex enough you may not have the ability to determine all permutations of your opponents strategy. As someone posted in another thread, the number of legal positions in Go is so high that, while calculating all possibilities is theoretically possible, actually doing so is effectively impossible. This requires you to limit your processing to the most likely scenarios, and a "trick play" becomes one that follows a highly unlikely strategy and banks on your opponent not realizing what you are actually doing.
    - Radim 11y ago
      Exactly. And it is the "highly unlikely strategy" bit that can be tailored to your specific opponent, opening a whole new can of research worms. Even in Go. For wodenokoto: you can read more complexity and the difference between "theoretical perfection" vs "practical execution" here [1] (especially the Scott Aaronson’s essay linked there). [1] http://rare-technologies.com/go_games_life/ http://rare-technologies.com/go_games_life/
iofj 11y ago
The answer for most of these problems where humans say "humans do X and X is difficult because ..." is either : 1) it's not complex at all. 2) same as 1, with "in 99.99...% of the cases" appended to it. Most of the problems in AI are a scaling problem. The reason you're not yet seeing androids taking over the economy are twofold: Portable energy. Our best technologies are not the equal of the human body when it comes to how much power can be stored. But they are constantly improving, even if we'll need another tenfold increase in energy density (less if we make mobile robots gasoline powered, which is impractical for other reasons, more if do the battery + electrical motors thing everyone wants). This means humans are cheaper and easier for a lot of tasks. Control. The human body (not including face and face-adjecent muscles) has about 300 actuators. That means controlling a human body means controlling 300 individual motors at the same time in a useful way (and every motor affects every other motor. Moving your hand forward means adjusting the power your little toe is applying to the ground to maintain balance). The state of the art is maybe 10-dimensional control (10 interdependent actuators), which can be increased to maybe 20 if the problem can be split into subproblems (e.g. cooperating robots, or parts of the robot that are attached, so imbalance cannot occur). In some ways every extra dimension adds an order of magnitude to the complexity, so it's not like we'll get there in 10 years. But in a century ... probably. The thing about these problems is that they are problems of degree. Just like today's AI algorithms were known in the 1960's. So why didn't we have live speech transcription in the 1960s ? We all know the answer : processing power was 50 orders of magnitude less than we needed. It was a problem of degree. We knew the problem, and had at least a good guess at what to do, but couldn't contemplate that actually acting on those ideas would make any serious progress due to the limitations of processing speeds at those times. If you wanted a 1960s computer to do a 50x50 matrix multiplication, you'd be waiting days. Doing millions of them was therefore considered useless. Even this is being gentle. A good case can be made that every component of these algorithms was known once differentiation was formalized, but nobody put them together, not because they didn't realize it could work, but because it was useless : doing things this way would have been incredibly inefficient compared to the then normal ways of doing things. E.g. finding formulas and constants by small adjustments on large chains of partial derivatives (ie. "Deep learning") is something that Isaac Newton knew how to do. It's just he would declare you totally mad for doing that. One might criticize that there were a few holes in the mathematical understanding of matrices in Newton's day, I wouldn't doubt that had he had a reason to really look into those problems, he would have fixed them. But he was more than 50 orders of magnitudes of : doing a single 50x50 matrix multiplication with any amount of resources then available wouldn't have finished before he died. Now machine learning tutorials tell you to run unrolled 30x30 LSTM expansions on audio data and unroll for 50 datapoints. And then doing that millions of times (tens of thousands of times on a few thousand samples). You can expect this to run on the computer you're reading this on in a matter of hours.
wodenokoto 11y ago
I have a hard time putting it to words, but I do feel that while computers are pretty good at statistically parsing language, humans have an extra sanity check that parses "meaning" (what ever that may satisfactionally be defined as). Maybe it means we need to hook up our NLP pipeline to deep-visual networks and knowledge-graphs in order to ask "Did the message of that sentence make sense, and if not, should check a similarly sounding sentence or a different parsing of the sentence in order to make it more cohesive"
- bendykstra 11y ago
  The Wikipedia article on garden path sentences has a good description of that phenomenon. https://en.wikipedia.org/wiki/Garden_path_sentence https://en.wikipedia.org/wiki/Garden_path_sentence
  - wodenokoto 11y ago
    Thank you for the link. I do think we have NLP algorithms that can handle garden paths without understanding the sentence. A state of the art Japanese POS-tagger for example has a dictionary of all previously seen word/pos pairs and their transition probabilities. Sine there are no spaces in Japanese, the possible combinations of word/pos pairs you can lay out that matches the input sentence is quite big, and gives you a fairly large graph that you need to find the globally best path through. This means the algorithm does walk down the garden path, but at some point decides that there is a better path that will get you to the end of the sentence. If you go to http://www.atilika.org http://www.atilika.org, scroll down a bit, select the "viterbi" radio button, paste 私は日本人です into the text box and click tokenise, you can see an example of such a graph.
- n0us 11y ago
  I agree that great advances can be made from statistically parsing language but it is another thing entirely to have a computer (for example) read all of Shakespeare's sonnets and write an essay about the influence they have had on other authors. Even if it is as a highschool level. I kind of see statistical parsing as another "parlor trick" type of AI. Impressive and hard to achieve but somehow separate from true intelligence. It's hard to formally describe this distinction which is why I think that advances will come from cross cutting academic departments and reaching out from the "throw more math and more processors at the problem approach" To achieve a true AI we will have to advance not just our understanding of algorithms, but our understanding of what it means to be intelligent which is a concept that I think most people lack any coherent definition of.
YeGoblynQueenne 11y ago
>> One thing that I think will be challenging is that language has observer depending meaning. The idea with word embeddings is that they provide a context for words, which informs their meaning. It goes back to Zellig Harris and the observation that "constituents of the same type can be replaced by each other" [1], or in other words: words that occur in the same context have the same meaning. [1] Zellig Harris. 1951. Methods in Structural Linguistics.
- kajecounterhack 11y ago
  If anyone is interested in embeddings, they are really cool and here is a relevant paper: https://cs.stanford.edu/~quocle/paragraph_vector.pdf https://cs.stanford.edu/~quocle/paragraph_vector.pdf (This is to say there is such a thing as an unsupervised method for creating features that encode meaning, and it's being used today for various tasks :] )
- zardo 11y ago
  The meaning to the reader is not encoded. Maybe something like book to book-review or book to cliff notes mappings could try to capture that.
joe_the_user 11y ago
Per force, there will be some aspects of language that will be easier to solve than others. At the same time, I think that there's been a tendency for people to overestimate the value of just a part of language understanding. To me, Google search with NLP is at best barely better than search with some kind of query language (and often worse imo). Those phone-based robobilling system now can mostly understand what you say but are still essentially only a notch above keypad menus. The thing with complex syntax-based meaning in human language is that really is only worth the trouble if one is setting up an ongoing relationship with an other. A very smart human with vast knowledge still couldn't give great answers to single-sentence questions from people he'd never seen before and who would never see him again. And ongoing language interactions actually tend to connect multiple aspect of human activity making divisions harder. So metaphorically we may have a few very small fruit and more or less one very large one.
vonmoltke 11y ago
> Determining the meaning of a sentence is a problem where the real answer depends on observer dependent perspective and therefore will need a completely different way to measure success compared to more 'mathematical' tasks like Go. Trying to program a machine to account for this kind of personal experience that humans have, as well as for individual differences between people will be quite challenging I think. I also think that the most significant advances will come from cross cutting academic disciplines like Psychology, Linguistics, and Philosophy of Language. I have worked off-and-on on a project to detect metaphors and identify the concrete meaning the metaphor is attempting to convey. Amongst many other things that project taught me that it is nearly impossible to get more than five linguists to agree on what a metaphor is in the first place.

by n0us 11y ago

Radim 11y ago

But this is relevant even in Go AI! As we saw with the recent AlphaGo matches, not understanding your opponent (always assuming you're playing against a clone of yourself, as AlphaGo does), can cause objectively suboptimal play. You just have no concept of a "trick play" -- in your world, everything you see, the opponent can see too! You could argue that with a little bit better theory of mind [1], AlphaGo could have executed a better strategy for its comeback. Instead of playing out obviously broken ladders, hoping the opponent will make a trivial mistake (ha), it was objectively strong enough to devise a cleverer plan, playing to the human's actual weaknesses. It didn't, and lost. So what you write is true, and only more true in the imperfect, fuzzy and intrinsically deceitful world of NLP. [1] https://en.wikipedia.org/wiki/Theory_of_mind https://en.wikipedia.org/wiki/Theory_of_mind

wodenokoto 11y ago
Given that it won 4 out of 5 games, I think your argument is pretty weak. The only trick play you can do in a game of perfect information is exactly to hope that your opponent will make a trivial mistake in response to your move.
- vonmoltke 11y ago
  Not at all. If the game is complex enough you may not have the ability to determine all permutations of your opponents strategy. As someone posted in another thread, the number of legal positions in Go is so high that, while calculating all possibilities is theoretically possible, actually doing so is effectively impossible. This requires you to limit your processing to the most likely scenarios, and a "trick play" becomes one that follows a highly unlikely strategy and banks on your opponent not realizing what you are actually doing.
  - Radim 11y ago
    Exactly. And it is the "highly unlikely strategy" bit that can be tailored to your specific opponent, opening a whole new can of research worms. Even in Go. For wodenokoto: you can read more complexity and the difference between "theoretical perfection" vs "practical execution" here [1] (especially the Scott Aaronson’s essay linked there). [1] http://rare-technologies.com/go_games_life/ http://rare-technologies.com/go_games_life/

iofj 11y ago

The answer for most of these problems where humans say "humans do X and X is difficult because ..." is either : 1) it's not complex at all. 2) same as 1, with "in 99.99...% of the cases" appended to it. Most of the problems in AI are a scaling problem. The reason you're not yet seeing androids taking over the economy are twofold: Portable energy. Our best technologies are not the equal of the human body when it comes to how much power can be stored. But they are constantly improving, even if we'll need another tenfold increase in energy density (less if we make mobile robots gasoline powered, which is impractical for other reasons, more if do the battery + electrical motors thing everyone wants). This means humans are cheaper and easier for a lot of tasks. Control. The human body (not including face and face-adjecent muscles) has about 300 actuators. That means controlling a human body means controlling 300 individual motors at the same time in a useful way (and every motor affects every other motor. Moving your hand forward means adjusting the power your little toe is applying to the ground to maintain balance). The state of the art is maybe 10-dimensional control (10 interdependent actuators), which can be increased to maybe 20 if the problem can be split into subproblems (e.g. cooperating robots, or parts of the robot that are attached, so imbalance cannot occur). In some ways every extra dimension adds an order of magnitude to the complexity, so it's not like we'll get there in 10 years. But in a century ... probably. The thing about these problems is that they are problems of degree. Just like today's AI algorithms were known in the 1960's. So why didn't we have live speech transcription in the 1960s ? We all know the answer : processing power was 50 orders of magnitude less than we needed. It was a problem of degree. We knew the problem, and had at least a good guess at what to do, but couldn't contemplate that actually acting on those ideas would make any serious progress due to the limitations of processing speeds at those times. If you wanted a 1960s computer to do a 50x50 matrix multiplication, you'd be waiting days. Doing millions of them was therefore considered useless. Even this is being gentle. A good case can be made that every component of these algorithms was known once differentiation was formalized, but nobody put them together, not because they didn't realize it could work, but because it was useless : doing things this way would have been incredibly inefficient compared to the then normal ways of doing things. E.g. finding formulas and constants by small adjustments on large chains of partial derivatives (ie. "Deep learning") is something that Isaac Newton knew how to do. It's just he would declare you totally mad for doing that. One might criticize that there were a few holes in the mathematical understanding of matrices in Newton's day, I wouldn't doubt that had he had a reason to really look into those problems, he would have fixed them. But he was more than 50 orders of magnitudes of : doing a single 50x50 matrix multiplication with any amount of resources then available wouldn't have finished before he died. Now machine learning tutorials tell you to run unrolled 30x30 LSTM expansions on audio data and unroll for 50 datapoints. And then doing that millions of times (tens of thousands of times on a few thousand samples). You can expect this to run on the computer you're reading this on in a matter of hours.

wodenokoto 11y ago

I have a hard time putting it to words, but I do feel that while computers are pretty good at statistically parsing language, humans have an extra sanity check that parses "meaning" (what ever that may satisfactionally be defined as). Maybe it means we need to hook up our NLP pipeline to deep-visual networks and knowledge-graphs in order to ask "Did the message of that sentence make sense, and if not, should check a similarly sounding sentence or a different parsing of the sentence in order to make it more cohesive"

bendykstra 11y ago
The Wikipedia article on garden path sentences has a good description of that phenomenon. https://en.wikipedia.org/wiki/Garden_path_sentence https://en.wikipedia.org/wiki/Garden_path_sentence
- wodenokoto 11y ago
  Thank you for the link. I do think we have NLP algorithms that can handle garden paths without understanding the sentence. A state of the art Japanese POS-tagger for example has a dictionary of all previously seen word/pos pairs and their transition probabilities. Sine there are no spaces in Japanese, the possible combinations of word/pos pairs you can lay out that matches the input sentence is quite big, and gives you a fairly large graph that you need to find the globally best path through. This means the algorithm does walk down the garden path, but at some point decides that there is a better path that will get you to the end of the sentence. If you go to http://www.atilika.org http://www.atilika.org, scroll down a bit, select the "viterbi" radio button, paste 私は日本人です into the text box and click tokenise, you can see an example of such a graph.
n0us 11y ago
I agree that great advances can be made from statistically parsing language but it is another thing entirely to have a computer (for example) read all of Shakespeare's sonnets and write an essay about the influence they have had on other authors. Even if it is as a highschool level. I kind of see statistical parsing as another "parlor trick" type of AI. Impressive and hard to achieve but somehow separate from true intelligence. It's hard to formally describe this distinction which is why I think that advances will come from cross cutting academic departments and reaching out from the "throw more math and more processors at the problem approach" To achieve a true AI we will have to advance not just our understanding of algorithms, but our understanding of what it means to be intelligent which is a concept that I think most people lack any coherent definition of.

YeGoblynQueenne 11y ago

>> One thing that I think will be challenging is that language has observer depending meaning. The idea with word embeddings is that they provide a context for words, which informs their meaning. It goes back to Zellig Harris and the observation that "constituents of the same type can be replaced by each other" [1], or in other words: words that occur in the same context have the same meaning. [1] Zellig Harris. 1951. Methods in Structural Linguistics.

kajecounterhack 11y ago
If anyone is interested in embeddings, they are really cool and here is a relevant paper: https://cs.stanford.edu/~quocle/paragraph_vector.pdf https://cs.stanford.edu/~quocle/paragraph_vector.pdf (This is to say there is such a thing as an unsupervised method for creating features that encode meaning, and it's being used today for various tasks :] )
zardo 11y ago
The meaning to the reader is not encoded. Maybe something like book to book-review or book to cliff notes mappings could try to capture that.

joe_the_user 11y ago

Per force, there will be some aspects of language that will be easier to solve than others. At the same time, I think that there's been a tendency for people to overestimate the value of just a part of language understanding. To me, Google search with NLP is at best barely better than search with some kind of query language (and often worse imo). Those phone-based robobilling system now can mostly understand what you say but are still essentially only a notch above keypad menus. The thing with complex syntax-based meaning in human language is that really is only worth the trouble if one is setting up an ongoing relationship with an other. A very smart human with vast knowledge still couldn't give great answers to single-sentence questions from people he'd never seen before and who would never see him again. And ongoing language interactions actually tend to connect multiple aspect of human activity making divisions harder. So metaphorically we may have a few very small fruit and more or less one very large one.

vonmoltke 11y ago

> Determining the meaning of a sentence is a problem where the real answer depends on observer dependent perspective and therefore will need a completely different way to measure success compared to more 'mathematical' tasks like Go. Trying to program a machine to account for this kind of personal experience that humans have, as well as for individual differences between people will be quite challenging I think. I also think that the most significant advances will come from cross cutting academic disciplines like Psychology, Linguistics, and Philosophy of Language. I have worked off-and-on on a project to detect metaphors and identify the concrete meaning the metaphor is attempting to convey. Amongst many other things that project taught me that it is nearly impossible to get more than five linguists to agree on what a metaphor is in the first place.