4 ms·

Beyond Local Pattern Matching: Recent Advances in Machine Reading

80 points by andreyk 8y ago

kreelman 8y ago
This looks to be very useful. I wonder how long it will be before we can have one of these datasets being used around the house. Would help with cheating on homework!
palad1n 8y ago
At what point would we say that a machine/system actually understands natural language? Or is the Chinese Room Argument always going to apply?
- serioussecurity 8y ago
  Maybe we reckon with the issue that "understand" doesn't mean anything in the absence of it's implications and we rephrase the question. What do you view as consequences of deciding that it "understands"?
  - palad1n 8y ago
    This might go with the question of whether the Turing Test is a valid measure of "intelligence". If we can't tell the difference between a machine understanding something and a person, is that all that is required?
    - pas 8y ago
      There's a view that intelligence without context is meaningless. (You can't do pattern matching without definition of what your signal is -- and if you just use a trivial measure, like entropy, then white noise is the best signal!) So, you can't be generally intelligent. You can be great at a lot of things, but that means you have priors for those things, which inherently predisposes you against other priors. (Though there are of course basically infinite dimensions, in which you can have priors, but let's just assume some dimensions are more important than others.) What we want is adaptability under goal permanence, meta-goals, usually called alignment. We want an intelligence that is very similar to us. And we like to talk a lot, and we can talk about everything, so we can represent our intelligence through talking, so a Turing test could be a great way to measure how human-similar two "intelligences" are.
- mjburgess 8y ago
  The machine has no reference for its terms, "salt" does not refer. To a machine "salt" is just a distributional pattern within a large body of text, it isnt salt. Ie., what computer scientists call "semantics" is again metaphorical BS, as much as calling an if-statement a "decision maker". Denotation is the most fundamental part of what a natural language is (ie., it is about the world, and caused by the world in a fundamental way). Machines processing terms without their denotations are not using language, they are shuffling it aorund.
nopinsight 8y ago
It is noticeable that all of the top 4 results in the recent Stanford Conversational Question Answering (CoQA) challenge are from groups based in China (the top 2 are from Microsoft Research Asia; the 3th and 4th, Sogou and Fudan; the 5th, anonymous): https://stanfordnlp.github.io/coqa/ https://stanfordnlp.github.io/coqa/ Chinese universities are ranked at 1st, 3th, and 4th in the world for papers published in top AI conferences in 2018. (Tsinghua at the top; followed by CMU, Beijing, CAS, and Stanford) Although some people might disagree with the methodology (e.g. larger departments have an advantage), it is still a decent indicator for China’s progress in the field. http://csrankings.org/#/fromyear/2018/toyear/2018/index?ai&vision&mlmining&nlp&ir&world http://csrankings.org/#/fromyear/2018/toyear/2018/index?ai&v... Perhaps it is time for the US as a nation to spend more resources to develop and attract more people into AI research. See also an Economist’s review of a recent book: “In the struggle for AI supremacy, China will prevail; Or so reckons Kai-Fu Lee, a tech insider, in “AI Superpowers”” https://www.economist.com/books-and-arts/2018/09/27/in-the-struggle-for-ai-supremacy-china-will-prevail https://www.economist.com/books-and-arts/2018/09/27/in-the-s...
- theblackcat1002 8y ago
  FYI, the top 6 results are mainly based on BERT which is proposed by Google[1]. [1] https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805
  - nopinsight 8y ago
    Yes, BERT is a significant advance. I think most would agree that many key advances still originate from the US. (And the US is probably still ahead overall in research.) China is apparently faster at adapting and deploying these changes in some areas and they are very focused on excelling at the technology, which has many major implications on society.
- antpls 8y ago
  Those rankings based on countries don't say much IMHO. It depends on : - the score. If the top 5 achieves 85/100 and the the rest of the top 20 achieves 83/100, they are all on par. - the field. If it's academic with open source code and scientific results shared with everyone on Earth, who cares who is in the top 5? It's a global effort
andreyk 8y ago
Some other fun recent-ish developments in QA datasets: - Natural Questions (https://ai.googleblog.com/2019/01/natural-questions-new-corpus-and.html https://ai.googleblog.com/2019/01/natural-questions-new-corp...) - SearchQA (https://arxiv.org/abs/1704.05179 https://arxiv.org/abs/1704.05179) Many AI researchers worry the community is overfitting to getting good numbers and not thinking big enough, but it seems to me datasets evolve fast enough to keep up with doing interesting things.

80 points by andreyk 8y ago

This looks to be very useful. I wonder how long it will be before we can have one of these datasets being used around the house. Would help with cheating on homework!

palad1n 8y ago

At what point would we say that a machine/system actually understands natural language? Or is the Chinese Room Argument always going to apply?

serioussecurity 8y ago
Maybe we reckon with the issue that "understand" doesn't mean anything in the absence of it's implications and we rephrase the question. What do you view as consequences of deciding that it "understands"?
- palad1n 8y ago
  This might go with the question of whether the Turing Test is a valid measure of "intelligence". If we can't tell the difference between a machine understanding something and a person, is that all that is required?
  - pas 8y ago
    There's a view that intelligence without context is meaningless. (You can't do pattern matching without definition of what your signal is -- and if you just use a trivial measure, like entropy, then white noise is the best signal!) So, you can't be generally intelligent. You can be great at a lot of things, but that means you have priors for those things, which inherently predisposes you against other priors. (Though there are of course basically infinite dimensions, in which you can have priors, but let's just assume some dimensions are more important than others.) What we want is adaptability under goal permanence, meta-goals, usually called alignment. We want an intelligence that is very similar to us. And we like to talk a lot, and we can talk about everything, so we can represent our intelligence through talking, so a Turing test could be a great way to measure how human-similar two "intelligences" are.
mjburgess 8y ago
The machine has no reference for its terms, "salt" does not refer. To a machine "salt" is just a distributional pattern within a large body of text, it isnt salt. Ie., what computer scientists call "semantics" is again metaphorical BS, as much as calling an if-statement a "decision maker". Denotation is the most fundamental part of what a natural language is (ie., it is about the world, and caused by the world in a fundamental way). Machines processing terms without their denotations are not using language, they are shuffling it aorund.

nopinsight 8y ago

It is noticeable that all of the top 4 results in the recent Stanford Conversational Question Answering (CoQA) challenge are from groups based in China (the top 2 are from Microsoft Research Asia; the 3th and 4th, Sogou and Fudan; the 5th, anonymous): https://stanfordnlp.github.io/coqa/ https://stanfordnlp.github.io/coqa/ Chinese universities are ranked at 1st, 3th, and 4th in the world for papers published in top AI conferences in 2018. (Tsinghua at the top; followed by CMU, Beijing, CAS, and Stanford) Although some people might disagree with the methodology (e.g. larger departments have an advantage), it is still a decent indicator for China’s progress in the field. http://csrankings.org/#/fromyear/2018/toyear/2018/index?ai&vision&mlmining&nlp&ir&world http://csrankings.org/#/fromyear/2018/toyear/2018/index?ai&v... Perhaps it is time for the US as a nation to spend more resources to develop and attract more people into AI research. See also an Economist’s review of a recent book: “In the struggle for AI supremacy, China will prevail; Or so reckons Kai-Fu Lee, a tech insider, in “AI Superpowers”” https://www.economist.com/books-and-arts/2018/09/27/in-the-struggle-for-ai-supremacy-china-will-prevail https://www.economist.com/books-and-arts/2018/09/27/in-the-s...

theblackcat1002 8y ago
FYI, the top 6 results are mainly based on BERT which is proposed by Google[1]. [1] https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805
- nopinsight 8y ago
  Yes, BERT is a significant advance. I think most would agree that many key advances still originate from the US. (And the US is probably still ahead overall in research.) China is apparently faster at adapting and deploying these changes in some areas and they are very focused on excelling at the technology, which has many major implications on society.
antpls 8y ago
Those rankings based on countries don't say much IMHO. It depends on : - the score. If the top 5 achieves 85/100 and the the rest of the top 20 achieves 83/100, they are all on par. - the field. If it's academic with open source code and scientific results shared with everyone on Earth, who cares who is in the top 5? It's a global effort

andreyk 8y ago

Some other fun recent-ish developments in QA datasets: - Natural Questions (https://ai.googleblog.com/2019/01/natural-questions-new-corpus-and.html https://ai.googleblog.com/2019/01/natural-questions-new-corp...) - SearchQA (https://arxiv.org/abs/1704.05179 https://arxiv.org/abs/1704.05179) Many AI researchers worry the community is overfitting to getting good numbers and not thinking big enough, but it seems to me datasets evolve fast enough to keep up with doing interesting things.