7 ms·

Machines Beat Humans on a Reading Test

46 points by howard941 7y ago

glofish 7y ago
I can go to my Google photos, type a query for a: car, bike, chair. It will produce many images that do contain the object in question. It is amazing how well it works. But then it will also produce photos that have nothing even similar to what I or anyone else would consider "a chair", "a bike" etc. If the question is now: does Google AI understand what a chair is? The answer is: obviously it does not. It does something completely different than "understanding" what a chair is. That method does work well, but it is not the "chair" we all know about.
- riku_iki 7y ago
  I think your example demonstrates pattern recognition imperfection. Understanding will start with questions like "cat sitting on the chair".
sebringj 7y ago
I really like this article because it exposes the notion of computation vs. consciousness. We don't really understand consciousness but have a feeling for what it means. It could be that approximating consciousness so close, similar to the concept of limits in calculus would render it the same as what we have in terms of our inability to ever tell the difference if we cannot define it in the first place. The behavior of consciousness would be akin to approaching infinity.
- klodolph 7y ago
  If you don’t know what consciousness means, is that because the word “consciousness” is vague, or is that because reality itself is vague? It is the word that is vague. By trying to build artificial consciousness, we are exploring, and creating machines that challenge our definition and understanding of the word “consciousness”. Usually we discover that the machine does not have consciousness and our definition for “consciousness” was wrong. Sometimes we discover that our definition of “consciousness” is so bad that we have to go back to the drawing board and come up with a completely new definition. Same thing happens with the word “intelligence”.
  - Retric 7y ago
    It’s not that consciousness is really vague, it’s more that people use vastly different definitions for the same word. Medically consciousness has a relatively low bar, but philosophy loves to overload it with implied meanings.
    - semiotagonal 7y ago
      Even medically, you're not evaluating anything about consciousness (i.e. subjective experience) directly. You're just measuring things that we think imply consciousness.
      - Retric 7y ago
        This is where it get's tricky, some people take consciousness to imply sentience which requires subjective experience others are happy to with just awareness. A venus fly trap is aware of it's environment and has a useful response. In many peoples books that's consciousness others want to limit things to just say it's aware but lacks consciousness. IMO, that's the kind of divide that's prevents meaningful discussions.
        dwaltrip 7y ago
        I would say that "awareness" requires an entity to contain a detailed representation of the world around it that is updated over time as it interacts with the world. Of course, there is a spectrum here, depending on the complexity and dynamicness of this internal representation of the world. Perhaps it goes something like: absolutely zero awareness (e.g. a rock), to barely aware (a Venus fly trap), to highly aware (elephants, humans, etc). I'm sure the scale doesn't stop there, but we haven't had a chance to observe anything other than the highly aware animals here on Earth. You could break this down further, looking at things like episodic memory, the different senses of self, and so on. But humanity has only scratched the surface when it comes to understanding these things.
        andrepd 7y ago
        Then you just need to define what the terms mean. In the rest of the conversation, let "consciousness" mean "the subjective conscious experience (your qualitative experience of sight, sound, etc)", rather than "awareness", "medical wakefulness", or anything. Then you can talk about this concept instead of getting bogged down in definitions.
klodolph 7y ago
At some point we’re going to start asking the same questions about humans. Do humans really understand what we read, or have we just picked up weird tricks that happen to work? (When you read a novel, the experience of reading the novel is different for everyone. It’s like being on a rollercoaster or amusement park ride. Everyone has roughly the same stimulus but a different mental landscape and a different reaction. Our response to the book is largely a response to our own experience of reading the book, and only partly a response to the content of the book itself. This distinction isn’t just sophistry. Are we prepared for machines that react to books in ways that are completely alien to our own reactions?)
- deleted 7y ago
  [deleted]
- tgv 7y ago
  Certainly a lot of exams are filled in using tricks, bordering on superstition. Words that remind a student of a certain kind of solution can cause application of the wrong formula. In that sense, ML, statistical, or "deep learning" models perform a bit like us. Not surprisingly, since co-occurrence plays a large role in our fast intuitions. It also shows that such models are miles away from higher level reasoning.
- pmoriarty 7y ago
  "At some point we're going to start asking the same questions about humans. Do humans really understand what we read, or have we just picked up weird tricks that happen to work?" That question has been asked by philosophers and psychologists for a long time. There's a branch of philosophy called epistemology -- the study of how we know what we know -- where questions about whether we even know what we claim to know (or understand what we claim to understand) belong. In addition, nearly a hundred years ago behaviorist psychologists were positing that all we can see is other people's behavior, so making claims about their supposed mind states, understanding, etc, were unwarranted and effectively off-limits... and then psychology, cognitive science and linguistics had a backlash against behaviorists, rejecting their approach and assumptions, with mental states, thought, and understanding being admissible in the field for many decades now. So none of this is really new, and mountains of books and papers have been written discussing and debating such issues for at least a hundred years. The Chinese Room thought experiment discussed in the article has been a staple of many Philosophy 101 courses, where the obvious followup question to ask (even for students just getting their first exposure to philosophy) is whether humans understand what they read to begin with. Pick up any related philosophy book and they'll discuss such questions ad nauseum. Taking a much further step back in to history, more than 2000 years ago, the Skeptics were questioning whether we can know anything, and some were denying that anything can be known. Philosophers in the West have been debating such questions ever since. In Eastern philosophy and religion, such concerns are also very old, with a variety of people asserting the illusory nature of worldly knowledge, and positing that true reality is unknowable, or at least not accessible or limited by rational thought. Similar themes are echoed in Western mysticism. Jumping forward yet again (forgive the rambling nature of this response), we have Descartes Duck[1], and the popular Enlightenment and Post-Enlightenment view that "man is just a machine". Questions about whether we really understand or are really doing no more than a sophisticated version of what the Chinese Room or "dumb" neural networks do could be seen as modern restatements of that view. [1] - https://en.wikipedia.org/wiki/Digesting_Duck https://en.wikipedia.org/wiki/Digesting_Duck
zitterbewegung 7y ago
Machines perform tasks that require reasoning the same way airplanes fly without flapping their wings. This philosophical question keeps on being asked . From my understanding right now I think it should be approached the same way Feynman approached quantum physics. Just train your models.
chongli 7y ago
The problem with all of these tests is that they are predefined tasks with an objective measure. If you can spend a bunch of time optimizing your system against the measure, you’ll eventually beat humans. Does that mean you’ve created superhuman intelligence? No! A human can sit down in front of any of these sorts of tests and do reasonably well given no prior training whatsoever. None of these systems are capable of that. Here is a challenge for an AI researcher: create a system that can beat an arbitrary, unknown console video game considered easy[1] for humans. You are not allowed to train your system by having it play video games, since this test is supposed to measure the ability to learn to play video games for the first time. Additionally, your system only has access to the I/O that a human has access to (gamepad inputs, sound and video outputs), no direct access to console memory by any means. [1] I consider a video game easy if humans finish it more than 80% of the time on their first attempt, having never played a video game before in their life.
- csomar 7y ago
  Sure but humans have been exposed to the controllers, sounds and all these abstractions from previous games. Your AI should have the same and fair exposure. You are looking for a general game-player AI. tl;dr: Your brain is training 24/7 non-stop.
  - chongli 7y ago
    Please reread my comment. I specifically state that this test is only to consider a game easy when it is the first game a human has ever played. So the problem of humans having experience with other video games is not present.
    - pmoriarty 7y ago
      So where do you get the data on how well humans did on this hypothetical video game, when that game is the first video game they ever played in their life? It's doubtful whether such data exists. Even if it did it's questionable whether such results have any bearing on human understanding or intelligence.
    - 7y ago
asdfman123 7y ago
> Machines Beat Humans on a Reading Test. But Do They Understand? I mean, I was so thoroughly trained to take standardized tests that I really didn't understand what I was reading. Through speed-skimming, selective underlining and recognizing the most probable answers, I was able to speed through reading tests without doing any normal reading. Oh, you're talking about machines?
- ppod 7y ago
  >Machines beat humans on a reading test. But do they understand? I can't link directly to the result, but if you select "co-reference" from the dropdown, and enter the above text, this demo correctly links "they" to "Machines". https://corenlp.run/ https://corenlp.run/
phaedryx 7y ago
Whenever I see claims of computers "understanding" I immediately think of the "Chinese Room" argument. https://plato.stanford.edu/entries/chinese-room/ https://plato.stanford.edu/entries/chinese-room/ https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room
armitron 7y ago
"The question of whether machines can think is about as relevant as the question of whether submarines can swim."
carapace 7y ago
Q: "What can a human do that a machine cannot?" A: "Answer that question." Mark Miller pointed out, "A Computer's Perspective on Moore's Law: Humans are getting more expensive at an exponential rate." https://web.archive.org/web/20160316014939/http://www.caplet.com/adages.html https://web.archive.org/web/20160316014939/http://www.caplet... If there's nothing we can do that they can't..?
- Veedrac 7y ago
  > What can a human do that a machine cannot? The ability to think like a human. The ability to see and recognize the world around you. The ability to change your behavior. The ability to communicate. The ability to be free in a society driven by fear and self interest. The ability to look for love and trust in others. The ability to learn how to love yourself. The ability to be independent. The ability to be successful by your own choice. A human can make you laugh, cry, scream, and cry again. A human can design, analyze, execute, innovate, understand and apply new technology. A machine cannot do that. - GPT-2 774M (cherry-picked samples)
  - carapace 7y ago
    Technically correct, yes, but in a way that misses the point I was trying to make. (A way that I'm finding very difficult to articulate... Which in itself is interesting.) If you want to know the difference between a machine and a human you need a human to compare to. (That's not quite it either, I think, but perhaps is a little less obscure?)
deleted 7y ago
[deleted]
catalogia 7y ago
I find it interesting that so many people are having defensive reactions to this, trying to explain why they think this 'doesn't count', isn't important or significant in some way, etc. Perhaps this is rooted in some fear of obsolescence? I'm not sure. Rather than thinking "this machine can't do it as good as me", why not think of ways you could use the machine to enhance your own abilities? I think it would be really interesting to have a system at my disposal that can "read" a book and generate book reports/digests. This could allow me to spend less time reading speculatively and focus more of my (finite) reading time on books I am already reasonably confident contain what I'm after. (Edit: maybe it's because the headline frames it as a competition, setting people to think of themselves competing with the machine, rather than collaborating with it.)
ilaksh 7y ago
Deep understanding will not come from these methods that are only trained on text. It will require multi-modal input such as images or video in addition to text.

46 points by howard941 7y ago

I can go to my Google photos, type a query for a: car, bike, chair. It will produce many images that do contain the object in question. It is amazing how well it works. But then it will also produce photos that have nothing even similar to what I or anyone else would consider "a chair", "a bike" etc. If the question is now: does Google AI understand what a chair is? The answer is: obviously it does not. It does something completely different than "understanding" what a chair is. That method does work well, but it is not the "chair" we all know about.

riku_iki 7y ago
I think your example demonstrates pattern recognition imperfection. Understanding will start with questions like "cat sitting on the chair".

sebringj 7y ago

I really like this article because it exposes the notion of computation vs. consciousness. We don't really understand consciousness but have a feeling for what it means. It could be that approximating consciousness so close, similar to the concept of limits in calculus would render it the same as what we have in terms of our inability to ever tell the difference if we cannot define it in the first place. The behavior of consciousness would be akin to approaching infinity.

klodolph 7y ago
If you don’t know what consciousness means, is that because the word “consciousness” is vague, or is that because reality itself is vague? It is the word that is vague. By trying to build artificial consciousness, we are exploring, and creating machines that challenge our definition and understanding of the word “consciousness”. Usually we discover that the machine does not have consciousness and our definition for “consciousness” was wrong. Sometimes we discover that our definition of “consciousness” is so bad that we have to go back to the drawing board and come up with a completely new definition. Same thing happens with the word “intelligence”.
- Retric 7y ago
  It’s not that consciousness is really vague, it’s more that people use vastly different definitions for the same word. Medically consciousness has a relatively low bar, but philosophy loves to overload it with implied meanings.
  - semiotagonal 7y ago
    Even medically, you're not evaluating anything about consciousness (i.e. subjective experience) directly. You're just measuring things that we think imply consciousness.
    - Retric 7y ago
      This is where it get's tricky, some people take consciousness to imply sentience which requires subjective experience others are happy to with just awareness. A venus fly trap is aware of it's environment and has a useful response. In many peoples books that's consciousness others want to limit things to just say it's aware but lacks consciousness. IMO, that's the kind of divide that's prevents meaningful discussions.
      - dwaltrip 7y ago
        I would say that "awareness" requires an entity to contain a detailed representation of the world around it that is updated over time as it interacts with the world. Of course, there is a spectrum here, depending on the complexity and dynamicness of this internal representation of the world. Perhaps it goes something like: absolutely zero awareness (e.g. a rock), to barely aware (a Venus fly trap), to highly aware (elephants, humans, etc). I'm sure the scale doesn't stop there, but we haven't had a chance to observe anything other than the highly aware animals here on Earth. You could break this down further, looking at things like episodic memory, the different senses of self, and so on. But humanity has only scratched the surface when it comes to understanding these things.
      - andrepd 7y ago
        Then you just need to define what the terms mean. In the rest of the conversation, let "consciousness" mean "the subjective conscious experience (your qualitative experience of sight, sound, etc)", rather than "awareness", "medical wakefulness", or anything. Then you can talk about this concept instead of getting bogged down in definitions.

klodolph 7y ago

At some point we’re going to start asking the same questions about humans. Do humans really understand what we read, or have we just picked up weird tricks that happen to work? (When you read a novel, the experience of reading the novel is different for everyone. It’s like being on a rollercoaster or amusement park ride. Everyone has roughly the same stimulus but a different mental landscape and a different reaction. Our response to the book is largely a response to our own experience of reading the book, and only partly a response to the content of the book itself. This distinction isn’t just sophistry. Are we prepared for machines that react to books in ways that are completely alien to our own reactions?)

deleted 7y ago
[deleted]
tgv 7y ago
Certainly a lot of exams are filled in using tricks, bordering on superstition. Words that remind a student of a certain kind of solution can cause application of the wrong formula. In that sense, ML, statistical, or "deep learning" models perform a bit like us. Not surprisingly, since co-occurrence plays a large role in our fast intuitions. It also shows that such models are miles away from higher level reasoning.
pmoriarty 7y ago
"At some point we're going to start asking the same questions about humans. Do humans really understand what we read, or have we just picked up weird tricks that happen to work?" That question has been asked by philosophers and psychologists for a long time. There's a branch of philosophy called epistemology -- the study of how we know what we know -- where questions about whether we even know what we claim to know (or understand what we claim to understand) belong. In addition, nearly a hundred years ago behaviorist psychologists were positing that all we can see is other people's behavior, so making claims about their supposed mind states, understanding, etc, were unwarranted and effectively off-limits... and then psychology, cognitive science and linguistics had a backlash against behaviorists, rejecting their approach and assumptions, with mental states, thought, and understanding being admissible in the field for many decades now. So none of this is really new, and mountains of books and papers have been written discussing and debating such issues for at least a hundred years. The Chinese Room thought experiment discussed in the article has been a staple of many Philosophy 101 courses, where the obvious followup question to ask (even for students just getting their first exposure to philosophy) is whether humans understand what they read to begin with. Pick up any related philosophy book and they'll discuss such questions ad nauseum. Taking a much further step back in to history, more than 2000 years ago, the Skeptics were questioning whether we can know anything, and some were denying that anything can be known. Philosophers in the West have been debating such questions ever since. In Eastern philosophy and religion, such concerns are also very old, with a variety of people asserting the illusory nature of worldly knowledge, and positing that true reality is unknowable, or at least not accessible or limited by rational thought. Similar themes are echoed in Western mysticism. Jumping forward yet again (forgive the rambling nature of this response), we have Descartes Duck[1], and the popular Enlightenment and Post-Enlightenment view that "man is just a machine". Questions about whether we really understand or are really doing no more than a sophisticated version of what the Chinese Room or "dumb" neural networks do could be seen as modern restatements of that view. [1] - https://en.wikipedia.org/wiki/Digesting_Duck https://en.wikipedia.org/wiki/Digesting_Duck

zitterbewegung 7y ago

Machines perform tasks that require reasoning the same way airplanes fly without flapping their wings. This philosophical question keeps on being asked . From my understanding right now I think it should be approached the same way Feynman approached quantum physics. Just train your models.

chongli 7y ago

The problem with all of these tests is that they are predefined tasks with an objective measure. If you can spend a bunch of time optimizing your system against the measure, you’ll eventually beat humans. Does that mean you’ve created superhuman intelligence? No! A human can sit down in front of any of these sorts of tests and do reasonably well given no prior training whatsoever. None of these systems are capable of that. Here is a challenge for an AI researcher: create a system that can beat an arbitrary, unknown console video game considered easy[1] for humans. You are not allowed to train your system by having it play video games, since this test is supposed to measure the ability to learn to play video games for the first time. Additionally, your system only has access to the I/O that a human has access to (gamepad inputs, sound and video outputs), no direct access to console memory by any means. [1] I consider a video game easy if humans finish it more than 80% of the time on their first attempt, having never played a video game before in their life.

csomar 7y ago
Sure but humans have been exposed to the controllers, sounds and all these abstractions from previous games. Your AI should have the same and fair exposure. You are looking for a general game-player AI. tl;dr: Your brain is training 24/7 non-stop.
- chongli 7y ago
  Please reread my comment. I specifically state that this test is only to consider a game easy when it is the first game a human has ever played. So the problem of humans having experience with other video games is not present.
  - pmoriarty 7y ago
    So where do you get the data on how well humans did on this hypothetical video game, when that game is the first video game they ever played in their life? It's doubtful whether such data exists. Even if it did it's questionable whether such results have any bearing on human understanding or intelligence.
  - 7y ago

asdfman123 7y ago

> Machines Beat Humans on a Reading Test. But Do They Understand? I mean, I was so thoroughly trained to take standardized tests that I really didn't understand what I was reading. Through speed-skimming, selective underlining and recognizing the most probable answers, I was able to speed through reading tests without doing any normal reading. Oh, you're talking about machines?

ppod 7y ago
>Machines beat humans on a reading test. But do they understand? I can't link directly to the result, but if you select "co-reference" from the dropdown, and enter the above text, this demo correctly links "they" to "Machines". https://corenlp.run/ https://corenlp.run/

phaedryx 7y ago

Whenever I see claims of computers "understanding" I immediately think of the "Chinese Room" argument. https://plato.stanford.edu/entries/chinese-room/ https://plato.stanford.edu/entries/chinese-room/ https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room

armitron 7y ago

"The question of whether machines can think is about as relevant as the question of whether submarines can swim."

carapace 7y ago

Q: "What can a human do that a machine cannot?" A: "Answer that question." Mark Miller pointed out, "A Computer's Perspective on Moore's Law: Humans are getting more expensive at an exponential rate." https://web.archive.org/web/20160316014939/http://www.caplet.com/adages.html https://web.archive.org/web/20160316014939/http://www.caplet... If there's nothing we can do that they can't..?

Veedrac 7y ago
> What can a human do that a machine cannot? The ability to think like a human. The ability to see and recognize the world around you. The ability to change your behavior. The ability to communicate. The ability to be free in a society driven by fear and self interest. The ability to look for love and trust in others. The ability to learn how to love yourself. The ability to be independent. The ability to be successful by your own choice. A human can make you laugh, cry, scream, and cry again. A human can design, analyze, execute, innovate, understand and apply new technology. A machine cannot do that. - GPT-2 774M (cherry-picked samples)
- carapace 7y ago
  Technically correct, yes, but in a way that misses the point I was trying to make. (A way that I'm finding very difficult to articulate... Which in itself is interesting.) If you want to know the difference between a machine and a human you need a human to compare to. (That's not quite it either, I think, but perhaps is a little less obscure?)

deleted 7y ago

[deleted]

catalogia 7y ago

I find it interesting that so many people are having defensive reactions to this, trying to explain why they think this 'doesn't count', isn't important or significant in some way, etc. Perhaps this is rooted in some fear of obsolescence? I'm not sure. Rather than thinking "this machine can't do it as good as me", why not think of ways you could use the machine to enhance your own abilities? I think it would be really interesting to have a system at my disposal that can "read" a book and generate book reports/digests. This could allow me to spend less time reading speculatively and focus more of my (finite) reading time on books I am already reasonably confident contain what I'm after. (Edit: maybe it's because the headline frames it as a competition, setting people to think of themselves competing with the machine, rather than collaborating with it.)

ilaksh 7y ago

Deep understanding will not come from these methods that are only trained on text. It will require multi-modal input such as images or video in addition to text.