5 ms·

The problem with all of these tests is that they are predefined tasks with an objective measure. If you can spend a bunch of time optimizing your system against

The problem with all of these tests is that they are predefined tasks with an objective measure. If you can spend a bunch of time optimizing your system against the measure, you’ll eventually beat humans. Does that mean you’ve created superhuman intelligence? No! A human can sit down in front of any of these sorts of tests and do reasonably well given no prior training whatsoever. None of these systems are capable of that. Here is a challenge for an AI researcher: create a system that can beat an arbitrary, unknown console video game considered easy[1] for humans. You are not allowed to train your system by having it play video games, since this test is supposed to measure the ability to learn to play video games for the first time. Additionally, your system only has access to the I/O that a human has access to (gamepad inputs, sound and video outputs), no direct access to console memory by any means. [1] I consider a video game easy if humans finish it more than 80% of the time on their first attempt, having never played a video game before in their life.

csomar 7y ago
Sure but humans have been exposed to the controllers, sounds and all these abstractions from previous games. Your AI should have the same and fair exposure. You are looking for a general game-player AI. tl;dr: Your brain is training 24/7 non-stop.
- chongli 7y ago
  Please reread my comment. I specifically state that this test is only to consider a game easy when it is the first game a human has ever played. So the problem of humans having experience with other video games is not present.
  - pmoriarty 7y ago
    So where do you get the data on how well humans did on this hypothetical video game, when that game is the first video game they ever played in their life? It's doubtful whether such data exists. Even if it did it's questionable whether such results have any bearing on human understanding or intelligence.
  - csomar 7y ago
    > first game a human has ever played Like a human that never played a game, used a computer, used any physical buttons, learned a language, understands what a game is, knows what a sport like football is, etc... There is lots of stuff that you'll need to feed your AI to be equivalent to a 6 year-old brain. That and a 6 year-old interaction with a game will be more random than logically driven. If you are doing that you are already at the stage of AGI. It is not theoretically impossible but the impracticality of it right now is what makes scientist trying domain-specific AI.
    - chongli 7y ago
      There is lots of stuff that you'll need to feed your AI to be equivalent to a 6 year-old brain... If you are doing that you are already at the stage of AGI. That’s kind of my point. The article is asking the question “do they understand?” and my answer is yes, if you can do this. Read, to a 6-year-old, a kid’s book about a knight with sword and shield rescuing a princess and they will have no problem playing Zelda. Does the child understand the damsel in distress trope? I would argue yes. This sort of test would be extremely difficult to get an AI to pass. Heck, it doesn’t even have to be video games. Just do any test that involves translating a concept from one medium to another (text and video). It’s extremely simple for humans to do and extremely hard for an AI.
mcbrit 7y ago
I do not believe that game exists. There’s too much embedded knowledge w/r/t playing video games. I am happy to be proven wrong; did you have an example in mind?
- chongli 7y ago
  For a concrete example, I'll nominate the very first video game I ever played: The Legend of Zelda for the NES. This is by no means an easy game, however, so I'd alter the win condition: you don't need to beat the whole game, just collect a single piece of the Triforce. As a kid I was able to gather the first 6 pieces of the Triforce without any outside help. I think an AI should be able to get at least one, given that it's playing a video game for the first time. As for the "embedded knowledge", I'm happy to talk about context in terms of cultural references, heroes, the monomyth, etc. The Zelda series draws heavily on these concepts, though I am not sure how much I knew of them as a 6-year-old.
- alpaca128 7y ago
  I'd say a really simple game like Pong or Snake would already be enough. Only two(or four) possible inputs, simple graphics. The difficulty could also be decreased, but in the end the challenge would be to grasp how the game works, not to beat highscores.
randyrand 7y ago
With a general enough measure you’ll have created general intelligence. e.g. for humans, quantity of reproduction
SolaceQuantum 7y ago
I'd be interested in a system that can beat arbitrary games that are considered relatively simple, or complex across a genre. I'd like an AI that can play bejeweled and then easily transfer that information to an arbitrary merge game, or an AI that can play CoD and transfer that to Overwatch, or an AI that can play MapleStory and then transfer that information to Mario brothers. Or an AI that plays League of Legends and transfers that knowledge to DotA. Does such a thing exist?
telesilla 7y ago
Hm I don't know - I'm pretty bad at games and will try hard at Pong at still lose, my coordination is poor. Maybe you need to be more specific, that the AI can learn the game mechanics without training?
savanaly 7y ago
The funny thing is that even the easiest games are surprisingly difficult, even insurmountable, for otherwise capable humans who have not been "trained" on what to expect and learned domain knowledge through trial and error. See [0]. [0] https://www.youtube.com/watch?v=ax7f3JZJHSw https://www.youtube.com/watch?v=ax7f3JZJHSw
QuadmasterXLII 7y ago
This[1] paper does an excellent job gradually removing cultural priors from a simple videogame, and measuring the extent to which human and computer performance degrade. Their conclusion is that human ability to learn games fast depends on our ability to map them to real world concepts. Would it be permitted for entrants in your contest to pretrain on some other non-videogame task? [1] https://arxiv.org/pdf/1802.10217.pdf https://arxiv.org/pdf/1802.10217.pdf
- chongli 7y ago
  I didn’t state it very clearly but the whole point of my test is the ability to translate cultural priors into a new medium. Here’s another task I’d like to see: have a kid who has never seen a film before read a children’s book and then watch the film adaptation. Test the child on their ability to identify all of the characters in the film. This is the sort of task any kid could do with high accuracy but an AI would struggle with unless it were specifically trained on a corpus of books and their film adaptations.
  - meowface 7y ago
    But like the parent said, you still have to make it fair. The child may have never seen a film before, but they've probably read other books, or have had books read to them, or have had stories told to them (real or fictional). And they've probably seen TV shows or videos of some sort. And they've probably had an upbringing with lots of adults and kids speaking to and around them about topics, and have had (rudimentary) experience telling stories themselves, like what they did today. All of this helps them form a model of the world that makes understanding a book or a film, and the relation between them, easier. So maybe training on a corpus of books and film adaptations is cheating in this scenario, but the AI should still be allowed to train on a massive corpus of all of the things a child may be exposed to. Maybe instead of showing it films, show it thousands of hours of footage from head-mounted cameras that people have worn while going through daily life, and children's TV shows, books, and games, and have it talk to a bunch of humans.
    - irio9kfkf 7y ago
rojobuffalo 7y ago
DeepMind's AlphaZero? I think it's an incoherent premise to suggest that you could have a human with no contextual knowledge or experience that pertains to playing an arbitrary game. There's no blank-slate human that you could test like that. Even the act of finding food and eating, walking around and avoiding hazards, observing the movement and behavior of other people and animals..those experiences shape a person's ability to interpret an environment and model the agents within it. Those priors would be useful for any first-attempt at a game.
- ajna91 7y ago
  AlphaGo needs to be retrained for new games. The breakthrough was in having a generic training algorithm that could be set loose on many games and learn them.
- chongli 7y ago
  Oh, feel free to have your AI read all the books and watch all the films you want. Heck, you can even let your system read the game’s manual, just as a kid would do if they got stuck in the game. The only restriction is that it can’t play any video games as training. The point of my challenge is to demonstrate understanding by being able to translate cultural and contextual information into a new medium. So rather than just making connections between different texts, you need to be able to take a text and leverage it to understand what’s happening on the screen in a video game.
  - doctorpangloss 7y ago
    Maybe you've already learned this from the replies, but it's not really possible to have an intellectual dialog about the meaning or limitations of this research. So while I get what you're saying and I'm sympathetic to your point of view, it's too late. The cloud bucks and 5x VC multiples that can be made on this stuff is too good. Even academics, whose whole advantage is that they can resist market forces, have very lucrative careers awaiting them if they publish the right stuff. Then there's all the adjacent benefits. Like what good does it to be publicly skeptical of DeepMind? Google is everywhere in tech and science. Or even some random guy's paper, like where is he going to work? I can count the number of opportunities on my hand, and they all have a lot of influence in technology broadly. You don't even have to be doing AI for it to be rational to simply say, "Wow what an Amazing Discovery" and move along. There's literally no forum that profits from skepticism. Nobody wins when "AI," whatever it is attached to, loses.
jonshariat 7y ago
Humans do have lots and lots of prior training albeit not specific to that exact task but I still understand your overall point. The more generalized the AI the stronger the claim of being artificial intelligence.
Veedrac 7y ago
You're arguing against a position that nobody sensible holds. It's like you're yelling at the invention of the train because it can't fly to the moon. Well, sure, but it's a necessary step on the way. Researchers aren't not solving your problem because they've not realized it would be an impressive thing to do, they're not solving your problem because the science isn't ready to tackle it yet (and also because it's an incredibly biased metric).

csomar 7y ago

Sure but humans have been exposed to the controllers, sounds and all these abstractions from previous games. Your AI should have the same and fair exposure. You are looking for a general game-player AI. tl;dr: Your brain is training 24/7 non-stop.

chongli 7y ago
Please reread my comment. I specifically state that this test is only to consider a game easy when it is the first game a human has ever played. So the problem of humans having experience with other video games is not present.
- pmoriarty 7y ago
  So where do you get the data on how well humans did on this hypothetical video game, when that game is the first video game they ever played in their life? It's doubtful whether such data exists. Even if it did it's questionable whether such results have any bearing on human understanding or intelligence.
- csomar 7y ago
  > first game a human has ever played Like a human that never played a game, used a computer, used any physical buttons, learned a language, understands what a game is, knows what a sport like football is, etc... There is lots of stuff that you'll need to feed your AI to be equivalent to a 6 year-old brain. That and a 6 year-old interaction with a game will be more random than logically driven. If you are doing that you are already at the stage of AGI. It is not theoretically impossible but the impracticality of it right now is what makes scientist trying domain-specific AI.
  - chongli 7y ago
    There is lots of stuff that you'll need to feed your AI to be equivalent to a 6 year-old brain... If you are doing that you are already at the stage of AGI. That’s kind of my point. The article is asking the question “do they understand?” and my answer is yes, if you can do this. Read, to a 6-year-old, a kid’s book about a knight with sword and shield rescuing a princess and they will have no problem playing Zelda. Does the child understand the damsel in distress trope? I would argue yes. This sort of test would be extremely difficult to get an AI to pass. Heck, it doesn’t even have to be video games. Just do any test that involves translating a concept from one medium to another (text and video). It’s extremely simple for humans to do and extremely hard for an AI.

mcbrit 7y ago

I do not believe that game exists. There’s too much embedded knowledge w/r/t playing video games. I am happy to be proven wrong; did you have an example in mind?

chongli 7y ago
For a concrete example, I'll nominate the very first video game I ever played: The Legend of Zelda for the NES. This is by no means an easy game, however, so I'd alter the win condition: you don't need to beat the whole game, just collect a single piece of the Triforce. As a kid I was able to gather the first 6 pieces of the Triforce without any outside help. I think an AI should be able to get at least one, given that it's playing a video game for the first time. As for the "embedded knowledge", I'm happy to talk about context in terms of cultural references, heroes, the monomyth, etc. The Zelda series draws heavily on these concepts, though I am not sure how much I knew of them as a 6-year-old.
alpaca128 7y ago
I'd say a really simple game like Pong or Snake would already be enough. Only two(or four) possible inputs, simple graphics. The difficulty could also be decreased, but in the end the challenge would be to grasp how the game works, not to beat highscores.

randyrand 7y ago

With a general enough measure you’ll have created general intelligence. e.g. for humans, quantity of reproduction

SolaceQuantum 7y ago

I'd be interested in a system that can beat arbitrary games that are considered relatively simple, or complex across a genre. I'd like an AI that can play bejeweled and then easily transfer that information to an arbitrary merge game, or an AI that can play CoD and transfer that to Overwatch, or an AI that can play MapleStory and then transfer that information to Mario brothers. Or an AI that plays League of Legends and transfers that knowledge to DotA. Does such a thing exist?

telesilla 7y ago

Hm I don't know - I'm pretty bad at games and will try hard at Pong at still lose, my coordination is poor. Maybe you need to be more specific, that the AI can learn the game mechanics without training?

savanaly 7y ago

The funny thing is that even the easiest games are surprisingly difficult, even insurmountable, for otherwise capable humans who have not been "trained" on what to expect and learned domain knowledge through trial and error. See [0]. [0] https://www.youtube.com/watch?v=ax7f3JZJHSw https://www.youtube.com/watch?v=ax7f3JZJHSw

QuadmasterXLII 7y ago

This[1] paper does an excellent job gradually removing cultural priors from a simple videogame, and measuring the extent to which human and computer performance degrade. Their conclusion is that human ability to learn games fast depends on our ability to map them to real world concepts. Would it be permitted for entrants in your contest to pretrain on some other non-videogame task? [1] https://arxiv.org/pdf/1802.10217.pdf https://arxiv.org/pdf/1802.10217.pdf

chongli 7y ago
I didn’t state it very clearly but the whole point of my test is the ability to translate cultural priors into a new medium. Here’s another task I’d like to see: have a kid who has never seen a film before read a children’s book and then watch the film adaptation. Test the child on their ability to identify all of the characters in the film. This is the sort of task any kid could do with high accuracy but an AI would struggle with unless it were specifically trained on a corpus of books and their film adaptations.
- meowface 7y ago
  But like the parent said, you still have to make it fair. The child may have never seen a film before, but they've probably read other books, or have had books read to them, or have had stories told to them (real or fictional). And they've probably seen TV shows or videos of some sort. And they've probably had an upbringing with lots of adults and kids speaking to and around them about topics, and have had (rudimentary) experience telling stories themselves, like what they did today. All of this helps them form a model of the world that makes understanding a book or a film, and the relation between them, easier. So maybe training on a corpus of books and film adaptations is cheating in this scenario, but the AI should still be allowed to train on a massive corpus of all of the things a child may be exposed to. Maybe instead of showing it films, show it thousands of hours of footage from head-mounted cameras that people have worn while going through daily life, and children's TV shows, books, and games, and have it talk to a bunch of humans.
  - irio9kfkf 7y ago

rojobuffalo 7y ago

DeepMind's AlphaZero? I think it's an incoherent premise to suggest that you could have a human with no contextual knowledge or experience that pertains to playing an arbitrary game. There's no blank-slate human that you could test like that. Even the act of finding food and eating, walking around and avoiding hazards, observing the movement and behavior of other people and animals..those experiences shape a person's ability to interpret an environment and model the agents within it. Those priors would be useful for any first-attempt at a game.

ajna91 7y ago
AlphaGo needs to be retrained for new games. The breakthrough was in having a generic training algorithm that could be set loose on many games and learn them.
chongli 7y ago
Oh, feel free to have your AI read all the books and watch all the films you want. Heck, you can even let your system read the game’s manual, just as a kid would do if they got stuck in the game. The only restriction is that it can’t play any video games as training. The point of my challenge is to demonstrate understanding by being able to translate cultural and contextual information into a new medium. So rather than just making connections between different texts, you need to be able to take a text and leverage it to understand what’s happening on the screen in a video game.
- doctorpangloss 7y ago
  Maybe you've already learned this from the replies, but it's not really possible to have an intellectual dialog about the meaning or limitations of this research. So while I get what you're saying and I'm sympathetic to your point of view, it's too late. The cloud bucks and 5x VC multiples that can be made on this stuff is too good. Even academics, whose whole advantage is that they can resist market forces, have very lucrative careers awaiting them if they publish the right stuff. Then there's all the adjacent benefits. Like what good does it to be publicly skeptical of DeepMind? Google is everywhere in tech and science. Or even some random guy's paper, like where is he going to work? I can count the number of opportunities on my hand, and they all have a lot of influence in technology broadly. You don't even have to be doing AI for it to be rational to simply say, "Wow what an Amazing Discovery" and move along. There's literally no forum that profits from skepticism. Nobody wins when "AI," whatever it is attached to, loses.

jonshariat 7y ago

Humans do have lots and lots of prior training albeit not specific to that exact task but I still understand your overall point. The more generalized the AI the stronger the claim of being artificial intelligence.

Veedrac 7y ago

You're arguing against a position that nobody sensible holds. It's like you're yelling at the invention of the train because it can't fly to the moon. Well, sure, but it's a necessary step on the way. Researchers aren't not solving your problem because they've not realized it would be an impressive thing to do, they're not solving your problem because the science isn't ready to tackle it yet (and also because it's an incredibly biased metric).