3 ms·
Soon we’ll need a richer approach to Turing tests. I think some sort of Elo system for both testers and test subjects might be in order. Reward testers who are
by thom 6y ago
Soon we’ll need a richer approach to Turing tests. I think some sort of Elo system for both testers and test subjects might be in order. Reward testers who are good at catching AIs out, don’t overly reward AIs for tricking people who just ask a few simple questions and don’t push too hard.
- mmhsieh 6y agoElos would give a nice apples to apples comparison for something that is an inherently comparative skill. Ironically, Elo may be better for AI than for chess now; in chess, there used to be only a relative measure of strength, but with engines being what they are, there is an absolute set of optimal moves and valuations of mistakes. So while I do not know what an absolute standard for AI quality vs. a Turing test, there is one for chess, and a thing invented for chess might be better used for a non-chess thing.
- pgt 6y agoElo ratings for dueling AIs are a good idea! I wonder if it follows from having limited memory that some AIs will be superior at detecting AIs but struggle to pass the tests themselves.
- thom 6y agoI meant just as much giving human testers a rating. Duelling AIs is sort of already a common training practise in many domains, right?
- renewiltord 6y agoThat’s what GANs do.
- Miiko 6y agoI see a World Turing Championship in the future. Half of participants are humans, the other half is AIs. After a series of matches - each consisting of opponents trying to figure out whether its adversary is human or not, - two champions are declared, one human for best results in detecting AI, one AI for best pretending being human. ;)
- jobigoud 6y agoI can see the plot twist from there, where the protagonist was in fact not a human all along :-) It's interesting that in such a setup, all participants must pretend to be judges. I've always felt uncomfortable that the Turing test is rewarding good lying. The AI must lie to pass. But developing judge AI is very interesting I think. Even in your setup, immediately, the Judge-side winner is not guaranteed to be a human, since each match results in an evaluation of both sides and AIs can be trained to detect other AIs. And obviously once we have one that is very good at judging, we can fold it over itself for self-improvement. Actually I wonder if being a good Turing test judge is an easier goal than passing the Turing test itself. All you need is come up with interesting questions and understand enough of the answers to make a judgment. It should be relatively easy to create this dataset by combining GPT-3 turing test and Mechanical turk turing tests. Can AI beat humans at this task before we have full AGI? A while ago someone posted a Show HN thread where you would be matched with 50% probability with a human or AI and you could ask questions to guess which. In reality there was no AI, it was all humans vs humans, and they were harvesting Turing test questions!