3 ms·
Kurzweil's and Mitch Kapor's rules on how to judge a Turing Test are well thought out ( * 1), but I'm finding that it's actually biased against the computer (an
by computator 12y ago
Kurzweil's and Mitch Kapor's rules on how to judge a Turing Test are well thought out ( * 1), but I'm finding that it's actually biased against the computer (and therefore unfair to Kurzweil). There's a significant chance that a computer with a breathtaking performance will lose by probability alone.
Look at this rule: The Computer will be deemed to have passed the “Turing Test Human Determination Test” if the Computer has fooled two or more of the three Human Judges into thinking that it is a human.
Suppose the Computer is absolutely perfect its responses (i.e., it should pass the Turing Test). The judges know that they're speaking to 3 humans and 1 computer, so if the judges are chatting with 4 equally-good subjects, they'll decide that one of the four is a computer on a whim. There's a chance that Kurzweil will lose just by arbitrariness.
It's like being asked to sample 4 glasses of wine to pick the worst. Unbeknownst to you, all 4 glasses have the same wine. Even though they're equally good, you'll reject one glass by some arbitrary measure. Maybe you felt an itch on your neck while drinking from the second glass, so that one is the bad wine.
( * 1) http://www.kurzweilai.net/a-wager-on-the-turing-test-the-rules http://www.kurzweilai.net/a-wager-on-the-turing-test-the-rul...
- computator 12y agoCan someone please help with the probability calculation? 3 judges and 4 participants What's the probability that any 2 or all 3 judges will pick a particular participant ("the computer") out of 4 at random?
- xmonkee 12y agoall 3 = 1.6% any 2 = 14%
- thesteamboat 12y agoSuppose that the participants are all indistinguishable. Each judge has a 3/4 chance of "being fooled". Let p be the event that the computer is correctly determined, and let q be the event that the computer is not selected. We calculate (1/4p+3/4q)^3 = 1/64(p^3 +3p^2(3q) + 3p(3q)^2+(3q)^3) = 1/64 (p^3 + 9p^2q + 27pq^2 + 27q^3). The probability, then, that the computer is chosen by random chance is 10/64 or approximately 15%.
- nardi 12y agoWhich means in only 14 trials, there's a greater than 90% chance of passing the test. (Again, assuming the contestant is indistinguishable from the judges.)
- kbenson 12y agoIt only has to happen once for Kurzweil to win, so it seems fair to me he has a higher bar. Winning through chance is much less fair than losing through chance in this situation.
- falcor84 12y agoFrom what I understood, the ai would only need to win once. Therefore, after the computer reaches near-human intelligence, it'll be statistically sure to win in a few years. (And given Moore's law, in these few years the ai will continue to significantly improve)