In thinking of this as a binary question (pass/fail) it feels like we're missing the point. As Kurzweil points out when quoting himself, "By the time there is

In thinking of this as a binary question (pass/fail) it feels like we're missing the point. As Kurzweil points out when quoting himself, "By the time there is a broad consensus that the Turing test has been passed, the actual threshold will have long since been achieved." What we need is more of a Turing "score". The design would be a website (say) where each human participant will be presented with both "defend your humanity" challenges (as a subject) or "judge other's humanity". For the former, you'll try to convince an interviewer that you are human; for the latter, you'll be presented with two subjects, and asked to identify which is human (possibly with a "both human" or "both robots" option). Based on this, individuals will get an ELO score (like chess) on how often they "win" the contest as a subject, that is, are identified as human (or as "more likely to be human than their opponent"). Computer programs will participate as subjects; the requirement will be that the behavior is deterministic given a random seed presented (to them only) at the beginning of the conversation,is to prevent cheating and allow reproducibility. On an orthogonal level, participants acting as judges could be scored on how often they are correct in making the identification; and there's no reason that computer programs could not compete on this side as well. And this could even feed back into the score for humanity; you get more credit for fooling a good judge than fooling a bad judge, etc.