3 ms·

For Turing's test with the binary choice, the pass threshold is clear. If the machine and human are indistinguishable, then the probabilities that they're judge

For Turing's test with the binary choice, the pass threshold is clear. If the machine and human are indistinguishable, then the probabilities that they're judged human must be equal. Since they sum to 100%, they must both equal 50%, making that a meaningful pass threshold. (A slightly higher pass threshold should be used in practice for statistical convenience, since infinitely many trials are required to make a confidence interval exactly include 50%. I'd guess that's why Turing mentions 70% in his paper.) Without the binary choice, what do you think is the correct pass threshold? Those probabilities can now sum to anything. For GPT-4 in Jones and Bergen's paper they sum to 121%, though please nobody say 60.5%. The threshold now obviously depends on the interrogator's prior--I'd judge very differently if I were told the witnesses were 99% human than if I were told they were 1% human. In that paper, do you think the interrogators knew their witness had only a 25% chance of being human? If so, why? If not, how do you think that affected the result? In aggregate over all the witnesses, their interrogators seem to have judged correctly only 60% of the time, while always guessing "machine" would have scored 75%. How did they manage to score worse than chance? Turing's formulation is elegant, admitting meaningful statistical analysis with minimum assumptions. Most modifications are not, and that paper's is particularly bad. Turing's description may seem casual, but it's filled with mathematical depth that should not be missed.

fenomas 2y ago
The setup Turing describes isn't the "both must sum to 100%" setup you're presenting. He has two different games being played, one with two humans and one with human-vs-machine, and suggests comparing the results. E.g. if a man successfully imitates a woman only 25% of the time, then we'd ask whether the machine can pass as human equally often. But much more importantly, as I said Turing is clearly not describing a specific experimental methodology! That's not what the paper is about, and in fact it would be somewhat absurd to run the test precisely as he describes it (since detecting a man imitating a woman is quite a different task from detecting a machine). His point is that we should approach the question of machine intelligence with actual experiments rather than asking unanswerable questions, but he only limns the general premise of what such an experiment could look like. So I understand that you find a particular test setup better or more elegant than others, and that's fine. But you shouldn't claim that Turing's paper demands your preferred setup, or that other setups are at odds with his paper.
- tripletao 2y ago
  > The setup Turing describes isn't the "both must sum to 100%" setup you're presenting. He has two different games being played [...] Turing introduces the game as a man imitating a woman, then modifies it to a machine imitating a human. In both of those games, the interrogator makes a binary choice between two witnesses, one real and one imitating. So P(imitator-judged-real) and P(real-judged-real) sum to 100% in both games. So in both games, a score of 50% means the imitator and the real witness are indistinguishable. I believe that's the reason why a score of 50% is treated as significant. The "GPT-4 passes Turing test" paper uses that number as a pass threshold, and the linked site repeats it. I'm complaining that the paper changed the game so it's no longer a binary choice, but continued to treat the 50% threshold as significant. Do you see why that's wrong? Or do you disagree? I'm not saying that any change to Turing's formulation would be bad, just that the paper's variation is specifically bad. It would be bad in isolation too, but I believe the reason for their confusion is most understandable with reference to Turing's original formulation. If you haven't read that paper then we're probably talking past each other. The link to that one also looks broken in the site, but it's also in the source, https://arxiv.org/pdf/2405.08007 https://arxiv.org/pdf/2405.08007
  - fenomas 2y ago
    > I'm not saying that any change to Turing's formulation would be bad, just that the paper's variation is specifically bad. I understand that, and I'm saying there is no "Turing's formulation". His paper argues for a certain sort of test, and the study you're talking about is the sort of test he advocated. It's not a departure or a bastardization, it's a Turing test. As for your argument against the study, to be honest I don't see it? AFAICT the participants' goal was just to judge the humanness of a single witness, not to maximize their long-term likelihood of judging correctly over many trials, or some such thing. Even if they'd known the prior chances of speaking to an LLM, there's no obvious reason why that prior should hugely affect their conclusion after a five minute conversation - which also seems immaterial since they didn't know. Plus, the authors give a pretty straightforward rationale for their 50% threshold, and it has no connection to Turing's 3-player setup or whether the imitator and witness are indistinguishable. If they had wanted "indistinguishable" as a threshold, then obviously their pass criteria would have been for the machine and human pass rates to be equal within an error bar, right? So it's pretty implausible to imagine a connection between that and their 50% threshold.

fenomas 2y ago

The setup Turing describes isn't the "both must sum to 100%" setup you're presenting. He has two different games being played, one with two humans and one with human-vs-machine, and suggests comparing the results. E.g. if a man successfully imitates a woman only 25% of the time, then we'd ask whether the machine can pass as human equally often. But much more importantly, as I said Turing is clearly not describing a specific experimental methodology! That's not what the paper is about, and in fact it would be somewhat absurd to run the test precisely as he describes it (since detecting a man imitating a woman is quite a different task from detecting a machine). His point is that we should approach the question of machine intelligence with actual experiments rather than asking unanswerable questions, but he only limns the general premise of what such an experiment could look like. So I understand that you find a particular test setup better or more elegant than others, and that's fine. But you shouldn't claim that Turing's paper demands your preferred setup, or that other setups are at odds with his paper.

tripletao 2y ago
> The setup Turing describes isn't the "both must sum to 100%" setup you're presenting. He has two different games being played [...] Turing introduces the game as a man imitating a woman, then modifies it to a machine imitating a human. In both of those games, the interrogator makes a binary choice between two witnesses, one real and one imitating. So P(imitator-judged-real) and P(real-judged-real) sum to 100% in both games. So in both games, a score of 50% means the imitator and the real witness are indistinguishable. I believe that's the reason why a score of 50% is treated as significant. The "GPT-4 passes Turing test" paper uses that number as a pass threshold, and the linked site repeats it. I'm complaining that the paper changed the game so it's no longer a binary choice, but continued to treat the 50% threshold as significant. Do you see why that's wrong? Or do you disagree? I'm not saying that any change to Turing's formulation would be bad, just that the paper's variation is specifically bad. It would be bad in isolation too, but I believe the reason for their confusion is most understandable with reference to Turing's original formulation. If you haven't read that paper then we're probably talking past each other. The link to that one also looks broken in the site, but it's also in the source, https://arxiv.org/pdf/2405.08007 https://arxiv.org/pdf/2405.08007
- fenomas 2y ago
  > I'm not saying that any change to Turing's formulation would be bad, just that the paper's variation is specifically bad. I understand that, and I'm saying there is no "Turing's formulation". His paper argues for a certain sort of test, and the study you're talking about is the sort of test he advocated. It's not a departure or a bastardization, it's a Turing test. As for your argument against the study, to be honest I don't see it? AFAICT the participants' goal was just to judge the humanness of a single witness, not to maximize their long-term likelihood of judging correctly over many trials, or some such thing. Even if they'd known the prior chances of speaking to an LLM, there's no obvious reason why that prior should hugely affect their conclusion after a five minute conversation - which also seems immaterial since they didn't know. Plus, the authors give a pretty straightforward rationale for their 50% threshold, and it has no connection to Turing's 3-player setup or whether the imitator and witness are indistinguishable. If they had wanted "indistinguishable" as a threshold, then obviously their pass criteria would have been for the machine and human pass rates to be equal within an error bar, right? So it's pretty implausible to imagine a connection between that and their 50% threshold.