8 ms·

Passing the Turing test requires, for any human judge: 1. This judge is aware that they have to discern whether the 'bot' is real or a machine. 2. The judge c

Passing the Turing test requires, for any human judge: 1. This judge is aware that they have to discern whether the 'bot' is real or a machine. 2. The judge cannot discern whether the 'bot' is real or a machine better than random chance. This failed 1. And even given that advantage, might have failed 2 as well? Often I see headlines along the lines of "X fools people and beats the Turing test!". But the point of the Turing test isn't to trick a person, it's to make it functionally impossible for a person to distinguish between the real and simulated thing, no matter how hard they try. For something to pass a Turing test, it would need to be able to pass the following: "Anyone can play as judge any number of times. You can take as long as you want, and if you're successful in breaking it under controlled conditions (IE, you don't cheat and use an out-of-band communication protocol with the 'bot'/human), we'll give you a 10,000,000$." The "One Million Dollar Paranormal Challenge" (https://en.wikipedia.org/wiki/One_Million_Dollar_Paranormal_Challenge https://en.wikipedia.org/wiki/One_Million_Dollar_Paranormal_...) is a solid example of a Turing test for magic.

VikingCoder 4y ago
How often have we run a Turing Test, where we asked the judge how confident they were in their final answer, except both participants were humans?
- IshKebab 4y ago
  It doesn't really matter for the purposes of the test how confident they say they are. If you're asking "do people often imagine they can confidently distinguish things when they actually can't" then the answer is a solid yes - things like audiophile and wine testing have proven that again and again.
godelski 4y ago
I think another important factor here is that it is unclear if the OP cherry picked photos or used the first ones given. Dall-E 2 has a bias to be better at real world scenes since it can just pull (near) memorized images, but I also wouldn't be surprised if these images were downselected from a larger set.
- karpierz 4y ago
  Great point. I should've added a third restriction: the 'bot' should not be able to communicate with, or use the judgement of, humans once it's started.
  - mattkrause 4y ago
    The original version of the test allows for interaction, and I think that’s probably a good thing. Language models currently have a hard time staying consistent/on topic and that’s a potentially valuable tell. Instead, I think you don’t want any third parties “editing” or “curating” the exchange (beyond whatever blinding is needed to make it work).
    - godelski 4y ago
      Yeah I think the interaction is a key part. People are confusing AGI with useful AI. DALL-E is very clearly useful even if 90% of the images it produced were unusable. The time save is still large and cost reduced for simple tasks. Same with language models. They may have a hard time staying consistent over long sequences but adding human selective pressure to responses saves everyone time and money. But this is very clearly different from a machine that can think and form new ideas on its own. We're still a long way from that.
GuB-42 4y ago
Furthermore, the judge is supposed to be an expert. That is, not only his task is explicitly to tell between human and computer but he has to have a good idea on how to do it. Random people from the internet are not enough. In the "paranormal challenge", the juges usually include stage magicians, because they know the tricks that would fool ordinary people. James Randi himself is a magician.
- karpierz 4y ago
  Conceptually, to pass the Turing test you should be able to fool any human (including expert judges). It's just practically easier to choose an expert judge as opposed to test on every human on Earth, since you'd hope that if an expert can't do it, then neither can anyone else. Also, it shouldn't matter if the human is someone who worked on the AI, or has read the code, or has seen every previous Turing test that the AI underwent. There shouldn't be any information a person could know that would allow them to tell that it's an AI.
  - RogerL 4y ago
    I have acquaintances that that couldn't pass this based on their common texting skills. There's probably no point to argue about what the Turing test "is", but the definition in this comment chain is pretty uncompromising - I can't imagine fooling any (meaning all, I assume) humans; there is too much ambiguity in the signal. "Are you going to be at the blah blah because I need blah". response: "I really wqnt it". Nonresponsive, typos, what does 'want' refer to? Who knows. Is this a bad bot, someone spun out on meth, someone with cognitive processing issues, a busy mom texting while distracted, or ?
    - karpierz 4y ago
      > I can't imagine fooling any (meaning all, I assume) humans; there is too much ambiguity in the signal. I think you've inverted the expectation here. What I was saying is a machine passes if you can't find anyone in the world who can distinguish between it and an actual human. Meaning that if the "world's smartest person" can distinguish between it and an actual human, it fails, even if it can fool everyone else. A machine can pass the test by deliberately feigning to be a human with limited communication capabilities (ex: I think we can simulate 2 month old baby talking via text). But then all you've shown is that your machine is as capable of thought as a 2 month old baby, which probably isn't the bar that you're trying to reach.
dredmorbius 4y ago
The test is now live all the time. You must constantly be aware that images, or text, or voice, or other audio, or other signals or data, might be computer generated or altered. All the time. And you individually, or those about you, or societies at large, may be influenced in large or small ways by such signals, patterns, and records. Your elderly neighbour or relative might be scammed out of life savings. Investors of false product claims. Voters of some fake outrage --- particularly of the October Surprise variety. Soldiers and diplomats of mock attacks, or false depictions of a tranquil situation where in fact danger lurks. The test never ends. This is your final warning.
- _justinfunk 4y ago
  da da dummmm....
- jmmcd 4y ago
  The only good news is that this is not your final warning.
  - dredmorbius 4y ago
    Final advance warning might have been more accurate. Sense meant was "the threat is active, you can't presume otherwise, ignorance is not safety". That said, I suppose that message will be repeated occasionally in future. Possibly even by myself.
- hooande 4y ago
  this has been true since photoshop existed? you don't need an algorithm to make a fake picture or to fake a military attack, etc. humans generally have better grasp of appropriate details
  - pjc50 4y ago
    Fakes have played a key role in historical events: https://en.wikipedia.org/wiki/Zinoviev_letter https://en.wikipedia.org/wiki/Zinoviev_letter ; https://en.wikipedia.org/wiki/The_Protocols_of_the_Elders_of_Zion https://en.wikipedia.org/wiki/The_Protocols_of_the_Elders_of... ; Roosevelt's map faked by British intelligence https://www.jstor.org/stable/40257685 https://www.jstor.org/stable/40257685 and so on. It's a matter of quantity as well as quality. It gets easier to produce convincing forgeries all the time, but it doesn't get easier to debunk them. This allows for public discourse to be destroyed by flooding it with fakes.
a-dub 4y ago
if a human participates in interactive gamified social media, and this participation begins to change, shape, reinforce or otherwise mutate their beliefs, for the purposes of the test are they still actually a human? could the entire social media mechanism (from the builders to the participants) be considered a form of a sort of singleton autonomous intelligence in and of itself?
jameshart 4y ago
I mean, you can read Turing's own definiton if his test - 'the Imitation Game' - on the first page of his 1950 paper Computing Machinery and Intelligence[1]. There's nothing in there about repetition, duration, or $10,000,000 prizes. It's a party game. And he just frames his question (which will "replace our original, 'Can machines think?'") as "Will the interrogator decide wrongly as often when the game is played like this [with a human and a computer] as he does when the game is played between a man and a woman?" So, to perform the experiment, one must have some people play the game with humans a few times and then play with a human and a machine a few times, and look to see if the results are statistically significant. When they aren't, Turing posits, the question 'can machines think?' will have been answered in the affirmative. That is not to say that this DALL-E vacation photo social media post constitutes a rigorous 'passes the Turing test'. But I don't think it's fair to criticize someone for using 'the Turing Test' colloquially as a catchall for saying 'you probably didn't notice this output was machine generated, therefore you might want to adjust your priors on the answer to the question, "can machines think?"'. Because that's exactly the spirit that Turing was working in when he proposed using a party game as a test of intelligence. [1] https://www.csee.umbc.edu/courses/471/papers/turing.pdf https://www.csee.umbc.edu/courses/471/papers/turing.pdf
- karpierz 4y ago
  There's the literal definition of the Turing test, as described above, which doesn't actually work for proving any sort of intelligence. Then there's the conceptual argument of the Turing test, which we can turn into a test of intelligence. It relies on the idea that we can abstract the mind into a "thinking" black box which takes inputs and outputs. And then posits that any black box which can't be distinguished from an actual "thinking" black box may as well be "thinking". Passing the literal Turing test is a sign that some humans can be tricked for some small domain over some small period of time. Passing the conceptual argument that the Turing test relies on shows that there are non-human entities which cannot be separated from humans on the basis of how they interact with a human (through some limited medium). The repetition, duration and prizes are just practicality; prizes incentivize people to try, repetition ensures that the results are robust, and duration ensures that the domain the AI can be tested over is equivalent to a humans.
causality0 4y ago
If someone told me they found a magic lamp and one of their wishes was that every time someone misused the term "Turing test" they got smacked in the face by an opening door I'd think "not bad, not bad".
sullyj3 4y ago
As usual, the headline is more sensationalised than the actual article > It's likely that with a harder version of the Turing Test, in which real and fake images of the same content are presented side by side and people are told that one of them is fake, it would be much easier to detect the fake images.

VikingCoder 4y ago

How often have we run a Turing Test, where we asked the judge how confident they were in their final answer, except both participants were humans?

IshKebab 4y ago
It doesn't really matter for the purposes of the test how confident they say they are. If you're asking "do people often imagine they can confidently distinguish things when they actually can't" then the answer is a solid yes - things like audiophile and wine testing have proven that again and again.

godelski 4y ago

I think another important factor here is that it is unclear if the OP cherry picked photos or used the first ones given. Dall-E 2 has a bias to be better at real world scenes since it can just pull (near) memorized images, but I also wouldn't be surprised if these images were downselected from a larger set.

karpierz 4y ago
Great point. I should've added a third restriction: the 'bot' should not be able to communicate with, or use the judgement of, humans once it's started.
- mattkrause 4y ago
  The original version of the test allows for interaction, and I think that’s probably a good thing. Language models currently have a hard time staying consistent/on topic and that’s a potentially valuable tell. Instead, I think you don’t want any third parties “editing” or “curating” the exchange (beyond whatever blinding is needed to make it work).
  - godelski 4y ago
    Yeah I think the interaction is a key part. People are confusing AGI with useful AI. DALL-E is very clearly useful even if 90% of the images it produced were unusable. The time save is still large and cost reduced for simple tasks. Same with language models. They may have a hard time staying consistent over long sequences but adding human selective pressure to responses saves everyone time and money. But this is very clearly different from a machine that can think and form new ideas on its own. We're still a long way from that.

GuB-42 4y ago

Furthermore, the judge is supposed to be an expert. That is, not only his task is explicitly to tell between human and computer but he has to have a good idea on how to do it. Random people from the internet are not enough. In the "paranormal challenge", the juges usually include stage magicians, because they know the tricks that would fool ordinary people. James Randi himself is a magician.

karpierz 4y ago
Conceptually, to pass the Turing test you should be able to fool any human (including expert judges). It's just practically easier to choose an expert judge as opposed to test on every human on Earth, since you'd hope that if an expert can't do it, then neither can anyone else. Also, it shouldn't matter if the human is someone who worked on the AI, or has read the code, or has seen every previous Turing test that the AI underwent. There shouldn't be any information a person could know that would allow them to tell that it's an AI.
- RogerL 4y ago
  I have acquaintances that that couldn't pass this based on their common texting skills. There's probably no point to argue about what the Turing test "is", but the definition in this comment chain is pretty uncompromising - I can't imagine fooling any (meaning all, I assume) humans; there is too much ambiguity in the signal. "Are you going to be at the blah blah because I need blah". response: "I really wqnt it". Nonresponsive, typos, what does 'want' refer to? Who knows. Is this a bad bot, someone spun out on meth, someone with cognitive processing issues, a busy mom texting while distracted, or ?
  - karpierz 4y ago
    > I can't imagine fooling any (meaning all, I assume) humans; there is too much ambiguity in the signal. I think you've inverted the expectation here. What I was saying is a machine passes if you can't find anyone in the world who can distinguish between it and an actual human. Meaning that if the "world's smartest person" can distinguish between it and an actual human, it fails, even if it can fool everyone else. A machine can pass the test by deliberately feigning to be a human with limited communication capabilities (ex: I think we can simulate 2 month old baby talking via text). But then all you've shown is that your machine is as capable of thought as a 2 month old baby, which probably isn't the bar that you're trying to reach.

dredmorbius 4y ago

The test is now live all the time. You must constantly be aware that images, or text, or voice, or other audio, or other signals or data, might be computer generated or altered. All the time. And you individually, or those about you, or societies at large, may be influenced in large or small ways by such signals, patterns, and records. Your elderly neighbour or relative might be scammed out of life savings. Investors of false product claims. Voters of some fake outrage --- particularly of the October Surprise variety. Soldiers and diplomats of mock attacks, or false depictions of a tranquil situation where in fact danger lurks. The test never ends. This is your final warning.

_justinfunk 4y ago
da da dummmm....
jmmcd 4y ago
The only good news is that this is not your final warning.
- dredmorbius 4y ago
  Final advance warning might have been more accurate. Sense meant was "the threat is active, you can't presume otherwise, ignorance is not safety". That said, I suppose that message will be repeated occasionally in future. Possibly even by myself.
hooande 4y ago
this has been true since photoshop existed? you don't need an algorithm to make a fake picture or to fake a military attack, etc. humans generally have better grasp of appropriate details
- pjc50 4y ago
  Fakes have played a key role in historical events: https://en.wikipedia.org/wiki/Zinoviev_letter https://en.wikipedia.org/wiki/Zinoviev_letter ; https://en.wikipedia.org/wiki/The_Protocols_of_the_Elders_of_Zion https://en.wikipedia.org/wiki/The_Protocols_of_the_Elders_of... ; Roosevelt's map faked by British intelligence https://www.jstor.org/stable/40257685 https://www.jstor.org/stable/40257685 and so on. It's a matter of quantity as well as quality. It gets easier to produce convincing forgeries all the time, but it doesn't get easier to debunk them. This allows for public discourse to be destroyed by flooding it with fakes.

a-dub 4y ago

if a human participates in interactive gamified social media, and this participation begins to change, shape, reinforce or otherwise mutate their beliefs, for the purposes of the test are they still actually a human? could the entire social media mechanism (from the builders to the participants) be considered a form of a sort of singleton autonomous intelligence in and of itself?

jameshart 4y ago

I mean, you can read Turing's own definiton if his test - 'the Imitation Game' - on the first page of his 1950 paper Computing Machinery and Intelligence[1]. There's nothing in there about repetition, duration, or $10,000,000 prizes. It's a party game. And he just frames his question (which will "replace our original, 'Can machines think?'") as "Will the interrogator decide wrongly as often when the game is played like this [with a human and a computer] as he does when the game is played between a man and a woman?" So, to perform the experiment, one must have some people play the game with humans a few times and then play with a human and a machine a few times, and look to see if the results are statistically significant. When they aren't, Turing posits, the question 'can machines think?' will have been answered in the affirmative. That is not to say that this DALL-E vacation photo social media post constitutes a rigorous 'passes the Turing test'. But I don't think it's fair to criticize someone for using 'the Turing Test' colloquially as a catchall for saying 'you probably didn't notice this output was machine generated, therefore you might want to adjust your priors on the answer to the question, "can machines think?"'. Because that's exactly the spirit that Turing was working in when he proposed using a party game as a test of intelligence. [1] https://www.csee.umbc.edu/courses/471/papers/turing.pdf https://www.csee.umbc.edu/courses/471/papers/turing.pdf

karpierz 4y ago
There's the literal definition of the Turing test, as described above, which doesn't actually work for proving any sort of intelligence. Then there's the conceptual argument of the Turing test, which we can turn into a test of intelligence. It relies on the idea that we can abstract the mind into a "thinking" black box which takes inputs and outputs. And then posits that any black box which can't be distinguished from an actual "thinking" black box may as well be "thinking". Passing the literal Turing test is a sign that some humans can be tricked for some small domain over some small period of time. Passing the conceptual argument that the Turing test relies on shows that there are non-human entities which cannot be separated from humans on the basis of how they interact with a human (through some limited medium). The repetition, duration and prizes are just practicality; prizes incentivize people to try, repetition ensures that the results are robust, and duration ensures that the domain the AI can be tested over is equivalent to a humans.

causality0 4y ago

If someone told me they found a magic lamp and one of their wishes was that every time someone misused the term "Turing test" they got smacked in the face by an opening door I'd think "not bad, not bad".

sullyj3 4y ago

As usual, the headline is more sensationalised than the actual article > It's likely that with a harder version of the Turing Test, in which real and fake images of the same content are presented side by side and people are told that one of them is fake, it would be much easier to detect the fake images.