7 ms·

The chatbot Eugene Goostman did not pass Ray Kurzweil's version of the Turing Test. It passed the Turing Test as set up by the organizers. Turing never even me

The chatbot Eugene Goostman did not pass Ray Kurzweil's version of the Turing Test. It passed the Turing Test as set up by the organizers. Turing never even mentioned the criteria that participants should be aware that they are possibly talking to a computer. This single criteria should not matter for intelligence IMO, because the whole point of the test was to abstract away the appearance. The other way around: a chatbot doesn't pass the test if a participant mistakes a human for a machine. People fail the Turing Test all the time when unaware of chat scripts. Even some upvoted Hackernews comments may have been artificially generated without being detected as such. The best way for me to detect if something is up, is to ask the chatbot about Alan Turing, 42 and the Turing Test. Then to curse at it. Most chatbot makers can't resist adding lines specifically for these questions, or they show feigned annoyance that is easy to pick up on. I got Goostman to admit that he was a Turing Test and then we talked about bots some more. Eugene ended with: I call all these chatter-bots "chatter-nuts" due to their extremely high intelligence. I hope you recognize irony. Full conversation here: http://pastebin.com/Wf4uiCRf http://pastebin.com/Wf4uiCRf

DanBC 12y ago
> Turing never even mentioned the criteria that participants should be aware that they are possibly talking to a computer. What? Here's what he said. It seems pretty clear that this version of the game involves one human interrogator, one human and one computer, and the interrogator has to decide which is human and which isn't. > The new form of the problem can be described in terms of a game which we call the 'imitation game." It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart front the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either "X is A and Y is B" or "X is B and Y is A." The interrogator is allowed to put questions to A and B thus: > C: Will X please tell me the length of his or her hair? > Now suppose X is actually A, then A must answer. It is A's object in the game to try and cause C to make the wrong identification. His answer might therefore be: > "My hair is shingled, and the longest strands are about nine inches long." > In order that tones of voice may not help the interrogator the answers should be written, or better still, typewritten. The ideal arrangement is to have a teleprinter communicating between the two rooms. Alternatively the question and answers can be repeated by an intermediary. The object of the game for the third player (B) is to help the interrogator. The best strategy for her is probably to give truthful answers. She can add such things as "I am the woman, don't listen to him!" to her answers, but it will avail nothing as the man can make similar remarks. > We now ask the question, "What will happen when a machine takes the part of A in this game?" Will the interrogator decide wrongly as often when the game is played like this as he does when the game is played between a man and a woman? These questions replace our original, "Can machines think?"
- blauwbilgorgel 12y ago
  The specifics of the Turing Test have been debated for decades. It is clear that different versions and interpretations exist, especially now we have moved on from behaviorism that was popular during the time Turing wrote the paper. From your quotes it only says that player A is to be replaced with a machine, not that player C is to be made aware of this replacement. I do agree that that is the popular interpretation, but it isn't written down, leading to this ambiguity. >the interrogator has to decide which is human and which isn't. The interrogator has to decide which is female and which is male. Replace the female or male with a computer, not making the interrogator aware of this and see if the interrogator is wrong as often as before. Kurzweil questions this bot, as if it was a bot. He is aware that he is talking to a bot, and doesn't even have to choose between A or B. Kurzweil's line of questioning is not fair or normal or productive communication even if it was between two humans. A 13- year-old boy would have told Kurzweil to bugger off when tasked to answer 2+2 or wouldn't have responded at all. Repeat questions like that in a random chatroom and people will think that you are the chatbot.
  - dragonwriter 12y ago
    > Kurzweil's line of questioning is not fair or normal or productive communication even if it was between two humans. A 13- year-old boy would have told Kurzweil to bugger off when tasked to answer 2+2 or wouldn't have responded at all. That seems to make it a very good way to distinguish humans from bots even if it isn't "fair or normal or productive communication" -- if humans could detect that a line of questioning isn't "fair or normal or productive communication" and cut it off, and the chatbot can't, that would seem to be a manner in which a chatbot is readily distinguishable from a human by way of interaction. And, I would argue, recognizing abusive, unproductive lines of inquiry and either diverting them early or cutting them off completely is an important part of human communication.
    - blauwbilgorgel 12y ago
      Absolutely! This sword cuts both ways. Our bots are stuck in Searle's Chinese room, performing silly input-output matching tricks without understanding language on a deep human level. An intelligent bot would indeed recognize unproductive lines of inquiry and should act accordingly. It would be more realistic if a bot answered that it has gotten tired of answering stupid questions. Yet: If the Turing Test was made to avoid discrimination based on appearance, then applying the Turing Test while starting out with a tricky biased (unfair, unnatural, unproductive) line of questioning in the hopes of tripping up the machine... this doesn't seem in the spirit of the Test. The chatbots are at least polite enough to try to answer. Who not extend them the same courtesy as you would other humans, at least till you "figure" out that it may be a bot? Again: It doesn't say in the paper that the interrogator is to be made aware that player A is now a machine. The interrogator will continue its line of questioning trying to distinguish between male and female. This line of questioning should be far saner, and easier for bots to interpret and play along with. By starting out like Kurzweil he places the Test on the shoulders of the bot: The bot has to find out if it is talking to a sane human or someone spouting gibberish. Kurzweil becomes Eugene's Turing Test.

DanBC 12y ago

> Turing never even mentioned the criteria that participants should be aware that they are possibly talking to a computer. What? Here's what he said. It seems pretty clear that this version of the game involves one human interrogator, one human and one computer, and the interrogator has to decide which is human and which isn't. > The new form of the problem can be described in terms of a game which we call the 'imitation game." It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart front the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either "X is A and Y is B" or "X is B and Y is A." The interrogator is allowed to put questions to A and B thus: > C: Will X please tell me the length of his or her hair? > Now suppose X is actually A, then A must answer. It is A's object in the game to try and cause C to make the wrong identification. His answer might therefore be: > "My hair is shingled, and the longest strands are about nine inches long." > In order that tones of voice may not help the interrogator the answers should be written, or better still, typewritten. The ideal arrangement is to have a teleprinter communicating between the two rooms. Alternatively the question and answers can be repeated by an intermediary. The object of the game for the third player (B) is to help the interrogator. The best strategy for her is probably to give truthful answers. She can add such things as "I am the woman, don't listen to him!" to her answers, but it will avail nothing as the man can make similar remarks. > We now ask the question, "What will happen when a machine takes the part of A in this game?" Will the interrogator decide wrongly as often when the game is played like this as he does when the game is played between a man and a woman? These questions replace our original, "Can machines think?"

blauwbilgorgel 12y ago
The specifics of the Turing Test have been debated for decades. It is clear that different versions and interpretations exist, especially now we have moved on from behaviorism that was popular during the time Turing wrote the paper. From your quotes it only says that player A is to be replaced with a machine, not that player C is to be made aware of this replacement. I do agree that that is the popular interpretation, but it isn't written down, leading to this ambiguity. >the interrogator has to decide which is human and which isn't. The interrogator has to decide which is female and which is male. Replace the female or male with a computer, not making the interrogator aware of this and see if the interrogator is wrong as often as before. Kurzweil questions this bot, as if it was a bot. He is aware that he is talking to a bot, and doesn't even have to choose between A or B. Kurzweil's line of questioning is not fair or normal or productive communication even if it was between two humans. A 13- year-old boy would have told Kurzweil to bugger off when tasked to answer 2+2 or wouldn't have responded at all. Repeat questions like that in a random chatroom and people will think that you are the chatbot.
- dragonwriter 12y ago
  > Kurzweil's line of questioning is not fair or normal or productive communication even if it was between two humans. A 13- year-old boy would have told Kurzweil to bugger off when tasked to answer 2+2 or wouldn't have responded at all. That seems to make it a very good way to distinguish humans from bots even if it isn't "fair or normal or productive communication" -- if humans could detect that a line of questioning isn't "fair or normal or productive communication" and cut it off, and the chatbot can't, that would seem to be a manner in which a chatbot is readily distinguishable from a human by way of interaction. And, I would argue, recognizing abusive, unproductive lines of inquiry and either diverting them early or cutting them off completely is an important part of human communication.
  - blauwbilgorgel 12y ago
    Absolutely! This sword cuts both ways. Our bots are stuck in Searle's Chinese room, performing silly input-output matching tricks without understanding language on a deep human level. An intelligent bot would indeed recognize unproductive lines of inquiry and should act accordingly. It would be more realistic if a bot answered that it has gotten tired of answering stupid questions. Yet: If the Turing Test was made to avoid discrimination based on appearance, then applying the Turing Test while starting out with a tricky biased (unfair, unnatural, unproductive) line of questioning in the hopes of tripping up the machine... this doesn't seem in the spirit of the Test. The chatbots are at least polite enough to try to answer. Who not extend them the same courtesy as you would other humans, at least till you "figure" out that it may be a bot? Again: It doesn't say in the paper that the interrogator is to be made aware that player A is now a machine. The interrogator will continue its line of questioning trying to distinguish between male and female. This line of questioning should be far saner, and easier for bots to interpret and play along with. By starting out like Kurzweil he places the Test on the shoulders of the bot: The bot has to find out if it is talking to a sane human or someone spouting gibberish. Kurzweil becomes Eugene's Turing Test.