3 ms·

The Loebner prize completely misrepresents Turing’s paper

12 points by edw519 17y ago

blackrabbit 17y ago
You're second quote is horribly out of context. Turing says using a survey to describe machines as thinking beings in the commonly used sense of the words is absurd, not the question of "Can a machine think?".
- hughprime 17y ago
  To quote the full paragraph: I propose to consider the question, "Can machines think?" This should begin with definitions of the meaning of the terms "machine" and "think." The definitions might be framed so as to reflect so far as possible the normal use of the words, but this attitude is dangerous, If the meaning of the words "machine" and "think" are to be found by examining how they are commonly used it is difficult to escape the conclusion that the meaning and the answer to the question, "Can machines think?" is to be sought in a statistical survey such as a Gallup poll. But this is absurd. Instead of attempting such a definition I shall replace the question by another, which is closely related to it and is expressed in relatively unambiguous words. So what he's really saying is that the question "Can machines think?" is too ambiguous since the words "machine" and "think" aren't well-enough defined. We could use a Gallup poll to find out what people think the words mean, and then try to answer the question based on those definitions, but that would be absurd. What's really "wrong" with the Loebner Prize is that it's a very simplified Turing test (five minutes of conversation) which doesn't promote the development of actual thinking machines any more than a paper airplane competition promotes the development of transcontinental airliners. Eventually somebody will write a chatterbot which is capable of fooling most of the judges into thinking it's human for the duration of a short conversation (if you've read the transcripts they rarely go on for more than a dozen lines or so) but it will be a bunch of clever tricks rather than an AI, and a serious attempt to figure out whether it's intelligent (e.g. by telling it a story and then asking it basic reading-comprehension questions about the story) will quickly show that it isn't.
  - joe_the_user 17y ago
    What's really "wrong" with the Loebner Prize is that it's a very simplified Turing test (five minutes of conversation) which doesn't promote the development of actual thinking machines any more than a paper airplane competition promotes the development of transcontinental airliners. Really? If all we had to begin with were paper airplanes, then a paper airplane competition would certainly promote transcontinental airliners. Before powered flight, studying paper airplanes was certainly one valid way to make progress. And the limits of today's AI in many ways make it seem pretty close to paper airplanes - so seeing if they can every fly across a room seems entirely valid. SHEESH, it's hard to find even a metaphor in which this controversial prize doesn't advance our knowledge. Consider, human behavior is very ad-hoc. AI as a field has neglected ad-hoc interaction, preferring more logically specified activities. If a computer could hold an effective five minute chat, it would be so far above what currently exists as to be breath-taking. If that's done with 'tricks', it's time to start understanding the 'tricks' rather going on and on with the systematic or whatever approaches we might have thought were the proper way to do this. Oh, and the reading test sounds easier than the interaction test - MUCH easier.
    - hughprime 17y ago
      My suspicion that it wouldn't be _that_ hard to be convincing in the context of the competition comes from reading some of the chat transcripts -- see, e.g. http://loebner.net/Prizef/2008_Contest/loebner-prize-2008.html http://loebner.net/Prizef/2008_Contest/loebner-prize-2008.ht... for last year's winners. The conversations with actual humans seem fairly dull and formulaic, the judges don't seem to attempt any particularly sophisticated interactions. Already the winners are managing to convince about 30% of the judges, so getting that above 50% doesn't seem too hard. Incidentally, I just had a chat with last year's winner elbot (see elbot.com) and it's got some pretty clever special-case interactions built in. For instance: me: What is 4 times 4? elbot: The answer is: 17! me: Haha, very clever. elbot: I'm glad you agree, most people think the answer is 16.
      - joe_the_user 17y ago
        If you think that doing this isn't hard, why attempt to do it! So far everyone's failed, you would certain get some attention even if it was all just 'tricks'.
sp332 17y ago
Yeah, did you see what they did to the Turing Test? http://en.wikipedia.org/wiki/Loebner_Prize http://en.wikipedia.org/wiki/Loebner_Prize Chris McKinstry (the guy behind the MindPixel project) got fed up with them and made the Minimum Intelligent Signal Test (MIST), which is more objective than the original Turing test and may be the simplest way to test for general, common-sense intelligence. http://en.wikipedia.org/wiki/Minimum_Intelligent_Signal_Test http://en.wikipedia.org/wiki/Minimum_Intelligent_Signal_Test Edit: forgot link. d'oh.
- RiderOfGiraffes 17y ago
  Clickable link to save you time: http://en.wikipedia.org/wiki/Minimum_Intelligent_Signal_Test http://en.wikipedia.org/wiki/Minimum_Intelligent_Signal_Test
  - joe_the_user 17y ago
    Also, this test actually doesn't seem as valid a simple conversation, since dealing with the apparent unpredictability of conversation is one of the real challenges for "real AI".
- deleted 17y ago
  [deleted]
- joe_the_user 17y ago
  Uh, It's not as if the machines are acing the Loebner version of the test or anything. Obviously, the test is dumbed-down to allow Bronze metal winners. Seems like a logical even if so much else about the contest seems mired in the persona of Hugh Loebner.
- Liron 17y ago
  It's cool that the MIST gets around average human interrogators' variations and weaknesses. But what about the weaknesses of the human or algorithm that generates the MIST questions? Plus the MIST has no opportunities for followup questions. Turing's intent was to define a sufficient condition for labeling behavior "intelligent", and I think a high score on Loebner's setup with a competent interrogator does this better than a pre-defined set of MIST questions.
deleted 17y ago
[deleted]
aarongough 17y ago
My problem with the Loebner Prize these days is not that it's mis-representing Turing's intention (I don't think it is) but that they have moved to a hideous input/output system that each entrant is required to use. Each program, upon startup, must provide a “browse” function to select a directory. Communications shall be by means of the creation, detection, and deletion of sub-directories within the specified communications directory. To simulate a key press the entry program must create a sub-directory within the communications directory with the following format: “time.keypress-name.extension” where time is a monotonically increasing 18 digit number (in lexical and numerical order) (i.e. zero filled to the left) to be retrieved from the system clock and expressed as milliseconds past some initial time as defined by the system clock. “keypress-name” is either a single letter (case sensitive) or the name of the special character, as appended to these rules. The extension is “.other” For example: “000001234567890123.bracketleft.other” To detect a key press by the judge, the program must detect, within the communications directory a sub-directory with the same format, but extension “.judge” and then must remove or delete the judge’s sub-directory from the communications directory. It's not the end of the world, but at the same time it's a fairly nasty way of doing something that should be relatively easy. I entered in 2002 before they introduced this and have been hesitant to enter again in recent years partly because I don't want to have to deal with the hackery of the 'communications protocol'.

12 points by edw519 17y ago

You're second quote is horribly out of context. Turing says using a survey to describe machines as thinking beings in the commonly used sense of the words is absurd, not the question of "Can a machine think?".

hughprime 17y ago
To quote the full paragraph: I propose to consider the question, "Can machines think?" This should begin with definitions of the meaning of the terms "machine" and "think." The definitions might be framed so as to reflect so far as possible the normal use of the words, but this attitude is dangerous, If the meaning of the words "machine" and "think" are to be found by examining how they are commonly used it is difficult to escape the conclusion that the meaning and the answer to the question, "Can machines think?" is to be sought in a statistical survey such as a Gallup poll. But this is absurd. Instead of attempting such a definition I shall replace the question by another, which is closely related to it and is expressed in relatively unambiguous words. So what he's really saying is that the question "Can machines think?" is too ambiguous since the words "machine" and "think" aren't well-enough defined. We could use a Gallup poll to find out what people think the words mean, and then try to answer the question based on those definitions, but that would be absurd. What's really "wrong" with the Loebner Prize is that it's a very simplified Turing test (five minutes of conversation) which doesn't promote the development of actual thinking machines any more than a paper airplane competition promotes the development of transcontinental airliners. Eventually somebody will write a chatterbot which is capable of fooling most of the judges into thinking it's human for the duration of a short conversation (if you've read the transcripts they rarely go on for more than a dozen lines or so) but it will be a bunch of clever tricks rather than an AI, and a serious attempt to figure out whether it's intelligent (e.g. by telling it a story and then asking it basic reading-comprehension questions about the story) will quickly show that it isn't.
- joe_the_user 17y ago
  What's really "wrong" with the Loebner Prize is that it's a very simplified Turing test (five minutes of conversation) which doesn't promote the development of actual thinking machines any more than a paper airplane competition promotes the development of transcontinental airliners. Really? If all we had to begin with were paper airplanes, then a paper airplane competition would certainly promote transcontinental airliners. Before powered flight, studying paper airplanes was certainly one valid way to make progress. And the limits of today's AI in many ways make it seem pretty close to paper airplanes - so seeing if they can every fly across a room seems entirely valid. SHEESH, it's hard to find even a metaphor in which this controversial prize doesn't advance our knowledge. Consider, human behavior is very ad-hoc. AI as a field has neglected ad-hoc interaction, preferring more logically specified activities. If a computer could hold an effective five minute chat, it would be so far above what currently exists as to be breath-taking. If that's done with 'tricks', it's time to start understanding the 'tricks' rather going on and on with the systematic or whatever approaches we might have thought were the proper way to do this. Oh, and the reading test sounds easier than the interaction test - MUCH easier.
  - hughprime 17y ago
    My suspicion that it wouldn't be _that_ hard to be convincing in the context of the competition comes from reading some of the chat transcripts -- see, e.g. http://loebner.net/Prizef/2008_Contest/loebner-prize-2008.html http://loebner.net/Prizef/2008_Contest/loebner-prize-2008.ht... for last year's winners. The conversations with actual humans seem fairly dull and formulaic, the judges don't seem to attempt any particularly sophisticated interactions. Already the winners are managing to convince about 30% of the judges, so getting that above 50% doesn't seem too hard. Incidentally, I just had a chat with last year's winner elbot (see elbot.com) and it's got some pretty clever special-case interactions built in. For instance: me: What is 4 times 4? elbot: The answer is: 17! me: Haha, very clever. elbot: I'm glad you agree, most people think the answer is 16.
    - joe_the_user 17y ago
      If you think that doing this isn't hard, why attempt to do it! So far everyone's failed, you would certain get some attention even if it was all just 'tricks'.

sp332 17y ago

Yeah, did you see what they did to the Turing Test? http://en.wikipedia.org/wiki/Loebner_Prize http://en.wikipedia.org/wiki/Loebner_Prize Chris McKinstry (the guy behind the MindPixel project) got fed up with them and made the Minimum Intelligent Signal Test (MIST), which is more objective than the original Turing test and may be the simplest way to test for general, common-sense intelligence. http://en.wikipedia.org/wiki/Minimum_Intelligent_Signal_Test http://en.wikipedia.org/wiki/Minimum_Intelligent_Signal_Test Edit: forgot link. d'oh.

RiderOfGiraffes 17y ago
Clickable link to save you time: http://en.wikipedia.org/wiki/Minimum_Intelligent_Signal_Test http://en.wikipedia.org/wiki/Minimum_Intelligent_Signal_Test
- joe_the_user 17y ago
  Also, this test actually doesn't seem as valid a simple conversation, since dealing with the apparent unpredictability of conversation is one of the real challenges for "real AI".
deleted 17y ago
[deleted]
joe_the_user 17y ago
Uh, It's not as if the machines are acing the Loebner version of the test or anything. Obviously, the test is dumbed-down to allow Bronze metal winners. Seems like a logical even if so much else about the contest seems mired in the persona of Hugh Loebner.
Liron 17y ago
It's cool that the MIST gets around average human interrogators' variations and weaknesses. But what about the weaknesses of the human or algorithm that generates the MIST questions? Plus the MIST has no opportunities for followup questions. Turing's intent was to define a sufficient condition for labeling behavior "intelligent", and I think a high score on Loebner's setup with a competent interrogator does this better than a pre-defined set of MIST questions.

deleted 17y ago

[deleted]

aarongough 17y ago

My problem with the Loebner Prize these days is not that it's mis-representing Turing's intention (I don't think it is) but that they have moved to a hideous input/output system that each entrant is required to use. Each program, upon startup, must provide a “browse” function to select a directory. Communications shall be by means of the creation, detection, and deletion of sub-directories within the specified communications directory. To simulate a key press the entry program must create a sub-directory within the communications directory with the following format: “time.keypress-name.extension” where time is a monotonically increasing 18 digit number (in lexical and numerical order) (i.e. zero filled to the left) to be retrieved from the system clock and expressed as milliseconds past some initial time as defined by the system clock. “keypress-name” is either a single letter (case sensitive) or the name of the special character, as appended to these rules. The extension is “.other” For example: “000001234567890123.bracketleft.other” To detect a key press by the judge, the program must detect, within the communications directory a sub-directory with the same format, but extension “.judge” and then must remove or delete the judge’s sub-directory from the communications directory. It's not the end of the world, but at the same time it's a fairly nasty way of doing something that should be relatively easy. I entered in 2002 before they introduced this and have been hesitant to enter again in recent years partly because I don't want to have to deal with the hackery of the 'communications protocol'.