4 ms·

As mentioned in another comment there are also numerous other issues missing relative to the canonical experiment. Three big ones: - The whole point of the nam

As mentioned in another comment there are also numerous other issues missing relative to the canonical experiment. Three big ones: - The whole point of the name, "the imitation game", is to imitate a specific identity. The more precise an identity is, the more difficult it would be for an imposter to imitate it. Turing chose male vs female, but modern choices have generalized it down to 'human or not' which is of course vastly easier to imitate than a more specific choice. - Participants are expected to collaborate with the interrogator, break the 4th wall, and do everything possible to make it clear to the interrogator that they are the real person. Modern variants generally have participants acting adversarially and actively, and intentionally, giving responses that would be difficult to distinguish from those of a bot. - The interrogator is expected to intelligently interrogate. For example one [intentionally] naive idea Turing gave was that an interrogator might ask the person to perform some mathematical calculation. If the person even tries to answer 37167361 * 372 (let alone succeeds in a short time frame), then they are probably not human. Of course the bot could be programmed to respond accordingly, but it's the point of actively and intelligently trying to break the bot and have it reveal itself. Contemporary interrogators typically ask the participants random and inane questions like "Where are you from?" which is a complete and absolute waste of time, unless part of some more precise plan - but it never is. To my knowledge there have been no Turing Tests carried out with anything even vaguely resembling the rigor and purpose of the original test, but I think that's largely because the goal seems to be to create a test that can be passed, rather than actually evaluate the capabilities of the various LLMs.

Ukv 2y ago
> - The whole point of the name, "the imitation game", is to imitate a specific identity. The more precise an identity is, the more difficult it would be for an imposter to imitate it. Turing chose male vs female, but modern choices have generalized it down to 'human or not' which is of course vastly easier to imitate than a more specific choice. Turing did introduce the concept of the game by having it played between a human man and human woman, with the man pretending to be a woman, but to my understanding this was just a stepping stone to move on to having the game played between machine and human. I don't think the gender specifity was meant to stick around beyond that initial introductory example. If you mean how he says things like "imitation of the behaviour of a man", that's most likely intended generally rather than specifically male (particuarly as the "machine takes the part of A", which was the man pretending to be a woman).
- somenameforme 2y ago
  Here [1] is the original paper. Though he does not state as such, I'm sure the idea of man vs woman was just an example. It could be anything, but I think it inherently must be something. Generalizing this down to being human or not greatly simplifies the test, because the identity aspect is basically just free information for the interrogator. With or without the identity, he could still ask the exact same questions. The only difference is the domain of viable answers is greatly limited with identities. And the more specific the identity, the more the real person will be able to reveal themselves, and the more difficulty the imposter will have impersonating them. [1] - https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf
  - Ukv 2y ago
    I agree that "Man pretending to be woman, vs real woman" is just an example, used to introduce the question in the form of a party game between humans. I see the "something" it is replaced by as "Machine pretending to be human, vs real human". I don't see indication that the machine must pretend to be human in addition to some other characteristic of the second player. I think the reason you see others as having "generalized it down" is that your interpretation is not apparent in the text. > the more specific the identity, the more the real person will be able to reveal themselves, and the more difficulty the imposter will have impersonating them. Definitely makes for a more difficult problem (arbitrarily difficult, even) and a potentially interesting extension. Currently to me it doesn't seem as insightful as Turing's original proposal - there's no more inherent human benchmark of 50%, for instance, since humans can also be bad at impersonating some specific characteristic.
    - somenameforme 2y ago
      With no need to actually imitate anything in particular, you could simply chop away everything except the most basic linguistic functions and claim you are a non-native preteen. And who's to say otherwise? In fact that's literally the exact "trick" that yet another mockery of the Turing Test used when claiming they'd overcome the Turing Test. In fact shall we not just take it to the next level? You're 5 years old - and simply respond by randomly pounding various keys on the keyboard on occasion. Boom - didn't see that coming, now did ya Turing? Passing the test will not be a benchmark because the test has been passed, but because of what passing the test ought entail. People often complain about shifting goalposts on AI, but that's not the issue. The issue is doing exactly what you're doing here and creating worthless goalposts to begin with. And so of course when you cross them, the first thing that happens is that they get inched forward somewhere closer to something reasonable, before you even have time to uncork the champagne. Why not simply skip this nonsense, and start with a reasonable goalpost to begin with? Because it's too hard? Well obviously - that's why it's a goal, and not next month's scrimmage point!
tripletao 2y ago
I agree that those are also highly significant differences, though I'd consider them a reasonable "easy mode" while we wait for a machine capable of passing Turing's original test. I focused on the statistical issue because that one seems indefensible to me. The paper's result has no clear interpretation, depending completely on what assumption the interrogator makes about the unspecified prior probability that their witness is human. It's not clear to me whether the paper's authors even understand what they've broken. Just for fun, I tried a few LLMs and couldn't get them to recognize the statistical issue either. I guess they'll probably learn before social science professors do, though.

Ukv 2y ago

> - The whole point of the name, "the imitation game", is to imitate a specific identity. The more precise an identity is, the more difficult it would be for an imposter to imitate it. Turing chose male vs female, but modern choices have generalized it down to 'human or not' which is of course vastly easier to imitate than a more specific choice. Turing did introduce the concept of the game by having it played between a human man and human woman, with the man pretending to be a woman, but to my understanding this was just a stepping stone to move on to having the game played between machine and human. I don't think the gender specifity was meant to stick around beyond that initial introductory example. If you mean how he says things like "imitation of the behaviour of a man", that's most likely intended generally rather than specifically male (particuarly as the "machine takes the part of A", which was the man pretending to be a woman).

somenameforme 2y ago
Here [1] is the original paper. Though he does not state as such, I'm sure the idea of man vs woman was just an example. It could be anything, but I think it inherently must be something. Generalizing this down to being human or not greatly simplifies the test, because the identity aspect is basically just free information for the interrogator. With or without the identity, he could still ask the exact same questions. The only difference is the domain of viable answers is greatly limited with identities. And the more specific the identity, the more the real person will be able to reveal themselves, and the more difficulty the imposter will have impersonating them. [1] - https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf
- Ukv 2y ago
  I agree that "Man pretending to be woman, vs real woman" is just an example, used to introduce the question in the form of a party game between humans. I see the "something" it is replaced by as "Machine pretending to be human, vs real human". I don't see indication that the machine must pretend to be human in addition to some other characteristic of the second player. I think the reason you see others as having "generalized it down" is that your interpretation is not apparent in the text. > the more specific the identity, the more the real person will be able to reveal themselves, and the more difficulty the imposter will have impersonating them. Definitely makes for a more difficult problem (arbitrarily difficult, even) and a potentially interesting extension. Currently to me it doesn't seem as insightful as Turing's original proposal - there's no more inherent human benchmark of 50%, for instance, since humans can also be bad at impersonating some specific characteristic.
  - somenameforme 2y ago
    With no need to actually imitate anything in particular, you could simply chop away everything except the most basic linguistic functions and claim you are a non-native preteen. And who's to say otherwise? In fact that's literally the exact "trick" that yet another mockery of the Turing Test used when claiming they'd overcome the Turing Test. In fact shall we not just take it to the next level? You're 5 years old - and simply respond by randomly pounding various keys on the keyboard on occasion. Boom - didn't see that coming, now did ya Turing? Passing the test will not be a benchmark because the test has been passed, but because of what passing the test ought entail. People often complain about shifting goalposts on AI, but that's not the issue. The issue is doing exactly what you're doing here and creating worthless goalposts to begin with. And so of course when you cross them, the first thing that happens is that they get inched forward somewhere closer to something reasonable, before you even have time to uncork the champagne. Why not simply skip this nonsense, and start with a reasonable goalpost to begin with? Because it's too hard? Well obviously - that's why it's a goal, and not next month's scrimmage point!

tripletao 2y ago

I agree that those are also highly significant differences, though I'd consider them a reasonable "easy mode" while we wait for a machine capable of passing Turing's original test. I focused on the statistical issue because that one seems indefensible to me. The paper's result has no clear interpretation, depending completely on what assumption the interrogator makes about the unspecified prior probability that their witness is human. It's not clear to me whether the paper's authors even understand what they've broken. Just for fun, I tried a few LLMs and couldn't get them to recognize the statistical issue either. I guess they'll probably learn before social science professors do, though.