18 ms·

I think it's too early to declare the Turing test passed. You just need to have a conversation long enough to exhaust the context window. Less than that, since

I think it's too early to declare the Turing test passed. You just need to have a conversation long enough to exhaust the context window. Less than that, since response quality degrades long before you hit hard window limits. Even with compaction. Neuroplasticity is hard to simulate in a few hundred thousand tokens.

criley2 7mo ago
For as rigorous of a Turing test as you present, I believe many (or even most) humans would also fail it. How many humans seriously have the attention span to have a million "token" conversation with someone else and get every detail perfect without misremembering a single thing?
- nine_k 7mo ago
  But context window exhaustion does not look like mere forgetfulness, but more like loss of general coherence, like getting drunk.
- stickfigure 7mo ago
  Response quality degrades long before you hit a million tokens. But sure, let's say it doesn't. If you interact with someone day after day, you'll eventually hit a million tokens. Add some audio or images and you will exhaust the context much much faster. However, I'll grant you that Turing's original imitation game (text only, human typist, five minutes) is probably pretty close, and that's impressive enough to call intelligence (of a sort). Though modern LLMs tend to manifest obvious dead giveaways like "you're absolutely right!"
- dairem 7mo ago
  Doesn't the Turing test require a human too, to be compared to the AI?
- MadxX79 7mo ago
  How do you propose to do a Turing test on a human (in a sense that is different from a machine simply passing the Turing test)? Like failing to pick out all the motorcycles in a captcha, or a turing test where you have a guy chat with two people without knowing that one of them could be a computer, and the interrogator, unprompted, suggesting one of them might be a computer?
downboots 7mo ago
It was not meant as a pass/fail
zug_zug 7mo ago
"You're absolutely right!" I think for a while the test was passed. Then we learned the hallmark characteristics of these models, and now most of us can easily differentiate. That said -- these models are programmed specifically to be more helpful, more articulate, more friendly, and more verbose than people, so that may not be a fair expectation. Even so, I think if you took all of that away, you'd be able to differentiate the two, it just might take longer.
- drob518 7mo ago
  Right. I think the modern LLMs are quite good at mimicking human words, but we were initially taken in like we were in the 1960s by ELIZA. It’s a (increasingly sophisticated) magic trick, but it’s just a trick.
sillyfluke 7mo ago
It's weird, I don't know how normally pedantic comp sci. people let this meme that the Turing test is beaten by LLMs to spread so unchallenged. As far as I'm aware, there is no restriction in the Turing test that demands that the interrogator be ignorant of the latest state-of-art in computing (and AI tech), nor is there a strict time limit enforced for the questioning? Given these conditions, it should be relatively easy for the interrogator to expose the AI in this current day and age.
Morromist 7mo ago
There are a lot of difference kinds of LLMs. 0 of the ones I've encountered are good writers, in fact all of them are horrible at it. But I wonder if there's one out there that I don't know about with a different kind of training that actually is good at writing and fun to talk to for a long time. (granted somepeople love talking to gpt 4, but also some people loved talking to ELIZA so clearly some people have a super high tolerance for slop.)
Sol- 7mo ago
I don't know. Practically, LLMs are already better conversation partners on any topic compared to the average human I have access to. This also holds in reverse, of course - if someone wants me to explain something, usually they'd be better off asking an LLM.
tim333 7mo ago
>Consider first the more accurate form of the question. I believe that in about fifty years' time it will be possible, to programme computers, with a storage capacity of about 10^9, to make them play the imitation game so well that an average interrogator will not have more than 70 per cent chance of making the right identification after five minutes of questioning. (Turing 1950) That was the test as discussed by Turning - five minutes, <70% chance of getting it right. It's not that demanding. The test you mention could maybe be called an enhanced Turing test but the original one is pretty much passed. He was a bit off on the time taken and memory used. I think more like 75 years and 50 GB rather than 50 years and 125 MB.

criley2 7mo ago

For as rigorous of a Turing test as you present, I believe many (or even most) humans would also fail it. How many humans seriously have the attention span to have a million "token" conversation with someone else and get every detail perfect without misremembering a single thing?

nine_k 7mo ago
But context window exhaustion does not look like mere forgetfulness, but more like loss of general coherence, like getting drunk.
stickfigure 7mo ago
Response quality degrades long before you hit a million tokens. But sure, let's say it doesn't. If you interact with someone day after day, you'll eventually hit a million tokens. Add some audio or images and you will exhaust the context much much faster. However, I'll grant you that Turing's original imitation game (text only, human typist, five minutes) is probably pretty close, and that's impressive enough to call intelligence (of a sort). Though modern LLMs tend to manifest obvious dead giveaways like "you're absolutely right!"
dairem 7mo ago
Doesn't the Turing test require a human too, to be compared to the AI?
MadxX79 7mo ago
How do you propose to do a Turing test on a human (in a sense that is different from a machine simply passing the Turing test)? Like failing to pick out all the motorcycles in a captcha, or a turing test where you have a guy chat with two people without knowing that one of them could be a computer, and the interrogator, unprompted, suggesting one of them might be a computer?

downboots 7mo ago

It was not meant as a pass/fail

zug_zug 7mo ago

"You're absolutely right!" I think for a while the test was passed. Then we learned the hallmark characteristics of these models, and now most of us can easily differentiate. That said -- these models are programmed specifically to be more helpful, more articulate, more friendly, and more verbose than people, so that may not be a fair expectation. Even so, I think if you took all of that away, you'd be able to differentiate the two, it just might take longer.

drob518 7mo ago
Right. I think the modern LLMs are quite good at mimicking human words, but we were initially taken in like we were in the 1960s by ELIZA. It’s a (increasingly sophisticated) magic trick, but it’s just a trick.

sillyfluke 7mo ago

It's weird, I don't know how normally pedantic comp sci. people let this meme that the Turing test is beaten by LLMs to spread so unchallenged. As far as I'm aware, there is no restriction in the Turing test that demands that the interrogator be ignorant of the latest state-of-art in computing (and AI tech), nor is there a strict time limit enforced for the questioning? Given these conditions, it should be relatively easy for the interrogator to expose the AI in this current day and age.

Morromist 7mo ago

There are a lot of difference kinds of LLMs. 0 of the ones I've encountered are good writers, in fact all of them are horrible at it. But I wonder if there's one out there that I don't know about with a different kind of training that actually is good at writing and fun to talk to for a long time. (granted somepeople love talking to gpt 4, but also some people loved talking to ELIZA so clearly some people have a super high tolerance for slop.)

Sol- 7mo ago

I don't know. Practically, LLMs are already better conversation partners on any topic compared to the average human I have access to. This also holds in reverse, of course - if someone wants me to explain something, usually they'd be better off asking an LLM.

tim333 7mo ago

>Consider first the more accurate form of the question. I believe that in about fifty years' time it will be possible, to programme computers, with a storage capacity of about 10^9, to make them play the imitation game so well that an average interrogator will not have more than 70 per cent chance of making the right identification after five minutes of questioning. (Turing 1950) That was the test as discussed by Turning - five minutes, <70% chance of getting it right. It's not that demanding. The test you mention could maybe be called an enhanced Turing test but the original one is pretty much passed. He was a bit off on the time taken and memory used. I think more like 75 years and 50 GB rather than 50 years and 125 MB.