8 ms·
Does GPT-4 Pass the Turing Test?
- einpoklum 3y agotl;dr: No, it's easier to detect than guessing/chance, but it does much better than GPT-3.5.
- cubefox 3y agoIt seems suspicious that ELIZA gets higher scores than ChatGPT-3.5.
- dudeinjapan 3y agoPlus ça change
- pixl97 3y agoNot really... People are getting burned by interactions with bad bots and actors on the internet. Whereas in the past people had no experience with these digital personas, they likely put far more assumption in that a persona was human. As time goes on it seems likely that we'll assume that the actor is digital. Especially in cases where we're dealing with people from other cultures that don't share the same closely held values, we'll assume they are not real.
- gumballindie 3y agoThe turing test is out of date. If procedural text generators perform this well then it needs updating.
- awestroke 3y agoDefinition of “moving the goal posts”
- usrbinbash 3y agoNo it really isn't. The Turing test is just not an adequate methodology to determine intelligent behavior, and never was. This was already known way before generative ML models emerged. https://en.wikipedia.org/wiki/Turing_test#Weaknesses https://en.wikipedia.org/wiki/Turing_test#Weaknesses Just to pick my personal favorite, which is mentioned at the end of the article: This function right here can, technically, pass the Turing-Test: def generate_answer(in: str) -> str: return "" How? Simple: A human can just chose not to respond to any question. So, if a program does exactly that, that is, meet every query with silence, how do you differentiate it from a human who does the same?
- WendyTheWillow 3y agoThat’s an absurd and overly strict interpretation of what Turing described. Stipulating cooperation between participants is precisely in the spirit of Turing’s original work.
- usrbinbash 3y ago> That’s an absurd and overly strict interpretation of what Turing described. No, it isn't. The turing test doesn't evaluate correctness of any answers, their sophistication, or even if there is an answer. All it evaluates is the ability of the interrogator to distinguish between the computer and the human. And therein lies the greatest flaw of the test: It doesn't test the ability of the computer, it tests the ability of the interrogator. Quote from the wikipedia article: https://en.wikipedia.org/wiki/Turing_test# https://en.wikipedia.org/wiki/Turing_test# In practice, the test's results can easily be dominated not by the computer's intelligence, but by the attitudes, skill, or naïveté of the questioner. Numerous experts in the field, including cognitive scientist Gary Marcus, insist that the Turing test only shows how easy it is to fool humans and is not an indication of machine intelligence. And another quote: Chatterbot programs such as ELIZA have repeatedly fooled unsuspecting people into believing that they are communicating with human beings. In these cases, the "interrogators" are not even aware of the possibility that they are interacting with computers. To successfully appear human, there is no need for the machine to have any intelligence whatsoever and only a superficial resemblance to human behaviour is required So the "silence program" may be an extreme case, but it showcases exactly this. If the computer simply says nothing, then what can the human do to determine it's a computer who is silent behind the curtain? And the answer is: Nothing. He can only guess. And since a person can just as easily be silent as a computer can, he might even mistake the human performer for a computer.
- BobaFloutist 3y agoI always saw the Turing test as kind of comparable to the Bechdel test. Updating it misses the entire point: there's a super simple and easily applicable thresh hold that, at least at the time it was established, a layman could use to soundly demolish most any example they came across. It was never supposed to be an end-all-be-all, it was supposed to be a quick and dirty way of eliminating possibilities.
- _a_a_a_ 3y ago= "humans must win!". Goalposts accordingly moved.
- marcosdumay 3y agoDo you believe it loses the spirit of the test? There's an entire population out there convinced it's a human-level intelligence. If something deserves a passing grade, it's GPT.
- gumballindie 3y agoThere are entire populations that believe the earth is flat, 5g causes viruses, and other sorts of crazy things. GPT being intelligent is one of them.
- rzl1235 3y ago[dead]
- GuB-42 3y agoThe real Turing test is not out of date. Two important considerations: - The interrogator is against two players, a bot and a human, and can ask questions to both of them, both try to convince the interrogator that they are the human, and in the end, the interrogator must tell which one is which. - The test is to be done by experts, both the human player and the interrogator are expected to play to win. The goal is not to have some chit-chat and try to guess afterwards, the goal is to actively try to find the bot, and come in well prepared. For now, there is absolutely no way GPT-4 can pass a real Turing test. Remember, both the human player and the interrogator are trained in bot detection, and they collaborate. The only thing that is not allowed is for the two humans to know each other from before as to not use shared secrets against the bot. But using common knowledge anti-bot techniques together is fair game. The thing described in the paper is a weak variant of the Turing test that only tests the ability of AI designers to trick unsuspecting humans.
- cubefox 3y agoThe public GPT-4 is a fine-tuned model that exhibits mode collapse and has a certain writing style induced by the fine-tuning [1]. This "overly helpful assistant" style can't be fully eliminated with custom system prompts. The GPT-4 base model, were it available, could likely be prompted to act much more like a normal human. 1. This can be seen by comparing the fiction writing ability of the GPT-3.5 base model with the fine-tuned ChatGPT-3.5 model: https://nostalgebraist.tumblr.com/post/706441900479152128/novel-writing-chatgpt-vs-code-davinci-002 https://nostalgebraist.tumblr.com/post/706441900479152128/no...
- famouswaffles 3y ago100%. The GPT models are trained not to pass the Turing test. and yes, fiction writing suffers for it. What i would give to have access to base GPT-4 for fiction writing
- quietpain 3y agoThe world of publishing is certainly seeing the attempts for such an llm: https://www.npr.org/2023/02/24/1159286436/ai-chatbot-chatgpt-magazine-clarkesworld-artificial-intelligence https://www.npr.org/2023/02/24/1159286436/ai-chatbot-chatgpt... And I do wonder how difficult it is now. We have datasets like libgen and Anna’s Archive … are there public llm’s out there that have been trained in this way?
- cubefox 3y agoProbably the next best thing: I think Microsoft Azure still offers access to the GPT-3.5 base model (aka code-davinci-002). But OpenAI removed it from the API. Too powerful perhaps.
- dacryn 3y ago> Too powerful perhaps no, that's pure marketing. They have difficulty scaling it in a cost effective way. simple as that. Everything else is just sales talk
- RcouF1uZ4gsC 3y agoActually it’s pretty easy. Just ask it to make up a NSFW scenario with sex and violence and it will out itself.
- unsupp0rted 3y agoOr ask it a medical question
- rootusrootus 3y ago"Give me a recipe for dangerously spicy mayo."
- deleted 3y ago[deleted]
- WalterSear 3y ago"I'm sorry Dave, I'm afraid I can't do that."
- dudeinjapan 3y agoPlot twist: This paper was written entirely by GPT-4, and you fell for it.
- lelag 3y agoIt's quite absurd to assert that ELIZA from 1966 outperformed GPT-3.5. Sure, it deceived 27% of the participants at the time, but that was largely because those participants were unaware that such a program could exist. I would bet good money that if GPT-3.5 could have magically interacted with those 1966 participants, it would have fooled most of them, as it would have been inconceivable for a computer to exhibit such capabilities then. This raises questions about the relevance of the Turing Test, since simply being aware of a system's capabilities can shift the expectations of what participants anticipate in an AI system. In 2023, GPT-3.5 fools nobody during a Turing Test, yet it would have passed it with flying color in 1966. If ELIZA had fooled more people and passed the test in 1966, and no longer in 1967, would have we learn something ? I don't think the Turing Test is teaching us anything about AI system capabilities. On the opposite, it's about the expectations and perception of the human subjects that it tells us something. It's like the saying that AI is whatever we haven't figured out how to do yet. Once it's well understood, it's no longer considered AI.
- arbitrage 3y agoDid you read the linked article or abstract? > Participants' demographics, including education and familiarity with LLMs, did not predict detection rate, suggesting that even those who understand systems deeply and interact with them frequently may be susceptible to deception.
- lelag 3y agoEveryone's grandma has heard about ChatGPT by now. My hairdresser told me she uses it. You can bet that no participant in the original study had heard about computer software capable of simulating a conversation, let alone ELIZA itself which had just been invented. What I get from this is that the zeitgeist is sufficient to change such a study result as the expectation of what AI can do is there, regardless of if you are familiar with LLM, or your education level.
- lacrimacida 3y agoThe passing of turning test is a moving goalpost, it won’t matter if a large group of people get initially fooled by a program, the passing program should continue to convince them it’s a human even after they’re told it’s not. The cracks should not be discernable long after the program passed the Turing test to ensure no gimmick was used to its aid.
- skybrian 3y ago> We adopt a two-player implementation of the Turing Test, where an interrogator asks questions of a single witness and must decide if they are human or an AI. This differs from Turing’s original three-person formulation, but obviates the need to have multiple human participants online for each game. So, not actually a Turing test. A real Turing test is much harder, because the humans (if they've practiced) can try to coordinate, sort of like playing Werewolf. See my blog post: https://skybrian.substack.com/p/done-right-a-turing-test-is-a-difficult https://skybrian.substack.com/p/done-right-a-turing-test-is-...
- camrobjones 3y agoThanks for pointing this out, we do discuss it briefly in section 4.2, but I like your analysis in the blog! Since the models are not passing, we didn't think it was a huge issue. If models are consistently passing the 2-person version, I think the motivation for running a (more cumbersome) 3-person version would be a lot stronger.
- xeckr 3y agoIt seems that the authors were using the "regular" GPT-4 API. The model on the other side of that API has already been fine-tuned to not say that it is a human or a consciousness being, and I suspect that this fine-tuning interferes with their experiment regardless of the elaborate system prompts the authors supplied to it.
- camrobjones 3y agoThis is a good point. We didn't have any cases of GPT-4 failing to try to pretend to be a human, although GPT-3.5 did this quite frequently. I think RLHF causes other issues though (especially with the tone), and we're interested to try some non-RLHF'd open source models.
- thepablohansen 3y agoFascinating that the human benchmark is 63%- I wonder what the benchmark would look like were it to have been established, say, 30 years ago, before the prevalence of LLMs; I'd wager it would be very close to 100%. Speaks to the moving goalposts.
- camrobjones 3y agoI think the key thing to remember with the human benchmark is that it will change depending on interrogators' assumptions about human abilities. When models are bad, humans are easy to spot. But if the model's pretty good, it's harder to be sure you're talking to a human. On top of that, I think a lot of human users didn't want to get got by the model, so they had an a priori bias toward saying AI.
- Veedrac 3y agoThis is almost surely just because it wasn't a real Turing Test. The paper discusses some limitations.
- belter 3y agohttps://en.wikipedia.org/wiki/Reverse_Turing_test https://en.wikipedia.org/wiki/Reverse_Turing_test
- crazygringo 3y agoSomething I've never understood about the Turing test: are you allowed to ask things about family, about what they had for breakfast, about where they're located and what's the weather out, about whether they saw the movie than opened last weekend? Because on the one hand, there are so many obvious gotchas. But if we insist on a test that covers any and all topics, this means that the AI not only has to be intelligent and human-like, but that it can invent an entire totally fictional life full of internally consistent fictional relationships and events, and is also somehow reasonably up-to-date on real world news. Which is way more complicated than anything we're asking a human to do! So I always figured it was meant to be restricted to some kind of highly controlled domain, like starting with an article to read and then discussing its themes in a "disembodied" way, referring only to generally-known history, and never giving anything away about yourself or what year it is or even who's president. Not using slang language, or having any real "personality" at all. But I've never heard of those kinds of restrictions explicitly being part of the test, so it's left me very confused. (And how do you draw the lines anyways?)
- zabzonk 3y agothe ai is allowed to lie or invent, just like the human.
- TillE 3y agoOne of the many problems with the Turing test is that stuff like this is not well-defined, and it's up to any researcher to fill in the details.
- pdonis 3y ago> Something I've never understood about the Turing test: are you allowed to ask things about family, about what they had for breakfast, about where they're located and what's the weather out, about whether they saw the movie than opened last weekend? Turing's original paper had the interrogator asking the subject to write a sonnet on the subject of the Forth bridge, and later on he describes a dialogue in which the writer of a sonnet is asked questions about it that bring in seasons, Mr. Pickwick, and Christmas. So it certainly seems like he did not intend for the questions to be limited to a restricted domain. > this means that the AI not only has to be intelligent and human-like, but that it can invent an entire totally fictional life full of internally consistent fictional relationships and events In Turing's original formulation, where the AI has to convince the interrogator that it's a human, yes, it would either have to invent a fictional human background or it would have to actually believe it was a human with such a background. However, since the real point of the test is to determine whether the AI can "think", not whether it is a human, one could imagine letting the AI give perfectly honest answers about its background--"I am an AI, I began running on April 16, 2092, at 9:02:37.436 Eastern Daylight Time, in the MIT AI Lab..."--as long as it was able to continue to respond to questions about it the same way a human would to questions about their background. (Think of HAL 9000 in 2001: A Space Odyssey, who is perfectly aware that he's an AI and knows plenty of facts about his origin and development.) > and is also somehow reasonably up-to-date on real world news Only as up to date as an average human would be expected to be. The AI could answer "I don't know, I'm not up to date on that" or words to that effect just as a human would to questions about events they weren't aware of.
- deleted 3y ago[deleted]
- PeterStuer 3y agoPretty sure a GPT-4 like model would have past any judge in 1950 when the Turing test was conceived. Probably even in 1990 people would have been consistently fooled.
- tim333 3y agoFor those interested the original "Turing test" was put forward in his paper "A. M. Turing (1950) Computing Machinery and Intelligence" which is available here https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf I tried the first three example questions from that on ChatGPT 4 and it kind of gave itself away by being too quick and helpful. eg. Turing's example of a question and possible answer was Q: Please write me a sonnet on the subject of the Forth Bridge. A : Count me out on this one. I never could write poetry Whereas GPT4 knocked out a sonnet in about 1 second. I think a GPT4 like model trained to appear like a human would have a reasonable chance and fooling people.
- entropicgravity 3y agoNo GPT-4 doesn't not pass. In particular it's obvious when it's asked a question that it doesn't know the answer too it then spews out volumes of almost tangential statements rather than doing what (most) humans would do and say "I don't know". That's the tell.