5 ms·
1960s chatbot ELIZA beat OpenAI's GPT-3.5 in a recent Turing test study
- throwuwu 3y agoWhat a waste of time. Why wouldn’t they put in a tiny bit of effort to use an uncensored model like a dolphin 2.0 finetune? Not serious work at all.
- 3836293648 3y agoI thought ELIZA war lost and only clones remained?
- homarp 3y agohttps://news.ycombinator.com/item?id=27321906 https://news.ycombinator.com/item?id=27321906 it was eventually found
- gumballindie 3y agoTime for a new test as the turning test while fun is out of date.
- cykros 3y agoVoight-Kampff.
- gyudin 3y agoAnother meaningless study written for the sake of writing something. Didn’t even bother to prepare a proper role for GPT-4. And the whole article can be shortened down to “GPT 3.5 has been conditioned by OpenAI specifically not to present itself as a human.” and it does it with success :)
- deleted 3y ago[deleted]
- RandomLensman 3y agoGPT4 might not be great at the Turing test either, it seems (https://arxiv.org/abs/2310.20216 https://arxiv.org/abs/2310.20216 and that is the simple non-adverserial version of the test).
- deleted 3y ago[deleted]
- hfhdjdks 3y ago"are you a human?" "No, I'm a model trained by OpenAI" Nearly fooled me
- mewpmewp2 3y ago> "No, I'm a model trained by OpenAI" That's exactly what a human would say.
- Kim_Bruning 3y ago> Hello, I am Eliza. * Are you a human > Would you prefer if I were not a human? Eliza wins this round! %-P
- deleted 3y ago[deleted]
- TechRemarker 3y ago“ GPT-3.5, the base model behind the free version of ChatGPT, has been conditioned by OpenAI specifically not to present itself as a human, which may partially account for its poor performance.” So 3.5 isn’t bad at passing it compared to a 1960s test, it just has been design specifically not to, contrary to the implications of the title.
- taspeotis 3y ago> some interrogators reported thinking that ELIZA was “too bad” to be a current AI model, and therefore was more likely to be a human intentionally being uncooperative
- masswerk 3y agoGiven that ELIZA doesn't really work outside its supposed social setup, which makes some of its behavior acceptable and/or intelligible, it would be hard to believe that anyone would have been actually fooled by this. (Still, the performance of ELIZA in the bounds of its intended setup is remarkable, especially for its small rule set. But there is little doubt that this is insufficient for a sustained free conversation.) – Let's call it a "parasitic win". ;-)
- ithkuil 3y agoIt clearly shows that the Turing test is not only about the bot but also about the humans
- rvense 3y agoYou can think of a language as an psychological object, a thing that can be described by a grammar and a dictionary. That's basically where linguistics started, and it's part of it. But some things about language are really best understood and described by looking at it as a thing that people do together, a situated, interactice activity. And I think that part of the reason that both Eliza and GPT work so well is that the observer really, really wants it to, because that's a crucial part of how this activity is structured, as described by Grice and others: https://en.m.wikipedia.org/wiki/Cooperative_principle https://en.m.wikipedia.org/wiki/Cooperative_principle
- pyinstallwoes 3y agoI like to think of it as language is intelligence as a lifeform, and humans as a hostform. Language has operations and rules that construct the space of all possible thought, and not all languages are in the set of all possible thoughts. The host form navigates a coordinate space of a constrainted possible thoughts. Whenever a language is used to communicate across interface boundaries (bodies) then some level of mapping takes place to exchanging mutual possible thoughts.
- transfire 3y agoHow does that make you feel?
- anthk 3y agoIf you use Linux/BSD you have Eliza under Alt-x (or Esc-x) doctor for Emacs. Also, under Perl set up cpanminus if you don't want to play with Emacs: curl -L https://cpanmin.us | perl - App::cpanminus ~/perl5/bin/cpanm -n local::lib echo 'eval $(perl -I ~/perl5/lib/perl5/ -Mlocal::lib)' >> ~/.profile echo 'export PATH=$HOME/perl5/bin:$PATH' >> ~/.profile Logout and login again. Install rivescript: cpanm -n RiveSript Try Eliza under RiveScript: ~/perl5/bin/rivescript ~/perl5/lib/perl5/RiveScript/demo/ If you want to create your own chatbot in RiveScript, the language it's pretty easy and you can bind Perl functions, code and who knows to your bot, so if you ask for the weather, for instance, you could call an HTTP module for Perl to call wttr.in for instance. Happy Hacking.
- toss1 3y ago>> raises further questions about using the Turing test to evaluate AI model performance. Aside from the fact that ELIZA was tuned to present as human and GPT3.5 was not, the validity of the basic Turing Test still seems a key issue. Fooling ordinary humans is a common occurrence, as Richard Feynman pointed out, "... and the easiest one to fool is ourselves.".
- usrbinbash 3y agoThe turing test, as has been often discussed, is pointless. Reason being that the tests results are determined at least as much, if not more, by the interrogator and the human interrogee as they are by the machine. It doesn't test "Can a machine think" so much as it tests "Can a human be tricked". There is a reason why ML research mostly ignores the Turing "Test". But, alas, its name has a nice ring to it, it is known (by name at least) to pop culture, and easy enough to understand for everyone, and so it gets far more attention than it deserves. Pretty much like Asimovs three "laws" of robotics :D
- n2d4 3y agoThat doesn't mean it's pointless. For most practical (commercial?) purposes, "can a human be tricked" is a much more important property than "can it think". If a decent portion of humans can't distinguish the AI anymore, with trickery or not, THAT opens up space for plenty of applications, regardless of conscience or not. Think of support bots that sound like humans, sales, email drafts, etc. Like, most people would say that animals can think, but clearly it can't pass the Turing test. Would you rather have unconscious GPT-4 or a conscious pig "working" at your job?
- usrbinbash 3y ago> For most practical (commercial?) purposes, "can a human be tricked" is a much more important property than "can it think". I disagree. In my experience human customers don't care much about whether its obvious when they are communicating with a robot. In fact, in several use cases it's considered good practice, and may even be legally required, to tell them when they are. What they DO care about, is whether their requests, inquiries, complaints, etc. are handled quickly and efficiently. We don't create value for our customers by giving them good chatterbots. We create value by giving them highly functional automated systems.
- n2d4 3y agoI mean, if it can't handle requests like that, then it'll also not pass the Turing test (since asking the AI to do help with simple support request would get it to fail). I doubt that "thinking" systems are the only ones that can handle a support ticket (GPT-4 does quite alright on a lot of them already), at least if you require consciousness as a requirement for "thinking". Maybe we just disagree on what we mean when we say "think"?
- seydor 3y ago[flagged]
- deleted 3y ago[deleted]
- epistasis 3y agoPeople go wild for new code integrating with ChatGPT, but us eMacs users have been living in the future since first picking up the editor. M-x doctor can fix all your woes.
- facialwipe 3y agoEmacs and eMacs are two very different things.
- tonyarkles 3y agoLol and iOS autocorrect definitely prefers the latter.
- epistasis 3y agoI had forgotten that eMacs even exist. But there must have been another bad update to iOS's autocorrection wieghts because I definitely did not have an eMacs problem the past few years while writing comments on my phone...
- tonyarkles 3y agoAhhhh interesting, I have had that problem forever but maybe it's because I often shrug it off and let it keep the auto-correct as-is... further reinforcing to it that that's what I meant? On the other hand, it did offer (correctly) to use the word "fucking" the other day for me instead of trying to change that to "ducking", so maybe we're making a bit of progress!
- CoastalCoder 3y agoThere's a great Corecursive podcast on the history of ELIZA: [0] [0] https://corecursive.com/eliza-with-jeff-shrager/ https://corecursive.com/eliza-with-jeff-shrager/
- Tiberium 3y agoIs there no repository with all the prompts they've used for their study? In a research like this prompts are the main part, so it's kind of weird for me that they only show one prompt in the paper.
- hedora 3y agoThere are some nice tidbits in the article: > The study also showed that participants' education and familiarity with large language models (LLMs) did not significantly predict their success in detecting AI. Also, humans only score 61%, and the best chat gpt prompt gets 41% (but 49% in the graph?). I’m definitely having trouble distinguishing between content farm crap written by humans and chatgpt output. They’re usually of comparable accuracy. I’d be curious to see an extended study where they had a dozen human participants or so, and got statistically significant percentages for each individual. Is the worst human below 41%? Is the best close to 100%?
- bbor 3y agoThis is a great example of how everyone misunderstands the Turing test.
- fennecbutt 3y agoTuring test is broken if the examiners know they're running a Turing test. Better way to have done it would have been to put them in a chat room with x other humans and just have a normal conversation/meeting, without the humans being told that AI is even involved. GPT would have aced it, this article is just scammy clickbait.