3 ms·

How does this site make sense? It lists the "Turing test" as "original" at greater than 50% and the the AI that "beat" it at 46%. At that point I just stopped

How does this site make sense? It lists the "Turing test" as "original" at greater than 50% and the the AI that "beat" it at 46%. At that point I just stopped scrolling.

mattnewton 2y ago
the Turing test is scored by how often an interrogator can determine if they are talking to a machine or a human. It’s perhaps a confusing way to show it, and leaves out a lot of important information about the result they are citing, but they are saying before the interrogator did better than chance and after gpt the interrogator guesses right slightly worse than chance.
lxgr 2y ago
Presumably it means that the human detected the AI correctly less than 50% of the time, averaged over a repeated number of experiments.
bacheaul 2y ago
I read that as the pass mark being able to identify human vs machine more than 50% of the time. At 50% it's no better than randomly guessing.
deleted 2y ago
[deleted]
meltyness 2y ago
I wonder if there's a hyper-Turing test where an AI passes if the model, itself, cannot determine if it is talking to itself; or perhaps stated differently, maximizing some measure of control and processing duration to successfully conceal its identity under forced processing, discounting a solution that specifically learns to be silent or incoherent. I'm not sure what the value would be, just a passing thought. This is probably already happening within the parade of censorship systems trying to imbue the models with agency
junon 2y ago
Score is based on the interrogator, a human. If you read a Markov chain bot's text you'd guess it was a bot probably 80-100% of the time. With a real human, you'd guess it was a bot maybe 0-30% of the time, depending. I'm making up these figures, but the point is lower is better, or "more Human-Like". Test was specified as >50% meaning "accurately determined human vs. bot more than half the time". The site claims LLMs are now guessed correctly less than half, which is how the turing test was defined as per the site. It makes sense, even if you disagree it's significant.
casey2 2y ago
From TFA: >GPT-4 was judged to be a human 54% of the time, outperforming ELIZA (22%) but lagging behind actual humans (67%).
- tharkun__ 2y ago
  What article? The Turing box on the page has no links to anything and the entire page is just a bunch of these boxes. Sure it has the link icon in the two places like all the others. But the Turing one has no actual links on them. Even the little info icon that works on the GSM8K does nothing for the Turing one.
spookie 2y ago
It wasn't even the real Turing test but a lesser version of it. I'm hopeful about uses of this tech, but companies need to be more honest unless they want a second winter. The incentives don't align with honesty though.

How does this site make sense? It lists the "Turing test" as "original" at greater than 50% and the the AI that "beat" it at 46%. At that point I just stopped scrolling.

mattnewton 2y ago

the Turing test is scored by how often an interrogator can determine if they are talking to a machine or a human. It’s perhaps a confusing way to show it, and leaves out a lot of important information about the result they are citing, but they are saying before the interrogator did better than chance and after gpt the interrogator guesses right slightly worse than chance.

lxgr 2y ago

Presumably it means that the human detected the AI correctly less than 50% of the time, averaged over a repeated number of experiments.

bacheaul 2y ago

I read that as the pass mark being able to identify human vs machine more than 50% of the time. At 50% it's no better than randomly guessing.

deleted 2y ago

[deleted]

meltyness 2y ago

I wonder if there's a hyper-Turing test where an AI passes if the model, itself, cannot determine if it is talking to itself; or perhaps stated differently, maximizing some measure of control and processing duration to successfully conceal its identity under forced processing, discounting a solution that specifically learns to be silent or incoherent. I'm not sure what the value would be, just a passing thought. This is probably already happening within the parade of censorship systems trying to imbue the models with agency

junon 2y ago

Score is based on the interrogator, a human. If you read a Markov chain bot's text you'd guess it was a bot probably 80-100% of the time. With a real human, you'd guess it was a bot maybe 0-30% of the time, depending. I'm making up these figures, but the point is lower is better, or "more Human-Like". Test was specified as >50% meaning "accurately determined human vs. bot more than half the time". The site claims LLMs are now guessed correctly less than half, which is how the turing test was defined as per the site. It makes sense, even if you disagree it's significant.

casey2 2y ago

From TFA: >GPT-4 was judged to be a human 54% of the time, outperforming ELIZA (22%) but lagging behind actual humans (67%).

tharkun__ 2y ago
What article? The Turing box on the page has no links to anything and the entire page is just a bunch of these boxes. Sure it has the link icon in the two places like all the others. But the Turing one has no actual links on them. Even the little info icon that works on the GSM8K does nothing for the Turing one.

spookie 2y ago

It wasn't even the real Turing test but a lesser version of it. I'm hopeful about uses of this tech, but companies need to be more honest unless they want a second winter. The incentives don't align with honesty though.