4 ms·
In summary, humans win the Turing test ~2/3 of the time against current SOTA LLMs. One of the more interesting tactics used was to target a weakness of the LLM
by tdba 3y ago
In summary, humans win the Turing test ~2/3 of the time against current SOTA LLMs. One of the more interesting tactics used was to target a weakness of the LLMs themselves:
> ... participants posed questions that required an awareness of the letters within words. For example, they might have asked their chat partner to spell a word backwards, to identify the third letter in a given word, to provide the word that begins with a specific letter, or to respond to a message like "?siht daer uoy naC", which can be incomprehensible for an AI model, but a human can easily understand...
- deleted 3y ago[deleted]
- tornato7 3y agoThis is a pretty astute tactic. Because AI models operate on tokens and not characters, it's pretty easy to confuse them when you go down to the character level. Even asking "how many letters in _" is tough for LLMs.
- miohtama 3y agoSo not yet intelligent enough to pass Turing test, which is the goal of the test. I would not be worried about AI doomsday any time soon.
- TheObviousOne 3y agoInteresting tactics might going the other direction. Asking to generate in a super human capabilities... write a 65 pages of poem about X...
- choeger 3y agoOr just questions about three or four wildly different fields of science, sports and culture you happen to have more than a layman's understanding. If the answers are somewhat plausible, it's probably a model. Or your life partner.
- 8bitsrule 3y agoOr ask two or three historical questions that very few humans would know anything about ... but a well-trained model would.
- chasd00 3y ago> "?siht daer uoy naC" That took me a while to figure out and I’m a human… as far as a I know anyway.