3 ms·

You're trying the old primary school method of testing things based on the wrong criteria. Why does it need to play tic-tac-toe, hangman, or rock paper scissor

You're trying the old primary school method of testing things based on the wrong criteria. Why does it need to play tic-tac-toe, hangman, or rock paper scissors? Why do you think a language model would be good at those things? Similarly, why would you expect a thing trained on the context of text would be good at reversing strings? It's amazing it's as good as it is at these things because it doesn't really make sense that it could do these things unless they trained it on reversed strings to add diversity but it's hard to gain context from a string backwards. ALSO: your transcript doxs you and is hard to tell where your messages end and GPT begins. Just a heads up in case you don't want your full name leaked to whoever reads this

localplume 4y ago
Because those games are just a way to measure how an internal state changes with moves initiated by the ego and initiated by someone else. The point is that there is no consistant internal state because it hallucinates and spotaneously changes. its like telling the language model a story, and getting it to repeat certain facts about it or you make additions to the story. its the exact same thing. It needs quite a lot of "prompt engineering" to push it in the correct direction, and even then its frequently incorrect.
- famouswaffles 4y ago
  It can play tic-tac toe, chess just fine https://pastebin.com/cPwpZnZu https://pastebin.com/cPwpZnZu https://twitter.com/zswitten/status/1631107663500304384 https://twitter.com/zswitten/status/1631107663500304384
jhanschoo 4y ago
The idea for games is that these are simple games, presumably with complete descriptions from various sources already in their database. The test is of their reasoning ability and ability to generalize descriptions into tasks that aren't chimeras of examples already in their database, y'know, like a classic test of what we'd call raw reasoning and intelligence, not crystallized intelligence. With reversing strings unfortunately it's not likely to achieve that considering its tokenization.

localplume 4y ago

Because those games are just a way to measure how an internal state changes with moves initiated by the ego and initiated by someone else. The point is that there is no consistant internal state because it hallucinates and spotaneously changes. its like telling the language model a story, and getting it to repeat certain facts about it or you make additions to the story. its the exact same thing. It needs quite a lot of "prompt engineering" to push it in the correct direction, and even then its frequently incorrect.

famouswaffles 4y ago
It can play tic-tac toe, chess just fine https://pastebin.com/cPwpZnZu https://pastebin.com/cPwpZnZu https://twitter.com/zswitten/status/1631107663500304384 https://twitter.com/zswitten/status/1631107663500304384

jhanschoo 4y ago

The idea for games is that these are simple games, presumably with complete descriptions from various sources already in their database. The test is of their reasoning ability and ability to generalize descriptions into tasks that aren't chimeras of examples already in their database, y'know, like a classic test of what we'd call raw reasoning and intelligence, not crystallized intelligence. With reversing strings unfortunately it's not likely to achieve that considering its tokenization.