4 ms·
It would be interesting to see what happens when we describe the rules of chess to chatgpt as a completely new abstract game and ask it to play it. That should
by RyEgswuCsn 4y ago
It would be interesting to see what happens when we describe the rules of chess to chatgpt as a completely new abstract game and ask it to play it. That should show whether chargpt actually understood the rules or has been just scraping from past game records.
- swatcoder 4y agoThe challenge with this kind of thing is that ChatGPT (the interactive product at chat.openai.com) doesn't have a mechanism to enforce "These are the rules and you must remember and apply these no matter what". The more content a conversation accumulates since the rules are provided, the less it weighs them into what it next generates. And the more you repeat the rules back to it, (or worse -- call it out on a violation) the more the conversation becomes about the rules and the less about the game. These are probably engineering problems that can get improved some with other interfaces to the underlying LLM, but it's not something you can pull off with the ChatGPT/BingAssistant web interfaces as they're deployed right now.
- singularity2001 4y agowhat happens if you repeat the rules in the system channel of each message (in the API)
- famouswaffles 4y agoIf you simply describe addition as an algorithm and ask language models to perform said algorithm on 2 numbers, addition arithmetic accuracy even on very large numbers shoots up to 98.5% https://arxiv.org/abs/2211.09066 https://arxiv.org/abs/2211.09066 LLMs don't struggle with arithmetic because "they can't understand anything!". They struggle with arithmetic because it's really not something that's all that well encoded in language. Self supervision means you aren't necessarily always picking up the most accurate world models. There are models of arithmetic in there. They're just wrong or faulty.