I've actually been working on a similar eval task that uses grids of symbols to evaluate LLM's reasoning ability around the game Wordle. My initial results are:

I've actually been working on a similar eval task that uses grids of symbols to evaluate LLM's reasoning ability around the game Wordle. My initial results are: - Success: ChatGPT-4-November, Phind-November - Failures (via hallucination): GPT-3.5, Mistral-7b, Llama2-13b Interestingly, ChatGPT-4 wrote a short python script (unprompted) to develop a pseudo-CoT type generation and answered correctly. Phind, which also answered correctly, is also a code-based/code-enabled LLM. Non-code-based-LLM's appear to reason cogently when describing their games - what moves are illegal, what the best guess in certain conditions are, etc - but their actual answer, even in multiple choice is comically wrong. These are very initial results but my tentative hypothesis is symbolic logical reasoning - string manipulation / function calling type tasks - are greatly enhanced by having a code modality. How to switch between natural language and code-generation may offer a strong boost on tasks like these.