Based on our results and previous work, these LLMs appear to have brittle, inflexible abstractions of what represents an alphabet in the context of the letter-s

Based on our results and previous work, these LLMs appear to have brittle, inflexible abstractions of what represents an alphabet in the context of the letter-string analogy task, despite being given the ordered list of letters/symbols before each item. It appears that LLMs like GPT-4 can only perform this abstraction by creating and executing code to map the novel alphabet to new positions In contrast, in children, familiarity with letters or symbols does not seem to influence how well they solve letter-string analogies. As such, our results add to the accumulating evidence that questions whether reasoning actually occurs in these LLMs Not terribly surprising I would think, but a nice study to have in case someone has had to much Kool-Aid.