> The result shows that the LLM has a hard time generalizing to new (basic!) problems that were not in its training set; this suggests it is memorizing solution

> The result shows that the LLM has a hard time generalizing to new (basic!) problems that were not in its training set; this suggests it is memorizing solutions rather than understanding the mechanisms [...] I am once again reminded of Searle's "Chinese room" thought experiment [0] in which it is argued that computers cannot understand Chinese, merely memorize and regurgitate a canned set of responses to given prompts, and execute procedures by rote; that their computations cannot be "about" a subject matter (such as Chinese, or programming) the same way the operation and contents of our minds are "about" something. [0] https://plato.stanford.edu/entries/chinese-room/ https://plato.stanford.edu/entries/chinese-room/