This seems to be a comprehensive repeat of the "Rot13" and "Mystery Blocks world" experiments as described by Prof. Subbarao Kambhampati Rot13 meaning that LLM

This seems to be a comprehensive repeat of the "Rot13" and "Mystery Blocks world" experiments as described by Prof. Subbarao Kambhampati Rot13 meaning that LLMs can't do Rot 3, 4, ..., n except for Rot13 (because that' in the training data) Mystery Blocks World being a trivial "translation" (by direct replacement of terms) of a simple Blocks World. The LLMs can solve the original, but not the "translation" - susprisingly, even when provided with the term replacements! Both are discussed in Prof. Subbarao's Machine Learning Street Talks episode