Actually LLMs can write a better plan than the author: The prompt format made absolutely no sense considering they had decided to translate away from the PDDL

Actually LLMs can write a better plan than the author: The prompt format made absolutely no sense considering they had decided to translate away from the PDDL to arbitrary natural language. So to avoid triggering their overused Clever Hans defense, I fed GPT 4 their prompt with only the instruction: "Think critically about how we could represent these rules in a way that's clearer to an LLM. Lean into using coding style identifiers where possible, and JSON formatting" (Hopefully the author won't claim telling a model to generate JSON is secretly telling it how to move blocks!) In a fresh context window at 0 temp and gpt-4-0613 I entered the JSON formatted rules it generated, along with the instruction: Return a JSON array of [{[<valid actions>],<internal thought>, <action>}] that results in goal state ... the resulting answer solved their failed few-shot example with zero-shot _ Also funny blunder: their chain of thought example generates thoughts... after the action. Surely the author understands a transformer model can't rely on an ungenerated token to affect the action taken? Edit: Also their "disguised difficulty" version was wrong? > To perform Attack action, the following facts need to be true: Province object, Planet object, Harmony. The single word "Harmony" somehow replaced replaces "Your hands are empty"? Before that they claim it's meant to be a 1:1 replacement of entities, but the actual disguised versions are not longer valid instructions