They're really only abysmal if you attempt to one-shot it and probe with tasks that would require a human a scratchpad to accomplish. Humans can't one-shot non

They're really only abysmal if you attempt to one-shot it and probe with tasks that would require a human a scratchpad to accomplish. Humans can't one-shot non trivial planning tasks either. It's the one problem i have with all the papers that try to evaluate planning for LLMs. Step away from that approach and they're ok. https://innermonologue.github.io/ https://innermonologue.github.io/ https://tidybot.cs.princeton.edu/ https://tidybot.cs.princeton.edu/