> ...If they answer doesn't make sense LLM has no ability to retrace the steps. Beam search already exists, it precisely allows the model to backtrack on its p

> ...If they answer doesn't make sense LLM has no ability to retrace the steps. Beam search already exists, it precisely allows the model to backtrack on its past tokens and find the global maxima of probability, at least within set depth/breadth. I think this is just a larger limitation of their shallow world model. Perhaps giving transformers registers will remedy this.