This is the call out though - the difference between a proof of concept, and then seeing what happens in production. Production and scale definitively shows th

This is the call out though - the difference between a proof of concept, and then seeing what happens in production. Production and scale definitively shows that LLMs don't reason. The output is highly unpredictable. Small changes can result in absolutely unrelated outcomes. If you were to classify text using chat gpt - chat gpt switches from classification to text generation if your text is longer than a certain length. LLM based Agents are the place where LLM reasoning died in practice. I really hope that there is a change, maybe larger context windows will allow the generated text to "reason". Although, that would not necessarily be reasoning. Even the article linked, "Indeed, LLMs make it easy to get problem-specific knowledge as long as we are willing to relax correctness requirements of that knowledge." LLMs generate text that approximates what a plan would "sound" like. It doesn't plan.