I would ask it to justify its decisions in detail and ask an expert to judge the reasoning. I'm still routinely disappointed by LLMs failing basic reasoning tas

I would ask it to justify its decisions in detail and ask an expert to judge the reasoning. I'm still routinely disappointed by LLMs failing basic reasoning tasks. Most LLMs fail easy tests like "A is faster than B, B is faster than C, is C faster than A?" and questions about a ball in an upside-down cup. Better models get those but fail at other “common sense” tasks. One example from the very end of the OpenAI keynote, the demo when they granted credits first to five people and then to everyone: did the AI know not to credit the five lucky recipients twice?