Interesting, thanks. That doesn't seem like an entirely simple question, but it does demonstrate that the model is still not great at recognizing when it is ou

Interesting, thanks. That doesn't seem like an entirely simple question, but it does demonstrate that the model is still not great at recognizing when it is out of its league and should either hedge its answer, refuse altogether, or delegate to an appropriate external tool. This failure seems similar to a case that someone brought up earlier ( https://news.ycombinator.com/item?id=43466531 https://news.ycombinator.com/item?id=43466531 ). While better than expected at computation, the transformer model ultimately overestimates its own ability, running afoul of Dunning-Kruger much like humans tend to. Replying here due to rate-limiting: One interesting thing is that when one model fails spectacularly like that, its competitors often do not. If you were to cut/paste the same prompt and feed it to o1-pro, Claude 3.7, and Gemini 2.5, it's possible that they would all get it wrong (after all, I doubt they saw a lot of Irish tax law during training.) But if they do, they will very likely make different errors. Unfortunately it doesn't sound like that experiment can be run now, but I've run similar tests often enough to tell me that wrong answers or faulty reasoning are more likely model-specific shortcomings rather than technology-specific shortcomings. That's why I get triggered when people speak authoritatively on here about what AI models "can't do" or "will never be able to do." These people have almost always, almost without exception, been proven dead wrong in the past, but that never seems to bother them.