It wasn't so long ago that researchers were worried their chatbots said "I don't know" too much: https://arxiv.org/pdf/1510.03055.pdf https://arxiv.org/pdf/151

It wasn't so long ago that researchers were worried their chatbots said "I don't know" too much: https://arxiv.org/pdf/1510.03055.pdf https://arxiv.org/pdf/1510.03055.pdf >Sequence-to-sequence neural network models for generation of conversational responses tend to generate safe, commonplace responses (e.g., I don’t know) regardless of the input. We suggest that the traditional objective function, i.e., the likelihood of output (response) given input (message) is unsuited to response generation tasks. And now we find that these giant models don't even know when to say that they don't know! Of course, the old models weren't saying "I don't know" because they had some insightful introspection - it was just universally applicable response that had high likelihood. Sort of like a denoising function that turns a very noisy image into a gray blob. And the new models aren't avoiding "I don't know" because they think they know. It doesn't seem totally impossible that an even bigger language model could predict that "I don't know" or "the question is nonsensical" are the most probable answers to the questions in the article, even without having any true understanding (however you want to define that). Being able to handle these types of questions is a necessary but not sufficient criterion.