Given the business model, it's important to OpenAI that attempts to get a confidence metric out of LLMs be unsuccessful. Here's why. If you could have simple L

Given the business model, it's important to OpenAI that attempts to get a confidence metric out of LLMs be unsuccessful. Here's why. If you could have simple LLMs which reported failure when asked a question they can't answer, you're going to send queries to them first. It's going to be like customer support people who answer questions from a menu but can escalate. The first-tier LLM is probably running locally on a 1U server. Harder problems get escalated to a bigger, but local, LLM. Only the really hard ones require buying some OpenAI tokens and asking the really big model. That approach would reduce OpenAI's revenue by over 90%. What OpenAI wants is for simple models to not work reliably, so you have to use their giant models even on dumb questions. Pushback on this will come from Google, which has to have a tier-type system to cope with their volume of queries, most of which are dumb.