The level of correctness from LLMs is actually pretty impressive, given that they are most just attempting to predict the next word, based on the prompt. It sh

The level of correctness from LLMs is actually pretty impressive, given that they are most just attempting to predict the next word, based on the prompt. It shouldn't come as much of a surprise that we can easily formulate questions that it will get wrong, by wording questions in a particular way, or asking about subjects for which is has little to no training data. For me the absolutely terrifying thing isn't that LLMs get answers wrong, it's the confidence with which it express those answers and how much some people / companies do not care. We know that the LLMs will get some answers wrong, they will lie, they will make up facts to justify their answers, but if will only do those last two because we make them and insist that they answer all questions (expect those where the developers put in restriction as to not offend). In some way I feel like the model should be able to rely a confidence score to the user, mostly that might be an interface issue, because we insist on limiting ourselves to the chat bot interface. The confidence score should perhaps exist outside the answer box. So you'd get an answer, and next to it a score from 0 - 100 perhaps, 0 meaning that the model doesn't actually had the training data that would allow it to answer the question.