3 ms·

LLMs operate on word co-occurances. Given an initial prompt (consisting of words), it tries to predict the most likely next word. And then it goes on to predict

LLMs operate on word co-occurances. Given an initial prompt (consisting of words), it tries to predict the most likely next word. And then it goes on to predict the next most likely word based on the previous output. There is also a bit of randomization involved, so that it doesn't always produce the same results. Some people call these LLMs "statistical parrots", because the LLMs simply repeat the word combinations that they were trained with. But despite this very simple approach, we get very convincing outputs for things that do not require reasoning. LLMs might have encountered examples of logical reasoning in their training data, but they can't generalize it to arbitrary logical reasoning, because that's not something they are designed to do. Disclaimer: I'm not an expert in AI and I do not follow the developments on a deep technical level.

mrweasel 2y ago
The level of correctness from LLMs is actually pretty impressive, given that they are most just attempting to predict the next word, based on the prompt. It shouldn't come as much of a surprise that we can easily formulate questions that it will get wrong, by wording questions in a particular way, or asking about subjects for which is has little to no training data. For me the absolutely terrifying thing isn't that LLMs get answers wrong, it's the confidence with which it express those answers and how much some people / companies do not care. We know that the LLMs will get some answers wrong, they will lie, they will make up facts to justify their answers, but if will only do those last two because we make them and insist that they answer all questions (expect those where the developers put in restriction as to not offend). In some way I feel like the model should be able to rely a confidence score to the user, mostly that might be an interface issue, because we insist on limiting ourselves to the chat bot interface. The confidence score should perhaps exist outside the answer box. So you'd get an answer, and next to it a score from 0 - 100 perhaps, 0 meaning that the model doesn't actually had the training data that would allow it to answer the question.

mrweasel 2y ago

The level of correctness from LLMs is actually pretty impressive, given that they are most just attempting to predict the next word, based on the prompt. It shouldn't come as much of a surprise that we can easily formulate questions that it will get wrong, by wording questions in a particular way, or asking about subjects for which is has little to no training data. For me the absolutely terrifying thing isn't that LLMs get answers wrong, it's the confidence with which it express those answers and how much some people / companies do not care. We know that the LLMs will get some answers wrong, they will lie, they will make up facts to justify their answers, but if will only do those last two because we make them and insist that they answer all questions (expect those where the developers put in restriction as to not offend). In some way I feel like the model should be able to rely a confidence score to the user, mostly that might be an interface issue, because we insist on limiting ourselves to the chat bot interface. The confidence score should perhaps exist outside the answer box. So you'd get an answer, and next to it a score from 0 - 100 perhaps, 0 meaning that the model doesn't actually had the training data that would allow it to answer the question.