3 ms·

People want to listen to folks who are confident. And that sentence right there is an example of what I mean. I could write 10 words, 100 words or 1,000 words

People want to listen to folks who are confident. And that sentence right there is an example of what I mean. I could write 10 words, 100 words or 1,000 words adding caveats to "People want to listen to folks who are confident," but most people don't want to hear it and they'd tune out. But nine words, they'll listen to and use that, even if it's not right all the time. This isn't just an "online" issue. Anecdotally, I'd say it's in human nature. I've read plenty lamenting how men are (over)confident at work and garner (unwarranted) success relative to less confident women. And IME, confidence at work is pretty successful, if only because folks _try_ the confident suggestion. The person with a host of caveats might have a better suggestion, but they are less confident in their result, which folks sense and shy away from. And then there are casual situations (which most of "online" discourse is), where I regularly see strangers confidently offer one another advice which is usually received positively. A lot of the advice is wrong, but that doesn't really matter. > I sometimes wonder if that is why LLMs can so confidently hallucinate — because they were trained on piles of overconfident human texts. The LLMs that I have worked with have no concept of "true" and "false". They have no sense of confidence in what they sense. They _phrase_ it definitively because that's what we want. "What is the capital of Australia." "The capital of Australia is Timbuktu." The LLM doesn't know if that's true. It's just making a statement we asked it to make.

maxrecursion 2y ago
There is a reason con man is short for confidence man. People are extremely susceptible to someone who sounds confident.
caseyy 2y ago
Exactly right, they do not have a concept of true and false as unsupervised learning simply makes them good at predicting the next token. But I think there is an over-confidence bias in the training data sample. On top of that, instruction tuning wants definitive answers, as you say. And finally, RLHF probably favors over-confident answers because people like that. From start to finish, over-confidence bias is everywhere — we both produce over-confident training data, and tune for over-confident answers. Or, well... that's what I think. See, I've not trained an LLM, I have only read about it online, and very little in books I have on the topic. I did some machine learning exercises in university, and that's the extent of my practical knowledge. And as I say that, the impact of my words goes down, right? They are taken less seriously than if someone said all that stuff about LLMs but never said they don't have practical experience. And yet, this makes the information as it is presented more exact, the limitations are clear, so it is more useful. More useful, but far less appealing... This is a really interesting topic.

maxrecursion 2y ago

There is a reason con man is short for confidence man. People are extremely susceptible to someone who sounds confident.

caseyy 2y ago

Exactly right, they do not have a concept of true and false as unsupervised learning simply makes them good at predicting the next token. But I think there is an over-confidence bias in the training data sample. On top of that, instruction tuning wants definitive answers, as you say. And finally, RLHF probably favors over-confident answers because people like that. From start to finish, over-confidence bias is everywhere — we both produce over-confident training data, and tune for over-confident answers. Or, well... that's what I think. See, I've not trained an LLM, I have only read about it online, and very little in books I have on the topic. I did some machine learning exercises in university, and that's the extent of my practical knowledge. And as I say that, the impact of my words goes down, right? They are taken less seriously than if someone said all that stuff about LLMs but never said they don't have practical experience. And yet, this makes the information as it is presented more exact, the limitations are clear, so it is more useful. More useful, but far less appealing... This is a really interesting topic.