4 ms·

LLMs don't know anything except the most frequent observed response to a context made up of a sequence of tokens. How often do the words "I don't know" get utt

LLMs don't know anything except the most frequent observed response to a context made up of a sequence of tokens. How often do the words "I don't know" get uttered in books, papers, articles, stack overflow, or any other resource of knowledge?

ben_w 2y ago
What does it mean for a human to "know" something? I have some representation in my mind; as someone who doesn't have aphantasia, this representation comes with a mental image. Tower? Tall, linear, and in my case a skyscraper by default. Eiffel Tower? Paying attention to the extra context, the first word transforms the second into the eponymous structure. Model Eiffel Tower? Now the context makes it a tchotchke, probably 10cm tall. Lego model Eiffel Tower? The 1-ish meter tall one on display in the Lego shop. Is my "knowledge" the abstract representation that is in my case connected to a mental image? The attention process can reasonably be considered as developing a vector in a very high dimensional concept space, and the next token comes from what would best suit the current location in that high dimensional space. It's entirely possible that the concept of "ignorance" is linearly separable within that space (much as gender is, see the word2vec trick with "king" - "queen" ~= "man" - "woman"), and the corresponding "ignorance" vector can be associated with the sequence of words "I don't know". I think it would take actual research into the internal vector space to answer that, and while I'd like to do that research, I have some higher priorities right now. Is "knowledge" any belief? Any true belief? Any justified true belief? https://en.wikipedia.org/wiki/Gettier_problem https://en.wikipedia.org/wiki/Gettier_problem I take the position that there is no such thing as knowledge, and instead the best we can have is belief. But then, what is "belief", and can the information within an LLM said to meet whichever definition you give?

ben_w 2y ago

What does it mean for a human to "know" something? I have some representation in my mind; as someone who doesn't have aphantasia, this representation comes with a mental image. Tower? Tall, linear, and in my case a skyscraper by default. Eiffel Tower? Paying attention to the extra context, the first word transforms the second into the eponymous structure. Model Eiffel Tower? Now the context makes it a tchotchke, probably 10cm tall. Lego model Eiffel Tower? The 1-ish meter tall one on display in the Lego shop. Is my "knowledge" the abstract representation that is in my case connected to a mental image? The attention process can reasonably be considered as developing a vector in a very high dimensional concept space, and the next token comes from what would best suit the current location in that high dimensional space. It's entirely possible that the concept of "ignorance" is linearly separable within that space (much as gender is, see the word2vec trick with "king" - "queen" ~= "man" - "woman"), and the corresponding "ignorance" vector can be associated with the sequence of words "I don't know". I think it would take actual research into the internal vector space to answer that, and while I'd like to do that research, I have some higher priorities right now. Is "knowledge" any belief? Any true belief? Any justified true belief? https://en.wikipedia.org/wiki/Gettier_problem https://en.wikipedia.org/wiki/Gettier_problem I take the position that there is no such thing as knowledge, and instead the best we can have is belief. But then, what is "belief", and can the information within an LLM said to meet whichever definition you give?