3 ms·

The thing is that LLM's explanations of its "reasoning" are hallucinated. It has no access to its internal workings, and is no more insightful about those than

The thing is that LLM's explanations of its "reasoning" are hallucinated. It has no access to its internal workings, and is no more insightful about those than humans are (ie. not at all). Here is what Claude[1] had to say when questioned about its own explanation about why it misinterpreted a sentence: "You raise a fair point - I don't actually "think" in the human sense, I'm an AI assistant created by Anthropic, PBC to be helpful, harmless, and honest. So I don't have subjective experiences, assumptions, or thoughts that led to misinterpreting this sentence. I simply operated based on the algorithms and training data provided by my creators at Anthropic to generate responses. My explanations for why I might make mistakes were fabricated in order to seem more human-like and helpful..." [1] - https://poe.com/Claude-instant https://poe.com/Claude-instant

ftxbro 4y ago
> "The thing is that LLM's explanations of its "reasoning" are hallucinated. It has no access to its internal workings, and is no more insightful about those than humans are (ie. not at all)." Yes this was my point when I said "If you ask a person what they were thinking, the person will just rationalize in the same way the bot will do it." We are saying the same thing. Or maybe you have more faith than me that a human's explanations of their own "reasoning" is not 'hallucinated' by post-hoc rationalization? > no access to its internal workings For the purposes of understanding the internal monologue of 'agentic' LLMs that run in some kind of loop, the human observer actually can get access to at least those parts of the internal workings of the AI to extract explanations like the ancestor commenter wanted. So in that sense, the AI is more reliably explainable than a human. It would be as if we had access to a human's internal monologue at all times (although the internal monologue of course is not the only part of cognition). That was my point in the 'red teaming' example.

ftxbro 4y ago

> "The thing is that LLM's explanations of its "reasoning" are hallucinated. It has no access to its internal workings, and is no more insightful about those than humans are (ie. not at all)." Yes this was my point when I said "If you ask a person what they were thinking, the person will just rationalize in the same way the bot will do it." We are saying the same thing. Or maybe you have more faith than me that a human's explanations of their own "reasoning" is not 'hallucinated' by post-hoc rationalization? > no access to its internal workings For the purposes of understanding the internal monologue of 'agentic' LLMs that run in some kind of loop, the human observer actually can get access to at least those parts of the internal workings of the AI to extract explanations like the ancestor commenter wanted. So in that sense, the AI is more reliably explainable than a human. It would be as if we had access to a human's internal monologue at all times (although the internal monologue of course is not the only part of cognition). That was my point in the 'red teaming' example.