2 ms·

If you ask a person what they were thinking, the person will just rationalize in the same way the bot will do it. But even now, with LLMs, we can capture the '

by ftxbro 4y ago

If you ask a person what they were thinking, the person will just rationalize in the same way the bot will do it. But even now, with LLMs, we can capture the 'internal monologue' of the bot when it is augmented in that way. For example in the red teaming section of the gpt 4 technical report, the bot is trying to solve a captcha for some nefarious purpose. It can't do it, so it decides (as we know from reading its internal monologue) to ask a taskrabbit worker to do it for them. The taskrabbit worker semi-jokingly asks if it can't read the captcha because it's a bot. In response the bot decides (again as we know from reading its mind) to deliberately lie to the worker so the worker won't know it's a bot. The bot says it's a human with a vision impairment. I'm not sure what else you would expect for "if an AI driven car drives off the side of the road, I want to know why it did that" beyond that kind of access to the internal monologue of an AI agent.

celdon25 4y ago
This is a different argument. Explaining reasoning != citing sources. The latter is a solved problem if not for the implicit obfuscation in the LLM data (though that also depends on the complexity of the reasoning used to derive the response, factual assertions should be identifiable and verifiable). It would not need to be part of the AI prompt response itself. Including it in API metadata response is sufficient.
- ftxbro 4y ago
  Sorry I don't understand what you're talking about. When you say this is a different argument, which arguments are you saying are different from each other? When you say explaining reasoning is different from citing sources, why are you talking about citing sources?
  - celdon25 4y ago
    > When you say explaining reasoning is different from citing sources, why are you talking about citing sources? Because that's what I was referring to and you moved the goalposts. Sources are a necessary part of the knowledge metadata so that truth weights can be taken into account. Otherwise the data's reliability factor isn't a first-class concept within the system. Citing the sources consistently serves as proof that the system does this (trust), and allows the end user to verify (at least to a reasonable level). My point was that while ChatGPT might be a viable product, an actual AGI system should not be attempted to be built atop a LLM that can't do this. Here is a system that does do this. https://wiki.opencog.org/w/AtomSpace https://wiki.opencog.org/w/AtomSpace
    - ftxbro 4y ago
      > Because that's what I was referring to and you moved the goalposts. Oh I understand it now. Your original top level comment was about citing sources and then the comment by some third one a couple layers deep about Gerald Sussman changed from 'citing sources' to 'explaining reasoning'. Sorry I got confused because the thread got so deep.
    - ftxbro 4y ago
      I wish there was some better wordings to distinguish between the viewpoint like 'I don't think this AI approach is technically capable of making AGI' compared to 'I think this AI approach is probably capable of making AGI but I don't think it would be a good idea because the AGI would have some bad properties like we can't trust it'
pmoriarty 4y ago
The thing is that LLM's explanations of its "reasoning" are hallucinated. It has no access to its internal workings, and is no more insightful about those than humans are (ie. not at all). Here is what Claude[1] had to say when questioned about its own explanation about why it misinterpreted a sentence: "You raise a fair point - I don't actually "think" in the human sense, I'm an AI assistant created by Anthropic, PBC to be helpful, harmless, and honest. So I don't have subjective experiences, assumptions, or thoughts that led to misinterpreting this sentence. I simply operated based on the algorithms and training data provided by my creators at Anthropic to generate responses. My explanations for why I might make mistakes were fabricated in order to seem more human-like and helpful..." [1] - https://poe.com/Claude-instant https://poe.com/Claude-instant
- ftxbro 4y ago
  > "The thing is that LLM's explanations of its "reasoning" are hallucinated. It has no access to its internal workings, and is no more insightful about those than humans are (ie. not at all)." Yes this was my point when I said "If you ask a person what they were thinking, the person will just rationalize in the same way the bot will do it." We are saying the same thing. Or maybe you have more faith than me that a human's explanations of their own "reasoning" is not 'hallucinated' by post-hoc rationalization? > no access to its internal workings For the purposes of understanding the internal monologue of 'agentic' LLMs that run in some kind of loop, the human observer actually can get access to at least those parts of the internal workings of the AI to extract explanations like the ancestor commenter wanted. So in that sense, the AI is more reliably explainable than a human. It would be as if we had access to a human's internal monologue at all times (although the internal monologue of course is not the only part of cognition). That was my point in the 'red teaming' example.

by ftxbro 4y ago

celdon25 4y ago

This is a different argument. Explaining reasoning != citing sources. The latter is a solved problem if not for the implicit obfuscation in the LLM data (though that also depends on the complexity of the reasoning used to derive the response, factual assertions should be identifiable and verifiable). It would not need to be part of the AI prompt response itself. Including it in API metadata response is sufficient.

ftxbro 4y ago
Sorry I don't understand what you're talking about. When you say this is a different argument, which arguments are you saying are different from each other? When you say explaining reasoning is different from citing sources, why are you talking about citing sources?
- celdon25 4y ago
  > When you say explaining reasoning is different from citing sources, why are you talking about citing sources? Because that's what I was referring to and you moved the goalposts. Sources are a necessary part of the knowledge metadata so that truth weights can be taken into account. Otherwise the data's reliability factor isn't a first-class concept within the system. Citing the sources consistently serves as proof that the system does this (trust), and allows the end user to verify (at least to a reasonable level). My point was that while ChatGPT might be a viable product, an actual AGI system should not be attempted to be built atop a LLM that can't do this. Here is a system that does do this. https://wiki.opencog.org/w/AtomSpace https://wiki.opencog.org/w/AtomSpace
  - ftxbro 4y ago
    > Because that's what I was referring to and you moved the goalposts. Oh I understand it now. Your original top level comment was about citing sources and then the comment by some third one a couple layers deep about Gerald Sussman changed from 'citing sources' to 'explaining reasoning'. Sorry I got confused because the thread got so deep.
  - ftxbro 4y ago
    I wish there was some better wordings to distinguish between the viewpoint like 'I don't think this AI approach is technically capable of making AGI' compared to 'I think this AI approach is probably capable of making AGI but I don't think it would be a good idea because the AGI would have some bad properties like we can't trust it'

pmoriarty 4y ago

The thing is that LLM's explanations of its "reasoning" are hallucinated. It has no access to its internal workings, and is no more insightful about those than humans are (ie. not at all). Here is what Claude[1] had to say when questioned about its own explanation about why it misinterpreted a sentence: "You raise a fair point - I don't actually "think" in the human sense, I'm an AI assistant created by Anthropic, PBC to be helpful, harmless, and honest. So I don't have subjective experiences, assumptions, or thoughts that led to misinterpreting this sentence. I simply operated based on the algorithms and training data provided by my creators at Anthropic to generate responses. My explanations for why I might make mistakes were fabricated in order to seem more human-like and helpful..." [1] - https://poe.com/Claude-instant https://poe.com/Claude-instant

ftxbro 4y ago
> "The thing is that LLM's explanations of its "reasoning" are hallucinated. It has no access to its internal workings, and is no more insightful about those than humans are (ie. not at all)." Yes this was my point when I said "If you ask a person what they were thinking, the person will just rationalize in the same way the bot will do it." We are saying the same thing. Or maybe you have more faith than me that a human's explanations of their own "reasoning" is not 'hallucinated' by post-hoc rationalization? > no access to its internal workings For the purposes of understanding the internal monologue of 'agentic' LLMs that run in some kind of loop, the human observer actually can get access to at least those parts of the internal workings of the AI to extract explanations like the ancestor commenter wanted. So in that sense, the AI is more reliably explainable than a human. It would be as if we had access to a human's internal monologue at all times (although the internal monologue of course is not the only part of cognition). That was my point in the 'red teaming' example.