3 ms·

This line sounds rather congruent with what Gerald Sussman has been saying for years. For example, from: https://dustycloud.org/blog/sussman-on-ai/ https://dust

by tjr 3y ago

This line sounds rather congruent with what Gerald Sussman has been saying for years. For example, from: https://dustycloud.org/blog/sussman-on-ai/ https://dustycloud.org/blog/sussman-on-ai/ At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were not interesting to him, because they were about building up a solid AI foundation, then the AI system runs as a sort of black box. "I'm not interested in that. I want software that's accountable." Accountable? "Yes, I want something that can express its symbolic reasoning. I want to it to tell me why it did the thing it did, what it thought was going to happen, and then what happened instead." He then said something that took me a long time to process, and at first I mistook for being very science-fiction'y, along the lines of, "If an AI driven car drives off the side of the road, I want to know why it did that. I could take the software developer to court, but I would much rather take the AI to court."

kenjackson 3y ago
The code and weights can be made available. He can even single step through the code. His ability to understand it though might be limited by his own intelligence.
- drsnow 3y ago
  You can't be serious. As if going line by line would make it any clearer why the software is doing what it's doing when it's obfuscated with so many neurons and layers that there's no way to grasp the entirety of the system through code alone.
  - kenjackson 3y ago
    My point is that it’s not really a black box. You need to understand it like you need to understand the brain. Our inability to understand the brain says less about the brain and more about our own intelligence (which is housed in the brain, so it gets a little circular here).
ftxbro 3y ago
If you ask a person what they were thinking, the person will just rationalize in the same way the bot will do it. But even now, with LLMs, we can capture the 'internal monologue' of the bot when it is augmented in that way. For example in the red teaming section of the gpt 4 technical report, the bot is trying to solve a captcha for some nefarious purpose. It can't do it, so it decides (as we know from reading its internal monologue) to ask a taskrabbit worker to do it for them. The taskrabbit worker semi-jokingly asks if it can't read the captcha because it's a bot. In response the bot decides (again as we know from reading its mind) to deliberately lie to the worker so the worker won't know it's a bot. The bot says it's a human with a vision impairment. I'm not sure what else you would expect for "if an AI driven car drives off the side of the road, I want to know why it did that" beyond that kind of access to the internal monologue of an AI agent.
- celdon25 3y ago
  This is a different argument. Explaining reasoning != citing sources. The latter is a solved problem if not for the implicit obfuscation in the LLM data (though that also depends on the complexity of the reasoning used to derive the response, factual assertions should be identifiable and verifiable). It would not need to be part of the AI prompt response itself. Including it in API metadata response is sufficient.
  - ftxbro 3y ago
    Sorry I don't understand what you're talking about. When you say this is a different argument, which arguments are you saying are different from each other? When you say explaining reasoning is different from citing sources, why are you talking about citing sources?
    - celdon25 3y ago
      > When you say explaining reasoning is different from citing sources, why are you talking about citing sources? Because that's what I was referring to and you moved the goalposts. Sources are a necessary part of the knowledge metadata so that truth weights can be taken into account. Otherwise the data's reliability factor isn't a first-class concept within the system. Citing the sources consistently serves as proof that the system does this (trust), and allows the end user to verify (at least to a reasonable level). My point was that while ChatGPT might be a viable product, an actual AGI system should not be attempted to be built atop a LLM that can't do this. Here is a system that does do this. https://wiki.opencog.org/w/AtomSpace https://wiki.opencog.org/w/AtomSpace

by tjr 3y ago

kenjackson 3y ago

The code and weights can be made available. He can even single step through the code. His ability to understand it though might be limited by his own intelligence.

drsnow 3y ago
You can't be serious. As if going line by line would make it any clearer why the software is doing what it's doing when it's obfuscated with so many neurons and layers that there's no way to grasp the entirety of the system through code alone.
- kenjackson 3y ago
  My point is that it’s not really a black box. You need to understand it like you need to understand the brain. Our inability to understand the brain says less about the brain and more about our own intelligence (which is housed in the brain, so it gets a little circular here).

ftxbro 3y ago

If you ask a person what they were thinking, the person will just rationalize in the same way the bot will do it. But even now, with LLMs, we can capture the 'internal monologue' of the bot when it is augmented in that way. For example in the red teaming section of the gpt 4 technical report, the bot is trying to solve a captcha for some nefarious purpose. It can't do it, so it decides (as we know from reading its internal monologue) to ask a taskrabbit worker to do it for them. The taskrabbit worker semi-jokingly asks if it can't read the captcha because it's a bot. In response the bot decides (again as we know from reading its mind) to deliberately lie to the worker so the worker won't know it's a bot. The bot says it's a human with a vision impairment. I'm not sure what else you would expect for "if an AI driven car drives off the side of the road, I want to know why it did that" beyond that kind of access to the internal monologue of an AI agent.

celdon25 3y ago
This is a different argument. Explaining reasoning != citing sources. The latter is a solved problem if not for the implicit obfuscation in the LLM data (though that also depends on the complexity of the reasoning used to derive the response, factual assertions should be identifiable and verifiable). It would not need to be part of the AI prompt response itself. Including it in API metadata response is sufficient.
- ftxbro 3y ago
  Sorry I don't understand what you're talking about. When you say this is a different argument, which arguments are you saying are different from each other? When you say explaining reasoning is different from citing sources, why are you talking about citing sources?
  - celdon25 3y ago
    > When you say explaining reasoning is different from citing sources, why are you talking about citing sources? Because that's what I was referring to and you moved the goalposts. Sources are a necessary part of the knowledge metadata so that truth weights can be taken into account. Otherwise the data's reliability factor isn't a first-class concept within the system. Citing the sources consistently serves as proof that the system does this (trust), and allows the end user to verify (at least to a reasonable level). My point was that while ChatGPT might be a viable product, an actual AGI system should not be attempted to be built atop a LLM that can't do this. Here is a system that does do this. https://wiki.opencog.org/w/AtomSpace https://wiki.opencog.org/w/AtomSpace