3 ms·

I'm curious, how do you determine whether ChatGPT "knows" or does not "know" anything? What would an AI model have to demonstrate to convince you that it "know

by 1lint 4y ago

I'm curious, how do you determine whether ChatGPT "knows" or does not "know" anything? What would an AI model have to demonstrate to convince you that it "knows" something, or that it is "intelligent", or that it is "a liar"?

swatcoder 4y ago
That's a great question. I'll admit that I haven't even figured out what it means for anything to "know" something. But everything I've learned about, witnessed, and experienced with ChatGPT is consistent with a far more mechanical process than I experience with myself, my friends, or my community. I find it easy to spot the wires above the marionette and find there workings to be pretty intuitive. I have a comfortable sense of which ones to tug to make it dance the way I want, and when it's producing something unusual I've found that it doesn't take a lot of work to come up with a consistent, mechanical theory of how. With people, that sense of clarity and comprehensibility hasn't really become apparent despite many decades of engaging with a bunch of them. There seems to be something far more sophisticated going on under the hood there, and they consistently surprise me and confound me in ways that ChatGPT doesn't even approach. Maybe some AI model will achieve the same thing someday, and maybe that day will be soon, but for now ChatGPT looks and acts exactly like what the associated research papers would suggest.
- 1lint 4y ago
  > I'll admit that I haven't even figured out what it means for anything to "know" something. Same! That's the overhanging concern I have when hearing comparisons made between artificial and human intelligence. I don't feel that human knowledge/intelligence is well defined enough to the point where we can meaningfully compare it to alternative systems. But my gut instinct is that our neural architecture is composed of mechanistic biological compounds that gradually evolved sufficiently complex levels of organization to achieve our current level of cognition. Capacity for knowledge/intelligence/consciousness did not suddenly appear at some instant in our evolutionary history, rather it gradually emerged from increasing levels of neural complexity. I would rationalize the development of AI capacity for cognition in a similar light, that it will gradually emerge from increasing levels of complexity.
  - candiodari 4y ago
    But ChatGPT doesn't have any knowledge. It has never interacted with anything other than a training corpus of conversations and all it's ever trying to do is keep you talking. It's also very much NOT interacting with you when it's talking to you (it's not learning/training, it's merely inferring). It's just so good at, frankly, at being a troll (keeping you talking while not caring about truth or being deceitful), that it has learned a bunch of truths, with emphasis on truths that keep people hooked on it's conversations.
    - 1lint 4y ago
      I feel that on some level, human knowledge/interaction can be similarly reduced to a basic mechanistic nature. Our prior experiences play the part of training data, and our responses can be reduced to mechanistic processing of stimuli. I believe integrating reinforcement learning would address the last point. I have to say though, humans are much better at trolling.
- okamiueru 4y ago
  > I'll admit that I haven't even figured out what it means for anything to "know" something. I think that a core part of what it means to "know" something, is also an understanding of "not knowing". If you provide ChatGPT with insufficient data, it doesn't ask you to clarify anything to fill in the dots or try to eliminate a possible misunderstanding. It just, as you nicely put it, completes the sentence as best it can. Ask a curious human a logical question or puzzle, and they'll often reply with a bunch of follow ups to fix what they assume to be a lack of knowing.
  - dns_snek 4y ago
    I've had an opposite experience when trying to get it to solve a puzzle. First it said that it didn't have a certain piece of information which I promptly provided, then it started going in circles, bullshitting about different pieces of "missing" information, all of which had been provided. It's really good at faking logical reasoning abilities up to a certain point but it very quickly falls apart and starts returning nonsense.
    - okamiueru 4y ago
      I see. I suppose what I mean is that it doesn't have a conceptual model of the what it knows and doesn't know. It doesn't try to improve upon that conceptual model because it knows that it needs to, it just maps sentences that typically require a follow up question, with a follow up question.
sublinear 4y ago
In a word: comprehension. I should be able to query specific details about a topic and get a consistent response with respect to its own and my own answers complete with citations of its sources. When I lie to it, it should know I'm lying. When I pose an argument it should actually debate instead of giving canned answers. When I ask something unfalsifiable it should recognize it as such instead of hallucinating. When I ask something in a deliberately vague way it should be able to pinpoint exactly what I made vague and ask for me to clarify that part explicitly. When I contradict myself it should ask me questions and continue the conversation instead of me doing all the heavy lifting.
- 1lint 4y ago
  Based on convos I see on reddit, twitter, social media at large, that standard sounds like a pretty high bar that could produce a lot of false negatives. I don't see many people citing sources, canned answers seems rather common, hallucination and self-contradiction seems disturbingly common as well. Of course many of those comments may come from legitimate bots, but I'm fairly certain the stuff I write came from a bona-fide human.
hansvm 4y ago
Ignoring the philosophical discussion, one thing that ChatGPT lacks is the fact that I absolutely have to be able to independently verify the results in the context of my intended application, whereas with my favorite YouTube cooking channel I can trust that it's good enough and move on with my day. There's no grounding in reality. Addressing the philosophy a little, it's pretty damning when the AI fails at extremely basic logical tasks. If even logical primitives are broken, how could it possibly prove to itself, much less to us, that its more complicated ideas are true? That still sidesteps what it means to "know" something, but I think it's a meaningful prerequisite.

by 1lint 4y ago

swatcoder 4y ago

That's a great question. I'll admit that I haven't even figured out what it means for anything to "know" something. But everything I've learned about, witnessed, and experienced with ChatGPT is consistent with a far more mechanical process than I experience with myself, my friends, or my community. I find it easy to spot the wires above the marionette and find there workings to be pretty intuitive. I have a comfortable sense of which ones to tug to make it dance the way I want, and when it's producing something unusual I've found that it doesn't take a lot of work to come up with a consistent, mechanical theory of how. With people, that sense of clarity and comprehensibility hasn't really become apparent despite many decades of engaging with a bunch of them. There seems to be something far more sophisticated going on under the hood there, and they consistently surprise me and confound me in ways that ChatGPT doesn't even approach. Maybe some AI model will achieve the same thing someday, and maybe that day will be soon, but for now ChatGPT looks and acts exactly like what the associated research papers would suggest.

1lint 4y ago
> I'll admit that I haven't even figured out what it means for anything to "know" something. Same! That's the overhanging concern I have when hearing comparisons made between artificial and human intelligence. I don't feel that human knowledge/intelligence is well defined enough to the point where we can meaningfully compare it to alternative systems. But my gut instinct is that our neural architecture is composed of mechanistic biological compounds that gradually evolved sufficiently complex levels of organization to achieve our current level of cognition. Capacity for knowledge/intelligence/consciousness did not suddenly appear at some instant in our evolutionary history, rather it gradually emerged from increasing levels of neural complexity. I would rationalize the development of AI capacity for cognition in a similar light, that it will gradually emerge from increasing levels of complexity.
- candiodari 4y ago
  But ChatGPT doesn't have any knowledge. It has never interacted with anything other than a training corpus of conversations and all it's ever trying to do is keep you talking. It's also very much NOT interacting with you when it's talking to you (it's not learning/training, it's merely inferring). It's just so good at, frankly, at being a troll (keeping you talking while not caring about truth or being deceitful), that it has learned a bunch of truths, with emphasis on truths that keep people hooked on it's conversations.
  - 1lint 4y ago
    I feel that on some level, human knowledge/interaction can be similarly reduced to a basic mechanistic nature. Our prior experiences play the part of training data, and our responses can be reduced to mechanistic processing of stimuli. I believe integrating reinforcement learning would address the last point. I have to say though, humans are much better at trolling.
okamiueru 4y ago
> I'll admit that I haven't even figured out what it means for anything to "know" something. I think that a core part of what it means to "know" something, is also an understanding of "not knowing". If you provide ChatGPT with insufficient data, it doesn't ask you to clarify anything to fill in the dots or try to eliminate a possible misunderstanding. It just, as you nicely put it, completes the sentence as best it can. Ask a curious human a logical question or puzzle, and they'll often reply with a bunch of follow ups to fix what they assume to be a lack of knowing.
- dns_snek 4y ago
  I've had an opposite experience when trying to get it to solve a puzzle. First it said that it didn't have a certain piece of information which I promptly provided, then it started going in circles, bullshitting about different pieces of "missing" information, all of which had been provided. It's really good at faking logical reasoning abilities up to a certain point but it very quickly falls apart and starts returning nonsense.
  - okamiueru 4y ago
    I see. I suppose what I mean is that it doesn't have a conceptual model of the what it knows and doesn't know. It doesn't try to improve upon that conceptual model because it knows that it needs to, it just maps sentences that typically require a follow up question, with a follow up question.

sublinear 4y ago

In a word: comprehension. I should be able to query specific details about a topic and get a consistent response with respect to its own and my own answers complete with citations of its sources. When I lie to it, it should know I'm lying. When I pose an argument it should actually debate instead of giving canned answers. When I ask something unfalsifiable it should recognize it as such instead of hallucinating. When I ask something in a deliberately vague way it should be able to pinpoint exactly what I made vague and ask for me to clarify that part explicitly. When I contradict myself it should ask me questions and continue the conversation instead of me doing all the heavy lifting.

1lint 4y ago
Based on convos I see on reddit, twitter, social media at large, that standard sounds like a pretty high bar that could produce a lot of false negatives. I don't see many people citing sources, canned answers seems rather common, hallucination and self-contradiction seems disturbingly common as well. Of course many of those comments may come from legitimate bots, but I'm fairly certain the stuff I write came from a bona-fide human.

hansvm 4y ago

Ignoring the philosophical discussion, one thing that ChatGPT lacks is the fact that I absolutely have to be able to independently verify the results in the context of my intended application, whereas with my favorite YouTube cooking channel I can trust that it's good enough and move on with my day. There's no grounding in reality. Addressing the philosophy a little, it's pretty damning when the AI fails at extremely basic logical tasks. If even logical primitives are broken, how could it possibly prove to itself, much less to us, that its more complicated ideas are true? That still sidesteps what it means to "know" something, but I think it's a meaningful prerequisite.