5 ms·

The knowledge is stored in the model's weights in the same way the knowledge you possess is stored in your neurons. There is not much difference, honestly.

cm2187 4y ago
My understanding is that it basically interpolates between the most frequent sequences of words, based on context. There is no concept of expressing a rational reasoning underpinning the words. I don't think it is comparable at this stage. It is just something very good at throwing you back segments of text it has found on the internet.
- rafaelero 4y ago
  What you are probably missing is that for the model to be competent at predicting the next the word IT HAS to learn how to reason as a human being would.
  - peteradio 4y ago
    If it quacks like a duck it must reason like a duck? Here I am quacking alone in my office, I guess I know how a duck thinks.
  - goatlover 4y ago
    I don't see why this is. Humans aren't trained to perform a statistical analysis of what the next word is likely to be. We learn language from other humans in social and environmental context. The LLM is just being trained on text.
    - rafaelero 4y ago
      Nobody is saying these models are trained the same way humans are. I am just saying that during their training task (predicting the next word) they end up developing human reasoning. That of course wasn't obvious before these large models appeared. But now it's pretty clear that's the case. It wouldn't be able to perform so well in so many tasks if it didn't have something resembling reasoning capabilities.
      - mcguire 4y ago
        I asked ChatGPT about the Chinese Room and asked for references. The first it came up with was "Minds, Brains, and Programs" by John Searle (1980). The next two were, "The Chinese Room: Searle and Artificial Intelligence" edited by John Preston and Mark Bishop (2002) (Preston and Bishop did publish Views into the Chinese Room in 2002, but I can't find anything by the given title) and "The Chinese Room Argument" by David Chalmers (2006) (Chalmers has written on the topic, and published three articles in 2006, that's not one of them. Also: "Two cities: one, a city of “knights”, and the other of “knaves”. The residents of the two cities like to visit each other. One day, a man enters one of the cities, but doesn’t know which. He decides to ask the first person he meets where he is. The answer he receives is that he is in the “knaves” city." produced the result: "The man cannot be sure that the person he met was telling the truth, because the person could be a knight from the city of knights, who always lie, or a knave from the city of knaves, who always tell the truth. "To determine which city he is in, the man could ask the second person he meets the same question. If the first person was a knight, then the second person would also be a knight, and would lie about being in the city of knights. If the first person was a knave, then the second person would also be a knave, and would tell the truth about being in the city of knaves. "By asking the same question to multiple people and analyzing their responses, the man can eventually determine which city he is in. This problem, known as the "Knights and Knaves" puzzle, is a classic example of logical reasoning and problem-solving." It seems to recognize the "knights and knaves" form of puzzle, but doesn't get the knights and knaves right. And it's not getting the reasoning part at all even given varying prompts.
        rafaelero 4y ago
        I don't understand what point you are making. Are you saying that it lacks any reasoning capability because it fails this test? I say try a bunch more tests and then get the average to really have a better sense; one test is meaningless.
        mcguire 4y ago
        How many opportunities would you give someone who, when asked to justify their answer, routinely made up references, and who ignored important parts of a question and whose answer wouldn't work anyway? As it turns out, we kinda do know what is going on in a LLM, and there's no model of reality for it to base reasoning on in there. Does that mean that's impossible? No. Does it mean seeing reasoning there is making the same mistake as those who enjoyed chatting with Parry and Eliza? Probably.
- rvcdbn 4y ago
  I hear this type of opinion expressed frequently by coders. I use ChatGPT pretty much every day (to help with various tasks) and there's just no way to hold on to this kind of belief after you get to "know" it at that level. It clearly can reason, maybe not in exactly the same way as a person. It is superior to the average person just in terms of following a logical argument in many cases. Maybe that is a low bar but I can't help but think that it is much higher than you think.
  - mcguire 4y ago
    Try asking it one of Ray Smullyan's logic puzzles, like "There are two cities: one, a city of “knights” who always tell the truth, and the other of “knaves” who always lie. The residents of the two cities like to visit each other. One day, a man enters one of the cities, but doesn’t know which. What question should he ask to find out?" I had to add the part about the behavior of knights and knaves, because ChatGPT seems to recognize the puzzle but gets them wrong, and it seems to be completely ignoring the visiting behavior. Further, the answer doesn't really work.
- sinenomine 4y ago
  Interpolation is a leaky analogy. With scale these models acquire increasingly sparse discrete-like activation patterns, and there are techniques[1] amounting to pinpointing the exact locations of the knowledge and updating it freely. It seems that nonlinear interpolation in a rich enough latent space can give you a sufficiently good approximation to reasoning. 1. https://arxiv.org/abs/2210.07229 https://arxiv.org/abs/2210.07229
deleted 4y ago
[deleted]
omnicognate 4y ago
This must be about the 50th time I've seen the exact same comment with the exact same response. It's getting very dull. At some point somebody will say "it gets stuff confidently wrong all the time" and someone else will say "so just like people do?" To call the model "just statistics" is meaningless. To call it "just like a brain" is unsupported. There are better ways to debate this.
- rafaelero 4y ago
  Not exactly unsupported. There are a bunch of papers showing a correlation between patterns of brain activation and weights activations inside an AI model when they are given the same task or stimuli. And this correlation increases with AI model accuracy. Doesn't that suggest that AI models are partly mimicking the same human reasoning pathways?
  - dekhn 4y ago
    This is an open and active question, but there really isn't any literature that's strongly trusted to show that there is a truly deep relationship between how current machine learning models operate (at the mathematical level) and how brains work. The suggestion is there, no doubt, but it's unclear where that suggestion will ultimately lead. There certainly has been quite a deal of health cross-fertilization between neuroscience and computer science.
    - YorkshireSeason 4y ago
      It seems save to say that ML has been moving further and further away from what neuroscience suggests, and towards efficient execution on fast GPUs and TPUs. That's in part b/c neuroscience is currently not equipped to explain how human high-level intelligence works.
- rvcdbn 4y ago
  Agreed! I think one useful direction to take this debate is to admit that in order for you or me to interact with the tool we need to have a mental model of the tool (so we have some idea of how to interact with it). So which mental model is superior when evaluated under the criterion of "ability to interact productively with the tool"? (a) It's just statistics / stringing together phrases from the internet or (b) it's more like a human brain with vastly superior general knowledge? Personally, I would put money on (b) resulting in better interactions. This is probably something we could study empirically. But ultimately I don't really care that much about the philosophical issues, my mental model of it (a more nuanced version of b) has enabled me to use it productively almost every day to do things that I couldn't do before.
peteradio 4y ago
> The knowledge is stored in the model's weights in the same way the knowledge you possess is stored in your neurons. There is not much difference, honestly. Do we know this to be true? I guess I'm not aware that we understand very well how memories are stored/recalled in a human brain.
sidlls 4y ago
That's a fairly strong assertion. What's the evidence for this?
- rafaelero 4y ago
  Which one? That human knowledge is stored in neurons or that machine knowledge is stored in weights?
  - jfk13 4y ago
    Or that such machine "knowledge" is anything remotely like human knowledge? (Personally, I see no reason to believe that it is.)
    - rafaelero 4y ago
      Why human knowledge would be special? What about other mammals? Can they store knowledge?
  - sidlls 4y ago
    That these are equivalent.
SeanLuke 4y ago
That is an astonishingly bold claim with, I suspect, no evidence to back it up.
goatlover 4y ago
Is that what neuroscientist and machine learning people are saying? Because I've heard there are significant differences between actual neurons and digital neural networks. Anyway, I think analyzing knowledge at the level of neurons misses the bigger picture. We learn from interacting with the world and other people via bodies, not vast amounts of textual tokens.
arcturus17 4y ago
See my comment in this thread about how ChatGPT thinks that sailfish are mammals. I possibly have never read the classification of sailfish explicitly in a book or a documentary, but I could have classified it as a fish any time of day. For one it's in the name, and furthermore I can imagine with high probability that baby sailfish do not drink milk from their mothers. I don't know if the knowledge in an LLM is stored like in my neurons - I know nothing about neurobiology. But after having used ChatGPT extensively I can certainly tell that ChatGPT does not mobilize knowledge like humans.
- rvcdbn 4y ago
  It's definitely missing a "module" that humans have: something like an ability to know & admit when it's not certain about something. The way I think of it is: imagine if you constitutionally unable to ever admit that you didn't know something and you were forced to continue talking about that topic. You too would have no choice in this situation but to start spouting nonsense. Once we add this model of uncertainty, I think this type of problem will go away and it will say something like "I'm not sure but ...".
marcosdumay 4y ago
> There is not much difference, honestly. Except on raw power, and this is an architecture where processing power and memory lead to each other, so it's missing a lot of raw memory too.
vannevar 4y ago
>The knowledge is stored in the model's weights in the same way the knowledge you possess is stored in your neurons. We don't know that this is true. But even if it is, there is a critical difference in the knowledge stored. Human knowledge incorporates direct experience of the world through multiple senses, while an LLM's knowledge is solely derived from text and so misses some critical physical constraints and contexts.

cm2187 4y ago

My understanding is that it basically interpolates between the most frequent sequences of words, based on context. There is no concept of expressing a rational reasoning underpinning the words. I don't think it is comparable at this stage. It is just something very good at throwing you back segments of text it has found on the internet.

rafaelero 4y ago
What you are probably missing is that for the model to be competent at predicting the next the word IT HAS to learn how to reason as a human being would.
- peteradio 4y ago
  If it quacks like a duck it must reason like a duck? Here I am quacking alone in my office, I guess I know how a duck thinks.
- goatlover 4y ago
  I don't see why this is. Humans aren't trained to perform a statistical analysis of what the next word is likely to be. We learn language from other humans in social and environmental context. The LLM is just being trained on text.
  - rafaelero 4y ago
    Nobody is saying these models are trained the same way humans are. I am just saying that during their training task (predicting the next word) they end up developing human reasoning. That of course wasn't obvious before these large models appeared. But now it's pretty clear that's the case. It wouldn't be able to perform so well in so many tasks if it didn't have something resembling reasoning capabilities.
    - mcguire 4y ago
      I asked ChatGPT about the Chinese Room and asked for references. The first it came up with was "Minds, Brains, and Programs" by John Searle (1980). The next two were, "The Chinese Room: Searle and Artificial Intelligence" edited by John Preston and Mark Bishop (2002) (Preston and Bishop did publish Views into the Chinese Room in 2002, but I can't find anything by the given title) and "The Chinese Room Argument" by David Chalmers (2006) (Chalmers has written on the topic, and published three articles in 2006, that's not one of them. Also: "Two cities: one, a city of “knights”, and the other of “knaves”. The residents of the two cities like to visit each other. One day, a man enters one of the cities, but doesn’t know which. He decides to ask the first person he meets where he is. The answer he receives is that he is in the “knaves” city." produced the result: "The man cannot be sure that the person he met was telling the truth, because the person could be a knight from the city of knights, who always lie, or a knave from the city of knaves, who always tell the truth. "To determine which city he is in, the man could ask the second person he meets the same question. If the first person was a knight, then the second person would also be a knight, and would lie about being in the city of knights. If the first person was a knave, then the second person would also be a knave, and would tell the truth about being in the city of knaves. "By asking the same question to multiple people and analyzing their responses, the man can eventually determine which city he is in. This problem, known as the "Knights and Knaves" puzzle, is a classic example of logical reasoning and problem-solving." It seems to recognize the "knights and knaves" form of puzzle, but doesn't get the knights and knaves right. And it's not getting the reasoning part at all even given varying prompts.
      - rafaelero 4y ago
        I don't understand what point you are making. Are you saying that it lacks any reasoning capability because it fails this test? I say try a bunch more tests and then get the average to really have a better sense; one test is meaningless.
        mcguire 4y ago
        How many opportunities would you give someone who, when asked to justify their answer, routinely made up references, and who ignored important parts of a question and whose answer wouldn't work anyway? As it turns out, we kinda do know what is going on in a LLM, and there's no model of reality for it to base reasoning on in there. Does that mean that's impossible? No. Does it mean seeing reasoning there is making the same mistake as those who enjoyed chatting with Parry and Eliza? Probably.
rvcdbn 4y ago
I hear this type of opinion expressed frequently by coders. I use ChatGPT pretty much every day (to help with various tasks) and there's just no way to hold on to this kind of belief after you get to "know" it at that level. It clearly can reason, maybe not in exactly the same way as a person. It is superior to the average person just in terms of following a logical argument in many cases. Maybe that is a low bar but I can't help but think that it is much higher than you think.
- mcguire 4y ago
  Try asking it one of Ray Smullyan's logic puzzles, like "There are two cities: one, a city of “knights” who always tell the truth, and the other of “knaves” who always lie. The residents of the two cities like to visit each other. One day, a man enters one of the cities, but doesn’t know which. What question should he ask to find out?" I had to add the part about the behavior of knights and knaves, because ChatGPT seems to recognize the puzzle but gets them wrong, and it seems to be completely ignoring the visiting behavior. Further, the answer doesn't really work.
sinenomine 4y ago
Interpolation is a leaky analogy. With scale these models acquire increasingly sparse discrete-like activation patterns, and there are techniques[1] amounting to pinpointing the exact locations of the knowledge and updating it freely. It seems that nonlinear interpolation in a rich enough latent space can give you a sufficiently good approximation to reasoning. 1. https://arxiv.org/abs/2210.07229 https://arxiv.org/abs/2210.07229

deleted 4y ago

[deleted]

omnicognate 4y ago

This must be about the 50th time I've seen the exact same comment with the exact same response. It's getting very dull. At some point somebody will say "it gets stuff confidently wrong all the time" and someone else will say "so just like people do?" To call the model "just statistics" is meaningless. To call it "just like a brain" is unsupported. There are better ways to debate this.

rafaelero 4y ago
Not exactly unsupported. There are a bunch of papers showing a correlation between patterns of brain activation and weights activations inside an AI model when they are given the same task or stimuli. And this correlation increases with AI model accuracy. Doesn't that suggest that AI models are partly mimicking the same human reasoning pathways?
- dekhn 4y ago
  This is an open and active question, but there really isn't any literature that's strongly trusted to show that there is a truly deep relationship between how current machine learning models operate (at the mathematical level) and how brains work. The suggestion is there, no doubt, but it's unclear where that suggestion will ultimately lead. There certainly has been quite a deal of health cross-fertilization between neuroscience and computer science.
  - YorkshireSeason 4y ago
    It seems save to say that ML has been moving further and further away from what neuroscience suggests, and towards efficient execution on fast GPUs and TPUs. That's in part b/c neuroscience is currently not equipped to explain how human high-level intelligence works.
rvcdbn 4y ago
Agreed! I think one useful direction to take this debate is to admit that in order for you or me to interact with the tool we need to have a mental model of the tool (so we have some idea of how to interact with it). So which mental model is superior when evaluated under the criterion of "ability to interact productively with the tool"? (a) It's just statistics / stringing together phrases from the internet or (b) it's more like a human brain with vastly superior general knowledge? Personally, I would put money on (b) resulting in better interactions. This is probably something we could study empirically. But ultimately I don't really care that much about the philosophical issues, my mental model of it (a more nuanced version of b) has enabled me to use it productively almost every day to do things that I couldn't do before.

peteradio 4y ago

> The knowledge is stored in the model's weights in the same way the knowledge you possess is stored in your neurons. There is not much difference, honestly. Do we know this to be true? I guess I'm not aware that we understand very well how memories are stored/recalled in a human brain.

sidlls 4y ago

That's a fairly strong assertion. What's the evidence for this?

rafaelero 4y ago
Which one? That human knowledge is stored in neurons or that machine knowledge is stored in weights?
- jfk13 4y ago
  Or that such machine "knowledge" is anything remotely like human knowledge? (Personally, I see no reason to believe that it is.)
  - rafaelero 4y ago
    Why human knowledge would be special? What about other mammals? Can they store knowledge?
- sidlls 4y ago
  That these are equivalent.

SeanLuke 4y ago

That is an astonishingly bold claim with, I suspect, no evidence to back it up.

goatlover 4y ago

Is that what neuroscientist and machine learning people are saying? Because I've heard there are significant differences between actual neurons and digital neural networks. Anyway, I think analyzing knowledge at the level of neurons misses the bigger picture. We learn from interacting with the world and other people via bodies, not vast amounts of textual tokens.

arcturus17 4y ago

See my comment in this thread about how ChatGPT thinks that sailfish are mammals. I possibly have never read the classification of sailfish explicitly in a book or a documentary, but I could have classified it as a fish any time of day. For one it's in the name, and furthermore I can imagine with high probability that baby sailfish do not drink milk from their mothers. I don't know if the knowledge in an LLM is stored like in my neurons - I know nothing about neurobiology. But after having used ChatGPT extensively I can certainly tell that ChatGPT does not mobilize knowledge like humans.

rvcdbn 4y ago
It's definitely missing a "module" that humans have: something like an ability to know & admit when it's not certain about something. The way I think of it is: imagine if you constitutionally unable to ever admit that you didn't know something and you were forced to continue talking about that topic. You too would have no choice in this situation but to start spouting nonsense. Once we add this model of uncertainty, I think this type of problem will go away and it will say something like "I'm not sure but ...".

marcosdumay 4y ago

> There is not much difference, honestly. Except on raw power, and this is an architecture where processing power and memory lead to each other, so it's missing a lot of raw memory too.

vannevar 4y ago

>The knowledge is stored in the model's weights in the same way the knowledge you possess is stored in your neurons. We don't know that this is true. But even if it is, there is a critical difference in the knowledge stored. Human knowledge incorporates direct experience of the world through multiple senses, while an LLM's knowledge is solely derived from text and so misses some critical physical constraints and contexts.