8 ms·

It's unclear to me how you could separate knowledge and reasoning: - Reasoning typically requires base knowledge to work from. A side effect of training reaso

by vikp 4y ago

It's unclear to me how you could separate knowledge and reasoning: - Reasoning typically requires base knowledge to work from. A side effect of training reasoning is embedding knowledge into the model parameters. - Even if you offload the search portion (either through outputting special tokens that are postprocessed, or applying the model in multiple steps with postprocessing), you still need embedded knowledge for the model to decide what to search for, and then to successfully integrate that knowledge (in the multi-step case). Maybe some kind of post-facto pruning of model weights?

kazinator 4y ago
Reasoning is that which knows that it lacks some necessary knowledge, whereas knowledge isn't aware that it lacks some necessary reasoning.
- rvbissell 4y ago
  This has to be one of the most insightful sentences I've ever read.
  - synu 4y ago
    Would you mind expanding on it a bit? I do sincerely appreciate its pithiness, but curious to read it explained a bit further.
    - anon35 4y ago
      Think of it as: reasoning=computation, knowledge=data. Data alone doesn’t say it must be computed. But computation, by definition, is attempting to create data (the result) that doesn’t exist. Thus: knowledge isn’t aware it must be reasoned about, but reasoning knows it’s trying to find (deduce, compute) knowledge it lacks.
      - kreeben 4y ago
        In the world of large language models, what part of "reasoning" is hard-coded and what part, if any, is learnt? Is reasoning simply a scan/search of your vector space (i.e. your knowledge) according to some hard-coded algo?
      - synu 4y ago
        Thanks for that!
      - NoToP 4y ago
        I disagree. If you have knowledge, and you don't try to do anything other than compress it to make space, reasoning about that knowledge will come about by sheer unintended consequence once the patterns your compressing on reach some threshold of sophistication. By definition, an optimal compression algo is a dimensionality reduction algo. A dimensionality reduction algo lets you do a bunch of machine learning tasks.
- lovecg 4y ago
  Fascinating - makes me think of the distinction between dreaming and being awake. Only in the latter state one can tell the difference.
  - dunefox 4y ago
    Not quite true, see lucid dreaming.
    - bheadmaster 4y ago
      To be fair to the GP comment, I've personally never experienced lucid dreaming, so for me the distinction holds - when I'm dreaming, I never know I'm dreaming, and mostly I don't even remember my life awake, I only remember whatever's in the context of the dream.
    - lovecg 4y ago
      That would be a third, distinct state from each. Dreaming while knowing that you are is quite a thrill!
Gigachad 4y ago
Doesn't seem to unreasonable to me. If I asked you "How old is Obama" and you had a data source which had the ages of every person, you don't need to know the answer from memory. Your reasoning tells you that to find out how old someone is you need to check the external resource and what to do with the info once you get it.
- PartiallyTyped 4y ago
  The required knowledge is around entity recognition, with "Obama" referring to the 44th POTUS, and not somebody else who happens to have the same surname (and there are multiple of them actually, at least 4 given his family).
  - Gigachad 4y ago
    Fair, although I’d think it acceptable if the response to your prompt was “which one? I found 4”
    - squeaky-clean 4y ago
      It would have way more than 4 people in the search results though. GP said there's at least 4 because there's him, his wife and his kids. Even knowing Obama is a person is a knowledge-based leap. (To us humans) it's obvious the question means Barack Obama because he's the most notable subject for that name. But how do you prevent your AI from responding that the "Obama JS library is 5 years old" https://github.com/rgbkrk/obama https://github.com/rgbkrk/obama
      - pixl97 4y ago
        If it's based on frequency of training data, the ex-president will have far more hits in the training corpus. Now, on less talked about topics it doesn't sound any different than what happens with people Q "How old is Tim?" A "Which Tim are you talking about, you didn't give me crap to work with?"
jfoutz 4y ago
> It's unclear to me how you could separate knowledge and reasoning yeah, me too. There are a few cases I'd point your attention to. 1, preschool toys, kids somehow manage to put the square peg in the square hole. I mean, they may chew on them or push them around, but there's a "moment of magic" when they make it all click together. Maybe there's some implicit knowledge there, I know I played games like that, but I don't remember. 2, sudoku. you don't really need to know anything, just make each line row and box different. no memorization, just look. but what about the rules? does that count as knowledge? I've been reading some math books lately, and I think we're not alone. Coping with sets of sets is a hard question that people have been wondering about for, as far as I can tell, a long time. For now, it's probably safe to say, knowledge about knowledge is different that just knowledge, and having one layer work on k1 and another layer work on k2 is ok. maybe someday add k3...kn. Other fields do that. Worth checking out. I think, we could both get very fussy about what exactly that _means_. But for now, I'm happy to be charitable in my reading. I'd also expect them to run into some really thorny problems when they try to pin down exactly what's going on, just like everybody else does. For today, good for them. Seems like a nice win.
- Closi 4y ago
  Different people might think diffrent, but when solving complex problems I do think I have seperate "think about / gather the facts" and "formulate the solution" phases. I don't think about the totality of facts in the world - I think my brain is mentally extracting the facts that are relevant to the problem and then reason about those facts. There is certinaly back/forth though, but I think I go "here is a bit of information, how does that apply? ok but what about this fact? ok here is how that would apply considering something else..." but I think this is still a gather -> solve -> gather -> solve
popinman322 4y ago
You can already coax ChatGPT into interacting with external systems today; I set up a prompt where the model pretended to be a factory system on a communication bus. It could access its "inventory" by posting a prefixed message to the communication bus. After a bit of prompt engineering the model could query inventory, "manufacture" various recipes, and store the end products in inventory. It might be possible to look at the weight activations as it reasons through contacting the external system over the emulated communication bus? For a suitably varied set of commands you might be able to find a subset of weights that are most correlated to the task and prune the others. Then you'd be left with a model that can retrieve and store information, as well as perform reasoning tasks. Still has problems with working memory (the input token limit, since the model is auto-regressive) given all the external information is coming back in via the prompt, but ChatGPT seems to handle that gracefully right now.
rocqua 4y ago
Humans (or at least I) have the ability to look up knowledge. What about an architecture where the main model has a background conversation with a knowledge model, just like I need to look up things I remember exist, but forget details off. Heck, such a knowledge model would have great value for people aswell. Using written conversation as an interface between language models feels natural and completely bonkers at the same time.
usrbinbash 4y ago
Reasoning can be defined in the abstract. Knowledge cannot. For example, If I need A & B | D & E to get C I can reason that if I have B and want C, I need A or D & E. Once I aquired this reasoning skill, I can apply it to any kind of "bool-sequence X required for Y" situation, regardless of what specificly X and Y are, or how many entities X encompasses. Whereas if I know that a rocket engine requires an oxygen/methane mix to function, I cannot transfer that to the knowledge that I need a raincoat or umbrealla in order to avoid getting wet in the rain.
krisoft 4y ago
I don’t know why you would want to separate them completely. You have some knowledge in your head, store some other in notes on your desk and some in a library (the kind with books in it). You can make the model write to a local wiki when it encounters new information and read from its own wiki when it feels it needs to store stuff. You can also make it spend time randomly browsing the knowledge base reconsolidating it, reorganising and labeling it. The architecture of the wiki doesn’t have to be “clever” in any way. It is just an old fasioned database with query function the model can write to and query from.

by vikp 4y ago

kazinator 4y ago

Reasoning is that which knows that it lacks some necessary knowledge, whereas knowledge isn't aware that it lacks some necessary reasoning.

rvbissell 4y ago
This has to be one of the most insightful sentences I've ever read.
- synu 4y ago
  Would you mind expanding on it a bit? I do sincerely appreciate its pithiness, but curious to read it explained a bit further.
  - anon35 4y ago
    Think of it as: reasoning=computation, knowledge=data. Data alone doesn’t say it must be computed. But computation, by definition, is attempting to create data (the result) that doesn’t exist. Thus: knowledge isn’t aware it must be reasoned about, but reasoning knows it’s trying to find (deduce, compute) knowledge it lacks.
    - kreeben 4y ago
      In the world of large language models, what part of "reasoning" is hard-coded and what part, if any, is learnt? Is reasoning simply a scan/search of your vector space (i.e. your knowledge) according to some hard-coded algo?
    - synu 4y ago
      Thanks for that!
    - NoToP 4y ago
      I disagree. If you have knowledge, and you don't try to do anything other than compress it to make space, reasoning about that knowledge will come about by sheer unintended consequence once the patterns your compressing on reach some threshold of sophistication. By definition, an optimal compression algo is a dimensionality reduction algo. A dimensionality reduction algo lets you do a bunch of machine learning tasks.
lovecg 4y ago
Fascinating - makes me think of the distinction between dreaming and being awake. Only in the latter state one can tell the difference.
- dunefox 4y ago
  Not quite true, see lucid dreaming.
  - bheadmaster 4y ago
    To be fair to the GP comment, I've personally never experienced lucid dreaming, so for me the distinction holds - when I'm dreaming, I never know I'm dreaming, and mostly I don't even remember my life awake, I only remember whatever's in the context of the dream.
  - lovecg 4y ago
    That would be a third, distinct state from each. Dreaming while knowing that you are is quite a thrill!

Gigachad 4y ago

Doesn't seem to unreasonable to me. If I asked you "How old is Obama" and you had a data source which had the ages of every person, you don't need to know the answer from memory. Your reasoning tells you that to find out how old someone is you need to check the external resource and what to do with the info once you get it.

PartiallyTyped 4y ago
The required knowledge is around entity recognition, with "Obama" referring to the 44th POTUS, and not somebody else who happens to have the same surname (and there are multiple of them actually, at least 4 given his family).
- Gigachad 4y ago
  Fair, although I’d think it acceptable if the response to your prompt was “which one? I found 4”
  - squeaky-clean 4y ago
    It would have way more than 4 people in the search results though. GP said there's at least 4 because there's him, his wife and his kids. Even knowing Obama is a person is a knowledge-based leap. (To us humans) it's obvious the question means Barack Obama because he's the most notable subject for that name. But how do you prevent your AI from responding that the "Obama JS library is 5 years old" https://github.com/rgbkrk/obama https://github.com/rgbkrk/obama
    - pixl97 4y ago
      If it's based on frequency of training data, the ex-president will have far more hits in the training corpus. Now, on less talked about topics it doesn't sound any different than what happens with people Q "How old is Tim?" A "Which Tim are you talking about, you didn't give me crap to work with?"

jfoutz 4y ago

> It's unclear to me how you could separate knowledge and reasoning yeah, me too. There are a few cases I'd point your attention to. 1, preschool toys, kids somehow manage to put the square peg in the square hole. I mean, they may chew on them or push them around, but there's a "moment of magic" when they make it all click together. Maybe there's some implicit knowledge there, I know I played games like that, but I don't remember. 2, sudoku. you don't really need to know anything, just make each line row and box different. no memorization, just look. but what about the rules? does that count as knowledge? I've been reading some math books lately, and I think we're not alone. Coping with sets of sets is a hard question that people have been wondering about for, as far as I can tell, a long time. For now, it's probably safe to say, knowledge about knowledge is different that just knowledge, and having one layer work on k1 and another layer work on k2 is ok. maybe someday add k3...kn. Other fields do that. Worth checking out. I think, we could both get very fussy about what exactly that _means_. But for now, I'm happy to be charitable in my reading. I'd also expect them to run into some really thorny problems when they try to pin down exactly what's going on, just like everybody else does. For today, good for them. Seems like a nice win.

Closi 4y ago
Different people might think diffrent, but when solving complex problems I do think I have seperate "think about / gather the facts" and "formulate the solution" phases. I don't think about the totality of facts in the world - I think my brain is mentally extracting the facts that are relevant to the problem and then reason about those facts. There is certinaly back/forth though, but I think I go "here is a bit of information, how does that apply? ok but what about this fact? ok here is how that would apply considering something else..." but I think this is still a gather -> solve -> gather -> solve

popinman322 4y ago

You can already coax ChatGPT into interacting with external systems today; I set up a prompt where the model pretended to be a factory system on a communication bus. It could access its "inventory" by posting a prefixed message to the communication bus. After a bit of prompt engineering the model could query inventory, "manufacture" various recipes, and store the end products in inventory. It might be possible to look at the weight activations as it reasons through contacting the external system over the emulated communication bus? For a suitably varied set of commands you might be able to find a subset of weights that are most correlated to the task and prune the others. Then you'd be left with a model that can retrieve and store information, as well as perform reasoning tasks. Still has problems with working memory (the input token limit, since the model is auto-regressive) given all the external information is coming back in via the prompt, but ChatGPT seems to handle that gracefully right now.

rocqua 4y ago

Humans (or at least I) have the ability to look up knowledge. What about an architecture where the main model has a background conversation with a knowledge model, just like I need to look up things I remember exist, but forget details off. Heck, such a knowledge model would have great value for people aswell. Using written conversation as an interface between language models feels natural and completely bonkers at the same time.

usrbinbash 4y ago

Reasoning can be defined in the abstract. Knowledge cannot. For example, If I need A & B | D & E to get C I can reason that if I have B and want C, I need A or D & E. Once I aquired this reasoning skill, I can apply it to any kind of "bool-sequence X required for Y" situation, regardless of what specificly X and Y are, or how many entities X encompasses. Whereas if I know that a rocket engine requires an oxygen/methane mix to function, I cannot transfer that to the knowledge that I need a raincoat or umbrealla in order to avoid getting wet in the rain.

krisoft 4y ago

I don’t know why you would want to separate them completely. You have some knowledge in your head, store some other in notes on your desk and some in a library (the kind with books in it). You can make the model write to a local wiki when it encounters new information and read from its own wiki when it feels it needs to store stuff. You can also make it spend time randomly browsing the knowledge base reconsolidating it, reorganising and labeling it. The architecture of the wiki doesn’t have to be “clever” in any way. It is just an old fasioned database with query function the model can write to and query from.