5 ms·

I do wonder if we'll see the rise of symbolic AI to give deep learning a sense of common sense? I've been thinking a lot about this interview on Cyc: https://w

I do wonder if we'll see the rise of symbolic AI to give deep learning a sense of common sense? I've been thinking a lot about this interview on Cyc: https://www.youtube.com/watch?v=3wMKoSRbGVs https://www.youtube.com/watch?v=3wMKoSRbGVs

drdeca 5y ago
And here I thought one of the big benefits of DL was that it could handle the complexities which would be too hard to specify symbolically in order to give symbolic AI “common sense”. The following argument comes to mind, but I don’t really buy it (it just came to mind as something that one might say next): ‘ Perhaps there is an analogy between the solutions of “we just need to get better (more varied, better fitting the desired behavior, etc.) training data, and maybe better training procedures” and “we just need to add more/better inference rules and symbolic ways to encode statements, and add more facts about the world”. Similar in that both will produce the specific improvements they target, but where solving “the real/whole/big problem” that way is infeasible. If so, then maybe this indicates that a practical full-solution to artificial “common sense” would require something fundamentally different than both of them, if it is even possible at all. ‘ Again, I don’t really buy that line of reasoning, just expressing my inner GPT2 I guess, haha. Ok, but I presented an argument (or something like an argument) which I made up, and said that I don’t buy it. So, I should say why I don’t buy it, right? Like many of the things I write, it is chock-full of qualifiers like “perhaps” and “maybe”, to the point that one might say that it hardly makes any claims at all. But ignoring that part of it, one major difference is that the DL style architectures, seem to be working? And it isn’t clear what kinds of (practically speaking) hard limits it could run into. Now, on the other hand, perhaps at the time that symbolic AI was all the rage, it appeared the same way. (Is this what people mean when they talk about inside view vs outside view?). Why should these two things not be especially analogous? Well, saying “proposed solution X to the problem says to just [do more of what X is/do X better], and that is just like how proposed solution Y says to just [do more of what Y is/do Y better]” is kind of a fully generalize argument for dismissing any proposed type of solution where partial solutions of that type have been tried, but the whole problem hasn’t been solved that way yet, and another proposed kind of solution has already lost favor. This doesn’t seem like a generally valid line of reasoning. Sometimes you really do just need more dakka (spelling? I mean “more of the thing you already tried some of”). Of course, if one is convinced that it really was right for the older proposed kind of solution to be discarded, that probably should say something about the currently popular kind of solution. Especially if there have been many proposed kinds of solutions which have been discarded. But, it seems like much of what it says is just that the problem is hard. And, sure, that may mean an increased probability that the currently popular proposed kind of solution also doesn’t end up being satisfactory, that doesn’t mean one should be too quick to discard it. Tautologically: if no known alternative is currently at least as promising as the type of solution currently being considered, then, the current one is the most promising of the currently known options. Whether it is promising enough to actively pursue may be a different question, but it shouldn’t be marked as discarded until something else (perhaps something previously discarded, or something novel) becomes more promising.
airstrike 5y ago
From Wikipedia (https://en.wikipedia.org/wiki/Cyc#Criticisms https://en.wikipedia.org/wiki/Cyc#Criticisms) > ... A similar sentiment was expressed by Marvin Minsky: "Unfortunately, the strategies most popular among AI researchers in the 1980s have come to a dead end," said Minsky. So-called “expert systems,” which emulated human expertise within tightly defined subject areas like law and medicine, could match users’ queries to relevant diagnoses, papers and abstracts, yet they could not learn concepts that most children know by the time they are 3 years old. “For each different kind of problem,” said Minsky, “the construction of expert systems had to start all over again, because they didn’t accumulate common-sense knowledge.” Only one researcher has committed himself to the colossal task of building a comprehensive common-sense reasoning system, according to Minsky. Douglas Lenat, through his Cyc project, has directed the line-by-line entry of more than 1 million rules into a commonsense knowledge base."
- ypcx 5y ago
  And then, GPT-3 came along and rendered Cyc a wasted effort.
  - goatlover 5y ago
    Cyc was trying to encode common knowledge about the world in a bunch of rules. That goes well beyond what GPT-3 does with text.
    - ypcx 5y ago
      GPT-3 learns these rules by itself.
      - johnthescott 5y ago
        can you automate your fate with gpt-3? that is where the money lives. will sublime conclusions made by GPT-3 EVER be trusted if the reasoning to its conclusion is not understood by a human? perhaps the gestalt of gpt3 implies a meta gpt3 that could derive human grokkable explains of its "dumber" self. or maybe not.
williamtrask 5y ago
From what I can tell, many of the best thinkers agree with this idea but we haven’t cracked it yet.
mr_toad 5y ago
I doubt that gains from symbolic AI or any algorithmic improvement will outpace Moore’s law in the long run. I certainly wouldn’t want to spend decades working to improve AI, only to have my work undercut by cheaper silicon. http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
singularity2001 5y ago
this is misguided since all networks represent symbolic sense in an abstract way

drdeca 5y ago

And here I thought one of the big benefits of DL was that it could handle the complexities which would be too hard to specify symbolically in order to give symbolic AI “common sense”. The following argument comes to mind, but I don’t really buy it (it just came to mind as something that one might say next): ‘ Perhaps there is an analogy between the solutions of “we just need to get better (more varied, better fitting the desired behavior, etc.) training data, and maybe better training procedures” and “we just need to add more/better inference rules and symbolic ways to encode statements, and add more facts about the world”. Similar in that both will produce the specific improvements they target, but where solving “the real/whole/big problem” that way is infeasible. If so, then maybe this indicates that a practical full-solution to artificial “common sense” would require something fundamentally different than both of them, if it is even possible at all. ‘ Again, I don’t really buy that line of reasoning, just expressing my inner GPT2 I guess, haha. Ok, but I presented an argument (or something like an argument) which I made up, and said that I don’t buy it. So, I should say why I don’t buy it, right? Like many of the things I write, it is chock-full of qualifiers like “perhaps” and “maybe”, to the point that one might say that it hardly makes any claims at all. But ignoring that part of it, one major difference is that the DL style architectures, seem to be working? And it isn’t clear what kinds of (practically speaking) hard limits it could run into. Now, on the other hand, perhaps at the time that symbolic AI was all the rage, it appeared the same way. (Is this what people mean when they talk about inside view vs outside view?). Why should these two things not be especially analogous? Well, saying “proposed solution X to the problem says to just [do more of what X is/do X better], and that is just like how proposed solution Y says to just [do more of what Y is/do Y better]” is kind of a fully generalize argument for dismissing any proposed type of solution where partial solutions of that type have been tried, but the whole problem hasn’t been solved that way yet, and another proposed kind of solution has already lost favor. This doesn’t seem like a generally valid line of reasoning. Sometimes you really do just need more dakka (spelling? I mean “more of the thing you already tried some of”). Of course, if one is convinced that it really was right for the older proposed kind of solution to be discarded, that probably should say something about the currently popular kind of solution. Especially if there have been many proposed kinds of solutions which have been discarded. But, it seems like much of what it says is just that the problem is hard. And, sure, that may mean an increased probability that the currently popular proposed kind of solution also doesn’t end up being satisfactory, that doesn’t mean one should be too quick to discard it. Tautologically: if no known alternative is currently at least as promising as the type of solution currently being considered, then, the current one is the most promising of the currently known options. Whether it is promising enough to actively pursue may be a different question, but it shouldn’t be marked as discarded until something else (perhaps something previously discarded, or something novel) becomes more promising.

airstrike 5y ago

From Wikipedia (https://en.wikipedia.org/wiki/Cyc#Criticisms https://en.wikipedia.org/wiki/Cyc#Criticisms) > ... A similar sentiment was expressed by Marvin Minsky: "Unfortunately, the strategies most popular among AI researchers in the 1980s have come to a dead end," said Minsky. So-called “expert systems,” which emulated human expertise within tightly defined subject areas like law and medicine, could match users’ queries to relevant diagnoses, papers and abstracts, yet they could not learn concepts that most children know by the time they are 3 years old. “For each different kind of problem,” said Minsky, “the construction of expert systems had to start all over again, because they didn’t accumulate common-sense knowledge.” Only one researcher has committed himself to the colossal task of building a comprehensive common-sense reasoning system, according to Minsky. Douglas Lenat, through his Cyc project, has directed the line-by-line entry of more than 1 million rules into a commonsense knowledge base."

ypcx 5y ago
And then, GPT-3 came along and rendered Cyc a wasted effort.
- goatlover 5y ago
  Cyc was trying to encode common knowledge about the world in a bunch of rules. That goes well beyond what GPT-3 does with text.
  - ypcx 5y ago
    GPT-3 learns these rules by itself.
    - johnthescott 5y ago
      can you automate your fate with gpt-3? that is where the money lives. will sublime conclusions made by GPT-3 EVER be trusted if the reasoning to its conclusion is not understood by a human? perhaps the gestalt of gpt3 implies a meta gpt3 that could derive human grokkable explains of its "dumber" self. or maybe not.

williamtrask 5y ago

From what I can tell, many of the best thinkers agree with this idea but we haven’t cracked it yet.

mr_toad 5y ago

I doubt that gains from symbolic AI or any algorithmic improvement will outpace Moore’s law in the long run. I certainly wouldn’t want to spend decades working to improve AI, only to have my work undercut by cheaper silicon. http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html

singularity2001 5y ago

this is misguided since all networks represent symbolic sense in an abstract way