3 ms·

A very interesting and worthwhile article (better than the comments here would suggest). However, I find it a bit of a pity that the author places so much empha

by Rochus 2y ago

A very interesting and worthwhile article (better than the comments here would suggest). However, I find it a bit of a pity that the author places so much emphasis on the assumption that the project has failed. The approach has not simply failed because the search for a solution has been going on for forty years. It took even more than forty years and costed at least as much before neural networks became really useful, and nobody would claim that the project failed because of that. And today's LLM are not really that intelligent yet. Maybe Cycorp's knowledge base will be made generally accessible at some point, so that it can be used to train LLMs. Perhaps then a greater benefit of this data will become apparent.

smoyer 2y ago
> on the assumption the project has failed. My daughter's PhD thesis was largely negative results. Even if the project had failed, we could learn from it if it wasn't so secretive. It could be much more open without being OSS!
- Rochus 2y ago
  It is at least as important to know which approaches do not work, but this gets significantly less press, which is not that attractive for scientists in the age of "publish or perish".
- YeGoblynQueenne 2y ago
  >> My daughter's PhD thesis was largely negative results. Well, give us a link man! :)
  - smoyer 2y ago
    You're forcing the proud dad function: https://pubmed.ncbi.nlm.nih.gov/36995257/ https://pubmed.ncbi.nlm.nih.gov/36995257/
    - YeGoblynQueenne 2y ago
      Absolutely. The proud nerd dad function :D Congratulations to your daughter for her PhD. I'm guessing she has got it by now. Sonic hedgehog signalling pathway! And what a date to submit a thesis. Why is that a negative result, btw?
zozbot234 2y ago
> Maybe Cycorp's knowledge base will be made generally accessible at some point, so that it can be used to train LLMs. More likely, it will be made increasingly irrelevant as open alternatives to it are developed instead. The Wikipedia folks are working on some sort of openly developed interlingua that can be edited by humans, in order to populate Wikipedias in underrepresented languages with basic encyclopedic text. (Details very much TBD, but see https://en.wikipedia.org/wiki/Abstract_Wikipedia https://en.wikipedia.org/wiki/Abstract_Wikipedia and https://meta.wikimedia.org/wiki/Abstract_Wikipedia https://meta.wikimedia.org/wiki/Abstract_Wikipedia ) This will probably be roughly as powerful as the system OP posits at some point in the article, that can generate text in both English and Japanese but only if fed with the right "common sense" to begin with. It's not clear exactly how useful logical inference on such statements might turn out to be, but the potential will definitely exist for something like that too, if it's found to be genuinely worthwhile in some way.
- Rochus 2y ago
  > made increasingly irrelevant as open alternatives to it are developed instead Certainly interesting what these projects are going for, but it's unlikely an "open alternative", given that the degree of formalization and rigor achieved by Cyc's higher-order logic specification is likely not achievable by statistical learning, and a symbolic approach is barely achievable in a shorter time than Cyc.
- yellowapple 2y ago
  https://www.wikidata.org/wiki/Wikidata:Main_Page https://www.wikidata.org/wiki/Wikidata:Main_Page, for those curious about the interlingua in question.
  - zozbot234 2y ago
    Strictly speaking, Wikidata is an existing project which only provides a rather restrictive model for its assertions; they are not fully compositional, thus are quite far from being able to express general encyclopedic text, especially in a way that can be 'seamlessly' translated to natural language. It does provide a likely foundation for these further planned developments, though.
Paracompact 2y ago
> Maybe Cycorp's knowledge base will be made generally accessible at some point I would sooner hold my breath waiting for OpenAI to open up than Cycorp :) > It took even more than forty years and costed at least as much before neural networks became really useful The correct class of comparison to make with "neural networks" would be "symbolic AI" writ large. Symbolic AIs have been working quite well in some areas. Just not at all in terms of common sense reasoning, or anything approaching AGI. If you want to keep "Cyc" in the comparison, then I would argue there is no comparison possible. Without exaggeration, there has never been a single project in AI as expensive as Cyc before 2020. Only with GPT-2 did the cost start to exceed the million USD mark. (Without exact figures, AlphaGo and Deep Blue probably also cost millions of dollars, but they unambiguously worked.) It's also just not true that it took 40 years. Consider e.g. LeNet-5, which was up and running in 1998, and was used in ATMs to read real cheques. The main cost was 10 years of engineering stamina by LeCun's research group at Bell Labs. The finished version could be trained "for about 20 epoches over MNIST. It took 2 to 3 days of CPU time on a Silicon Graphics Origin 2000 server, using a single 200 MHz R10000 processor." (1998 might technically be 40 years out from e.g. the inception of the perceptron in the 1950s, but if that is supposed to be our reference point for neural networks, then Cyc's reference point should be the inception of logical AIs in the same decade. And really, what use was Cyc in industry in 1998?)

by Rochus 2y ago

smoyer 2y ago

> on the assumption the project has failed. My daughter's PhD thesis was largely negative results. Even if the project had failed, we could learn from it if it wasn't so secretive. It could be much more open without being OSS!

Rochus 2y ago
It is at least as important to know which approaches do not work, but this gets significantly less press, which is not that attractive for scientists in the age of "publish or perish".
YeGoblynQueenne 2y ago
>> My daughter's PhD thesis was largely negative results. Well, give us a link man! :)
- smoyer 2y ago
  You're forcing the proud dad function: https://pubmed.ncbi.nlm.nih.gov/36995257/ https://pubmed.ncbi.nlm.nih.gov/36995257/
  - YeGoblynQueenne 2y ago
    Absolutely. The proud nerd dad function :D Congratulations to your daughter for her PhD. I'm guessing she has got it by now. Sonic hedgehog signalling pathway! And what a date to submit a thesis. Why is that a negative result, btw?

zozbot234 2y ago

> Maybe Cycorp's knowledge base will be made generally accessible at some point, so that it can be used to train LLMs. More likely, it will be made increasingly irrelevant as open alternatives to it are developed instead. The Wikipedia folks are working on some sort of openly developed interlingua that can be edited by humans, in order to populate Wikipedias in underrepresented languages with basic encyclopedic text. (Details very much TBD, but see https://en.wikipedia.org/wiki/Abstract_Wikipedia https://en.wikipedia.org/wiki/Abstract_Wikipedia and https://meta.wikimedia.org/wiki/Abstract_Wikipedia https://meta.wikimedia.org/wiki/Abstract_Wikipedia ) This will probably be roughly as powerful as the system OP posits at some point in the article, that can generate text in both English and Japanese but only if fed with the right "common sense" to begin with. It's not clear exactly how useful logical inference on such statements might turn out to be, but the potential will definitely exist for something like that too, if it's found to be genuinely worthwhile in some way.

Rochus 2y ago
> made increasingly irrelevant as open alternatives to it are developed instead Certainly interesting what these projects are going for, but it's unlikely an "open alternative", given that the degree of formalization and rigor achieved by Cyc's higher-order logic specification is likely not achievable by statistical learning, and a symbolic approach is barely achievable in a shorter time than Cyc.
yellowapple 2y ago
https://www.wikidata.org/wiki/Wikidata:Main_Page https://www.wikidata.org/wiki/Wikidata:Main_Page, for those curious about the interlingua in question.
- zozbot234 2y ago
  Strictly speaking, Wikidata is an existing project which only provides a rather restrictive model for its assertions; they are not fully compositional, thus are quite far from being able to express general encyclopedic text, especially in a way that can be 'seamlessly' translated to natural language. It does provide a likely foundation for these further planned developments, though.

Paracompact 2y ago

> Maybe Cycorp's knowledge base will be made generally accessible at some point I would sooner hold my breath waiting for OpenAI to open up than Cycorp :) > It took even more than forty years and costed at least as much before neural networks became really useful The correct class of comparison to make with "neural networks" would be "symbolic AI" writ large. Symbolic AIs have been working quite well in some areas. Just not at all in terms of common sense reasoning, or anything approaching AGI. If you want to keep "Cyc" in the comparison, then I would argue there is no comparison possible. Without exaggeration, there has never been a single project in AI as expensive as Cyc before 2020. Only with GPT-2 did the cost start to exceed the million USD mark. (Without exact figures, AlphaGo and Deep Blue probably also cost millions of dollars, but they unambiguously worked.) It's also just not true that it took 40 years. Consider e.g. LeNet-5, which was up and running in 1998, and was used in ATMs to read real cheques. The main cost was 10 years of engineering stamina by LeCun's research group at Bell Labs. The finished version could be trained "for about 20 epoches over MNIST. It took 2 to 3 days of CPU time on a Silicon Graphics Origin 2000 server, using a single 200 MHz R10000 processor." (1998 might technically be 40 years out from e.g. the inception of the perceptron in the 1950s, but if that is supposed to be our reference point for neural networks, then Cyc's reference point should be the inception of logical AIs in the same decade. And really, what use was Cyc in industry in 1998?)