8 ms·
I would argue that Lenat was at least directionally correct in understanding that sheer volume of data (in Cyc's case, rules and facts) was the key in eventuall
by vannevar 2y ago
I would argue that Lenat was at least directionally correct in understanding that sheer volume of data (in Cyc's case, rules and facts) was the key in eventually achieving useful intelligence. I have to confess that I once criticized the Cyc project for creating an ever-larger pile of sh*t and expecting a pony to emerge, but that's sort of what has happened with LLMs.
- cmrdporcupine 2y agoI suspect at some point the pendulum will again swing back the other way and symbolic approaches will have some kind of breakthrough and become trendy again. And, I bet it will likely have something to do with accelerating these systems with hardware, much like GPUs have done for neural networks, in order to crunch really large quantities of facts
- whiplash451 2y agoOr maybe program synthesis combined by LLMs might be the way?
- cmrdporcupine 2y agoIt does seem like the Cyc people hit the wall with simply collecting facts. Having to have a human in the loop. The problem I think is if you have LLMs figuring out the propositions, the whole system is just as prone to garbage-in-garbage-out as LLMs are.
- luma 2y agoThe Bitter Lesson has a few things to say about this. http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- deleted 2y ago[deleted]
- wzdd 2y agoThe Bitter Lesson says "general methods that leverage computation are ultimately the most effective". That doesn't seem to rule out symbolic approaches. It does rule out anything which relies on having humans in the loop, because terabytes of data plus a dumb learning process works better than megabytes of data plus expert instruction. (I know your message wasn't claiming that The Bitter Lesson was explicitly a counterpoint, I just thought it was interesting.)
- bcoates 2y agoImho, this is wrong. Even independent of access to vast amounts of compute, symbolic methods seem to consistently underperform statistical/numerical ones across a wide variety of domains. I can't help but think that there's more to it than just brute force.
- YeGoblynQueenne 2y agoI've lost count how many times I've written the same words in this thread but: SAT Solving, Automated Theorem Proving, Program Verification and Model Checking, Planning and Scheduling. These are not domains where symbolic methods "consistently underperform" anything. You guys really need to look into what's been going on in classical AI in the last 20-30 years. There are two large conferences that are mainly about symbolic AI, IJCAI and AAAI. Then there's all the individual conferences on the above sub-fields, like the International Conference on Automated Planning and Scheduling (ICAPS). Don't expect to hear about symbolic AI on social media or press releases from Alpha and Meta, but there's plenty of material online if you're interested.
- kevin_thibedeau 2y agoReal AGI will need a way to reason about factual knowledge. An ontology is a useful framework for establishing facts without inferring them from messy human language.
- deleted 2y ago[deleted]
- IshKebab 2y agoThese guys are trying to combine symbolic reasoning with LLMs somehow: https://www.symbolica.ai/ https://www.symbolica.ai/
- specialgoodness 2y agocheck out Imandra's platform for neurosymbolic AI - https://www.imandra.ai/ https://www.imandra.ai/
- deleted 2y ago[deleted]
- chubot 2y agoThat’s hilarious, but at least Llama was trained on libgen, an archive of most books and publications by humanity, no? Except for the ones which were not digitized I guess So there is probably a big pile of Reddit comments, twitter messages, and libgen and arxiv PDFs I imagine So there is some shit, but also painstakingly encoded knowledge (ie writing), and yeah it is miraculous that LLMs are right as often as they are
- ChadNauseam 2y agoIt's a miracle, but it's all thanks to the post-training. When you think of it, for so-called "next token predictors", LLMs talk in a way that almost no one actually talks, with perfect spelling and use of punctuation. The post-training somehow is able to get them to predict something along the lines of what a reasonably intelligent assistant with perfect grammar would say. LLMs are probably smarter than is exposed through their chat interface, since it's unlikely the post-training process is able to get them to impersonate the smartest character they'd be capable of impersonating.
- chubot 2y agoI dunno I actually think say Claude AI SOUNDS smarter than it is, right now It has a phenomenal recall. I just asked it about "SmartOS", something I knew about, vaguely, in ~2012, and it gave me a pretty darn good answer. On that particular subject, I think it probably gave a better answer than anyone I could e-mail, call, or text right now It was significantly more informative than wikipedia - https://en.wikipedia.org/wiki/SmartOS https://en.wikipedia.org/wiki/SmartOS But I still find it easy to stump it and get it to hallucinate, which makes it seem dumb It is like a person with good manners, and a lot of memory, and which is quite good at comparisons (although you have to verify, which is usually fine) But I would not say it is "smart" at coming up with new ideas or anything I do think a key point is that a "text calculator" is doing a lot of work ... i.e. summarization and comparison are extremely useful things. They can accelerate thinking
- cratermoon 2y agolibgen is far from an archive of "most" books and publications, not even close. The most recent numbers from libgen itself are 2.4 million non-fiction books and 80 million science journal articles. The Atlantic's database published in 2025 has 7.5 million books.[0] The publishing industry estimates that many books are published each year. As of 2010, Google counted over 129 million books[1]. At best an LLM like Llama will have have 20% of all books in its training set. 0. https://www.theatlantic.com/technology/archive/2025/03/libgen-meta-openai/682093/ https://www.theatlantic.com/technology/archive/2025/03/libge... 1. https://booksearch.blogspot.com/2010/08/books-of-world-stand-up-and-be-counted.html https://booksearch.blogspot.com/2010/08/books-of-world-stand...
- baq 2y agohttps://ai-2027.com/ https://ai-2027.com/ postulates that a good enough LLM will rewrite itself using rules and facts... sci-fi, but so is chatting with a matrix multiplication.
- josephg 2y agoI doubt it. The human mind is a probabilistic computer, at every level. There’s no set definition for what a chair is. It’s fuzzy. Some things are obviously in the category, and some are at the periphery of it. (Eg is a stool a chair? Is a log next to a campfire a chair? How about a tree stump in the woods? Etc). This kind of fuzzy reasoning is the rule, not the exception when it comes to human intuition. There’s no way to use “rules and facts” to express concepts like “chair” or “grass”, or “face” or “justice” or really anything. Any project trying to use deterministic symbolic logic to represent the world fundamentally misunderstands cognition.
- veqq 2y agoSo you're just ignoring all the probabilistic, fuzzy etc. Prologs etc. which do precisely that? https://github.com/lab-v2/pyreason https://github.com/lab-v2/pyreason
- jgalt212 2y ago> The human mind is a probabilistic computer, at every level. Fair enough, but an airplane's wing is not very similar to a bird's wing.
- josephg 2y agoThat argument would hold a lot more weight if Cyc could fly. But as this article points out, decades of work and millions of dollars have utterly failed to get it off the ground.
- jgalt212 2y agoright, but as others have pointed out the amount of money invested in Cyc is approx 2 orders of magnitude less than what was invested in LLMs. So maybe the method was OK, but it was insufficiently resourced.
- TechDebtDevin 2y agoThe dataset for lots of LLMs is literally called "The Pile" lmao
- deleted 2y ago[deleted]
- jibal 2y agoBut a) The pile of LLM training data is vastly larger. b) The data is actual human utterances in situ--these are ponies, not pony shit. c) LLMs have no intelligence ... they channel the intelligence of a vast number of humans by pattern matching their utterances to a query. This has indeed proved useful because of how extremely well the statistical apparatus works, but the fact that LLMs have no cognitive states puts great limits on what this technology can achieve. With Cyc, OTOH, it's not even clear what you can get out of it. The thing may well be useful if combined with LLMs, but it's under lock and key. The big conclusions about symbolic AI that the author reaches based on this one system and approach are unwarranted. As he himself notes, "Even Ernest Davis and Gary Marcus, highly sympathetic to the symbolic approach to AI, found little evidence for the success of Cyc, not because Cyc had provably failed, but simply because there was too little evidence in any direction, success or failure."
- YeGoblynQueenne 2y ago>> they channel the intelligence of a vast number of humans by pattern matching their utterances to a query. Just a little problem with that: to understand the utterances of a vast number of humans you need to channel it to something that can understand the utterances of humans in the first place. Just channeling it around from statistic to statistic doesn't do the trick.
- jibal 2y agoUm, the "something" is the person reading the LLM's output. I'm afraid that you have completely missed the context and point of the discussion, which was not about LLMs understanding things--they understand nothing ("LLMs have no cognitive states"). But again, "because of how extremely well the statistical apparatus works", their outputs are useful to intelligent consumers who do have cognitive states--us.