4 ms·

I'll keep shilling the idea that someone should find a way to glue LLMs and prolog together to create better reasoning agents. I say this as someone who cant wr

I'll keep shilling the idea that someone should find a way to glue LLMs and prolog together to create better reasoning agents. I say this as someone who cant write a single prolog program, but i get the gist of it and afaik it hasn't been done yet. The idea is based on the following simple logic: LLMs are bad at counting the number of "r's" in "strawberry" but good at writing code that counts letters in a string. It follows that while LLMs are also bad at solving reasoning problems, maybe they're good at writing the prolog code (or driving a hypothetical prolog engine) that solves those types of problems instead. Maybe one day ill stop being lazy and do it myself, but until then I'm putting the idea out there - to OP's request, that's one reason to learn prolog.

johnecheck 1y ago
This idea feels obvious enough that people must be trying it. It's probably just hard to train such a model that performs as you'd want it to.
dmd 1y ago
http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- sirwhinesalot 1y ago
  This is a very important article but not one that applies to what the person is saying. They're not suggesting hardcoding knowledge into the system (i.e. through a predefined prolog program) but by having the LLM learn to use it as a tool, same as how an LLM will write and run Python instead of guessing the output of a program. I don't think Prolog is the right tool for this, actually for the very reason stated in the Bitter Lesson (effective Prolog use requires imperative constructs like cuts). It makes much more sense to have the LLM learn how to use a constraint solver or (automated) theorem prover which have really powerful search algorithms (the search part of the bitter lesson), same as how humans use them.
  - disambiguation 1y ago
    Good take, yeah my naive idea of prolog is as a kind of "solver", so I suppose the better the solver the better implementation.
conception 1y ago
Would be pretty easy to write an mcp that interfaces with prolog apps. Probably an hour or two of vibe coding. ;)
- disambiguation 1y ago
  Eh I don't think I would commit to vibe coding something I don't understand. I don't know enough about prolog to know if the idea is truly plausible - or as another commenter suggested, less than ideal compared to other solvers. And I don't know enough to steer it in the right direction either. I'd rather sit down and learn it the right way first.
tannhaeuser 1y ago
Indeed, generating Prolog for robotic and other discrete planning has been pursued. From [1]: > Rather than having LLMs perform planning itself, it's thus only natural to complement LLMs with the raw combinatorial power of Prolog, much like a human would be reaching out to a pocket calculator. [1]: https://quantumprolog.sgml.net/llm-demo/part1.html https://quantumprolog.sgml.net/llm-demo/part1.html
- disambiguation 1y ago
  seems interesting, though i'm not sure I fully understand - its mainly a demo where LLMs generate prolog code for this specific port problem? I was imagining my idea on a larger scale, 1. using prolog to improve general "reasoning" benchmarks (for ex. these kinds of problems https://www.llm-reasoning-benchmark.com/ https://www.llm-reasoning-benchmark.com/) and 2. using prolog in the context of an agent, as a way to model its environment and make better plans towards goals.
  - tannhaeuser 1y ago
    It's a guide for prompting LLMs to generate Prolog for any problem that can be solved by a STRIPS-like formulation with state and robotic actions like the container packing and container shipping problems (and a few artificial ones from AI competitions). A main predicate encoding a goal selection strategy is omitted in generation and a generic one is supplied instead. The point is using language models to translate text descriptions into Prolog knowledge and state change representations guided by in-context examples, which was found to work really well in practice, and is also quite natural given Prolog's NLP roots.
    - disambiguation 1y ago
      Very interesting, TIL about STRIPS - but if this works as well as claimed then without knowing more it seems too good to be true, and further surprising that it isn't more widely adopted. I'm curious if you have plans for this site and code, or was it your own PoC? Like I mentioned I'm curious how it would hold up to more standard benchmarks. When I finally start my own prolog journey I'll remember to revisit your site so I can fully appreciate it, nonetheless thanks for sharing and explaining.

johnecheck 1y ago

This idea feels obvious enough that people must be trying it. It's probably just hard to train such a model that performs as you'd want it to.

dmd 1y ago

http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html

sirwhinesalot 1y ago
This is a very important article but not one that applies to what the person is saying. They're not suggesting hardcoding knowledge into the system (i.e. through a predefined prolog program) but by having the LLM learn to use it as a tool, same as how an LLM will write and run Python instead of guessing the output of a program. I don't think Prolog is the right tool for this, actually for the very reason stated in the Bitter Lesson (effective Prolog use requires imperative constructs like cuts). It makes much more sense to have the LLM learn how to use a constraint solver or (automated) theorem prover which have really powerful search algorithms (the search part of the bitter lesson), same as how humans use them.
- disambiguation 1y ago
  Good take, yeah my naive idea of prolog is as a kind of "solver", so I suppose the better the solver the better implementation.

conception 1y ago

Would be pretty easy to write an mcp that interfaces with prolog apps. Probably an hour or two of vibe coding. ;)

disambiguation 1y ago
Eh I don't think I would commit to vibe coding something I don't understand. I don't know enough about prolog to know if the idea is truly plausible - or as another commenter suggested, less than ideal compared to other solvers. And I don't know enough to steer it in the right direction either. I'd rather sit down and learn it the right way first.

tannhaeuser 1y ago

Indeed, generating Prolog for robotic and other discrete planning has been pursued. From [1]: > Rather than having LLMs perform planning itself, it's thus only natural to complement LLMs with the raw combinatorial power of Prolog, much like a human would be reaching out to a pocket calculator. [1]: https://quantumprolog.sgml.net/llm-demo/part1.html https://quantumprolog.sgml.net/llm-demo/part1.html

disambiguation 1y ago
seems interesting, though i'm not sure I fully understand - its mainly a demo where LLMs generate prolog code for this specific port problem? I was imagining my idea on a larger scale, 1. using prolog to improve general "reasoning" benchmarks (for ex. these kinds of problems https://www.llm-reasoning-benchmark.com/ https://www.llm-reasoning-benchmark.com/) and 2. using prolog in the context of an agent, as a way to model its environment and make better plans towards goals.
- tannhaeuser 1y ago
  It's a guide for prompting LLMs to generate Prolog for any problem that can be solved by a STRIPS-like formulation with state and robotic actions like the container packing and container shipping problems (and a few artificial ones from AI competitions). A main predicate encoding a goal selection strategy is omitted in generation and a generic one is supplied instead. The point is using language models to translate text descriptions into Prolog knowledge and state change representations guided by in-context examples, which was found to work really well in practice, and is also quite natural given Prolog's NLP roots.
  - disambiguation 1y ago
    Very interesting, TIL about STRIPS - but if this works as well as claimed then without knowing more it seems too good to be true, and further surprising that it isn't more widely adopted. I'm curious if you have plans for this site and code, or was it your own PoC? Like I mentioned I'm curious how it would hold up to more standard benchmarks. When I finally start my own prolog journey I'll remember to revisit your site so I can fully appreciate it, nonetheless thanks for sharing and explaining.