14 ms·

Use Prolog to improve LLM's reasoning

379 points by shchegrikovich 2y ago

pjmlp 2y ago
So we are back to Japanese Fifth Generation plan from 1980's. :)
- thelastparadise 2y ago
  Watson did it too, a while back.
- tokinonagare 2y ago
  Missing some LISP but yeah it's funny how old things are new again (same story with wasm, RISC archs, etc.)
  - nxobject 2y ago
    Lots of GOFAI being implemented again – decision trees, goal searching and planning, agent-based strategies... just not symbolic representations, and that might be the key. I figure you might get an interesting contribution out of skimming old AI laboratory publications and seeing whether you could find a way of implementing it through a single LLM, multiple LLM agents, methods of training, etc.
    - anthk 2y ago
      https://en.m.wikipedia.org/wiki/Constraint_satisfaction_problem https://en.m.wikipedia.org/wiki/Constraint_satisfaction_prob...
    - nmadden 2y ago
      Indeed, modern ML has been a validation of (some of) GOFAI: https://neilmadden.blog/2024/06/30/machine-learning-and-the-triumph-of-gofai/ https://neilmadden.blog/2024/06/30/machine-learning-and-the-...
- linguae 2y ago
  This time around we have all sorts of parallel processing capabilities in the form of GPUs. If I recall correctly, the Fifth Generation project envisioned highly parallel machines performing symbolic AI. From a hardware standpoint, those researchers were way ahead of their time.
  - nxobject 2y ago
    And they had a self-sustaining video game industry too... if only someone had had the wild thought of implementing perceptrons and tensor arithmetic on the same hardware!
- postepowanieadm 2y ago
  and winter is coming.
- metadat 2y ago
  For the uninitiated (like me): The Japanese Fifth Generation Project https://www.sjsu.edu/faculty/watkins/5thgen.htm https://www.sjsu.edu/faculty/watkins/5thgen.htm
sgt101 2y ago
Building on this idea people have grounded LLM generated reasoning logic with perceptual information from other networks : https://web.stanford.edu/~joycj/projects/left_neurips_2023 https://web.stanford.edu/~joycj/projects/left_neurips_2023
a1j9o94 2y ago
I tried an experiment with this using a Prolog interpreter with GPT-4 to try to answer complex logic questions. I found that it was really difficult because the model didn't seem to know Prolog well enough to write a description of any complexity. It seems like you used an interpreter in the loop which is likely to help. I'd also be interested to see how o1 would do in a task like this or if it even makes sense to use something like prolog if the models can backtrack during the "thinking" phase
- lukasb 2y ago
  I bet one person could probably build a pretty good synthetic NL->Prolog dataset. ROI for paying that person would be high if you were building a foundation model (ie benefits beyond being able to output Prolog.)
  - mcswell 2y ago
    I'm not exactly sure what you're referring to, but Fernando Pereira's dissertation included a natural language (English) program for querying a "database". Both the NLP part and the database were written in Prolog. Mid-1980s, I think. Of course both parts were "toy" in the sense that they would need to be hugely expanded to be of real world use, but they did handle some interesting things (like quantifiers, graded adjectives etc.).
- hendler 2y ago
  I also wrote wrote an LLM to Prolog interpreter for a hackathon called "Logical". With a few hours effort I'm sure it could be improved. https://github.com/Hendler/logical https://github.com/Hendler/logical I think while LLMs may approach completeness here, it's good to have an interpretable system to audit/verify and reproduce results.
  - shchegrikovich 2y ago
    This is really cool!
    - 2y ago
baq 2y ago
Patiently waiting for z3-guided generation, but this is a welcome, if obvious, development. Results are a bit surprising and sound too optimistic, though.
nonamepcbrand1 2y ago
This is why GitHub CodeQL and Co-Pilot assistance is working better for everyone? basically codeql uses variant of Prolog (datalog) to query source code to generate better results.
z5h 2y ago
i've come to appreciate, over the past 2 years of heavy Prolog use, that all coding should be (eventually) be done in Prolog. It's one of few languages that is simultaneously a standalone logical formalism, and a standalone representation of computation. (With caveats and exceptions, I know). So a Prolog program can stand in as a document of all facts, rules and relations that a person/organization understands/declares to be true. Even if AI writes code for us, we should expect to have it presented and manipulated as a logical formalism. Now if someone cares to argue that some other language/compiler is better at generating more performant code on certain architectures, then that person can declare their arguments in a logical formalism (Prolog) and we can use Prolog to translate between language representations, compile, optimize, etc.
- dmead 2y ago
  It's taken ages for anything from functional programming to penetrate general use. Do you think uptake of logic stuff will be any faster?
  - johnnyjeans 2y ago
    Prolog (and logic programming in general) is much older than you think. In fact, if we take modern functional programming to have been born with John Backus' Turing Award presentation[1], then it even predates it. Many advancements to functional programming were implemented on top of Prolog! Erlang's early versions were built on top of a Prolog-derived language who's name escapes me. It's the source of Erlang's unfamiliar syntax for more unlearned programmers. It's very much like writing Prolog if you had return values and no cuts or complex terms. As for penetrating general use, probably not without a major shift in the industry. But it's a very popular language just on the periphery, even to this day. [1] - https://dl.acm.org/doi/10.1145/359576.359579 https://dl.acm.org/doi/10.1145/359576.359579
    - dmead 2y ago
      Did you just answer me with chatgpt?
anthk 2y ago
Use Constraint Satisfaction Problem Solvers. It commes up with Common Lisp with ease.
mise_en_place 2y ago
I really enjoyed tinkering with languages like Prolog and Coq. Interactive theorem proving with LLMs would be awesome to try out, if possible.
fsndz 2y ago
This is basically the LLM modulo approach recommended by Prof. Subbarao Kambhampati. Interesting but only works mostly for problems that have some math/first degree logic puzzle at their heart. Will fail at improving perf at ARC-AGI for example... Difficult to mimic reasoning by basic trial and error then hoping for the best: https://www.lycee.ai/blog/why-sam-altman-is-wrong https://www.lycee.ai/blog/why-sam-altman-is-wrong
YeGoblynQueenne 2y ago
That's not going to work. Garbage in - Garbage out is success-set equivalent to Garbage in - Prolog out. Garbage is garbage and failure to reason is failure to reason no matter the language. If your LLM can't translate your problem to a Prolog program that solves your problem- Prolog can't solve your problem.
- Philpax 2y ago
  This is a shallow critique that does not engage with the core idea. Specifying the problem is not the same as solving the problem.
  - YeGoblynQueenne 2y ago
    I've programmed in Prolog for ~13 years and my PhD thesis is in machine learning of Prolog programs. How deep would you like me to go?
    - Philpax 2y ago
      As deep as is required to actually make your argument!
      - YeGoblynQueenne 2y ago
        You'll have to be more specific than that. For me what I point out is obvious: Prolog is not magick. Your program won't magickally reason if you write it in Prolog, much less reason correctly. If an LLM translates a Problem to the wrong Prolog program, Prolog won't magickally turn it into a correct program. And that's just rephrasing what I said in my comment above. There's really not much more to say. Here's just one more observation: the problems where translating reasoning to Prolog will work best are problems where there are a lot of examples of Prolog to be found on the web, e.g. wolf-cabbage-goat problems and the like. With problems like that it is much easier for an LLM to generate a correct translation of the problem to Prolog and get a correct solution just because there's lots of examples. But if you choose a problem that's rarely attacked with Prolog code, like, I don't know, some mathematical problem that obtains in nuclear physics as a for instance, then an LLM will be much more likely to generate garbage Prolog, while e.g. Fortran would be a better target language. From what I can see, the papers linked in the article above concentrate on the Prolog-friendly kind of problem, like logical puzzles and the like. That smells like cherry picking to me, or just good, old confirmation bias. Again, Prolog is not magick. The article above and the papers it links to seem to take this attitude of "just add Prolog" and that will make LLMs suddenly magickally reason with fairy dust on top. Ain't gonna happen.
arjun_khamkar 2y ago
Would Creating a prolog dataset would be beneficial, so that future LLM's can be trained on it and then they would be able to output prolog code.
- shchegrikovich 2y ago
  I discussed this a few weeks back. The idea is to take a Python dataset, as Python is the most popular language, and write a transpiler to Prolog with the help of llms. So, creating this synthetic dataset is not a huge problem.
DeborahWrites 2y ago
You're telling me the seemingly arbitrary 6 weeks of Prolog on my comp sci course 11yrs ago is suddenly about to be relevant? I did not see this one coming . . .
- fullstackwife 2y ago
  Is there any need to look at this generated Prolog code?
- aitchnyu 2y ago
  I rolled my eyes when iterating a list meant splitting it into first and rest and recursing on the rest. I would do side projects in Scala a few years later of my own interest.
_yb2s 2y ago
I think this general idea is going to be the key to really making LLMs widely useful for solving real problems. I’ve been playing with using GPT-4 together with the Wolfram Alpha plugin, and the combo of the two can reliably solve difficult quantitative problems that neither can individually by working together, much like a human using a calculator.
de6u99er 2y ago
I always thought that Prolog is great for reasoning in the semantic web. It doesn't surprise me that LLM people stumble on it.
bytebach 2y ago
An application I am developing for a customer needed to read constraints around clinical trials and essentially build a query from them. Constraints involve prior treatments, biomarkers, type of disease (cancers) etc. Using just an LLM did not produce reliable queries, despite trying many many prompts, so being an old Prolog hacker I wondered if using it might impose more 'logic' on the LLM. So we precede the textual description of the constraints with the following prompt: ------------- Now consider the following Prolog predicates: biomarker(Name, Status) where Status will be one of the following integers - Wildtype = 0 Mutated = 1 Methylated = 2 Unmethylated = 3 Amplified = 4 Deleted = 5 Positive = 6 Negative = 7 tumor(Name, Status) where Status will be one of the following integers if know else left unbound - Newly diagnosed = 1 Recurrence = 2 Metastasized = 3 Progression = 4 chemo(Name) surgery(Name) Where Name may be an unbound variable other_treatment(Name) radiation(Name) Where Name may be an unbound variable Assume you are given predicate atMost(T, N) where T is a compound term and N is an integer. It will return true if the number of 'occurences' of T is less than or equal N else it will fail. Assume you are given a predicate atLeastOneOf(L) where L is a list of compound terms. It will succeed if at least one of the compound terms, when executed as a predicate returns true. Assume you are given a predicate age(Min, Max) which will return true if the patient's age is in between Min and Max. Assume you have a predicate not(T) which returns true if predicate T evaluates false and vice versa. i.e. rather than '\\+ A' use not(A). Do not implement the above helper functions. VERY IMPORTANT: Use 'atLeastOneOf()' whenever you would otherwise use ';' to represent 'OR'. i.e. rather than 'A ; B' use atLeastOneOf([A, B]). EXAMPLE INPUT: Patient must have recurrent GBM, methylated MGMT and wildtype EGFR. Patient must not have mutated KRAS. EXAMPLE OUTPUT: tumor('gbm', 2), biomarker('MGMT', 2), biomarker('EGFR', 0), not(biomarker('KRAS', 1)) ------------------ The Prolog predicates, when evaluated generate the required underlying query (of course the Prolog is itself a form of query). Anyway - the upshot was a vast improvement in the accuracy of the generated query (I've yet to see a bad one). Somewhere in its bowels, being told to generate Prolog 'focused' the LLM. Perhaps LLMs are happier with declarative languages rather than imperative ones (I know I am :) ).
- int_19h 2y ago
  I find that having GPT-4 write SQL queries to query the data source as needed to solve a complex task step-by-step also works pretty well (and you can give it the schema in form of CREATE TABLE). It's not exactly good at writing fast queries, but it can do some hella complex ones with nesting and joins to get exactly what it needs in one go.
gorkempacaci 2y ago
The generated programs are only technically Prolog programs. They use CLPFD, which makes these constraint programs. Prolog programs are quite a bit more tricky with termination issues. I wouldn’t have nitpicked if it wasn’t in the title. Also, the experiment method has some flaws. Problems are hand-picked out of a random subset of the full set. Why not run the full set?
- bbor 2y ago
  Yeah I’m a huge proponent of this general philosophy, but after being introduced to prolog itself for a third of a semester back in undergrad I decided to stay far, far away. The vision never quite came through as clearly as it did for the other wacky languages, namely the functional family (Lisp and Haskell in my case). I believe you on the fundamental termination issues, but just basic phrasing seemed unnecessarily convoluted… Since you seem like an expert: is there a better technology for logical/constraint programming? I loved predicate calculus in school so it seems like there should be something out there for me, but so far no dice. This seems kinda related to the widely-discussed paradigm of “Linear Programming”, but I’ve also failed to find much of interest there behind all the talk of “Management Theory” and detailed mathematical efficiency comparisons. I guess Curry (from above) might be the go-to these days?
  - gorkempacaci 2y ago
    Curious to know what part of syntax you found convoluted. If you remember any examples I’d appreciate it. Maybe you want a constraint programming environment instead. As example check out Conjure from St Andrews: https://conjure.readthedocs.io/en/latest/tutorials-notebook.html https://conjure.readthedocs.io/en/latest/tutorials-notebook.... More generally there are the theorem provers like Coq, etc., but their use cases are even more specific.
- YeGoblynQueenne 2y ago
  >> Why not run the full set? Most likely cherry-picking. The approach is only going to work well in domains where Prolog is commonly used to write solutions to problems, like logical puzzles or constraint problems etc.
ianbicking 2y ago
I made a pipeline using Z3 (another prover language) to get LLMs to solve very specific puzzle problems: https://youtu.be/UjSf0rA1blc https://youtu.be/UjSf0rA1blc (and a presentation: https://youtu.be/TUAmfi8Ws1g https://youtu.be/TUAmfi8Ws1g) Some thoughts: 1. Getting an LLM to model a problem accurately is a significant prompting exercise. Bridging casual logical statements and formal logic is difficult. E.g., "or" statements in English usually mean "xor" in logic. 2. Domains usually have their own language expectations. I was doing Zebra puzzles (https://en.wikipedia.org/wiki/Zebra_Puzzle https://en.wikipedia.org/wiki/Zebra_Puzzle) and they have a very specific pattern and language. I don't think it's fair to really call it intuitive or even entirely unambiguous, it's something you have to learn. The LLM has to learn it too. They have seen this kind of puzzle (and I think most can reproduce the original Zebra puzzle from memory), but they lack a really firm familiarity. 3. Arguably some of the familiarity is about contextualizing the problem, which is itself a prompting task. People don't naturally solve Zebra puzzles that we find organically, it's something we encounter in specific contexts (like a puzzle book) which is not so dissimilar from prompting. 4. Incidentally Claude Sonnet 3.5 has a substantial lead. And GPT o1 is not much better than GPT 4o. In some sense I think o1 is a kind of self-prompting, an attempt to create its own context; so if you already have a well-worded prompt with instructions then o1 isn't that good at improving performance over 4o. 5. A lot of the prompting is really intended to slow down the LLM, to keep it from jumping to conclusions or solving a task too quickly (and incorrectly). Which again is a case of the prompt doing what o1 tries to do generally. 6. I'm not sure what tasks call for this kind of logical reasoning. Not that I don't think they exist, I just don't know how to recognize them. Planning tasks? Highly formalized and artificially constructed problems don't seem all that interesting... and the whole point of adding an LLM to the process is to formalize the informal. 7. Perhaps it's hard to see because real-world problems seldom have conveniently exact solutions. But that's not a blocker... Prolog (and Z3) can take constraints as a form of elimination, providing lists of possible answers, and maybe just reducing the search space is enough to move forward on some kinds of problems. 8. For instance when I give my pipeline really hard Zebra problems it usually doesn't succeed; one bug in one rule will kill the whole thing. Also I think the LLMs have a hard time keeping track of large problems; a context size problem, even though the problems don't approach their formal context limits. But I can imagine building the pipeline so it also tries to mark low-confidence rules. Given that I can imagine removing those rules, sampling the resulting (non-unique, sometimes incorrect) answers and using that to revisit and perhaps correct some of those rules. Really I'd be most interested to hear thoughts on where this logic programming might actually be applied... artificial puzzles are an interesting exercise, but I can't really motivate myself to go too deep.
luke_galea 2y ago
Super cool. I dig generating rules from within the LLM, but I'm not sure Prolog is the right choice in 2024. I love Prolog and had the opportunity to use it "in anger" years ago to handle temporal logic in a scheduling app. Great experience, but I've found that more modern rules engines like Drools (anything using the Rete algorithm) are a MUCH better fit for most use cases these days. If you are into this stuff, you might like the talk I gave on rules engines, prolog and how it led to erlang & elixir. https://www.youtube.com/watch?v=mDnntrhk-8g&t=1s https://www.youtube.com/watch?v=mDnntrhk-8g&t=1s
- OutOfHere 2y ago
  The choice is limited to the languages that LLMs already know really well. Fwiw, here is GPT's self-rating out of 10: Python: 9, Prolog: 7, Datalog: 6, Mercury: 6, Curry: 5, Drools: 4 This is not even the full set of what the LLM might like to use. It may also like pyDatalog, SymPy, Haskell, Clingo ASP, ECLiPSe CLP, etc.
- YeGoblynQueenne 2y ago
  Drools is a rules engine, but Prolog is a fully-fledged, general-purpose language, yes? For example SWI-Prolog has a bunch of http libraries and can be used as a web development language (using Prolog's clause database itself in place of some SQL). I don't think that'd be a sensible use case for Drools.
  - riku_iki 2y ago
    You can use drools from Java as a library probably.
timonoko 2y ago
Interest in Prolog always ends with the "!". It is ugly and like smack in the head, "you are thinking too much".
- sgdfhijfgsdfgds 2y ago
  The course I did at uni, decades ago now, set us a Prolog assessment where we were not allowed to use the cut operator. Code that backtracks is hard to reason about.
treetalker 2y ago
Does anyone know why US attorneys and law firms are not using Prolog-based apps to automate the low-hanging fruit of issue-spotting?
- shchegrikovich 2y ago
  Or something like Catala language - https://catala-lang.org/ https://catala-lang.org/? Catala: A Programming Language for the Law - https://arxiv.org/abs/2103.03198 https://arxiv.org/abs/2103.03198
- sgdfhijfgsdfgds 2y ago
  Because Prolog is difficult, and expressing fuzzy real-world facts and nuances in it is harder.
- samatman 2y ago
  Law was particularly badly burned by the hype wave of "expert systems" in the 1970s and 80s, many of them coded in Prolog. https://en.wikipedia.org/wiki/Expert_system https://en.wikipedia.org/wiki/Expert_system The Wiki link is nearly hagiographic in its studied avoidance of the topic of how the field crashed and burned, and the term "expert system" fell into disrepute, but these are things which happened. Which isn't to say that the legal field can't benefit from software which uses Prolog, in fact, I have a strong hunch that the number of such products currently in use is not zero. But if you wrote a new one, you would do well to make sure that no senior partners hear the word "Prolog" in the sales pitch.
sitkack 2y ago
Check out https://github.com/mthom/scryer-prolog https://github.com/mthom/scryer-prolog https://www.scryer.pl/ https://www.scryer.pl/ If you are close to Vienna Nov 7th and 8th there is a community meeting https://www.digitalaustria.gv.at/eng/insights/Digital-Austria-Events-EN/Scryer-Prolog-Meetup-2024.html https://www.digitalaustria.gv.at/eng/insights/Digital-Austri...
TyrianPurple 2y ago
Ah Prolog. So the full circle back to expert systems is complete now, yeah?
- namaria 2y ago
  They tried brute forcing it, now it's back to programming it directly.
lynx23 2y ago
I implemented Prolog and Z3 as a function tool for my little OpenAI Assistants API client. Z3 (SMT-LIB, actually) seemed even more promising. The model speaks both languages. However, I didn't find any convincing use-cases yet. But it seems a logical extension of the idea to provide a programming language via a function tool to solve problems. So you start thinking "What else has text input and output, and a powerful engine in between?"
johnisgood 2y ago
ChatGPT seems to do awful with Prolog. Do you guys have any experiences with using an LLM to write Prolog?
- tannhaeuser 2y ago
  See my top-level comment for an example. Though different, there's also casual experimentation described in [1] (and additional posts linking to academic research on the SWI Prolog forum). Do you mind sharing your experience with ChatGPT (which version)? [1]: https://swi-prolog.discourse.group/t/chatgpt-prompts-prolog-any-implementation/6258 https://swi-prolog.discourse.group/t/chatgpt-prompts-prolog-...
  - johnisgood 2y ago
    I do not think that I have the conversation / chat anymore, but I was trying to get it to make a monthly schedule for N workers with specific constraints[1]. I thought Prolog is suitable for this (is it?), but the generated code got stuck / hung up. [1] https://news.ycombinator.com/item?id=41756679 https://news.ycombinator.com/item?id=41756679
shchegrikovich 2y ago
The draft version of this blog was called - 'Renaissance of Programming Languages'. With so much hidden gems we are re-discovering the full potential of formal languages.
tannhaeuser 2y ago
While the paper is about helping an LLM using Prolog, to give an idea for a realistic application for generating Prolog, here's the zero-shot prompt and response of a basic local Instruct LLM on a challenge to a classic 1998 AI Planning Competition "Logistics" problem re-formulated in English from PDDL, running on a very basic local 8B LLM quant'd to run on 12GB and containing even typos, but not showing the system prompt portion though. I mean, it's quite impressive and the generated state and action predicates can be run right away in the browser on eg. Quantum Prolog much like the container planning problem [1], with the main clause replaced by a generic STRIPS-like indeterministic/backtracking planning routine. Using additional training and/or in-context techniques and prompt reformulation can improve the result further. Since the action predicates use retract/assert primitives to change state, Quantum Prolog's vardb feature [2] can also be put to good use by transforming the resulting program into an equivalent form that can be backtracked over, even in parallel, to perform combinatorical search (whereas assertz/retract in Prolog is destructive and cannot be "undone" automatically to search new states). But as you also can see from the plan description text, a natural-language description of the problem that an LLM can comprehend only helps so much since it's highly repetitive in stating hundreds of facts about eg. current and desired package locations. This is where Prolog comes in since you can state those ground facts in clause format, or a DSL format, or a database (as in realistic applications where Prolog facts are generated or Prolog is deployed alongside a backend for integration with other services for eg. messaging in a a logistics app). Still, the combination of an LLM (excelling at language tasks) and Prolog (for actually searching a combinatorical space spanned by clauses extracted in a natural way), just as a human reach out to a pocket calculator for certain tasks, sure is powerful and has been also explored recently in one form or another in academia eg [3] also referenced elsewhere in this thread. And this isn't even using tool calling and iterative re-prompting, etc. [1]: https://quantumprolog.sgml.net/container-planning-demo/part1.html https://quantumprolog.sgml.net/container-planning-demo/part1... [2]: https://quantumprolog.sgml.net/container-planning-demo/part2.html https://quantumprolog.sgml.net/container-planning-demo/part2... [3]: "LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks" at <https://arxiv.org/abs/2402.01817 https://arxiv.org/abs/2402.01817> PROMPT (submitted as-is, including typos): Write a Prolog program to solve the following logistics problem: Packages must be transported from and to locations inside the six cities named city1, city2, city3, city4, city5, and city6) using trucks, and between those citiesusing airplanes. Each city has two locations: city1 has locations city11 and city12, city2 has locations city21 and city22, city3 has locations city31 and city32, city4 has locations city41 and city42, city5 has locations city51 and city52, andcity6 has locations city61 and city62. Initially, package1 is located at city21, package2 is located at city12, package3 is located in city11, package4 is located in city11, package5 is located in city42, and package6 is located in city31. The aim is to move package1 to city21, package2 to city62, package3 to city61, package4 to city32, package5 to city62, and package6 to city12. In each city, at the locations city11, city21, city31, city41, city51, and city61, there's a truck to transport packages to other locations within the city. There are moreover two airplanes, named plane1 and plane2, located at the airport of city1 at location11 and at the airport of city2 at location22. The airports of the other cities have the locations location32 in city3, location42 in city4, location52 in city5, and location62 in city6. To move packages, a sequence of actions as follows must be performed until the packages have reached their destination. The action load_truck will place a package in a truck at a given location. The action unload_truck will unload a package from a truck at a given location. The action fly_airplane will fly an airplane from one airport to another. The action unload_airplane will unload a package from an airplane at a given location. The action load_airplane will load an airplane with a package at a given location. RESPONSE: % State facts % Initial state of the packages package(package1, city21). package(package2, city12). package(package3, city11). package(package4, city11). package(package5, city42). package(package6, city31). % Initial state of the trucks truck(city11, city11). truck(city21, city21). truck(city31, city31). truck(city41, city41). truck(city51, city51). truck(city61, city61). % Initial state of the airplanes airplane(plane1, city11). airplane(plane2, city22). % Action clauses % load_truck :- load a package into a truck at a given location load_truck :- % If the given location is valid for loading a truck location(Loc, City), package(Pkg, Loc), truck(T, Loc), % Remove the package from its current location retract(package(Pkg, Loc)), % Add the package to the truck assertz(package(Pkg, T)), % Assert the new state assertz(truck(T, Loc)). % unload_truck :- unload a package from a truck at a given location unload_truck :- % If the given location is valid for unloading a truck location(Loc, City), package(Pkg, T), truck(T, Loc), % Remove the package from the truck retract(package(Pkg, T)), % Add the package to its destination location assertz(package(Pkg, Loc)), % Assert the new state assertz(truck(T, Loc)). % fly_airplane :- fly an airplane from one airport to another fly_airplane :- % If the airplane is at a valid airport airplane(plane, Loc), % If there is a valid destination airport airport(Dest, DestCity), % Remove the airplane from its current location retract(airplane(plane, Loc)), % Add the airplane to the destination airport assertz(airplane(plane, Dest)). % unload_airplane :- unload a package from an airplane at a given location unload_airplane :- % If the given location is valid for unloading an airplane location(Loc, City), package(Pkg, A), airplane(plane, A), % Remove the package from the airplane retract(package(Pkg, A)), % Add the package to its destination location assertz(package(Pkg, Loc)), % Assert the new state assertz(airplane(plane, Loc)). % load_airplane :- load a package into an airplane at a given location load_airplane :- % If the given location is valid for loading an airplane location(Loc, City), package(Pkg, Loc), airplane(plane, A), % Remove the package from its current location retract(package(Pkg, Loc)), % Add the package to the airplane assertz(package(Pkg, A)), % Assert the new state assertz(airplane(plane, Loc)). % Target state target :- package(package1, city22), package(package2, city62), package(package3, city61), package(package4, city32), package(package5, city62), package(package6, city12). % Main predicate to solve the problem solve :- ... % BOGUS MAIN AND HELPER CLAUSES OMITTED
lincpa 2y ago
[dead]
sgdfhijfgsdfgds 2y ago
This is magical thinking. If an LLM can’t reason it isn’t going to be able to express itself clearly in Prolog. Suggesting otherwise is intellectually on the same level as trying to make up a small consistent per-sale loss with volume.
- sgdfhijfgsdfgds 2y ago
  I know we're not supposed to comment on downvotes but I really question the logic of anyone who thinks that a thing that cannot reason can write a prolog program that is really going to be much more successful. Prolog is actually pretty difficult to do right, even if you are skilled. It actually requires reasoning. You don't just write out facts and have the system do the work. And many of the examples in the training set will be wrong, naturally simplistic or be full of backtracking that is itself difficult for a person to comprehend at a glance; why should an LLM be better at it? There can't even be that much data in the training set. Ultimately, though: stop believing in magical solutions to fundamental problems. This is nuts.
  - shchegrikovich 2y ago
    I have another example - just a few people believed that you can apply 'a simple next token prediction algorithm' and achieve what we know as LLM. From my perspective, in the past few years, we've tried a lot of different approaches to improve LLM reasoning; some of them were good, others not so good. We need to keep trying and researching. 'Prolog + LLM' is not the answer to all questions, but it looks like a good step to move us forward.
    - sgdfhijfgsdfgds 2y ago
      > 'Prolog + LLM' is not the answer to all questions, but it looks like a good step to move us forward. Or it's a thing people can write papers about, and chase reproducibility on afterwards, as the shell game of claiming LLM reasoning continues.
vaguetruth 2y ago
guys! I'm into logic and philosophy of language applied to psychology. stumbled on this thread from googling prolog! please, can you recommend a programming Prolog video intro for me? I'm interested in programming (my undergrad is cs) immediately and applying Prolog to write psycho-philosophical case studies.

So we are back to Japanese Fifth Generation plan from 1980's. :)

thelastparadise 2y ago
Watson did it too, a while back.
tokinonagare 2y ago
Missing some LISP but yeah it's funny how old things are new again (same story with wasm, RISC archs, etc.)
- nxobject 2y ago
  Lots of GOFAI being implemented again – decision trees, goal searching and planning, agent-based strategies... just not symbolic representations, and that might be the key. I figure you might get an interesting contribution out of skimming old AI laboratory publications and seeing whether you could find a way of implementing it through a single LLM, multiple LLM agents, methods of training, etc.
  - anthk 2y ago
    https://en.m.wikipedia.org/wiki/Constraint_satisfaction_problem https://en.m.wikipedia.org/wiki/Constraint_satisfaction_prob...
  - nmadden 2y ago
    Indeed, modern ML has been a validation of (some of) GOFAI: https://neilmadden.blog/2024/06/30/machine-learning-and-the-triumph-of-gofai/ https://neilmadden.blog/2024/06/30/machine-learning-and-the-...
linguae 2y ago
This time around we have all sorts of parallel processing capabilities in the form of GPUs. If I recall correctly, the Fifth Generation project envisioned highly parallel machines performing symbolic AI. From a hardware standpoint, those researchers were way ahead of their time.
- nxobject 2y ago
  And they had a self-sustaining video game industry too... if only someone had had the wild thought of implementing perceptrons and tensor arithmetic on the same hardware!
postepowanieadm 2y ago
and winter is coming.
metadat 2y ago
For the uninitiated (like me): The Japanese Fifth Generation Project https://www.sjsu.edu/faculty/watkins/5thgen.htm https://www.sjsu.edu/faculty/watkins/5thgen.htm

Building on this idea people have grounded LLM generated reasoning logic with perceptual information from other networks : https://web.stanford.edu/~joycj/projects/left_neurips_2023 https://web.stanford.edu/~joycj/projects/left_neurips_2023

a1j9o94 2y ago

I tried an experiment with this using a Prolog interpreter with GPT-4 to try to answer complex logic questions. I found that it was really difficult because the model didn't seem to know Prolog well enough to write a description of any complexity. It seems like you used an interpreter in the loop which is likely to help. I'd also be interested to see how o1 would do in a task like this or if it even makes sense to use something like prolog if the models can backtrack during the "thinking" phase

lukasb 2y ago
I bet one person could probably build a pretty good synthetic NL->Prolog dataset. ROI for paying that person would be high if you were building a foundation model (ie benefits beyond being able to output Prolog.)
- mcswell 2y ago
  I'm not exactly sure what you're referring to, but Fernando Pereira's dissertation included a natural language (English) program for querying a "database". Both the NLP part and the database were written in Prolog. Mid-1980s, I think. Of course both parts were "toy" in the sense that they would need to be hugely expanded to be of real world use, but they did handle some interesting things (like quantifiers, graded adjectives etc.).
hendler 2y ago
I also wrote wrote an LLM to Prolog interpreter for a hackathon called "Logical". With a few hours effort I'm sure it could be improved. https://github.com/Hendler/logical https://github.com/Hendler/logical I think while LLMs may approach completeness here, it's good to have an interpretable system to audit/verify and reproduce results.
- shchegrikovich 2y ago
  This is really cool!
  - 2y ago

baq 2y ago

Patiently waiting for z3-guided generation, but this is a welcome, if obvious, development. Results are a bit surprising and sound too optimistic, though.

nonamepcbrand1 2y ago

This is why GitHub CodeQL and Co-Pilot assistance is working better for everyone? basically codeql uses variant of Prolog (datalog) to query source code to generate better results.

z5h 2y ago

i've come to appreciate, over the past 2 years of heavy Prolog use, that all coding should be (eventually) be done in Prolog. It's one of few languages that is simultaneously a standalone logical formalism, and a standalone representation of computation. (With caveats and exceptions, I know). So a Prolog program can stand in as a document of all facts, rules and relations that a person/organization understands/declares to be true. Even if AI writes code for us, we should expect to have it presented and manipulated as a logical formalism. Now if someone cares to argue that some other language/compiler is better at generating more performant code on certain architectures, then that person can declare their arguments in a logical formalism (Prolog) and we can use Prolog to translate between language representations, compile, optimize, etc.

dmead 2y ago
It's taken ages for anything from functional programming to penetrate general use. Do you think uptake of logic stuff will be any faster?
- johnnyjeans 2y ago
  Prolog (and logic programming in general) is much older than you think. In fact, if we take modern functional programming to have been born with John Backus' Turing Award presentation[1], then it even predates it. Many advancements to functional programming were implemented on top of Prolog! Erlang's early versions were built on top of a Prolog-derived language who's name escapes me. It's the source of Erlang's unfamiliar syntax for more unlearned programmers. It's very much like writing Prolog if you had return values and no cuts or complex terms. As for penetrating general use, probably not without a major shift in the industry. But it's a very popular language just on the periphery, even to this day. [1] - https://dl.acm.org/doi/10.1145/359576.359579 https://dl.acm.org/doi/10.1145/359576.359579
  - dmead 2y ago
    Did you just answer me with chatgpt?

anthk 2y ago

Use Constraint Satisfaction Problem Solvers. It commes up with Common Lisp with ease.

mise_en_place 2y ago

I really enjoyed tinkering with languages like Prolog and Coq. Interactive theorem proving with LLMs would be awesome to try out, if possible.

fsndz 2y ago

This is basically the LLM modulo approach recommended by Prof. Subbarao Kambhampati. Interesting but only works mostly for problems that have some math/first degree logic puzzle at their heart. Will fail at improving perf at ARC-AGI for example... Difficult to mimic reasoning by basic trial and error then hoping for the best: https://www.lycee.ai/blog/why-sam-altman-is-wrong https://www.lycee.ai/blog/why-sam-altman-is-wrong

YeGoblynQueenne 2y ago

That's not going to work. Garbage in - Garbage out is success-set equivalent to Garbage in - Prolog out. Garbage is garbage and failure to reason is failure to reason no matter the language. If your LLM can't translate your problem to a Prolog program that solves your problem- Prolog can't solve your problem.

Philpax 2y ago
This is a shallow critique that does not engage with the core idea. Specifying the problem is not the same as solving the problem.
- YeGoblynQueenne 2y ago
  I've programmed in Prolog for ~13 years and my PhD thesis is in machine learning of Prolog programs. How deep would you like me to go?
  - Philpax 2y ago
    As deep as is required to actually make your argument!
    - YeGoblynQueenne 2y ago
      You'll have to be more specific than that. For me what I point out is obvious: Prolog is not magick. Your program won't magickally reason if you write it in Prolog, much less reason correctly. If an LLM translates a Problem to the wrong Prolog program, Prolog won't magickally turn it into a correct program. And that's just rephrasing what I said in my comment above. There's really not much more to say. Here's just one more observation: the problems where translating reasoning to Prolog will work best are problems where there are a lot of examples of Prolog to be found on the web, e.g. wolf-cabbage-goat problems and the like. With problems like that it is much easier for an LLM to generate a correct translation of the problem to Prolog and get a correct solution just because there's lots of examples. But if you choose a problem that's rarely attacked with Prolog code, like, I don't know, some mathematical problem that obtains in nuclear physics as a for instance, then an LLM will be much more likely to generate garbage Prolog, while e.g. Fortran would be a better target language. From what I can see, the papers linked in the article above concentrate on the Prolog-friendly kind of problem, like logical puzzles and the like. That smells like cherry picking to me, or just good, old confirmation bias. Again, Prolog is not magick. The article above and the papers it links to seem to take this attitude of "just add Prolog" and that will make LLMs suddenly magickally reason with fairy dust on top. Ain't gonna happen.

arjun_khamkar 2y ago

Would Creating a prolog dataset would be beneficial, so that future LLM's can be trained on it and then they would be able to output prolog code.

shchegrikovich 2y ago
I discussed this a few weeks back. The idea is to take a Python dataset, as Python is the most popular language, and write a transpiler to Prolog with the help of llms. So, creating this synthetic dataset is not a huge problem.

DeborahWrites 2y ago

You're telling me the seemingly arbitrary 6 weeks of Prolog on my comp sci course 11yrs ago is suddenly about to be relevant? I did not see this one coming . . .

fullstackwife 2y ago
Is there any need to look at this generated Prolog code?
aitchnyu 2y ago
I rolled my eyes when iterating a list meant splitting it into first and rest and recursing on the rest. I would do side projects in Scala a few years later of my own interest.

_yb2s 2y ago

I think this general idea is going to be the key to really making LLMs widely useful for solving real problems. I’ve been playing with using GPT-4 together with the Wolfram Alpha plugin, and the combo of the two can reliably solve difficult quantitative problems that neither can individually by working together, much like a human using a calculator.

de6u99er 2y ago

I always thought that Prolog is great for reasoning in the semantic web. It doesn't surprise me that LLM people stumble on it.

bytebach 2y ago

An application I am developing for a customer needed to read constraints around clinical trials and essentially build a query from them. Constraints involve prior treatments, biomarkers, type of disease (cancers) etc. Using just an LLM did not produce reliable queries, despite trying many many prompts, so being an old Prolog hacker I wondered if using it might impose more 'logic' on the LLM. So we precede the textual description of the constraints with the following prompt: ------------- Now consider the following Prolog predicates: biomarker(Name, Status) where Status will be one of the following integers - Wildtype = 0 Mutated = 1 Methylated = 2 Unmethylated = 3 Amplified = 4 Deleted = 5 Positive = 6 Negative = 7 tumor(Name, Status) where Status will be one of the following integers if know else left unbound - Newly diagnosed = 1 Recurrence = 2 Metastasized = 3 Progression = 4 chemo(Name) surgery(Name) Where Name may be an unbound variable other_treatment(Name) radiation(Name) Where Name may be an unbound variable Assume you are given predicate atMost(T, N) where T is a compound term and N is an integer. It will return true if the number of 'occurences' of T is less than or equal N else it will fail. Assume you are given a predicate atLeastOneOf(L) where L is a list of compound terms. It will succeed if at least one of the compound terms, when executed as a predicate returns true. Assume you are given a predicate age(Min, Max) which will return true if the patient's age is in between Min and Max. Assume you have a predicate not(T) which returns true if predicate T evaluates false and vice versa. i.e. rather than '\\+ A' use not(A). Do not implement the above helper functions. VERY IMPORTANT: Use 'atLeastOneOf()' whenever you would otherwise use ';' to represent 'OR'. i.e. rather than 'A ; B' use atLeastOneOf([A, B]). EXAMPLE INPUT: Patient must have recurrent GBM, methylated MGMT and wildtype EGFR. Patient must not have mutated KRAS. EXAMPLE OUTPUT: tumor('gbm', 2), biomarker('MGMT', 2), biomarker('EGFR', 0), not(biomarker('KRAS', 1)) ------------------ The Prolog predicates, when evaluated generate the required underlying query (of course the Prolog is itself a form of query). Anyway - the upshot was a vast improvement in the accuracy of the generated query (I've yet to see a bad one). Somewhere in its bowels, being told to generate Prolog 'focused' the LLM. Perhaps LLMs are happier with declarative languages rather than imperative ones (I know I am :) ).

int_19h 2y ago
I find that having GPT-4 write SQL queries to query the data source as needed to solve a complex task step-by-step also works pretty well (and you can give it the schema in form of CREATE TABLE). It's not exactly good at writing fast queries, but it can do some hella complex ones with nesting and joins to get exactly what it needs in one go.

gorkempacaci 2y ago

The generated programs are only technically Prolog programs. They use CLPFD, which makes these constraint programs. Prolog programs are quite a bit more tricky with termination issues. I wouldn’t have nitpicked if it wasn’t in the title. Also, the experiment method has some flaws. Problems are hand-picked out of a random subset of the full set. Why not run the full set?

bbor 2y ago
Yeah I’m a huge proponent of this general philosophy, but after being introduced to prolog itself for a third of a semester back in undergrad I decided to stay far, far away. The vision never quite came through as clearly as it did for the other wacky languages, namely the functional family (Lisp and Haskell in my case). I believe you on the fundamental termination issues, but just basic phrasing seemed unnecessarily convoluted… Since you seem like an expert: is there a better technology for logical/constraint programming? I loved predicate calculus in school so it seems like there should be something out there for me, but so far no dice. This seems kinda related to the widely-discussed paradigm of “Linear Programming”, but I’ve also failed to find much of interest there behind all the talk of “Management Theory” and detailed mathematical efficiency comparisons. I guess Curry (from above) might be the go-to these days?
- gorkempacaci 2y ago
  Curious to know what part of syntax you found convoluted. If you remember any examples I’d appreciate it. Maybe you want a constraint programming environment instead. As example check out Conjure from St Andrews: https://conjure.readthedocs.io/en/latest/tutorials-notebook.html https://conjure.readthedocs.io/en/latest/tutorials-notebook.... More generally there are the theorem provers like Coq, etc., but their use cases are even more specific.
YeGoblynQueenne 2y ago
>> Why not run the full set? Most likely cherry-picking. The approach is only going to work well in domains where Prolog is commonly used to write solutions to problems, like logical puzzles or constraint problems etc.

ianbicking 2y ago

I made a pipeline using Z3 (another prover language) to get LLMs to solve very specific puzzle problems: https://youtu.be/UjSf0rA1blc https://youtu.be/UjSf0rA1blc (and a presentation: https://youtu.be/TUAmfi8Ws1g https://youtu.be/TUAmfi8Ws1g) Some thoughts: 1. Getting an LLM to model a problem accurately is a significant prompting exercise. Bridging casual logical statements and formal logic is difficult. E.g., "or" statements in English usually mean "xor" in logic. 2. Domains usually have their own language expectations. I was doing Zebra puzzles (https://en.wikipedia.org/wiki/Zebra_Puzzle https://en.wikipedia.org/wiki/Zebra_Puzzle) and they have a very specific pattern and language. I don't think it's fair to really call it intuitive or even entirely unambiguous, it's something you have to learn. The LLM has to learn it too. They have seen this kind of puzzle (and I think most can reproduce the original Zebra puzzle from memory), but they lack a really firm familiarity. 3. Arguably some of the familiarity is about contextualizing the problem, which is itself a prompting task. People don't naturally solve Zebra puzzles that we find organically, it's something we encounter in specific contexts (like a puzzle book) which is not so dissimilar from prompting. 4. Incidentally Claude Sonnet 3.5 has a substantial lead. And GPT o1 is not much better than GPT 4o. In some sense I think o1 is a kind of self-prompting, an attempt to create its own context; so if you already have a well-worded prompt with instructions then o1 isn't that good at improving performance over 4o. 5. A lot of the prompting is really intended to slow down the LLM, to keep it from jumping to conclusions or solving a task too quickly (and incorrectly). Which again is a case of the prompt doing what o1 tries to do generally. 6. I'm not sure what tasks call for this kind of logical reasoning. Not that I don't think they exist, I just don't know how to recognize them. Planning tasks? Highly formalized and artificially constructed problems don't seem all that interesting... and the whole point of adding an LLM to the process is to formalize the informal. 7. Perhaps it's hard to see because real-world problems seldom have conveniently exact solutions. But that's not a blocker... Prolog (and Z3) can take constraints as a form of elimination, providing lists of possible answers, and maybe just reducing the search space is enough to move forward on some kinds of problems. 8. For instance when I give my pipeline really hard Zebra problems it usually doesn't succeed; one bug in one rule will kill the whole thing. Also I think the LLMs have a hard time keeping track of large problems; a context size problem, even though the problems don't approach their formal context limits. But I can imagine building the pipeline so it also tries to mark low-confidence rules. Given that I can imagine removing those rules, sampling the resulting (non-unique, sometimes incorrect) answers and using that to revisit and perhaps correct some of those rules. Really I'd be most interested to hear thoughts on where this logic programming might actually be applied... artificial puzzles are an interesting exercise, but I can't really motivate myself to go too deep.

luke_galea 2y ago

Super cool. I dig generating rules from within the LLM, but I'm not sure Prolog is the right choice in 2024. I love Prolog and had the opportunity to use it "in anger" years ago to handle temporal logic in a scheduling app. Great experience, but I've found that more modern rules engines like Drools (anything using the Rete algorithm) are a MUCH better fit for most use cases these days. If you are into this stuff, you might like the talk I gave on rules engines, prolog and how it led to erlang & elixir. https://www.youtube.com/watch?v=mDnntrhk-8g&t=1s https://www.youtube.com/watch?v=mDnntrhk-8g&t=1s

OutOfHere 2y ago
The choice is limited to the languages that LLMs already know really well. Fwiw, here is GPT's self-rating out of 10: Python: 9, Prolog: 7, Datalog: 6, Mercury: 6, Curry: 5, Drools: 4 This is not even the full set of what the LLM might like to use. It may also like pyDatalog, SymPy, Haskell, Clingo ASP, ECLiPSe CLP, etc.
YeGoblynQueenne 2y ago
Drools is a rules engine, but Prolog is a fully-fledged, general-purpose language, yes? For example SWI-Prolog has a bunch of http libraries and can be used as a web development language (using Prolog's clause database itself in place of some SQL). I don't think that'd be a sensible use case for Drools.
- riku_iki 2y ago
  You can use drools from Java as a library probably.

timonoko 2y ago

Interest in Prolog always ends with the "!". It is ugly and like smack in the head, "you are thinking too much".

sgdfhijfgsdfgds 2y ago
The course I did at uni, decades ago now, set us a Prolog assessment where we were not allowed to use the cut operator. Code that backtracks is hard to reason about.

treetalker 2y ago

Does anyone know why US attorneys and law firms are not using Prolog-based apps to automate the low-hanging fruit of issue-spotting?

shchegrikovich 2y ago
Or something like Catala language - https://catala-lang.org/ https://catala-lang.org/? Catala: A Programming Language for the Law - https://arxiv.org/abs/2103.03198 https://arxiv.org/abs/2103.03198
sgdfhijfgsdfgds 2y ago
Because Prolog is difficult, and expressing fuzzy real-world facts and nuances in it is harder.
samatman 2y ago
Law was particularly badly burned by the hype wave of "expert systems" in the 1970s and 80s, many of them coded in Prolog. https://en.wikipedia.org/wiki/Expert_system https://en.wikipedia.org/wiki/Expert_system The Wiki link is nearly hagiographic in its studied avoidance of the topic of how the field crashed and burned, and the term "expert system" fell into disrepute, but these are things which happened. Which isn't to say that the legal field can't benefit from software which uses Prolog, in fact, I have a strong hunch that the number of such products currently in use is not zero. But if you wrote a new one, you would do well to make sure that no senior partners hear the word "Prolog" in the sales pitch.

sitkack 2y ago

Check out https://github.com/mthom/scryer-prolog https://github.com/mthom/scryer-prolog https://www.scryer.pl/ https://www.scryer.pl/ If you are close to Vienna Nov 7th and 8th there is a community meeting https://www.digitalaustria.gv.at/eng/insights/Digital-Austria-Events-EN/Scryer-Prolog-Meetup-2024.html https://www.digitalaustria.gv.at/eng/insights/Digital-Austri...

TyrianPurple 2y ago

Ah Prolog. So the full circle back to expert systems is complete now, yeah?

namaria 2y ago
They tried brute forcing it, now it's back to programming it directly.

lynx23 2y ago

I implemented Prolog and Z3 as a function tool for my little OpenAI Assistants API client. Z3 (SMT-LIB, actually) seemed even more promising. The model speaks both languages. However, I didn't find any convincing use-cases yet. But it seems a logical extension of the idea to provide a programming language via a function tool to solve problems. So you start thinking "What else has text input and output, and a powerful engine in between?"

johnisgood 2y ago

ChatGPT seems to do awful with Prolog. Do you guys have any experiences with using an LLM to write Prolog?

tannhaeuser 2y ago
See my top-level comment for an example. Though different, there's also casual experimentation described in [1] (and additional posts linking to academic research on the SWI Prolog forum). Do you mind sharing your experience with ChatGPT (which version)? [1]: https://swi-prolog.discourse.group/t/chatgpt-prompts-prolog-any-implementation/6258 https://swi-prolog.discourse.group/t/chatgpt-prompts-prolog-...
- johnisgood 2y ago
  I do not think that I have the conversation / chat anymore, but I was trying to get it to make a monthly schedule for N workers with specific constraints[1]. I thought Prolog is suitable for this (is it?), but the generated code got stuck / hung up. [1] https://news.ycombinator.com/item?id=41756679 https://news.ycombinator.com/item?id=41756679

shchegrikovich 2y ago

The draft version of this blog was called - 'Renaissance of Programming Languages'. With so much hidden gems we are re-discovering the full potential of formal languages.

tannhaeuser 2y ago

While the paper is about helping an LLM using Prolog, to give an idea for a realistic application for generating Prolog, here's the zero-shot prompt and response of a basic local Instruct LLM on a challenge to a classic 1998 AI Planning Competition "Logistics" problem re-formulated in English from PDDL, running on a very basic local 8B LLM quant'd to run on 12GB and containing even typos, but not showing the system prompt portion though. I mean, it's quite impressive and the generated state and action predicates can be run right away in the browser on eg. Quantum Prolog much like the container planning problem [1], with the main clause replaced by a generic STRIPS-like indeterministic/backtracking planning routine. Using additional training and/or in-context techniques and prompt reformulation can improve the result further. Since the action predicates use retract/assert primitives to change state, Quantum Prolog's vardb feature [2] can also be put to good use by transforming the resulting program into an equivalent form that can be backtracked over, even in parallel, to perform combinatorical search (whereas assertz/retract in Prolog is destructive and cannot be "undone" automatically to search new states). But as you also can see from the plan description text, a natural-language description of the problem that an LLM can comprehend only helps so much since it's highly repetitive in stating hundreds of facts about eg. current and desired package locations. This is where Prolog comes in since you can state those ground facts in clause format, or a DSL format, or a database (as in realistic applications where Prolog facts are generated or Prolog is deployed alongside a backend for integration with other services for eg. messaging in a a logistics app). Still, the combination of an LLM (excelling at language tasks) and Prolog (for actually searching a combinatorical space spanned by clauses extracted in a natural way), just as a human reach out to a pocket calculator for certain tasks, sure is powerful and has been also explored recently in one form or another in academia eg [3] also referenced elsewhere in this thread. And this isn't even using tool calling and iterative re-prompting, etc. [1]: https://quantumprolog.sgml.net/container-planning-demo/part1.html https://quantumprolog.sgml.net/container-planning-demo/part1... [2]: https://quantumprolog.sgml.net/container-planning-demo/part2.html https://quantumprolog.sgml.net/container-planning-demo/part2... [3]: "LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks" at <https://arxiv.org/abs/2402.01817 https://arxiv.org/abs/2402.01817> PROMPT (submitted as-is, including typos): Write a Prolog program to solve the following logistics problem: Packages must be transported from and to locations inside the six cities named city1, city2, city3, city4, city5, and city6) using trucks, and between those citiesusing airplanes. Each city has two locations: city1 has locations city11 and city12, city2 has locations city21 and city22, city3 has locations city31 and city32, city4 has locations city41 and city42, city5 has locations city51 and city52, andcity6 has locations city61 and city62. Initially, package1 is located at city21, package2 is located at city12, package3 is located in city11, package4 is located in city11, package5 is located in city42, and package6 is located in city31. The aim is to move package1 to city21, package2 to city62, package3 to city61, package4 to city32, package5 to city62, and package6 to city12. In each city, at the locations city11, city21, city31, city41, city51, and city61, there's a truck to transport packages to other locations within the city. There are moreover two airplanes, named plane1 and plane2, located at the airport of city1 at location11 and at the airport of city2 at location22. The airports of the other cities have the locations location32 in city3, location42 in city4, location52 in city5, and location62 in city6. To move packages, a sequence of actions as follows must be performed until the packages have reached their destination. The action load_truck will place a package in a truck at a given location. The action unload_truck will unload a package from a truck at a given location. The action fly_airplane will fly an airplane from one airport to another. The action unload_airplane will unload a package from an airplane at a given location. The action load_airplane will load an airplane with a package at a given location. RESPONSE: % State facts % Initial state of the packages package(package1, city21). package(package2, city12). package(package3, city11). package(package4, city11). package(package5, city42). package(package6, city31). % Initial state of the trucks truck(city11, city11). truck(city21, city21). truck(city31, city31). truck(city41, city41). truck(city51, city51). truck(city61, city61). % Initial state of the airplanes airplane(plane1, city11). airplane(plane2, city22). % Action clauses % load_truck :- load a package into a truck at a given location load_truck :- % If the given location is valid for loading a truck location(Loc, City), package(Pkg, Loc), truck(T, Loc), % Remove the package from its current location retract(package(Pkg, Loc)), % Add the package to the truck assertz(package(Pkg, T)), % Assert the new state assertz(truck(T, Loc)). % unload_truck :- unload a package from a truck at a given location unload_truck :- % If the given location is valid for unloading a truck location(Loc, City), package(Pkg, T), truck(T, Loc), % Remove the package from the truck retract(package(Pkg, T)), % Add the package to its destination location assertz(package(Pkg, Loc)), % Assert the new state assertz(truck(T, Loc)). % fly_airplane :- fly an airplane from one airport to another fly_airplane :- % If the airplane is at a valid airport airplane(plane, Loc), % If there is a valid destination airport airport(Dest, DestCity), % Remove the airplane from its current location retract(airplane(plane, Loc)), % Add the airplane to the destination airport assertz(airplane(plane, Dest)). % unload_airplane :- unload a package from an airplane at a given location unload_airplane :- % If the given location is valid for unloading an airplane location(Loc, City), package(Pkg, A), airplane(plane, A), % Remove the package from the airplane retract(package(Pkg, A)), % Add the package to its destination location assertz(package(Pkg, Loc)), % Assert the new state assertz(airplane(plane, Loc)). % load_airplane :- load a package into an airplane at a given location load_airplane :- % If the given location is valid for loading an airplane location(Loc, City), package(Pkg, Loc), airplane(plane, A), % Remove the package from its current location retract(package(Pkg, Loc)), % Add the package to the airplane assertz(package(Pkg, A)), % Assert the new state assertz(airplane(plane, Loc)). % Target state target :- package(package1, city22), package(package2, city62), package(package3, city61), package(package4, city32), package(package5, city62), package(package6, city12). % Main predicate to solve the problem solve :- ... % BOGUS MAIN AND HELPER CLAUSES OMITTED

lincpa 2y ago

[dead]

sgdfhijfgsdfgds 2y ago

This is magical thinking. If an LLM can’t reason it isn’t going to be able to express itself clearly in Prolog. Suggesting otherwise is intellectually on the same level as trying to make up a small consistent per-sale loss with volume.

sgdfhijfgsdfgds 2y ago
I know we're not supposed to comment on downvotes but I really question the logic of anyone who thinks that a thing that cannot reason can write a prolog program that is really going to be much more successful. Prolog is actually pretty difficult to do right, even if you are skilled. It actually requires reasoning. You don't just write out facts and have the system do the work. And many of the examples in the training set will be wrong, naturally simplistic or be full of backtracking that is itself difficult for a person to comprehend at a glance; why should an LLM be better at it? There can't even be that much data in the training set. Ultimately, though: stop believing in magical solutions to fundamental problems. This is nuts.
- shchegrikovich 2y ago
  I have another example - just a few people believed that you can apply 'a simple next token prediction algorithm' and achieve what we know as LLM. From my perspective, in the past few years, we've tried a lot of different approaches to improve LLM reasoning; some of them were good, others not so good. We need to keep trying and researching. 'Prolog + LLM' is not the answer to all questions, but it looks like a good step to move us forward.
  - sgdfhijfgsdfgds 2y ago
    > 'Prolog + LLM' is not the answer to all questions, but it looks like a good step to move us forward. Or it's a thing people can write papers about, and chase reproducibility on afterwards, as the shell game of claiming LLM reasoning continues.

vaguetruth 2y ago

guys! I'm into logic and philosophy of language applied to psychology. stumbled on this thread from googling prolog! please, can you recommend a programming Prolog video intro for me? I'm interested in programming (my undergrad is cs) immediately and applying Prolog to write psycho-philosophical case studies.