4 ms·
I've only given a cursory reading to the paper, so my concerns may be addressed somewhere in it. This is admittedly more rigorous work than previous efforts to
by YeGoblynQueenne 6y ago
I've only given a cursory reading to the paper, so my concerns may be addressed somewhere in it. This is admittedly more rigorous work than previous efforts to train language models to perform automated theorem proving and it's admirable that the authors have actually reached out to the community around Metamath (the classical automated theorem prover used in the paper) to understand the extent to which their work is useful to that community.
That said, I don't understand the motivation of the work. Why is it necessary to teach a language model to perform automated theorm proving, when there are already approaches that can perform automated theorem proving, that outperform language models in theorem proving and that do not have the high costs associated with training language models? Indeed, Metamath is based on an extremly simple principle of variable substitution, which is just what it sounds like and comparing the complexity of a modern, GPT-based language model to such a simple approach is not flattering for the language model approach. Besides if I understand correctly, Metamath, was used as the "ground truth" for evaluation and testing of the trained language model and the language model didn't get anywhere near 100% performance, meaning it's far behind the ability of Metamath in automated theorem proving. So we have a very simple approach that works very well on one side and an extremely complex approach that doesn't work very well on the other. The comparison is, again, not flattering for the language model approach.
I can see that GPT-ƒ (the language model in the paper) has contributed some shorter proofs to some theorems in Metamath's database, which seems to be very much appreciated by the Metamath community, but on the other hand, it seems that GPT-ƒ was not able to prove new theorems (and a shorter version of a theorem proved by GPT-ƒ was found by hand, by one of the Metamath users, a little after the GPT-ƒ results were announced on the Metamath google group).
It seems to me it would be a long and arduous road for language models to achieve parity with classical automated theorem provers - if indeed such parity can be achieved at all. And all this hard work- for what, when there already exist perfectly serviceable automated theorem provers with none of the drawbacks of language models)? Why reinvent a wheel, only to come up with a new wheel that can only roll one way and only on Tuesdays?
Finally, despite the more rigorous approach in this paper, the trend of only comparing neural net approaches to other neural net approaches continues. An automated theorem prover approach of any stripe should be compared to the state of the art in automated theorm proving, not just to other efforts using a similar approach! Imagine if I wanted to present a new image recognition algorithm based on automated theorem proving and ommitted comparison with Convolutional Neural Nets!
- alexlyzhov 6y ago> Besides if I understand correctly, Metamath, was used as the "ground truth" for evaluation and testing of the trained language model and the language model didn't get anywhere near 100% performance, meaning it's far behind the ability of Metamath in automated theorem proving. I'm new to this but wiki says Metamath refers to language + proof checker. It doesn't refer do a method of proof search right? Isn't it just a database of proofs? I'm confused. Can you please shine light on traditional approaches that are better than DL approaches at proof search? And on how the setting they are evaluated in is different from the evaluation setting in this paper?
- YeGoblynQueenne 6y agoProof verification is traditionally considered an automated theorem proving task. In short, being able to automatically decide the validity of a proof is almost as useful as being able to generate a valid proof automatically. A lot of work has gone into automated theorem proving since the early 20th century. The wikipedia page is a place to start if you are interested in more details: https://en.wikipedia.org/wiki/Automated_theorem_proving https://en.wikipedia.org/wiki/Automated_theorem_proving This work includes proof checkers, for which wikipedia also has a page, though it doesn't have a lot of meat on its bones and you'll have to follow the links to the systems listed yourself: https://en.wikipedia.org/wiki/Proof_assistant https://en.wikipedia.org/wiki/Proof_assistant I'm not an expert on proof checkers, but I hear a lot about Coq and Lean, lately, so you might want to start there. Traditional approaches to automated theorem proving, in the sense of generating new proofs (rather than checking the validity of proofs) include, for example, Robinson's resolution algorithm that is at the heart of the logic programming language Prolog. Such traditional approaches are "better than DL" in the sense that they are provably sound (they only return correct results) and complete (they return all correct results) under certain assumptions. Given the impossibility of empirically checking the proof of every possible theorem, e.g. in first order logic, given finite resources, such system are typically evaluated theoretically, i.e. by proofs of their properties. So for instance, there are proofs of the soundness and completeness of resolution for the purpose of deciding entailment between FOL theories given as sets of Horn clauses. I keep returning to resolution because I happen to be familiar with it, but there are other approaches, of course, though I believe many use some kind of inference rule (e.g. Modus Ponens) and a search procedure usually assisted by some heuristics to find the best next step in a proof.