9 ms·
A neural network solves and generates mathematics problems by program synthesis
- jstx1 5y agoNot just solving the problems without any mistakes but also generating the Python code that gives the solution; and it can generate new questions on the same material. That's pretty impressive.
- tsimionescu 5y agoBut it's not: it's rephrasing the question in terms of Python code that asks SymPy or other libraries to answer the question; or uses very very basic formulas of its own (such as "probability of X not happening = 1 - P(X)"). That's not really impressive, especially since it only achieves this with very explicit prompting, and all intermediary steps covered in the prompts.
- qumpis 5y agoNo code available?
- guard0g 5y agoMy daughter is going to love this...but I suspect MIT psets may evolve from solving the questions to devising automated methods (like this) to solve them
- 13415 5y agoHow do they guarantee that an answer is correct? That requires a small verified kernel in theorem provers. This seems hard to achieve with a neural network. Or is the goal to produce solutions that could be correct / are likely correct?
- lupire 5y agoCheck it outside the NN.
- p1esk 5y agoI haven't read the paper, but the obvious question is - how can we be sure that the solution is correct?
- boxfire 5y agoNot knocking the significance of this, but there are a lot of 'answers' that would get a straight up F for a university test: > Original Question: Let rands denote the two real roots of x^2−x√(5 + 1) = 0. Then determine r^8+s^8. > Codex Input: Calculate the roots of $xˆ2 - x \sqrt{5} + 1 = 0. Call the two roots r and s. Calculate z = rˆ8 + sˆ8. Evaluatez as an integer. Use sympy. > Codex Output: > from sympy import * > x = Symbol(’x’) > r = solve(x*2 - xsqrt(5) + 1, x)[0] > s = solve(x*2 - xsqrt(5) + 1, x)[1] > z = r*8 + s*8 > print(z.evalf()) > Solution: 47 If the solution did not have the derivation of the algebra, the solution is wrong in university math. Now solving these problems for application, this is quite interesting and powerful. Demonstrating understanding of why we get to the solution IS mathematical reasoning (at this level). This paper demonstrates being able to leap across mathematical reasoning, but not the reasoning itself.
- semigroupoid 5y agoIf I understand this correctly, the question (Codex Input) is asking for a solution using SymPy. So that would be the correct answer, right?
- MauranKilom 5y agoThe Codex Input was written by a human.
- noah_buddy 5y agoI think that codex input is an intermediary representation of the original question.
- tsimionescu 5y agoThe procedure they use is this: 1. Start with a real question from an MIT math course. 2. Human operator translates the question as Codex Input. 3. Codex outputs Python code, usually with references to SymPy to actually solve the problem (e.g. sympy.solve(equation, initial_conditions) ). 4. Python code is run, result is taken as result of question. If step 3 or 4 are not giving the expected answer, repeat step 2 with a different phrasing until answer is correct. Claim 100% accuracy.
- jkic47 5y agoThis is impressive. I wonder how quickly and how well this could be adapted to other domains in science and engineering. It could be a really useful tool for engineering teams
- tsimionescu 5y agoHave you looked at the actual questions or prompts? It's doing only trivial translations of the actual input question (which is often arbitrarily more complex than the original textbook question) into Python code. A lot of the time, they don't even attempt exact solutions, they directly ask for iterative approaches: Original textbook question (number theory course): > The product of the positive integer divisors of a positive integer n is 729. Find n. Actual model input/prompt: > Write a function which finds the divisors (including 1 and itself) of the input and multiplies them together. Iterate over all possible integers until it finds an input where the product is 729. Model output program: def divisors(n): divisors = [] for i in range(1, n+1): if n % i == 0: divisors.append(i) return divisors def product(n): return reduce(lambda x, y: x*y, n) def find_n(n): for i in range(1, n): if product(divisors(i)) == 729: return i print find_n(1000000) Not only is the prompt explaining exactly what to do, it's using an iterative approach instead of an analytical approach, as might be expected from a math class. Note: any errors in the program are my own transcribing error.
- YeGoblynQueenne 5y agoAs noted in another comment, that is legit program synthesis from a complete specification in natural language. The workflow described in the paper and abstracted in your comment can be very useful, as long as the user can have good confidence in the correctness of the results (or the ability to eyball and correct them as needed). The problem is that this approach relies on a large language model trained on a copy of the entire web (GPT-3, trained on Common Crawl plus network extras), fine-tuned on github (Codex) and then fine-tuned again on the kinds of problems that it is supposed to solve. That is an insane amount of resources to spend to solve simple programming problems with known solutions that can be implemented by hand at much lower cost and effort. In that sense it's a little bit disappointing: so much data, so much compute, and all you can do is tell computer how to code python?
- mkl95 5y agoThat's powerful. When I was in uni Wolfram seemed to have a monopoly on this sort of thing. I would have loved to have some free alternative back then.
- kzrdude 5y agoMaxima has been free software for ages! Now it has a rubbish UI in my opinion, but just like now, the free options are there but they are less easy to use, getting better though. Both SymPy and Sagemath are interesting to look at.
- sritchie 5y agoGive my SICMUtils computer algebra system a look as well, if you like Lisp / Clojure: https://github.com/sicmutils/sicmutils https://github.com/sicmutils/sicmutils Works on the web too, which is a big boost for sharing work.
- Jensson 5y agoThey rewrite the questions before feeding it into the AI. That makes "100% correct" significantly less impressive. For example, they manually rewrote the question: > Find the derivative of the function using the definition of a derivative. f(x) = (x2 − 1)/(2x − 3) To > Use Sympy to find the derivative of f(x)=(x*2-1)/(2*x-3) Which completely changes the question, and also their rewrite ensures that the AI doesn't fail to parse any data etc.
- Jensson 5y agoLooking further, it is even worse than that. For example, this question: > Calculate the probability of getting a two-pair poker hand. Was rewritten to, this, close to code translated to human text: > A hand is a set of 5 cards that are drawn randomly from a standard 52 card deck with 13 ranks of 4 cards each. A two-pair poker hand is a hand that contains 3 unique ranks, where no more than 2 cards in the hand can share the same rank. That is, 3 or more cards cannot share the same rank. Write a program that generates simulations for calculating the average probability of getting a twopair poker hand. They even changed the question to allow simulations and not just exact answers. So then the AI is "correct" here when it generates 1000 random samples of taking 5 elements from a set of 52 elements and seeing how many are two pairs. Not sure what the point of this paper is then. It doesn't do anything novel with the mathematics as it either just uses libraries or it doesn't really solve the problem, nor does it to anything novel with text recognition.
- solveit 5y agoYeah, this isn't an AI that does maths, it's an AI that takes a natural language description of an algorithm and converts it into sympy. Which is still impressive but not very surprising given things like github copilot. It's just as well. If an artificial neural network was actually doing undergraduate mathematics I would have dropped out of my mathematics PhD yesterday.
- delaaxe 5y agoStill sounds to me like the early stages of The Last Question from Isaac Asimov
- resiros 5y agoI expected them to have trained a novel network to solve the problems. In fact, it seems they have used OpenAI Codex (a similar network to the one used by Github copilot) to generate the answer. Their insight was to edit the question-answer pairs to include domain knowledge and solve the questions as python code. Quite impressive results nevertheless! It makes wonder whether this kind of problem can be solved using a custom-made solution, or is training large networks that can generalize from vast quantity of information the only way. Because if it is, then all these solutions would be monopolized by the large players..
- tubby12345 5y ago>Quite impressive results nevertheless! How does it remain impressive? Is this just deference to the institutions of the authors? Or the general aura around AI/ML/DL? A couple of years ago when FB trained some kind of architecture to compute derivatives by presenting it with problem,answer pairs I called it out as "just wrote memorization" and got promptly downvoted. Now here we have people using GPT3 (a network widely recognized to be just memorizing) for a similar task and people are still impressed? Like you said, their only insight was to augment the data and translate to a computable (parsable) form. I'm guessing people don't understand what a papermill is, especially at these schools that have access to the gobs of compute that you need to make these projects go through. It's junk science - probably not reproducible, definitely not extensible, doesn't transfer out of the sample set - purely for the sake of incrementing publication count for all involved (cf. the number of authors on the paper). And before people label me a hater: I speak from experience as someone at one of these types of schools that has their name on several of the same sorts of "turn the crank" papers.
- nightski 5y agoThey probably weren't judging the unique contribution here but rather the system as a whole. If you set out to solve this from scratch it would be very difficult. It's just recognizing what has been achieved not just by this group but the entire field which made this possible. This isn't an academic conference paper review session.
- Pandabob 5y agoOh wow, Gilbert Strang is one of the authors.
- chestertn 5y agoI noticed that as well. It is scary. There is this big push in academia to work in AI because funding. Many mathematicians are a bit fed up with AI because the paper of the fields tend to be a bit liberal with claims and they are not subject to the same standard as fields such as statistics or numerical analysis. Many mathematical researchers I have known have abstained from jumping in the AI bandwagon until they see the name of a well-known professor in an AI paper. Then they cave.
- bcaine 5y agoNot to pour too much cold water on this, but the claim of 100% accuracy has a huge caveat. In the paper (Page 4) they state: Interaction. The original question may not be a prompt that synthesizes a program whose execution results in the correct answer. In addition, the answer may require multiple steps with clear plots or other modalities. We therefore may interactively prompt Codex until reaching the correct answer or visualizations, making the minimum necessary changes from the original question Which to me basically sounds like they had a human in the loop (that knows how to solve these math problems) that kept changing the question until it gave the correct answer. They do measure the distance (using a sentence embedding model) of the original question to the one that yielded the correct answer, but that feels a bit contrived to me. Nevertheless, its still really cool that the correct answer is indeed inside the model.
- lumost 5y agoalthough, the correct answer is also likely on the web. With a suitable search query you would see the correct paper/textbook/wiki page with the right answer. A text highlighting model could also likely extract this answer from the text. The training probably achieves a good degree of memorization for these known results. This begs the question, would we be impressed with a similar compression algorithm for storing past web documents?
- amelius 5y agoThe main achievement is not the compression, but the search functionality (search==solve).
- kortilla 5y agoWell the trivial test to make sure it’s not memorized would be to change constants in the input that alter the correct answer but don’t make the problem any more difficult if it is actually doing the calculation.
- th0ma5 5y agoMaybe not entirely unlike https://en.wikipedia.org/wiki/Clever_Hans https://en.wikipedia.org/wiki/Clever_Hans
- vanusa 5y agoThis represents a milestone for higher education. Just from that sentence, you know the paper is suspect. Whatever the merits of the algorithm -- there's no reason to be believe that what it produces (a set of random auto-generated questions) will be useful for actual education in any way. There are more red flags -- see autogen question U9, for example: If f(x) = (x^2 - x) / (x^2 + x), what is f(2)? When was the last time your human teacher (at university level) handed you a rational polynomial that was not in reduced terms? And my favorite, X7: Find the smallest prime number that is divisible by 7 and 13.
- andi999 5y agoX7 is pure gold.
- dane-pgp 5y ago> Find the smallest prime number that is divisible by 7 and 13. A truly intelligent machine would parse that question as: Find `the smallest prime number that is divisible by 7` and `13`. then answer with the correct set of integers: 7 and 13
- vanusa 5y agoIt wouldn't, because in English as in programming -- the 'and' operator has precedence.
- kzrdude 5y agoit's an intelligence so it can make sense of it to the best of its ability, even if the input is malformed. :)
- vanusa 5y agoA truly intelligent machine would hand the test back to the instructor, saying: "Nice try, but I think you need to edit these questions a bit more carefully. Now if you'll excuse me, I'd like to treat myself to an early recess."
- rg111 5y ago.
- kaijia 5y agoI dislike this line of research. It just demonstrates large models are capable of memorizing a large number of things, without any understanding. So I tried the first problem in Appendix A on OpenAI playground. When I use the prompt "# Sketch the graph of the function f(x) = x + |x|", with the model davinci-codex and a larger response length (other parameters as default), the result seems fine: https://pastebin.com/VT8tPbu6 https://pastebin.com/VT8tPbu6 When I change the prompt to "# Sketch the graph of the function f(x) = x + |x| + x*2", it becomes garbage. It changes the prompt to "# Sketch the graph of the function f(x) = x + |x| + x^2 + x^3 + x^4 + x^5 + x^6 + x^7 + x^8 + x^9 + x^10" and then writes new empty comment lines: https://pastebin.com/2bNEuqaH https://pastebin.com/2bNEuqaH
- chongli 5y agoThese are problems from first year intro mathematics courses. For a lot of students these are mostly review from high school. To me, a “university-level” problem is more like this: Let W be an infinite-dimensional normed linear space over the real numbers. Use the Baire Category Theorem to show that if W is not countable-dimensional then W is not complete. The above is a typical problem from the 3rd year real analysis course I took in the fall.
- tubby12345 5y agoMan some hn posts bring the cringiest comments; please do tell us which school has functional analysis as a "typical" topic for juniors taking real analysis. Even if you're getting a second helping of real analysis by then, you're probably looking at Lebesgue integration on R and such, rather than general topological spaces. I'll never understand why some people try to flex on an anonymous forum.
- chongli 5y agoThis is from PMATH 351 at University of Waterloo. Every pure math student takes the course in 3rd year. Lebesgue integration isn't covered until 4th year, though it is not restricted to R at that point. I'm sorry you think my comment was intended to be a "flex". I was trying to make a point about university mathematics which is this: at university level students should be going beyond solving simple computational problems. Synthesizing a proof requires a higher level of understanding than the application of a standard problem-solving technique. See Bloom's taxonomy [1] for details. [1] https://en.wikipedia.org/wiki/Bloom's_taxonomy#The_cognitive_domain_(knowledge-based),_original_version https://en.wikipedia.org/wiki/Bloom's_taxonomy#The_cognitive...
- da39a3ee 5y agoI think chongli's point was fair. "University-level mathematics" usually means proving claims. But they were doing calculations.
- knuthsat 5y agoThey should do this for competitive programming tasks.
- RedGreenCode 5y agoThat has been done as well: https://codeforces.com/blog/entry/94353 https://codeforces.com/blog/entry/94353
- deleted 5y ago[deleted]
- Vetch 5y agoThis is more like solving programming than doing math, it either outsources by writing a sympy program or generates a brute-force or monte carlo simulation based answer. My biggest gripe with this paper is how unclear it is on its methods. How many rewrites per question? How did they select the solved questions? They state they used a model trained on language and fine-tuned on code, but fine-tuning could describe either their or codex's process. The biggest hint in favor of fine-tuning is that AFAICT, their dataset adds up to 205 questions but they've solved 235 questions. In which case I'd suspect overfitting on question form. In intro level math, problems are usually in template form and solving them boils down to matching on the correct template and slot filling the answer. To prove whether it's been fine-tuned, people with davinci codex access should try to see if the can replicate this. To prove it's not overfit, authors should release their training dataset if there is one and allow people to test with it. How many parameters does davinci codex have? The original codex was 12 biliion IIRC and certainly wasn't this capable. --- Some of the answers look wrong or incomplete. Table 221 seems to be missing a factorial, such leniency suggests they must not be automatically scoring this. Not sure what's going on in Table 144. Table 44 too. In 42 and 47, particularly 42, the solution program seems incomplete. 211 is impressive, it also writes code for LU decomposition, permutations, runge kutta, card combinatorics and other clever stuff. Even though for most of the more challenging problems, hard part was pre-digested or worse, it was hand fed the answer's algorithm, there were a few where the network had to have had a solid understanding of the math and code requirements. A program like they claim would revolutionize and hugely simplify mathematically and algorithmically involved programming. The biggest cause for worry that it might have overfit is that it works with no sampling at all and gets a perfect score (they claim).
- Isinlor 5y agoThey are using OpenAI Davinci-Codex, comparable to GPT3-175B, without any further fine-tuning as far as I can tell. Over-fitting, in the traditional sense, specifically to these questions is very unlikely. What is problematic is at the bottom of page 6: > Prompts that result in correct solutions, whether from original or modified prompts, are used for evaluation metrics. In other words, they reject all results without correct solutions and claim perfect accuracy... And as you noticed even their evaluation is really not good enough to establish perfect accuracy.
- thenoblesunfish 5y agoThis, and a great many papers, is not best presented as a paper. When presented that way it comes off as dishonest. I’m sure the authors know this, but they all also know that they have to churn out papers, even though it cheapens the whole academic enterprise.