7 ms·
Minerva: Solving Quantitative Reasoning Problems with Language Models
- 331c8c71 4y agoWeird. There are some impressive examples and then it fails spectacularly e.g. by simply removing all square root signs at one step. Given that type of failure I must conclude the model didnt learn essential fundamentals of math reasoning (i guess it was clear from the start it is not built to do so). I would like to have an interactive demo where I can type tasks for it.
- visarga 4y ago> then it fails spectacularly Like a math professor never forgot to carry a sign or programmers don't make off by one errors. It's quite endearing to see it make human like errors.
- 331c8c71 4y agoOne of the errors the authors presented was > f(x) = f(y) + f(z) => x = y + z where f(.) denotes square root. This is very different in nature from flipping a sign or a one-off error.
- rhaen 4y agoI mean that’s pretty close to a mistake that students make so frequently it has a Wikipedia page https://en.m.wikipedia.org/wiki/Freshman%27s_dream https://en.m.wikipedia.org/wiki/Freshman%27s_dream
- 331c8c71 4y agolol. it's a funny name but I find it hard to believe it's actually a common mistake. my experience is that squaring and multiplying polynomials is done so often in high school that it's hard not to learn this is not true (i studied in an ordinary high school in russia, not any special one).
- visarga 4y agoMe too, I studied it in Romania and it was very basic and well drilled. But plenty of students just blocked out math. I bet the Minerva model learned it on the internet, some bad training example could explain it.
- cgearhart 4y agoWhat happens if you feed it the Riemann Hypothesis or the Collatz Conjecture? It’s impressive that the model is clearly learning a lot of facts in training that it just memorizes—like the variance of the first m even numbers, and remarkable to me that these models can generate correct output with high probability when they have no actual understanding of mathematics. But perhaps that’s just an indication that my own ideas are wrong—to wit, maybe math just isn’t that hard (in the sense that the sequence of correct outputs to a given prompt are a highly constrained pattern built around successive substitutions in common templates, rather than searching for a needle in a practically-infinite haystack).
- Konohamaru 4y agoSomebody needs to figure out how to merge language models with theorem proving. In fact, that might be the first step to AGI.
- globuous 4y agoThat’s the topic of my thesis ! Here’s some work by open ai doing exactly that ! :) https://arxiv.org/abs/2202.01344 https://arxiv.org/abs/2202.01344
- alphabetting 4y agoPretty insane. I thought AI had a ways to go before it could solve problems like these. https://minerva-demo.github.io/#category=Algebra&index=1 https://minerva-demo.github.io/#category=Algebra&index=1
- riku_iki 4y agoSo, I look at the internet for some questions, for example: https://minerva-demo.github.io/#category=Algebra&index=3 https://minerva-demo.github.io/#category=Algebra&index=3 And here you go: https://books.google.com/books?id=QutS5unZDLYC&pg=PA126&lpg=PA126&dq=1296%3D64%3D2434&source=bl&ots=HCvmdpH8JZ&sig=ACfU3U2s9UaZUJo1B1sB13zbVZHPgxiBhw&hl=en&sa=X&ved=2ahUKEwjcmr2hsNb4AhUYD0QIHWFqAQ4Q6AF6BAgCEAM#v=onepage&q=1296%3D64%3D2434&f=false https://books.google.com/books?id=QutS5unZDLYC&pg=PA126&lpg=... So, they may have data leak in training data. I am wondering what would be model answer if they slightly change numbers.
- ramasesh 4y agoThey have this plot in the paper. See Figure 5. Average accuracy (averaged over problems) does not change.
- riku_iki 4y agoThe question listed in paper (they modified 20 out of thousands in MATH dataset) as example is very far in calculating complexity from what I quoted in my comment above. Their question for your convenience: At each basketball practice last week, Jenny made twice as many free throws as she made at the previous practice. At her fifth practice she made 48 free throws. How many free throws did she make at the first practice?
- riku_iki 4y ago> accuracy (averaged over problems My understanding is that it is average over 64 solution attempts (model generated 64 possible answers, and they calculated average accuracy of this answers), which is not target metric of benchmark. Additionally, accuracy degradation is clearly visible on graphs, first graph has 4 problems out of 20 bellow the line, last graph has 14 problems bellow the line.
- sharemywin 4y agoLimitations Our approach to quantitative reasoning is not grounded in formal mathematics. Minerva parses questions and generates answers using a mix of natural language and LaTeX mathematical expressions, with no explicit underlying mathematical structure. This approach has an important limitation, in that the model’s answers cannot be automatically verified. Even when the final answer is known and can be verified, the model can arrive at a correct final answer using incorrect reasoning steps, which cannot be automatically detected. This limitation is not present in formal methods for theorem proving (e.g., see Coq, Isabelle, HOL, Lean, Metamath, and Mizar). On the other hand, an advantage of the informal approach is that it can be applied to a highly diverse set of problems which may not lend themselves to formalization.
- sharemywin 4y agoDon't get me wrong it's pretty amazon what it can do but, I get the impression there's no confidence in the answers so I'm not really sure what to do with it. As with a lot of these systems how cherry picked were the answers
- nharada 4y agoIf you can create a system that presents a limited number of novel solutions a human could double check each one.
- alphabetting 4y agosome interesting stuff from one of the authors. could only solve a third of MIT undergrad problems but did well on other tests. https://twitter.com/alewkowycz/status/1542559200051593219 https://twitter.com/alewkowycz/status/1542559200051593219
- riku_iki 4y agoLooks like Polish tests were included into training dataset.
- 4y ago