2 ms·
I remember there being a lot of skepticism about the performance metrics used in that paper last time this topic came up. Vaguely, I think they were comparing t
by whatshisface 7y ago
I remember there being a lot of skepticism about the performance metrics used in that paper last time this topic came up. Vaguely, I think they were comparing the runtime of Wolfram's perfect answer search to their own model which was not required to get the answer exactly right. I think there were also some other shenanigans.
- throwlaplace 7y agoI was a critic when this was first posted (and was promptly downvoted). I don't understand why this is getting traction (or even a research direction they decided to pursue). does anyone really believe this thing is doing some kind of symbolic reasoning? it's doing nothing but memorizing training examples! that it has nonzero performance in test is owing to the wonky metric.
- whatshisface 7y agoOn one hand you have 1980s symbolic AI that can produce true, coherent statements, and on the other you have GPT-2 which can produce really nifty gibberish. If you could put the two together maybe that would be some progress. I can see that getting transformers to emulate symbolic math might be a step in that direction.
- throwlaplace 7y agothat's not reasoning though. it's not even an expert system or prolog (because adversarial examples exist). it's very close to a million monkeys typing.
- deleted 7y ago[deleted]
- monocasa 7y agoWhere does a 100B neurons firing fit on to that spectrum?
- jfkebwjsbx 7y agoI am not one that hypes AI and ML, but I don't see the problem with this. If this new system can integrate some equations that other methods cannot, I welcome it. It is almost instantaneous to check the solutions, so what is the problem? I imagine the devs at Wolfram Mathematica are already working into adding this new kind of solvers into their overall integrator.
- martincmartin 7y agoThis is also required to get the answer exactly right. It's just that the problem domains they chose -- symbolic integration and differential equations -- can be checked easily. That is, if I have a guess F for the integral of f, I just need to symbolically differentiate F and see whether I get f.* So the paper does "guess and check," and only returns answers that pass the check, i.e. are exactly right. There are some problems where all the checks fail, i.e. the paper can't generate an answer. And there are also questions where Matlab / Maple can't find an answer either. * It's a little more complicated; e.g. if f = cos^2, and the derivative of F is 1 - sin^2, I need to recognize they're the same. But it's safe, in the sense that if I can't determine whether they're the same, I can reject the solution and move on to the next one.
- whatshisface 7y agoWait a minute, doesn't that mean your checker is doing symbolic algebra?
- jfkebwjsbx 7y agoYes, what is the issue?
- thomasahle 7y agoYes, but the algebra required to check the equations is much less than what's required to solve them.
- martincmartin 7y agoYes, but symbolic differentiation is straight forward. There are simple rules to apply to differentiate any expression: the chain rule, the product rule, the quotient rule, derivatives of known single argument functions, etc. Integration is a completely different ball game. Given an arbitrary expression, most likely it's not symbolically integrable, i.e. it's integral can't be expressed as a combination of known functions. That's why there are lots of functions defined as "the integral of blah." Even given an expression that's known to be symbolically integrable, it's quite the puzzle to figure out what the integral is. There's no straight forward algorithm, as with differentiation. It boils down to "try integration by parts; try this; try that; maybe do some clever algebraic manipulation to get it into a form where you can try one of those things; look it up in a book of integrals to see if it's there, or something similar is there; if all else fails, guess and check."
- fcharton 7y agoSince the paper was presented, it was reviewed in ICLR and significantly expanded, and many questions raised in September were addressed. The performance metric is the number of equations/integrals correctly solved over a held out test set, generated randomly. This is done using an external tool (SymPy), which some might consider "cheating" (but you need a way to test, don't you?). Anyway, keep in mind that checking a solution for correctness is much easier than finding one. Another issue was about using timeout for Wolfram. This is now discussed in the appendix (it makes very little difference). During review, an interesting point was discussed: out of distribution generalization, or how performance depends on the random problem generator we used. This is now measured and discussed at the end of the appendix.
- whatshisface 7y ago>During review, an interesting point was discussed: out of distribution generalization, or how performance depends on the random problem generator we used. This is now measured and discussed at the end of the appendix. Your appendix E poses a really interesting question that I hope future research addresses. The distribution of integrals that appear on math tests are likely to closely match FWD, BWD and IBP because that's how professors write problems for students. However the distribution of integrals that arise in physics and engineering is quite different and includes a lot of ones that aren't solvable in closed form. I wonder what the performance of the system would be on the "engineering distribution." I also wonder what you would have to do to find out what the engineering distribution was...
- fcharton 7y agoTextbook problems are usually short, with short solutions, and demonstrating one specific rule. They are better handled by classical (rule-based) tools. Deep learning tools would either memorize them or resort to a rule based sub-module. For integrals, solvability depends on the function set you work with. Since we use elementary functions on the real domain, a lot of integrals have no solution. We could have gone for a larger set (adding erf, the Fresnels, up to Liouvillian functions). This would mean more solvable cases. As for the engineering distribution, no one knows what it is. The best we can do is to generate a broader training set, knowing that it will generalize better (this is the key takeaway of our appendix). BWD+IBP is a step in this direction, but to progress further, we need a better understanding of the problem space, and issues related to simplification. We are working on this now.