5 ms·

I find that MATH challenge “solved” by AI hard to believe. The reason given was “saturation”. Could anybody help explain it a bit? Also in my daily encounter, I

I find that MATH challenge “solved” by AI hard to believe. The reason given was “saturation”. Could anybody help explain it a bit? Also in my daily encounter, I stop find a lot of simple math problems all the frontier models could not solve: long logic puzzle, many cases reasoning, and particularly geometry problems. I don’t know where the 97% number for o1 does come from, but in my experience they are much lower than that and math, even elementary maths certainly can not be considered to be “solved”. As far as I can see, OpenAI has been trained their models on all these public problems, so testing on them to record a benchmark is tainted as best when not outright cheating.

Taek 2y ago
I've found o1 to be entirely useful at math problems that are beyond my own (admittedly modest) skills. I've had it write full proofs of correctness for me (one shot, verified), I've had it optimize equations to reduce necessary precision, I've had it optimize equations to remove specific expensive operations (making them computationally more efficient), and finally I've had it prove a handful of my conjectures, which was helpful for taking algorithmic shortcuts in a security sensitive environment. Mostly all algebra and calculus, but definitely all problems that most undergrads would struggle with. It's most useful because it has deep knowledge of related and adjacent conjectures that are well understood, even if you've never heard of them. So it can mix and match things with a lot more ease than a tinkering mathematician
- hgomersall 2y ago
  Yeah, I had (so far apparent, but still be verified) success with o1 teaching me the necessary physics and maths I need to solve my specific problem. This is definitely grad level stuff but well understood. My concern though is it's missing things that are more esoteric.
- gazchop 2y ago
  It can't handle trigonometric identities and any form of calculus at the same time without fucking it up. Also abstract stuff like symmetry groups, nope! And anything which involves vectors is a mess. The big problem is it confidently answers the questions utterly wrongly. This is stuff I expect a basic mathematics undergrad to be able to work out in their first or second year.
blinding-streak 2y ago
Scroll down on the page. It explains saturation.

Taek 2y ago

I've found o1 to be entirely useful at math problems that are beyond my own (admittedly modest) skills. I've had it write full proofs of correctness for me (one shot, verified), I've had it optimize equations to reduce necessary precision, I've had it optimize equations to remove specific expensive operations (making them computationally more efficient), and finally I've had it prove a handful of my conjectures, which was helpful for taking algorithmic shortcuts in a security sensitive environment. Mostly all algebra and calculus, but definitely all problems that most undergrads would struggle with. It's most useful because it has deep knowledge of related and adjacent conjectures that are well understood, even if you've never heard of them. So it can mix and match things with a lot more ease than a tinkering mathematician

hgomersall 2y ago
Yeah, I had (so far apparent, but still be verified) success with o1 teaching me the necessary physics and maths I need to solve my specific problem. This is definitely grad level stuff but well understood. My concern though is it's missing things that are more esoteric.
gazchop 2y ago
It can't handle trigonometric identities and any form of calculus at the same time without fucking it up. Also abstract stuff like symmetry groups, nope! And anything which involves vectors is a mess. The big problem is it confidently answers the questions utterly wrongly. This is stuff I expect a basic mathematics undergrad to be able to work out in their first or second year.

blinding-streak 2y ago

Scroll down on the page. It explains saturation.