8 ms·
I've been using a math puzzle as a way to benchmark the different models. The math puzzle took me ~3 days to solve with a computer. A math major I know took abo
by malisper 2y ago
I've been using a math puzzle as a way to benchmark the different models. The math puzzle took me ~3 days to solve with a computer. A math major I know took about a day to solve it by hand.
Gemini 2.5 is the first model I tested that was able to solve it and it one-shotted it. I think it's not an exaggeration to say LLMs are now better than 95+% of the population at mathematical reasoning.
For those curious the riddle is: There's three people in a circle. Each person has a positive integer floating above their heads, such that each person can see the other two numbers but not his own. The sum of two of the numbers is equal to the third. The first person is asked for his number, and he says that he doesn't know. The second person is asked for his number, and he says that he doesn't know. The third person is asked for his number, and he says that he doesn't know. Then, the first person is asked for his number again, and he says: 65. What is the product of the three numbers?
- bbstats 2y agoCould you explain "The sum of two of the numbers is equal to the third"??
- deleted 2y ago[deleted]
- malisper 2y agoif the three numbers are a, b, and c, then either a+b=c, a+c=b, or b+c=a
- bena 2y agoAnd they must all be positive integers. So A + B = C and A + C = B. But we know that A + B = C, so we can replace C with (A + B). So we know that A + A + B = B. So 2A + B = B. Or 2A = 0. And this holds any way you slice it. Even if you were to try and brute force it. A = 1 B = 2 Then C = 3. But A + C has to equal B. That's 1 + 3 = 2? That's not true. I don't see a case where you can add to the sum of two numbers one of the numbers and get the other number. I'm guessing that's a misreading of the problem. Because it looks like the third number is the sum of the first two.
- refulgentis 2y agoOne of the cases has to be true, not all 3. (as you show, they're mutually exclusive for positive integers) i.e. "either" is important in the parent comment.
- bena 2y agoWhich is why I indicated that it would be a misreading of the problem. The original problem is a little ambiguously worded. You could say "one of their numbers is the sum of the other two" and it would be a little clearer.
- deleted 2y ago[deleted]
- thaumasiotes 2y ago> The original problem is a little ambiguously worded. No it isn't. If it said "the sum of any two of the numbers is equal to the third", that would be a contradiction. What it says is "the sum of two of the numbers is equal to the third".
- bena 2y agoI have three items. Buying two of the items gets you the third for free. The implication is any two. It’s ok that it’s ambiguous. It happens. In most cases, we clarify and move on. There’s no need to defend it.
- thaumasiotes 2y agoWhy look for ambiguity that isn't there?
- refulgentis 2y agoThere's a certain mind that either doesn't realize they're sidestepping the problem and turning it into a editing review, or realizes it, and doesn't understand why it seems off-topic/trivial to others. What's especially strange here is, they repeatedly demonstrate if you interpret it that way, the problem is obviously, trivially, unsolvable, in a way that a beginner in algebra could intuit. (roughly 12 years old, at least, we started touching algebra in 7th grade) I really don't get it. When I've seen this sort of thing play out this way, the talking-down is usually for the benefit of demonstrating something to an observer (i.e. I am smart look at this thing I figured out; I can hold my own when the haters chirp; look they say $INTERLOCUTOR is a thinker but they can't even understand me!), but ~0 of that would apply here, at least traditionally.
- refulgentis 2y agoGiven #s x,y, and z, either x + y = z, x + z = y, or y + z = x.
- rappatic 2y agoI think: Call the three numbers a, b, and c. This means c = a + b, but we still don’t know to which person each number belongs. When person 1 (p1) is asked what his number is, he has no way to know whether he has a, b, or c, so he says he doesn’t know. Same goes for p2 and p3. Clearly p1 somehow gains information by p2 and p3 passing. Either he realizes that he must be either a or b, and such his number is the difference between p2 and p3’s numbers, or he realizes that he must be c and so his number is the sum of p2 and p3’s numbers. That’s all I have so far. Anyone have other ideas?
- deleted 2y ago[deleted]
- aardvarkr 2y agoI think it has something to do with applying the lower bound of 1. If p1 KNOWS that he’s the largest then he has to have gained some other piece of information. Say the numbers he sees are 32 and 33. His number would have to be either 1 or 65. If p1 was 1 then the other two would have known p1 couldn’t be the sum of the other two
- oezi 2y agoBut p2 and p3 don't yet know what they are themselves just because they see a 1: If p2 sees 1 and 33, s/he would wonder if s/he is 32 or 34. P3 would consider 31 or 33.
- bena 2y agoThe answer is online and it's clever. P1 knows that P2 and P3 are not equal. So they know that the set isn't [2A, A, A]. P2 knows that P1 and P3 are not equal. So they know that the set isn't [A, 2A, A]. They also know that if P1 doesn't know, then they were able to make the same deduction. So they now know that both [2A, A, A] and [A, 2A, A] aren't correct. Since they know that [2A, A, A] isn't correct, they can also know that [2A, 3A, A] isn't correct either. Because they'd be able to see if P1 = 2A and P3 = A, and if that were true and P1 doesn't know their number, it would have to be because P2 isn't A. And if P2 isn't A, they'd have to be 3A. P3 knows that P1 and P2 aren't equal. Eliminates [A, A, 2A]. Knows that [2A, A, A], [A, 2A, A], and [2A, 3A, A], are eliminated. Using the same process as P2, they can eliminate [2A, A, 3A], [A, 2A, 3A], and also [2A, 3A, 5A]. Because they can see the numbers and they know if P1 is 2A and P2 is 3A. Now we're back at P1. Who now knows. So P2 and P3 are in the eliminated sets. Which means we're one of these [2A, A, A]; [3A, 2A, A]; [4A, 3A, A]; [3A, A, 2A]; [4A, A, 3A]; [5A, 2A, 3A]; [8A, 3A, 5A] We know his number is 65. To find the set, we can factor 65: (5 * 13). We can check the other numbers 2(13) = 26. 3(13) = 39. And technically, you don't need to find the other numbers. The final answer is 5A * 2A * 3A or (A^3) * 30.
- g105b 2y agoPlease can you enlighten me, I'm a mathematic plebian?
- sebzim4500 2y agoThis is a great riddle. Unfortunately, I was easily able to find the exact question with a solution (albeit with a different number) online, thus it will have been in the training set.
- deleted 2y ago[deleted]
- varispeed 2y agoSeems like we might need a section of internet that is off limits to robots.
- Centigonal 2y agoeveryone with limited bandwidth has been trying to limit site access to robots. the latest generation of AI web scrapers are brutal and do not respect robots.txt
- varispeed 2y agoThere are websites where you can only register to in person and have two existing members vouch for you. Probably still can be gamed, but sounds like a great barrier to entry for robots (for now).
- tmpz22 2y agoWhat prevents someone from getting access and then running an authenticated headless browser to scoop the data?
- varispeed 2y agoAdmins will see unusual traffic from that account and then take action. Of course it will not be perfect as there could be a way to mimic human traffic and slowly scrape the data anyway, that's why there is element of trust (two existing members to vouch).
- hmottestad 2y agoIs the answer somehow {65, 20, 45} with the product 58,500? That’s one-shot for o1 pro.
- sebzim4500 2y agoThat's wrong. From player 1's perspective {25 20 45} is entirely consistent with the calls made in the first three rounds.
- refulgentis 2y agoIn general I find commentary here too negative on AI, but I'm a bit squeamish about maximalist claims re: AI mathematical reasoning vs. human population based off this, even setting aside lottery-ticket-hypothesis-like concerns. It's a common logic puzzle, Google can't turn up an exact match to the wording you have, but ex. here: https://www.futilitycloset.com/2018/03/03/three-hat-problem/ https://www.futilitycloset.com/2018/03/03/three-hat-problem/
- TrackerFF 2y agoThe riddle certainly nerd-sniped GPT 4.5 After a couple of minutes it decided on the answer being 65000. (S = {65, 40, 25)}
- semiinfinitely 2y agoI love how the entire comment section is getting one-shotted by your math riddle instead of the original post topic.
- hmottestad 2y agoThis looks like it’s been posted on Reddit 10 years ago: https://www.reddit.com/r/math/comments/32m611/logic_question_that_has_me_stumped/ https://www.reddit.com/r/math/comments/32m611/logic_question... So it’s likely that it’s part of the training data by now.
- canucker2016 2y agoYou'd think so, but both Google's AI Overview and Bing's CoPilot output wrong answers. Google spits out: "The product of the three numbers is 10,225 (65 * 20 * 8). The three numbers are 65, 20, and 8." Whoa. Math is not AI's strong suit... Bing spits out: "The solution to the three people in a circle puzzle is that all three people are wearing red hats." Hats??? Same text was used for both prompts (all the text after 'For those curious the riddle is:' in the GP comment), so Bing just goes off the rails.
- moritzwarhier 2y agoThat's a non-sequitur, they would be stupid to run ab expensive _L_LM for every search query. This post is not about Google Search being replaced by Gemini 2.5 and/or a chatbot.
- canucker2016 2y agoGoogle placed its "AI overview" answer at the top of the page. The second result is this reddit.com answer, https://www.reddit.com/r/math/comments/32m611/logic_question_that_has_me_stumped/ https://www.reddit.com/r/math/comments/32m611/logic_question..., where at least the numbers make sense. I haven't examined the logic portion of the answer. Bing doesn't list any reddit posts (that Google-exclusive deal) so I'll assume no stackexchange-related sites have an appropriate answer (or bing is only looking for hat-related answers for some reason).
- moritzwarhier 2y agoI might have been phrasing poorly. With _L_ (or L as intended), I meant their state-of-the-art model, which I presume Gemini 2.5 is (didn't come around to TFA yet). Not sure if this question is just about model size. I'm eagerly awaiting an article about RAG caching strategies though!
- ototot 2y agoI also tried one-shot. https://g.co/gemini/share/badd00a824d2 https://g.co/gemini/share/badd00a824d2
- yifanl 2y agoYou'd have better results if you had prompted it with the actual answer and asked how the first person came to the conclusion. Giving a number in the training set is very easy. i.e. You observe three people in a magical room. The first person is standing underneath a 65, the second person is standing underneath a 26 and the third person is standing underneath a 39. They can see the others numbers but not the one they are directly under. You tell them one of the three numbers is the sum of the other two and all numbers are positive integers. You ask the first person for their number, they respond that they don't know. You ask the second person for their number, they respond that they don't know. You ask the third person, they respond that they don't know. You ask the first person again and they respond with the correct value, how did they know? And of course, if it responds with a verbatim answer in the line of https://www.reddit.com/r/math/comments/32m611/logic_question_that_has_me_stumped/cqckk8a/ https://www.reddit.com/r/math/comments/32m611/logic_question..., we can be pretty confident what's happening under the hood.
- deleted 2y ago[deleted]
- SwayStar123 2y agoYou can also put the AI in the first person's shoes. Prompt: You are standing in a circle, there are 2 other people in the circle with you, everyone in the circle, has a positive integer above their head, no one knows what the number above their own head is but can see the numbers above the heads of the other people. You see that the person infront of you on the left has 26 above their head. The person on the right has 39 above their head. You are told that the sum of two of the numbers is the third number. You are asked what the number above your head is, the option is the sum, 65, or 13, as 26 + 13 = 39. You don't know which one it is, and you say so. The second person is asked the number above their head. They also say they dont know, the third person also says they dont know. What is your number? Gemini 2.5 and claude 3.7 thinking get it right, o3 mini and 4o get it wrong
- adpirz 2y agoInteractive playground for the puzzle: https://claude.site/artifacts/832e77d7-5f46-477c-a411-bdad10059304 https://claude.site/artifacts/832e77d7-5f46-477c-a411-bdad10... (All state is stored in localStorage so you can come back to it :) ).
- doener 2y agoChatGPT 4.5: Let’s carefully analyze this puzzle step-by-step: ⸻ Step 1: Define the Problem Clearly We have three people in a circle. Let’s call them: • Person A (first person) • Person B (second person) • Person C (third person) Each has a positive integer above their head. Each sees the other two numbers but not their own. The numbers satisfy: A + B = C \quad \text{or} \quad B + C = A \quad \text{or} \quad C + A = B Given the puzzle scenario, we need to find out the exact numbers given the clues, and ultimately find the product of the three numbers. ⸻ Step 2: Logic of the Puzzle (Step-by-Step Reasoning) Initial conditions: The numbers are positive integers. The puzzle implies a situation of ambiguity at first, but after multiple rounds of “I don’t know,” Person A finally knows their number is 65. Let’s denote clearly the conditions again: • Exactly one of the following three equations is true: • A + B = C, or • B + C = A, or • C + A = B. Person A sees B and C, but not A. Similarly for Persons B and C. ⸻ Step 3: Logical deductions from the repeated “I don’t know” First round deductions: • Person A initially doesn’t know their number. This means, from Person A’s perspective, both scenarios A = B + C and A + B = C (or A + C = B) are still possible. If Person A had seen identical numbers, for instance, Person A could have instantly known their number. Thus, Person A sees two different numbers. • Similarly, Person B doesn’t know either, implying that Person B also sees ambiguity in their own number. Thus, Person B also sees two numbers that could result in at least two possible solutions. • Person C also doesn’t know, same logic as above. Thus, after one full round, we understand that initially, all three numbers must create ambiguity. No trivial case (like two identical numbers) is present. Second round deductions: After hearing that no one knows their number, each person gains information: they now know the scenario is one in which none of the other two people could have immediately solved it. Crucially, in the second round: • Person A now knows their number is 65 after considering the previous statements. This implies that Person A sees numbers B and C, and from these numbers plus the statements of uncertainty, Person A deduces that their own number can only be 65. ⸻ Step 4: Deriving the Numbers (Critical deduction) Let’s denote clearly again: Person A’s number is now known to be 65. We have three possible equations. Let’s test each carefully given A = 65: 1. Case 1: A + B = C Then we have: • 65 + B = C. Person A sees numbers B and C, so they would see B and 65 + B. Person A sees numbers B and 65+B. If Person A sees these two numbers, they might initially consider their number could be either the sum or difference of these two. • However, if Person A sees two numbers, and one is exactly 2× the other, Person A could immediately guess their number is the difference (since if you see numbers X and 2X, your number is clearly X). So this scenario of B and 65+B is okay only if these two numbers don’t exhibit an immediate trivial scenario. • Let’s hold this scenario and continue examining others first. 2. Case 2: B + C = A (So, B + C = 65) • Then Person A sees numbers B and C, adding exactly to 65. • Initially, Person A sees two numbers summing exactly to 65. Could Person A know their number immediately at first glance? No, since their number could either be the sum (65) or the difference of the two numbers. Ambiguity here initially makes sense. • Initially, Person B and Person C also see ambiguous scenarios. They both don’t know their numbers either at first round. This scenario is plausible. 3. Case 3: C + A = B (So, C + 65 = B) • Then Person A sees numbers B = C + 65 and C. • Person A sees C and C+65. Similar reasoning as Case 1. ⸻ Step 5: Identifying the Correct Scenario The puzzle’s logic typically aligns best with Case 2: B + C = A = 65. Here’s why clearly stated: • Initially, Person A sees two numbers adding to exactly 65. At first glance, Person A’s number could either be 65 (sum) or the difference between the two numbers. • Person B also initially sees two numbers and cannot distinguish immediately their own number, nor can Person C. • After everyone states “I don’t know,” Person A gains critical insight: if the difference between the two numbers (that A sees) allowed Person B or Person C to know their number immediately, they’d have known right away. The fact that B and C didn’t know means the difference is not immediately obvious to B or C. Thus, the difference cannot be trivial (such as identical numbers or easy multiples). Therefore, the two numbers must be such that their difference doesn’t immediately clue in the other persons. This occurs most clearly with pairs of numbers which have multiple factorization possibilities or aren’t trivial multiples. But we must carefully test quickly to find a suitable, plausible integer pair whose sum is 65 and whose ambiguity would logically survive through multiple rounds. The puzzle is a classic logic puzzle with a known structure: A common scenario in these classic puzzles is the use of Fibonacci-like or similar “golden” sets. Let’s quickly check the following well-known ambiguous triple: • Numbers: 65, 39, 26 • Check quickly: indeed, 39 + 26 = 65. Checking logic quickly: • Person A sees 39 and 26. Initially ambiguous (number could be 65 or 13). • Person B sees 65 and 26. Could be either 39 or 91 at first glance. Initially ambiguous. • Person C sees 65 and 39. Could be either 26 or 104 initially. Initially ambiguous. After each person says they don’t know: • Now, Person A considers: If my number was 13, then Person B would see numbers 13 and 26. Person B would then immediately know their number is 39 (13+26=39), no ambiguity. But Person B didn’t know! So Person A’s number cannot be 13. Hence, Person A confidently concludes their number is 65. Thus, the numbers 65, 39, and 26 satisfy the puzzle conditions perfectly! ⸻ Step 6: Final Answer (Product of the Numbers) The three numbers identified clearly are: • A = 65 • B = 39 • C = 26 Then, the product is: 65 \times 39 \times 26 = 65 \times 1014 = 65,910 ⸻ Final Answer: \boxed{65,910}
- _cs2017_ 2y agoThis is solvable in roughly half an hour on pen and paper by a random person I picked with no special math skills (beyond a university). This is far from a difficult problem. The "95%+" in math reasoning is a meaningless standard, it's like saying a model is better than 99.9% of world population in Albanian language, since less than 0.1% bother to learn Albanian. Even ignoring the fact that this or similar problem may have appeared in the training data, it's something a careful brute-force math logic should solve. It's neither difficult, nor interesting, nor useful. Yes, it may suggest a slight improvement on the basic logic, but no more so than a million other benchmarks people quote. This goes to show that evaluating models is not a trivial problem. In fact, it's a hard problem (in particular, it's a far far harder than this math puzzle).
- windowshopping 2y agoThe "random person" you picked is likely very, very intelligent and not at all a good random sample. I'm not saying this is difficult to the extent that it merits academic focus, but it is NOT a simple problem and I suspect less than 1% of the population could solve this in half an hour "with no special math skills." You have to be either exceedingly clever or trained in a certain type of reasoning or both.
- sundarurfriend 2y agoI agree with your general point that this "random person" is probably not representative of anything close to an average person off the street, but I think the phrasing "very very intelligent" and "exceedingly clever" is kinda misleading. In my experience, the difference between someone who solves this type of logic puzzle and someone who doesn't, has more to do with persistence and ability to maintain focus, rather than "intelligence" in terms of problem-solving ability per se. I've worked with college students helping them learn to solve these kinds of problems (eg. as part of pre-interview test prep), and in most cases, those who solve it and those who don't have the same rate of progress towards the solution as long as they're actively working at it. The difference comes in how quickly they get frustrated (at themselves mostly), decide they're not capable of solving it, and give up on working on it further. I mention this because this frustration itself comes from a belief that the ability to solve these belongs some "exceedingly clever" people only, and not someone like them. So, this kind of thinking ends up being a vicious cycle that keeps them from working on their actual issues.
- highfrequency 2y agoFun puzzle! I’m curious how you managed to structure the problem such that a computer could solve it but it took 3 days of computation?
- dkjaudyeqooe 2y ago> I think it's not an exaggeration to say LLMs are now better than 95+% of the population at mathematical reasoning. It's not an exaggeration it's a non-sequitur, you first have to show that the LLMs are reasoning in the same way humans do.
- r0fl 2y agoWow Tried this in deepseek and grok and it kept thunking in loops for a while and I just turned it off I haven’t seen a question loop this long ever. Very impressed
- deepboy2 2y agoJust tried it on Deepseek (not R1, maybe V3-0324) and got the correct answer after 7-8 pages of reasoning. Incredible!
- z2 2y agoDeepseek R1 got the right answer after a whopping ~10 minutes of thinking. I'm impressed and feel kind of dirty, I suspect my electricity use from this could have been put to better use baking a frozen pizza.
- utopcell 2y agoSame here: My problem of choice is the 100 prisoners problem [1]. I used to ask simple reasoning questions in the style of "what is the day three days before the day after tomorrow", but nowadays when I ask such questions, I can almost feel the the NN giggling at the naivety of its human operator. [1] https://en.wikipedia.org/wiki/100_prisoners_problem https://en.wikipedia.org/wiki/100_prisoners_problem
- mitko 2y agoLoved that puzzle, thanks for sharing it. I’ve solved a lot of math problems in the past but this one had a unique flavor of interleaving logical reasoning, partial information and a little bit of arithmetic.
- eru 2y agoI use an algorithmic question that I'd been working on for years and that I'm finally writing up the answer to. It's basically: given a sequence of heap operations (insert element, delete minimum element), can you predict the left-over elements (that are in the heap at the end) in linear time in the comparison model? (The answer is surprisingly: Yes.)
- integralof5y 2y agoA prolog program, swipl (it takes less than a second to solve your puzzle) N is number of turns of don't know answers. the bad predicate means that the person can know its number at turn N. bad(_,_,_,-1) :- !,false. bad(_,A,A,0) :- !. bad(A,_,A,0) :- !. bad(A,A,_,0) :- !. bad(B,C,A,N) :- D is abs(B-A),D<C,N1 is N-1, bad(B,D,A,N1),!. bad(C,A,B,N) :- D is abs(B-A),D<C,N1 is N-1, bad(D,A,B,N1),!. bad(A,B,C,N) :- D is abs(B-A),D<C,N1 is N-1, bad(A,B,D,N1),!. solve(X,Y,Z) :- Y1 is X-1, between(1,Y1,Y), between(0,2,N), Z is X-Y,bad(X,Y,Z,N). ?- solve(65,X,Y). X = 26, Y = 39 ; X = 39, Y = 26 .
- drewbeck 2y agoI just asked it this twice and it gave me 65×65×130=549250. Both times. The first time I made it about ducks instead of people and mentioned that there was a thunderstorm. The second time I c/p your exact text and it gave me the same answer. Again we find that the failure state of LLMs is a problem – yeah, when you know the answer already and it gets it right, that's impressive! When it fails, it still acts the same exact way and someone who doesn't already know the answer is now a lil stupider.