11 ms·

Out of curiosity, I gave it the latest project euler problem published on 11/16/2025, very likely out of the training data Gemini thought for 5m10s before givi

by lairv 11mo ago

Out of curiosity, I gave it the latest project euler problem published on 11/16/2025, very likely out of the training data Gemini thought for 5m10s before giving me a python snippet that produced the correct answer. The leaderboard says that the 3 fastest human to solve this problem took 14min, 20min and 1h14min respectively Even thought I expect this sort of problem to very much be in the distribution of what the model has been RL-tuned to do, it's wild that frontier model can now solve in minutes what would take me days

orly01 11mo ago
Wow. Sounds pretty impressive.
qsort 11mo ago
To be fair a lot of the impressive Elo scores models get are simply due to the fact that they're faster: many serious competitive coders could get the same or better results given enough time. But seeing these results I'd be surprised if by the end of the decade we don't have something that is to these puzzles what Stockfish is to chess. Effectively ground truth and often coming up with solutions that would be absolutely ridiculous for a human to find within a reasonable time limit.
- nerdsniper 11mo ago
  I’d love if anyone could provide examples of such AND(“ground truth”, “absolutely ridiculous”) solutions! Even if they took clever humans a long time to create. I’m curious to explore such fun programming code. But I’m also curious to explore what knowledgeable humans consider to be both “ground truth” as well as “absolutely ridiculous” to create within the usual time constraints.
  - qsort 11mo ago
    I'm not explaining myself right. Stockfish is a superhuman chess program. It's routinely used in chess analysis as "ground truth": if Stockfish says you've made a mistake, it's almost certain you did in fact make a mistake[0]. Also, because it's incomparably stronger than even the very best humans, sometimes the moves it suggests are extremely counterintuitive and it would be unrealistic to expect a human to find them in tournament conditions. Obviously software development in general is way more open-ended, but if we restrict ourselves to puzzles and competitions, which are closed game-like environments, it seems plausible to me that a similar skill level could be achieved with an agent system that's RL'd to death on that task. If you have base models that can get there, even inconsistently so, and an environment where making a lot of attempts is cheap, that's the kind of setup that RL can optimize to the moon and beyond. I don't predict the future and I'm very skeptical of anybody who claims to do so, correctly predicting the present is already hard enough, I'm just saying that given the progress we've already made I would find plausible that a system like that could be made in a few years. The details of what it would look like are beyond my pay grade. --- [0] With caveats in endgames, closed positions and whatnot, I'm using it as an example.
    - pclmulqdq 11mo ago
      Yeah, it is often pointed out as a brilliance in game analysis if a GM makes a move that an engine says is bad and turns out to be good. However, it only happens in very specific positions.
      - emodendroket 11mo ago
        Does that happen because the player understands some tendency of their opponent that will cause them to not play optimally? Or is it genuinely some flaw in the machine’s analysis?
        pclmulqdq 11mo ago
        It can be either one. In closed positions, it is often the latter.
        thomasahle 11mo ago
        It's only the later if it's a weak browser engine, and it's early enough in the game that the player had studied the position with a cloud engine.
        nerdsniper 11mo ago
        Both, but perhaps more often neither. From what I've seen, sometimes the computer correctly assesses that the "bad" move opens up some kind of "checkmate in 45 moves" that could technically happen, but requires the opponent to see it 45 moves ahead of time and play something that would otherwise appear to be completely sub-optimal until something like 35 moves in, at which point normal peak grandmasters would finally go "oh okay now I get the point of all of that confusing behavior, and I can now see that I'm going to get mated in 10 moves". So, the computer is "right" - that move is worse if you're playing a supercomputer. But it's "wrong" because that same move is better as long as you're playing a human, who will never be able to see an absurd thread-the-needle forced play 45-75 moves ahead. That said, this probably isn't what GP was referring to, as it wouldn't lead to an assignment of a "brilliant" move simply for failing to see the impossible-to-actually-play line.
- vjerancrnjak 11mo ago
  How are they faster? I don’t think any ELO report actually comes from participating at a live coding contest on previously unseen problems.
  - qsort 11mo ago
    My background is more on math competitions, but all of those things are essentially speed contests. The skill comes from solving hard problems within a strict time limit. If you gave people twice the time, they'd do better, but time is never going to be an issue for a computer. Comparing raw Elo ratings isn't very indicative IMHO, but I do find it plausible that in closed, game-like environments models could indeed achieve the superhuman performance the Elo comparison implies, see my other comment in this thread.
thomasahle 11mo ago
I tried it with gpt-5.1 thinking, and it just searched and found a solution online :p
- lairv 11mo ago
  Is there a solution to this exact problem, or to related notions (renewal equation etc.)? Anyway seems like nothing beats training on test
thomasahle 11mo ago
I also used Gemini 3 Pro Preview. It finished it 271s = 4m31s. Sadly, the answer was wrong. It also returned 8 "sources", like stackexchange.com, youtube.com, mpmath.org, ncert.nic.in, and kangaroo.org.pk, even though I specifically told it not to use websearch. Still a useful tool though. It definitely gets the majority of the insights. Prompt: https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%5B%221_Uc2_gHGoYEZ8r1DeoPVDhDyg5a2PEBM%22%5D,%22action%22:%22open%22,%22userId%22:%22102648868016956856396%22,%22resourceKeys%22:%7B%7D%7D&usp=sharing https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%...
- JBiserkov 11mo ago
  The requested prompt does not exist or you do not have access. If you believe the request is correct, make sure you have first allowed AI Studio access to your Google Drive, and then ask the owner to share the prompt with you.
  - junon 11mo ago
    I thought this was a joke at first. It actually needs drive access to run someone else's prompt. Wild.
    - ashdksnndck 11mo ago
      On iOS safari, it just says “Allow access to Google Drive to load this Prompt”. When I run into that UI, my first instinct is that the poster of the link is trying to phish me. That they’ve composed some kind of script that wants to read my Google Drive so it can send info back to them. I’m only going to click “allow” if I trust the sender with my data. IMO, if that’s not what is happening, this is awful product design.
    - dormento 11mo ago
      Imagine the metrics though. "this quarter we've had a 12% increase on people using AI solutions in their google drive".
      - 11mo ago
id 11mo ago
gpt-5.1 gave me the correct answer after 2m 17s. That includes retrieving the Euler website. I didn't even have to run the Python script, it also did that.
sedatk 11mo ago
Just to clarify the context for future readers: the latest problem at the moment is #970: https://projecteuler.net/problem=970 https://projecteuler.net/problem=970
- deleted 11mo ago
  [deleted]
- tails4e 11mo ago
  I just had chatgpt explain that problem to me (I was unfamiliar with the mathematical background). It showed how to solve closed form answers for H(2) and H(3) and then numerical solutions using RK4 for higher values. Truly impressive, and it explained the derivations beautifully. There are few maths experts I've encountered who could have hand-held me through it as good.
  - lionkor 11mo ago
    Was the explanation correct?
    - philipwhiuk 11mo ago
      He has no idea because he's unfamiliar with the background.
      - tails4e 11mo ago
        I didn't understand the background before the explanation, but afterwards I did. Inl walked me through the mathematical steps and each was logical and ok to follow if you have a basic calculus knowledge.
    - tails4e 11mo ago
      It was. I asked it to give more details on parts of the derivation I didn't quite follow and it did that. Overall it was able to build from the ground up to the solution and solve it both numerically and analytically (for smaller values of x)
j2kun 11mo ago
Did it search the web?
rbjorklin 11mo ago
Your post made me curious to try a problem I have been coming back to ever since ChatGPT was first released: https://open.kattis.com/problems/low https://open.kattis.com/problems/low I have had no success using LLM's to solve this particular problem until trying Gemini 3 just now despite solutions to it existing in the training data. This has been my personal litmus test for testing out LLM programming capabilities and a model finally passed.
- deleted 11mo ago
  [deleted]
- kenjackson 11mo ago
  ChatGPT solves this problem now as well with 5.1. Time for a new litmus test.
irthomasthomas 11mo ago
Are you sure it did not retrieve the answer using websearch?
bumling 11mo ago
I asked Grok to write a Python script to solve this and it did it in slightly under ten minutes, after one false start where I'd asked it using a mode that doesn't think deeply enough. Impressive.
jamilton 11mo ago
Yeah, LLMs used to not be up to par for new Project Euler problems, but GPT-5 was able to do a few of the recent ones which I tried a few weeks ago.
panarky 11mo ago
[flagged]
- Xss3 11mo ago
  This comment was written by an AI specifically instructed to be more concise than usual.
- thejarren 11mo ago
  [flagged]
  - sindriava 11mo ago
    You seem very comfortable making unfounded claims. I don't think this is very constructive or adds much to the discussion. While we can debate the stylistic changes of the previous commenter, you seem to be discounting the rate at which the writing style of various LLMs has backpropagated into many peoples' brains.
    - roncesvalles 11mo ago
      Also discounting the fact that people actually do talk like that. In fact, these days I have to modify my prose to be intentionally less LLM-like lest the reader thinks it's LLM output.
    - ewoodrich 11mo ago
      I can sympathize with being mistakingly accused of using LLM output, but as a reader the above format of "Its not x - it's y" repeated multiple times for artificial dramatic emphasis to make a pretty mundane point that could use 1/3 the length grates on me like reading LinkedIn or marketing voice whether it's AI or not (and it's almost always AI anyway). I've seen fairly niche subreddits go from enjoyable and interesting to ruined by being clogged with LLM spam that sounds exactly like this so my tolerance for reading it is incredibly low, especially on HN, and I'll just dismiss it. I probably lose the occasionally legitimate original observation now and then but in a world where our attention is being hijacked by zero effort spam everywhere you look I just don't have the time or energy to avoid that heuristic.
bgwalter 11mo ago
Does it matter if it is out of the training data? The models integrate web search quite well. What if they have an internal corpus of new and curated knowledge that is constantly updated by humans and accessed in a similar manner? It could be active even if web search is turned off. They would surely add the latest Euler problems with solutions in order to show off in benchmarks.
- NaomiLehman 11mo ago
  you can disable search. just create a different problem if you don't believe it.
ivape 11mo ago
So when does the developer admit defeat? Do we have a benchmark for that yet?
- bgwalter 11mo ago
  According to a bunch of philosophers (https://ai-2027.com/ https://ai-2027.com/), doom is likely imminent. Kokotajlo was on Breaking Points today. Breaking Points is usually less gullible, but the top comment shows that "AI" hype strategy detection is now mainstream (https://www.youtube.com/watch?v=zRlIFn0ZIlU https://www.youtube.com/watch?v=zRlIFn0ZIlU): AI researcher: "Just another trillion dollars. This time we'll reach superintelligence, I swear."
  - kmeisthax 11mo ago
    Every Ai researcher calls it quits one YOLO run away from inventing a machine that turns all matter in the Universe into paperclips
    - whiteboardr 11mo ago
      Do NOT follow this link: https://www.decisionproblem.com/paperclips/ https://www.decisionproblem.com/paperclips/
lofaszvanitt 11mo ago
The problem is these models are optimized to solve the benchmarks, not real world problems.
motbus3 11mo ago
We need to wait and see. According to Google they have solved AI 10 years ago with Google Duo but somehow they keep smashing records despite being the worst coding tool until Gemini 2.5. Google internal benchmarks are irrelevant
NaomiLehman 11mo ago
definitely uses a lot of tooling. From "thinking": > I'm now writing a Python script to automate the summation computation. I'm implementing a prime sieve and focusing on functions for Rm and Km calculation [...]
mistercheph 11mo ago
If using through the chat interface are these models not doing some RAG?
blubber 11mo ago
Is this a problem for which the (human) solution is well documented an known and was learned during the training phase? Or is it a novel problem? I personally think anthropomorphizing LLMs is a bad idea.

by lairv 11mo ago

orly01 11mo ago

Wow. Sounds pretty impressive.

qsort 11mo ago

To be fair a lot of the impressive Elo scores models get are simply due to the fact that they're faster: many serious competitive coders could get the same or better results given enough time. But seeing these results I'd be surprised if by the end of the decade we don't have something that is to these puzzles what Stockfish is to chess. Effectively ground truth and often coming up with solutions that would be absolutely ridiculous for a human to find within a reasonable time limit.

nerdsniper 11mo ago
I’d love if anyone could provide examples of such AND(“ground truth”, “absolutely ridiculous”) solutions! Even if they took clever humans a long time to create. I’m curious to explore such fun programming code. But I’m also curious to explore what knowledgeable humans consider to be both “ground truth” as well as “absolutely ridiculous” to create within the usual time constraints.
- qsort 11mo ago
  I'm not explaining myself right. Stockfish is a superhuman chess program. It's routinely used in chess analysis as "ground truth": if Stockfish says you've made a mistake, it's almost certain you did in fact make a mistake[0]. Also, because it's incomparably stronger than even the very best humans, sometimes the moves it suggests are extremely counterintuitive and it would be unrealistic to expect a human to find them in tournament conditions. Obviously software development in general is way more open-ended, but if we restrict ourselves to puzzles and competitions, which are closed game-like environments, it seems plausible to me that a similar skill level could be achieved with an agent system that's RL'd to death on that task. If you have base models that can get there, even inconsistently so, and an environment where making a lot of attempts is cheap, that's the kind of setup that RL can optimize to the moon and beyond. I don't predict the future and I'm very skeptical of anybody who claims to do so, correctly predicting the present is already hard enough, I'm just saying that given the progress we've already made I would find plausible that a system like that could be made in a few years. The details of what it would look like are beyond my pay grade. --- [0] With caveats in endgames, closed positions and whatnot, I'm using it as an example.
  - pclmulqdq 11mo ago
    Yeah, it is often pointed out as a brilliance in game analysis if a GM makes a move that an engine says is bad and turns out to be good. However, it only happens in very specific positions.
    - emodendroket 11mo ago
      Does that happen because the player understands some tendency of their opponent that will cause them to not play optimally? Or is it genuinely some flaw in the machine’s analysis?
      - pclmulqdq 11mo ago
        It can be either one. In closed positions, it is often the latter.
      - thomasahle 11mo ago
        It's only the later if it's a weak browser engine, and it's early enough in the game that the player had studied the position with a cloud engine.
      - nerdsniper 11mo ago
        Both, but perhaps more often neither. From what I've seen, sometimes the computer correctly assesses that the "bad" move opens up some kind of "checkmate in 45 moves" that could technically happen, but requires the opponent to see it 45 moves ahead of time and play something that would otherwise appear to be completely sub-optimal until something like 35 moves in, at which point normal peak grandmasters would finally go "oh okay now I get the point of all of that confusing behavior, and I can now see that I'm going to get mated in 10 moves". So, the computer is "right" - that move is worse if you're playing a supercomputer. But it's "wrong" because that same move is better as long as you're playing a human, who will never be able to see an absurd thread-the-needle forced play 45-75 moves ahead. That said, this probably isn't what GP was referring to, as it wouldn't lead to an assignment of a "brilliant" move simply for failing to see the impossible-to-actually-play line.
vjerancrnjak 11mo ago
How are they faster? I don’t think any ELO report actually comes from participating at a live coding contest on previously unseen problems.
- qsort 11mo ago
  My background is more on math competitions, but all of those things are essentially speed contests. The skill comes from solving hard problems within a strict time limit. If you gave people twice the time, they'd do better, but time is never going to be an issue for a computer. Comparing raw Elo ratings isn't very indicative IMHO, but I do find it plausible that in closed, game-like environments models could indeed achieve the superhuman performance the Elo comparison implies, see my other comment in this thread.

thomasahle 11mo ago

I tried it with gpt-5.1 thinking, and it just searched and found a solution online :p

lairv 11mo ago
Is there a solution to this exact problem, or to related notions (renewal equation etc.)? Anyway seems like nothing beats training on test

thomasahle 11mo ago

I also used Gemini 3 Pro Preview. It finished it 271s = 4m31s. Sadly, the answer was wrong. It also returned 8 "sources", like stackexchange.com, youtube.com, mpmath.org, ncert.nic.in, and kangaroo.org.pk, even though I specifically told it not to use websearch. Still a useful tool though. It definitely gets the majority of the insights. Prompt: https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%5B%221_Uc2_gHGoYEZ8r1DeoPVDhDyg5a2PEBM%22%5D,%22action%22:%22open%22,%22userId%22:%22102648868016956856396%22,%22resourceKeys%22:%7B%7D%7D&usp=sharing https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%...

JBiserkov 11mo ago
The requested prompt does not exist or you do not have access. If you believe the request is correct, make sure you have first allowed AI Studio access to your Google Drive, and then ask the owner to share the prompt with you.
- junon 11mo ago
  I thought this was a joke at first. It actually needs drive access to run someone else's prompt. Wild.
  - ashdksnndck 11mo ago
    On iOS safari, it just says “Allow access to Google Drive to load this Prompt”. When I run into that UI, my first instinct is that the poster of the link is trying to phish me. That they’ve composed some kind of script that wants to read my Google Drive so it can send info back to them. I’m only going to click “allow” if I trust the sender with my data. IMO, if that’s not what is happening, this is awful product design.
  - dormento 11mo ago
    Imagine the metrics though. "this quarter we've had a 12% increase on people using AI solutions in their google drive".
    - 11mo ago

id 11mo ago

gpt-5.1 gave me the correct answer after 2m 17s. That includes retrieving the Euler website. I didn't even have to run the Python script, it also did that.

sedatk 11mo ago

Just to clarify the context for future readers: the latest problem at the moment is #970: https://projecteuler.net/problem=970 https://projecteuler.net/problem=970

deleted 11mo ago
[deleted]
tails4e 11mo ago
I just had chatgpt explain that problem to me (I was unfamiliar with the mathematical background). It showed how to solve closed form answers for H(2) and H(3) and then numerical solutions using RK4 for higher values. Truly impressive, and it explained the derivations beautifully. There are few maths experts I've encountered who could have hand-held me through it as good.
- lionkor 11mo ago
  Was the explanation correct?
  - philipwhiuk 11mo ago
    He has no idea because he's unfamiliar with the background.
    - tails4e 11mo ago
      I didn't understand the background before the explanation, but afterwards I did. Inl walked me through the mathematical steps and each was logical and ok to follow if you have a basic calculus knowledge.
  - tails4e 11mo ago
    It was. I asked it to give more details on parts of the derivation I didn't quite follow and it did that. Overall it was able to build from the ground up to the solution and solve it both numerically and analytically (for smaller values of x)

j2kun 11mo ago

Did it search the web?

rbjorklin 11mo ago

Your post made me curious to try a problem I have been coming back to ever since ChatGPT was first released: https://open.kattis.com/problems/low https://open.kattis.com/problems/low I have had no success using LLM's to solve this particular problem until trying Gemini 3 just now despite solutions to it existing in the training data. This has been my personal litmus test for testing out LLM programming capabilities and a model finally passed.

deleted 11mo ago
[deleted]
kenjackson 11mo ago
ChatGPT solves this problem now as well with 5.1. Time for a new litmus test.

irthomasthomas 11mo ago

Are you sure it did not retrieve the answer using websearch?

bumling 11mo ago

I asked Grok to write a Python script to solve this and it did it in slightly under ten minutes, after one false start where I'd asked it using a mode that doesn't think deeply enough. Impressive.

jamilton 11mo ago

Yeah, LLMs used to not be up to par for new Project Euler problems, but GPT-5 was able to do a few of the recent ones which I tried a few weeks ago.

panarky 11mo ago

[flagged]

Xss3 11mo ago
This comment was written by an AI specifically instructed to be more concise than usual.
thejarren 11mo ago
[flagged]
- sindriava 11mo ago
  You seem very comfortable making unfounded claims. I don't think this is very constructive or adds much to the discussion. While we can debate the stylistic changes of the previous commenter, you seem to be discounting the rate at which the writing style of various LLMs has backpropagated into many peoples' brains.
  - roncesvalles 11mo ago
    Also discounting the fact that people actually do talk like that. In fact, these days I have to modify my prose to be intentionally less LLM-like lest the reader thinks it's LLM output.
  - ewoodrich 11mo ago
    I can sympathize with being mistakingly accused of using LLM output, but as a reader the above format of "Its not x - it's y" repeated multiple times for artificial dramatic emphasis to make a pretty mundane point that could use 1/3 the length grates on me like reading LinkedIn or marketing voice whether it's AI or not (and it's almost always AI anyway). I've seen fairly niche subreddits go from enjoyable and interesting to ruined by being clogged with LLM spam that sounds exactly like this so my tolerance for reading it is incredibly low, especially on HN, and I'll just dismiss it. I probably lose the occasionally legitimate original observation now and then but in a world where our attention is being hijacked by zero effort spam everywhere you look I just don't have the time or energy to avoid that heuristic.

bgwalter 11mo ago

Does it matter if it is out of the training data? The models integrate web search quite well. What if they have an internal corpus of new and curated knowledge that is constantly updated by humans and accessed in a similar manner? It could be active even if web search is turned off. They would surely add the latest Euler problems with solutions in order to show off in benchmarks.

NaomiLehman 11mo ago
you can disable search. just create a different problem if you don't believe it.

ivape 11mo ago

So when does the developer admit defeat? Do we have a benchmark for that yet?

bgwalter 11mo ago
According to a bunch of philosophers (https://ai-2027.com/ https://ai-2027.com/), doom is likely imminent. Kokotajlo was on Breaking Points today. Breaking Points is usually less gullible, but the top comment shows that "AI" hype strategy detection is now mainstream (https://www.youtube.com/watch?v=zRlIFn0ZIlU https://www.youtube.com/watch?v=zRlIFn0ZIlU): AI researcher: "Just another trillion dollars. This time we'll reach superintelligence, I swear."
- kmeisthax 11mo ago
  Every Ai researcher calls it quits one YOLO run away from inventing a machine that turns all matter in the Universe into paperclips
  - whiteboardr 11mo ago
    Do NOT follow this link: https://www.decisionproblem.com/paperclips/ https://www.decisionproblem.com/paperclips/

lofaszvanitt 11mo ago

The problem is these models are optimized to solve the benchmarks, not real world problems.

motbus3 11mo ago

We need to wait and see. According to Google they have solved AI 10 years ago with Google Duo but somehow they keep smashing records despite being the worst coding tool until Gemini 2.5. Google internal benchmarks are irrelevant

NaomiLehman 11mo ago

definitely uses a lot of tooling. From "thinking": > I'm now writing a Python script to automate the summation computation. I'm implementing a prime sieve and focusing on functions for Rm and Km calculation [...]

mistercheph 11mo ago

If using through the chat interface are these models not doing some RAG?

blubber 11mo ago

Is this a problem for which the (human) solution is well documented an known and was learned during the training phase? Or is it a novel problem? I personally think anthropomorphizing LLMs is a bad idea.