5 ms·
I'm sure this is a very impressive model, but gemini-3-pro-preview is failing spectacularly at my fairly basic python benchmark. In fact, gemini-2.5-pro gets a
by prodigycorp 11mo ago
I'm sure this is a very impressive model, but gemini-3-pro-preview is failing spectacularly at my fairly basic python benchmark. In fact, gemini-2.5-pro gets a lot closer (but is still wrong).
For reference: gpt-5.1-thinking passes, gpt-5.1-instant fails, gpt-5-thinking fails, gpt-5-instant fails, sonnet-4.5 passes, opus-4.1 passes (lesser claude models fail).
This is a reminder that benchmarks are meaningless – you should always curate your own out-of-sample benchmarks. A lot of people are going to say "wow, look how much they jumped in x, y, and z benchmark" and start to make some extrapolation about society, and what this means for others. Meanwhile.. I'm still wondering how they're still getting this problem wrong.
edit: I've a lot of good feedback here. I think there are ways I can improve my benchmark.
- m00dy 11mo agothat's why everyone using AI for code should code in rust only.
- Filligree 11mo agoWhat's the benchmark?
- petters 11mo agoGood personal benchmarks should be kept secret :)
- ahmedfromtunis 11mo agoI don't think it would be a good idea to publish it on a prime source of training data.
- Hammershaft 11mo agoHe could post an encrypted version and post the key with it to avoid it being trained on?
- shawabawa3 11mo agobut they've asked all the AI models this question. Whatever you tell an AI model is also in its training data
- mlrtime 11mo agoThis sounds like paranoia to me to be honest. Please tell me I'm wrong. I could have easily come up with just the same claim, without seeing the benchmark, it doesn't exist. Maybe if we weren't anonymous and your profile leads to credentials that you have experience in this field, otherwise I don't believe it without seeing/testing myself.
- prodigycorp 11mo agonice try!
- ankit219 11mo agoyou already sent the prompt to gemini api - and they likely recorded it. So in a way they can access it anyway. Posting here or not would not matter in that aspect.
- GuB-42 11mo agoNIBBLES.BAS maybe [1] If you make some assumptions about the species of the snake, it can count as a basic python benchmark ;) [1] https://en.wikipedia.org/wiki/Nibbles_(video_game) https://en.wikipedia.org/wiki/Nibbles_(video_game)
- mupuff1234 11mo agoCould also just be rollout issues.
- prodigycorp 11mo agoCould be. I'll reply to my comment later with pass/fail results of a re-run.
- ddalex 11mo agoI moved to using the model from python coding to golang coding and got incredible speedups in writing the correct version of the code
- layer8 11mo agoIs observed speed meaningful for a model preview? Isn’t it likely to go down once usage goes up?
- mring33621 11mo agoI agree that benchmarks are noise. I guess, if you're selling an LLM wrapper, you'd care, but as a happy chat end-user, I just like to ask a new model about random stuff that I'm working on. That helps me decide if I like it or not. I just chatted with gemini-3-pro-preview about an idea I had and I'm glad that I did. I will definitely come back to it. IMHO, the current batch of free, free-ish models are all perfectly adequate for my uses, which are mostly coding, troubleshooting and learning/research. This is an amazing time to be alive and the AI bubble doomers that are costing me some gains RN can F-Off!
- testartr 11mo agoand models are still pretty bad at playing tic-tac-toe, they can do it, but think way too much it's easy to focus on what they can't do
- deleted 11mo ago[deleted]
- big-and-small 11mo agoEverything is about context. When you just ask non-concrete task it's still have to parse your input and figure what is tic-tac-toe in this context and what exactly you expect it to do. This is why all "thinking". Ask it to implement tic-tac-toe in Python for command line. Or even just bring your own tic-tac toe code. Then make it imagine playing against you and it's gonna be fast and reliable.
- testartr 11mo agoprompt was very concrete: draw a tic tac toe ASCII table and let's play. gemini 2.5 thought for pages particular moves
- benterix 11mo ago> This is a reminder that benchmarks are meaningless – you should always curate your own out-of-sample benchmarks. Yeah I have my own set of tests and the results are a bit unsettling in the sense that sometimes older models outperform newer ones. Moreover, they change even if officially the model doesn't change. This is especially true of Gemini 2.5 pro that was performing much better on the same tests several months ago vs. now.
- Iulioh 11mo agoA lot of newer models are geared towards efficency and if you add the fact that more efficent models are trained on the output of less efficent (but more accurate) models.... GPT4/3o might be the best we will ever have
- adastra22 11mo agoI maintain a set of prompts and scripts for development using Claude Code. They are still all locked to using Sonnet 4 and Opus 4.1, because Sonnet 4.5 is flaming hot garbage. I’ve stopped trusting the benchmarks for anything.
- Archer6621 11mo agoI wonder whether it could be related to some kind of over-fitting, i.e. a prompting style that tends to work better with the older models, but performs worse with the newer ones.
- WhitneyLand 11mo ago>>benchmarks are meaningless No they’re not. Maybe you mean to say they don’t tell the whole story or have their limitations, which has always been the case. >>my fairly basic python benchmark I suspect your definition of “basic” may not be consensus. Gpt-5 thinking is a strong model for basic coding and it’d be interesting to see a simple python task it reliably fails at.
- deleted 11mo ago[deleted]
- NaomiLehman 11mo agothey are not meaningless, but when you work a lot with LLMs and know them VERY well, then a few varied, complex prompts tell you all you need to know about things like EQ, sycophancy, and creative writing. I like to compare them using chathub using the same prompts Gemini still calls me "the architect" in half of the prompts. It's very cringe.
- beepbooptheory 11mo agoI get that one can perhaps have an intuition about these things, but doesn't this seem like a somewhat flawed attitude to have all things considered? That is, saying something to the effect of "well I know its not too sycophantic, no measurement needed, I have some special prompts of my own and it passed with flying colors!" just sounds a little suspect on first pass, even if its not like totally unbelievable I guess.
- mpalmer 11mo agoGemini still calls me "the architect" in half of the prompts. It's very cringe. Can't say I've ever seen this in my own chats. Maybe it's something about your writing style?
- NaomiLehman 11mo agoit absolutely does. and human employees don't call me "the architect." that's the point.
- Rover222 11mo agocurious if you tried grok 4.1 too
- sosodev 11mo agoHow can you be sure that your benchmark is meaningful and well designed? Is the only thing that prevents a benchmark from being meaningful publicity?
- prodigycorp 11mo agoI didn't tell you what you should think about the model. All I said is that you should have your own benchmark. I think my benchmark is well designed. It's well designed because it's a generalization of a problem I've consistently had with LLMs on my code. Insofar that it encapsulates my coding preferences and communication style, that's the proper benchmark for me.
- gregsadetsky 11mo agoI asked a semi related question in a different thread [0] -- is the basic idea behind your benchmark that you specifically keep it secret to use it as an "actually real" test that was definitely withheld from training new LLMs? I've been thinking about making/publishing a new eval - if it's not public, presumably LLMs would never get better at them. But is your fear that generally speaking, LLMs tend to (I don't want to say cheat but) overfit on known problems, but then do (generally speaking) poorly on anything they haven't seen? Thanks [0] https://news.ycombinator.com/item?id=45968665 https://news.ycombinator.com/item?id=45968665
- adastra22 11mo ago> if it's not public, presumably LLMs would never get better at them. Why? This is not obvious to me at all.
- gregsadetsky 11mo agoYou're correct of course - LLMs may get better at any task of course, but I meant that publishing the evals might (optimistically speaking) help LLMs get better at the task. If the eval was actually picked up / used in the training loop, of course.
- thefourthchime 11mo agoI like to ask "Make a pacman game in a single html page". No model has ever gotten a decent game in one shot. My attempt with Gemini3 was no better than 2.5.
- ofa0e 11mo agoYour benchmarks should not involve IP.
- ComplexSystems 11mo agoWhy? This seems like a reasonable task to benchmark on.
- 11mo ago
- dekhn 11mo agoUsing a single custom benchmark as a metric seems pretty unreliable to me. Even at the risk of teaching future AI the answer to your benchmark, I think you should share it here so we can evaluate it. It's entirely possible you are coming to a wrong conclusion.
- prodigycorp 11mo agoafter taking a walk for a bit i decided you’re right. I came to the wrong conclusion. Gemini 3 is incredibly powerful in some other stuff I’ve run. This probably means my test is a little too niche. The fact that it didn’t pass one of my tests doesn’t speak to the broader intelligence of the model per se. While i still believe in the importance of a personalized suite of benchmarks, my python one needs to be down weighted or supplanted. my bad to the google team for the cursory brush off.
- chermi 11mo agoWalks are magical. But also this reads partially like you got sent to a reeducation camp lol.
- deleted 11mo ago[deleted]
- nomel 11mo ago> This probably means my test is a little too niche. > my python one needs to be down weighted or supplanted. To me, this just proves your original statement. You can't know if an AI can do your specific task based on benchmarks. They are relatively meaningless. You must just try. I have AI fail spectacularly, often, because I'm in a niche field. To me, in the context of AI, "niche" is "most of the code for this is proprietary/not in public repos, so statistically sparse".
- relaytheurgency 11mo agoI feel similarly. If you're working with some relatively niche APIs on services that don't get seen by the public, the AI isn't one-shotting anything. But I still find it helpful to generate some crap that I can then feel good about fixing.
- luckydata 11mo agoI'm dying to know what you're giving to it that's choking on. It's actually really impressive if that's the case.
- nomel 11mo agoI find this hard to understand. I have AI completely choke on my code constantly. What are you doing where it performs so well? Web? I constantly see failures in trivial vectors projections, broken bash scripts that don't properly quote variables (fail if space in filenames), and near completely inability to do relatively basic image processing tasks (if they don't rely on template matches). I accidentally spent $50 on Gemeni 2.5 Pro last week, with Roo, trying to make a simple Mock interface for some lab equipment. The result: it asks permission to delete everything it did and start over...
- t0mas88 11mo agoGoogle reports a lower score for Gemini 3 Pro on SWEBench than Claude Sonnet 4.5, which is comparing a top tier model with a smaller one. Very curious to see whether there will be an Opus 4.5 that does even better.