3 ms·
blog post is up- https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/ https://blog.google/innovation-and-ai/models-and-resear
by sigmar 8mo ago
blog post is up- https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/ https://blog.google/innovation-and-ai/models-and-research/ge...
edit: biggest benchmark changes from 3 pro:
arc-agi-2 score went from 31.1% -> 77.1%
apex-agents score went from 18.4% -> 33.5%
- sho_hn 8mo agoThe touted SVG improvements make me excited for animated pelicans.
- takoid 8mo agoI just gave it a shot and this is what I got: https://codepen.io/takoid/pen/wBWLOKj https://codepen.io/takoid/pen/wBWLOKj The model thought for over 5 minutes to produce this. It's not quite photorealistic (some parts are definitely "off"), but this is definitely a significant leap in complexity.
- makeavish 8mo agoLooks great!
- onionisafruit 8mo agoGood to see it wearing a helmet. Their safety team must be on their game.
- BrokenCogs 8mo agoYes but why would a pelican need a helmet? If it falls over it can just fly away... Common sense 1 Gemini 0
- throwa356262 8mo agoObviously these domestic pelicans can't fly, otherwise why would they need a bike?
- Gander5739 8mo agoWhy would a pelican be riding a bicycle at all, for that matter?
- BrokenCogs 8mo agoBecause the user asked for it
- benatkin 8mo agoHere's what I got from Gemini Pro on gemini.google.com, it thought for under a minute...might you have been using AI studio? https://jsbin.com/zopekaquga/edit?html,output https://jsbin.com/zopekaquga/edit?html,output It does say 3.1 in the Pro dropdown box in the message sending component.
- tasuki 8mo agoThat's a good pelican. What I like the most is that the SVG is nice and readable. If only Inkscape could output nice SVG like this!
- aoeusnth1 8mo agoI imagine they're also benchgooning on SVG generation
- james2doyle 8mo agoThe blog post includes a video showcasing the improvements. Looks really impressive: https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/ https://blog.google/innovation-and-ai/models-and-research/ge...
- vunderba 8mo agoSVG is an under-rated use case for LLMs because it gives you the scalability of vector graphics along with CSS-style interactivity (hover effects, animations, transitions, etc.).
- rdtsc 8mo agoMy perennial joke is as soon as that got on HN front page Google went and hired some interns and they spend a 100% of the time on pelicans.
- DonHopkins 8mo agoHow about STL files for 3d printing pelicans!
- ripbozo 8mo agoDoes the arc-agi-2 score more than doubling in a .1 release indicate benchmark-maxing? Though i dont know what arc-agi-2 actually tests
- blinding-streak 8mo agoI assume all the frontier models are benchmaxxing, so it would make sense
- maxall4 8mo agoTheoretically, you can’t benchmaxx ARC-AGI, but I too am suspect of such a large improvement, especially since the improvement on other benchmarks is not of the same order.
- moffkalast 8mo agohttps://arcprize.org/arc-agi/1/ https://arcprize.org/arc-agi/1/ It's a sort of arbitrary pattern matching thing that can't be trained on in the sense that the MMLU can be, but you can definitely generate billions of examples of this kind of task and train on it, and it will not make the model better on any other task. So in that sense, it absolutely can be. I think it's been harder to solve because it's a visual puzzle, and we know how well today's vision encoders actually work https://arxiv.org/html/2407.06581v1 https://arxiv.org/html/2407.06581v1
- km144 8mo agoThe real question is: Why are people designing benchmarks that, if a model is trained on them, it won't improve the performance of the model at any real-world tasks? Why would anyone care about such benchmarks?
- moffkalast 8mo agoPeople are like typewriter monkeys, if something is possible to make it'll eventually be made.
- 8mo ago