10 ms·
Gemini 3 Pro: the frontier of vision AI
- deleted 11mo ago[deleted]
- simonw 11mo agoIn case the article author sees this, the "HTML transcription" link is broken - it goes to https://aistudio-preprod.corp.google.com/prompts/1GUEWbLIlpXrE_ams-GBg3e7ZG7AcYgbr https://aistudio-preprod.corp.google.com/prompts/1GUEWbLIlpX... which is a Google-employee-only URL.
- TechRemarker 11mo agoLove how employee portals for many companies essentially never get updated design wise over the decades, lol. That page styling and the balls certainly take me back.
- jamiek88 11mo agoWow yeah. Flashbacks to when Gmail Invites were cool! Google too.
- inerte 11mo agoI used to work for a company where the SSO screen had a nice corporate happy people at the office type of image. 25mb. I was in Brazil on a crappy roaming 2g service and couldn't login at all. I know most of the work happens on desktop but geee..... Oh speaking on mobile, I remember when I tried to use Jira mobile web to move a few tickets up on priority by drag and dropping and ended up closing the Sprint. That stuff was horrible.
- dekhn 11mo agoLiterally decades: the login page looked like that when I joined google in 2007.
- cubefox 11mo agoExcept for the updated Google logo.
- ed 11mo agoSame with "See prompt in Google AI Studio" which links to an unpublished prompt in AI Studio.
- rohanlikesai 11mo agohey, it's Rohan (the author of the article) - appreciate you catching this, we just fixed this!
- sumedh 11mo agoYou should try using AI to check such things :)
- buildbot 11mo agoI’m a little surprised how open the help links are… I guess that if need help logging in you can’t be expected to well, log in.
- ch2026 11mo agowhat framework is being utilized for computer use here?
- causal 11mo agoOkay maybe this one isn't an exaggeration when they say leap forward
- fngjdflmdflg 11mo agoThese OCR improvements will almost certainly be brought to google books, which is great. Long term it can enable compressing all non-digital rare books into a manageable size that can be stored for less than $5,000.[0] It would also be great for archive.org to move to this from Tesseract. I wonder what the cost would be, both in raw cost to run, and via a paid API, to do that. [0] https://annas-archive.org/blog/critical-window.html https://annas-archive.org/blog/critical-window.html
- kridsdale3 11mo agoMore Data for the Data Gods!
- levocardia 11mo agoThis is a really interesting "data flywheel" -- better model >> more usable data >> even better model
- tills13 11mo agosurely there's an upper limit to this though with models literally eating themselves.
- jeffbee 11mo agoWhen a human students learns to read more carefully we don't consider that a negative.
- Choco31415 11mo agoWe can wait for that to start appearing in tests or benchmarks first.
- Workaccount2 11mo agoThey already purposely train them on their own output, it's called synthetic training data.
- visarga 11mo ago
- djoldman 11mo agoInteresting "ScreenSpot Pro" results: 72.7% Gemini 3 Pro 11.4% Gemini 2.5 Pro 49.9% Claude Opus 4.5 3.50% GPT-5.1 ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use https://arxiv.org/abs/2504.07981 https://arxiv.org/abs/2504.07981
- agentifysh 11mo agoimpressive.....most impressive its going to reach low 90s very soon if trends continue
- jasonjmcghee 11mo agoThat is... astronomically different. Is GPT-5.1 downscaling and losing critical information or something? How could it be so different?
- ericd 11mo agoI found much better results with smallish UI elements in large screenshots on GPT by slicing it up manually and feeding them one at a time. I think it does severely lossy downscaling.
- zubiaur 11mo agoIt has a rather poor max resolution. Higher resolution images get tiled up to a point. 512 x 512, I think is the max tile size, 2048 x 2048 the max canvas.
- energy123 11mo agoThis is my default explanation for visual impairments in LLMs, they're trying to compress the image into about 3000 tokens, you're going to lose a lot in the name of efficiency.
- simonw 11mo agoI was surprised at how poorly GPT-5 did in comparison to Opus 4.1 and Gemini 2.5 on a pretty simple OCR task a few months ago - I should run that again against the latest models and see how they do. https://simonwillison.net/2025/Aug/29/the-perils-of-vibe-coding/ https://simonwillison.net/2025/Aug/29/the-perils-of-vibe-cod...
- jonplackett 11mo agoGoogle really are a fully woken sleeping giant. More code reds being issued today I expect.
- agentifysh 11mo agoim realizing how much of a bottleneck vision models are im just a glorified speedreadin' promptin' QA at this point with codex once it replaces the QA layer its truly over for software dev jobs future would be a software genie where on aistudio you type: "go make counterstrike 1.6 clone, here is $500, you have two hours" edit: saw the Screenspot benchmark and holy ** this is an insane jump!!! 11% to 71% even beating Opus 4.5's 50%...chatgpt is at 3.5% and it matches my experience with codex
- alex1138 11mo ago> once it replaces the QA layer its truly over for software dev jobs Maybe. However, with CYA requirements being everywhere in industry, there would have to be 100 waiver forms signed. I-promise-not-to-sue-company-if-AI-deletes-the-entire-database It won't happen for that reason alone. Oh who am I kidding of course it will
- hklrekeclhkle 11mo ago[dead]
- iamjackg 11mo agoCurious how this will fare when playing Pokemon Red.
- euvin 11mo agoYeah the "High frame rate understanding" feature caught my eye, actual real time analysis of live video feeds seems really cool. Also wondering what they mean by "video reasoning/thinking"?
- skybrian 11mo agoI don’t think it’s real time? The videos were likely taken previously.
- minimaxir 11mo agoGemini 3 Pro has been playing Pokemon Crystal (which is significantly harder than Red) in a race against Gemini 2.5 Pro: https://www.twitch.tv/gemini_plays_pokemon https://www.twitch.tv/gemini_plays_pokemon Gemini 3 Pro has been making steady progress (12/16 badges) while Gemini 2.5 Pro is stuck (3/16 badges) despite using double the turns and tokens.
- theLiminator 11mo agoI think what would be interesting is if it could play the game with vision only inputs. That would represent a massive leap multimodal understanding.
- danso 11mo ago> 3. Turning long videos into action: Gemini 3 Pro bridges the gap between video and code. It can extract knowledge from long-form content and immediately translate it into functioning apps or structured code I'm curious as to how close these models are to achieving that once long-ago mocked claim (by Microsoft I think?) that AIs could view gameplay video of long lost games and produce the code to emulate them.
- siva7 11mo agoInteresting. When i asked Gemini 3 Pro to generate a Infographic from my personal accounting sheet, it first failed to generate anything except a black background, then it generated something where it mixed different languages in a non-sensical way, with obvious typos and irrelevant information grouping. It's certainly a leap forward in OCR, rendering classic OCR useless.
- minimaxir 11mo agoThat's more of an issue with Nano Banana Pro than with Gemini 3 Pro.
- siva7 11mo agoWhat's the difference? I thought the vision ai component of gemini 3 is called nano banana?
- IanCal 11mo agoThat’s about generating images, the other side is about understanding images.
- brokensegue 11mo agoi assumed nano banana was just a tool that gemini 3 used though i don't know
- minimaxir 11mo agoGemini 3 Pro's text encoder powers Nano Banana Pro, but it has its own image decoding model that decodes the generated image tokens into an actual image, which appears to be the more pertinent issue in this case.
- andy12_ 11mo agoThat's weird, from my own tests Nano banana pro has no problem generating complex infographics with legible text.
- deleted 11mo ago[deleted]
- stego-tech 11mo agoThe document is paints a super impressive picture, but the core constraint of “network connection to Google required so we can harvest your data” is still a big showstopper for me (and all cloud-based AI tooling, really). I’d be curious to see how well something like this can be distilled down for isolated acceleration on SBCs or consumer kit, because that’s where the billions to be made reside (factories, remote sites, dangerous or sensitive facilities, etc).
- bgwalter 11mo agoArpanet was supposed to be decentralized. Now everyone wants to centralize everything so in a war it is sufficient to strike 100 data centers and the whole tethered economy collapses. That is called progress. EDIT: You can downvote the truth but still no one wants your "AI" slop.
- stego-tech 11mo agoAh, the fond memories of telnetting to NCSA to upload the raw HTML of my first website, written on an OG Macintosh computer and ported via floppy to a PowerMac for network connectivity. Simple, elegant. I do miss those days.
- oklahomasports 11mo agoPeople with your concerns probably make up 1% of the market if that. Also I don’t upload stuff I’m worried about Google seeing. I wonder if they will allows special plans for corporations
- stego-tech 11mo agoI’m very curious where you get that number from, because I thought the same thing until I got a job inside that market and realized how much more vast it actually is. The revenue numbers might not be as big as Big Tech, but the product market is shockingly vast. My advice is not to confuse Big Tech revenues for total market size, because they bring in such revenue by catering to everyone, rather than specific segments or niches; a McDonald’s will always do more volume than a steakhouse, but it doesn’t mean the market for steakhouses is small enough to ignore. As for this throwaway line: > Also I don’t upload stuff I’m worried about Google seeing. You do realize that these companies harvest even private data, right? Like, even in places you think you own, or that you pay for, they’re mining for revenue opportunities and using you as the product even when you’re a customer, right? > I wonder if they will allows special plans for corporations They do, but no matter how much redlining Legal does to protect IP interests, the consensus I keep hearing is “don’t put private or sensitive corporate data into third-parties because no legal agreement will sufficiently protect us from harm if they steal our IP or data”. Just look at the glut of lawsuits against Apple, Google, Microsoft, etc from smaller companies that trusted them to act in good faith but got burned for evidence that you cannot trust these entities.
- empressplay 11mo agoYes, but can it play PacMan yet?
- ed 11mo agoWhat’s new here? I believe this is just gemini 3 which was released last month (the model id hasn’t changed AFAICT)
- minimaxir 11mo agoNothing new, it's just highlighting practical vision use cases.
- hodder 11mo ago"Gemini 3 Pro represents a generational leap from simple recognition to true visual and spatial reasoning." Prompt: "wine glass full to the brim" Image generated: 2/3 full wine glass. True visual and spatial reasoning denied.
- minimaxir 11mo agoGemini 3 Pro is not Nano Banana Pro, and the image generation/model that decodes the generated image tokens may not be as robust. The thinking step of Nano Banana Pro can refine some lateral steps (i.e. the errors in the homework correction and where they are spatially in the image) but it isn't perfect and can encounter some of the typical pitfalls. It's a lot better than Nano Banana base, though.
- hodder 11mo agoAs a consumer I typed this into "Gemini". The behind the scenes model selection just adds confusion. If "AI" trust is the big barrier for widespread adoption to these products, Alphabet soup isn't the solution (pun intended).
- iknowstuff 11mo agoNano Banana generates images. This article is about understanding images. Your task is unrelated to the article.
- JacobAsmuth 11mo agoIt works fine for me. https://imgur.com/a/MKNufm1 https://imgur.com/a/MKNufm1
- spchampion2 11mo agoI actually did this prompt and found that it worked with a single nudge on a followup prompt. My first shot got me a wine glass that was almost full but not quite. I told it I wanted it full to the top - another drop would overflow. The second shot was perfectly full.
- knollimar 11mo agoI do some electrical drafting work for construction and throw basic tasks at LLMs. I gave it a shitty harness and it almost 1 shotted laying out outlets in a room based on a shitty pdf. I think if I gave it better control it could do a huge portion of my coworkers jobs very soon
- amorzor 11mo agoCan you give an example of the sort of harness you used for that? Would love to play around with it
- knollimar 11mo agoI've been using pyrevit inside revit so I just threw a basic loop in there. There's already a building model and the coworkers are just placing and wiring outlets, switches, etc. The harness wasn't impressive enough to share (alos contains vibe coded UI since I didn't want to learn XAML stuff on a friday night). Nothing fancy; I'm not very skilled (I work in construction) I gave it some custom methods it could call, including "get_available_families", "place family instance", "scan_geometry" (reads model walls into LLM by wall endpoint), and "get_view_scale". The task is basically copy the building engineer's layout onto the architect model by placing my families. It requires reading the symbol list, and you give it a pdf that contains the room. Notably, it even used a GFCI family when it noticed it was a bathroom (I had told it to check NEC code, implying outlet spacing).
- ftcHn 11mo agoI'm going to try to get it to generate extrusions in Revit based on images of floor plans. I've tried doing this in bunch of models without success so far.
- knollimar 11mo agoYou might want to give it some guidance based on edge centers? It'll have a hard time thinking of wall thickness and have it draw points if you're trying to copy floor plans. for clarity now that I'm rereading: it understands vectors a lot better than areas. Encoding it like that seems to work better for me.
- devinprater 11mo agoAudio described Youtube please? That'd be so amazing! Even if I couldn't play Zelda yet, I could listen to a playthrough with Gemini describing it.
- SXX 11mo agoHey, I just made simple test on 5 minute downloaded YouTube video uploading it to Gemini app. Source video title: Zelda: Breath of the Wild - Opening five minutes of gameplay https://www.youtube.com/watch?v=xbt7ZYdUXn8 https://www.youtube.com/watch?v=xbt7ZYdUXn8 Prompt: Please describe what happening in each scene of this video. List scenes with timestamp, then describe separately: - Setup and background, colors - What is moving, what appear - What objects in this scene and what is happening, Basically make desceiption of 5 minutes video for a person who cant watch it. Result on github gist since there too much text: https://gist.github.com/ArseniyShestakov/43fe8b8c1dca45eadab329273ed835bb https://gist.github.com/ArseniyShestakov/43fe8b8c1dca45eadab... I'd say thi is quite accurate.
- SXX 11mo agoAnother example with completely random 10 minute benchmark video from Tears of Kingdom: https://gist.github.com/ArseniyShestakov/47123ce2b6b19a8e6b317f784916c952 https://gist.github.com/ArseniyShestakov/47123ce2b6b19a8e6b3...
- SXX 11mo agoBTW I asked detailed narrative descriprion of other purely benchmarking Zelda video with 5 second snapshots: Video: Zelda TOTK, R5 5600X, GTX 1650, 1080p 10 Minute Gameplay, No Commentary https://www.youtube.com/watch?v=wZGmgV-8Rbo https://www.youtube.com/watch?v=wZGmgV-8Rbo Here can be found narrative descriprion source and command: https://gist.github.com/ArseniyShestakov/47123ce2b6b19a8e6b317f784916c952 https://gist.github.com/ArseniyShestakov/47123ce2b6b19a8e6b3... Then I converted it into narrative voice over with Gemini 2.5 Pro TTS: https://drive.google.com/file/d/1Js2nDtM7sx14I43UY2PEoV5PuLMBanqd/view?usp=share_link https://drive.google.com/file/d/1Js2nDtM7sx14I43UY2PEoV5PuLM... It's somewhat desynced from original video and voice over take 9 and half minutes instead of 10 in video, but description of what happening on screen is quite accurate. PS: I used 144p video so details could be also messed up because of poor quality. And ofc I specifically asked for narrative-like descripription
- pseudosavant 11mo agoI'm really fascinate by the opportunities to analyze videos. The amount of tokens it compresses down to, and what you can reason across those tokens, is incredible.
- minimaxir 11mo agoThe actual token calculations with input videos for Gemini 3 Pro is...confusing. https://ai.google.dev/gemini-api/docs/media-resolution https://ai.google.dev/gemini-api/docs/media-resolution
- pseudosavant 11mo agoThat is because it isn't actually tokens that are fed into the model for non-text. For text, it is tokenized, and each token has a specific set of vectors. But with other media, they've trained encoders that analyze the media and produce a set of vectors that are the same "format" as the token's vectors, but it isn't actually ever a token. Most companies have rules for how many tokens the media should "cost", but they aren't usually exact.
- dmarzio 11mo agoSo we’re going to use this to make the maid from the Jetsons finally. Right?
- Workaccount2 11mo agoWell It is the first model to get partial-credit on an LLM image test I have. Which is counting the legs of a dog. Specifically, a dog with 5 legs. This is a wild test, because LLMs get really pushy and insistent that the dog only has 4 legs. In fact GPT5 wrote an edge detection script to see where "golden dog feet" met "bright green grass" to prove to me that there were only 4 legs. The script found 5, and GPT-5 then said it was a bug, and adjusted the script sensitivity so it only located 4, lol. Anyway, Gemini 3, while still being unable to count the legs first try, did identify "male anatomy" (it's own words) also visible in the picture. The 5th leg was approximately where you could expect a well endowed dog to have a "5th leg". That aside though, I still wouldn't call it particularly impressive. As a note, Meta's image slicer correctly highlighted all 5 legs without a hitch. Maybe not quite a transformer, but interesting that it could properly interpret "dog leg" and ID them. Also the dog with many legs (I have a few of them) all had there extra legs added by nano-banana.
- cancan 11mo agothis is hilarious and incredibly interesting at the same time! thanks for writing it up.
- reed1234 11mo agoCould you link the image? Interesting stuff.
- danielvaughn 11mo agoI don’t know much about AI, but I have this image test that everything has failed at. You basically just present an image of a maze and ask the LLM to draw a line through the most optimal path. Here’s how Nano Banana fared: https://x.com/danielvaughn/status/1971640520176029704?s=46 https://x.com/danielvaughn/status/1971640520176029704?s=46
- buildbot 11mo agoThat might be an interesting training set, a bunch of mazes…
- 11mo ago
- caseyf 11mo agoI'm playing with this and wondering if this is an actually good way to identify dominant colors and other features of a garment/product when using a photo where the item is styled and not isolated from the model or other garments
- sublimefire 11mo agoThere should be an existing simpler way to do it. Image contains a bunch of pixels so you could just group the to see main colors
- TheAceOfHearts 11mo agoSince I think it's interesting to highlight the jagged intelligence, I have a simple word search puzzle [0] that Nano Banana Pro stills struggles to solve correctly. Gemini 3 Pro with Code Execution is able to one-shot the problem and find the positions of each word (this is super impressive! one year ago it wasn't possible), but Nano Banana Pro fails to highlight the words correctly. Here's the output from two tests I ran: 1. Asking Nano Banana Pro to solve the word search puzzle directly [1]. 2. Asking Nano Banana Pro to highlight each word on the grid, with the position of every word included as part of the prompt [2]. The fact that it gets 2 words correct demonstrates meaningful progress, and it seems like we're really close to having a model that can one-shot this problem soon. There's actually a bit of nuance required to solve this puzzle correctly which an older Gemini model struggled to do without additional nudging. You have to convert the grid or word list to use matching casing (the grid uses uppercase, the word list uses lowercase), and you need to recognize that "soup mix" needs to have the space removed when doing the search. [0] https://imgur.com/ekwfHrN https://imgur.com/ekwfHrN [1] https://imgur.com/1nybezU https://imgur.com/1nybezU [2] https://imgur.com/18mK5i5 https://imgur.com/18mK5i5
- genrader 11mo agoIf you're using for instance the Gemini web app there may be a preference in the system prompt to immediately favor the fact that you said to create an image when in fact it may have been better to initially start with a regular chat prompt, making sure you're on Gemini 3 Pro thinking, and then give it exactly what you usually would. You can tell it that after it has an answer to the question then to create an image for it. This may even work if you tell it to do all that prior to figuring out what to create for the image,
- TheAceOfHearts 11mo agoI just used Nano Banana Pro from LMArena, but if you have access to a paid account I'd love to see you try it out! I just gave it the puzzle image as an input along with the prompt: "Please solve this word search puzzle". For generating the prompt which included the word positions I had Gemini 3 Pro do that using the following prompt: "Please try to solve this word search puzzle. Give me the position of each word in the grid. Then generate a prompt which I can pass to Nano Banana Pro, which I will pass along with the same input image to see if Nano Banana Pro is able to properly highlight all the words if given their correct position."
- drivebyhooting 11mo agoScreen understanding is huge for further automating dev work.
- k8sToGo 11mo agoWhen will we get Gemini 3 Flash?
- sumedh 11mo agoProbably waiting to see when OpenAI announces their new models.
- bovermyer 11mo agoI would be interested in seeing what G3P makes of the Dead Sea Scrolls or similarly old documents.
- ichik 11mo agoFrankly, it's insane how laughably bad under scrutiny their own examples are. It both distorted the data and made the chart less readable (labels placement, segments separation, missing labels, worse contrast). And it combined them into one, so you you'll have harder time comparing them compared to the original image! Isn't it amazing that it added a toggle? Post author seems to think it deserves an exclamation point even.
- aziis98 11mo ago> Pointing capability: Gemini 3 has the ability to point at specific locations in images by outputting pixel-precise coordinates. Sequences of 2D points can be strung together to perform complex tasks, such as estimating human poses or reflecting trajectories over time Does somebody know how to correctly prompt the model for these tasks or even better provide some docs? The pictures with the pretty markers are appreciated but that section is a bit vague and without references
- atonse 11mo agoFor my CMS I’d love to get an AI to nicely frame a picture in certain aspect ratios. Like of I provide an image, give me coordinates for a widescreen, square, portrait, and 4x3 using a photographers eye. Any model that can do that? I tried looking in huggingface but didn’t quite see anything.
- themanmaran 11mo agoSimon Wilson has some good blogs on this: https://simonwillison.net/2024/Aug/26/gemini-bounding-box-visualization https://simonwillison.net/2024/Aug/26/gemini-bounding-box-vi...
- inquirerGeneral 11mo ago[dead]
- MostlyStable 11mo agoGoing to compare this to our current solution of Amazon's Textract service for analyzing handwritten datasheets. Textract, when extracting tables (which is what we use it for) does not allow for providing any context or information about the tables and what we expect them to contain, but it is really good at correctly recognizing hand written characters. All of my attempts at less specialized, more general models allow me to provide that context, which is helpful in some ways, but fail at the basic part of almost always correctly getting the character. Hopefully Google pro marries the two together.
- lysecret 11mo ago2.5 pro is already excellent at this.
- a-dub 11mo agoi like to put it in live mode and point it at my plants and have conversations about how they're doing. it properly identifies them and flags any signs of disease and then provides correct next steps.
- inquirerGeneral 11mo ago[dead]
- axpy906 11mo agoSo Gemini was the most non-deterministic model of them all and now we get this one with temperature at 1 and max thinking. It’s so random that it’s hard to justify putting in my setup right now.
- genrader 11mo agoThis is an excellent short way to understand that what you give Gemini 3 Pro is substantial better in understanding the data. Making sure you ask correctly how it should give you the info is still lacking in many people's ability
- hackeruser741 11mo agoIt's fascinating how these models struggle with simple counting or novel configurations like a 5-legged dog or a 13-hour clock, despite excelling at complex language tasks. It highlights the difference between learning patterns from vast datasets and true conceptual understanding.
- Frannky 11mo agoIt's a good model. I worry that they will be able to win the game by offering the best service for free, thanks to selling users' data—kind of like search, email, etc. It's sad. Not that the alternatives are better... You either trust synchopathic ChatGPT backed by Scama, go with woke Claude (they once banned my account for asking how some news was trying to influence me), Grok that feels like a 20-year-old sure about stuff that don't work, and Chinese models that are agenda-aligned...
- kkukshtel 11mo agosounds awesome but too bad it is impossible to figure out how to actually use these models and what I have to pay for/where
- romanovcode 11mo agoI gotta say - processing video at 10fps is very impressive.
- mhl47 11mo agoWe are currently working on some christmas puzzle, that are - I would say - a bit more difficult from the visual side. GPT5.1 completely failed at all of them while Gemini 3 solved two till know that I would consider rather impressive. One was two screenshots of a phone screen with chats that are timestamped and it had to take the nth letter of the mth word based on the timestamp. While the type of riddle could be in the training data the ability to OCR this that well and understand the spatial relation to each object perfectly is something I have not seen from other models yet.
- devttyeu 11mo agoVisual puzzle solving is a pretty easily trainable problem due to it being simple to verify, so that skill getting really good is just a matter of time
- sora2video 11mo ago[dead]
- themafia 11mo ago"the frontier" I've never hated industry infatuation with a buzzword more.
- Spacecosmonaut 11mo agoHow about "bleeding edge"
- vharish 11mo agoNope... frontier takes the cup! :D
- OBELISK_ASI 11mo ago[dead]