13 ms·
O3 beats a master-level GeoGuessr player, even with fake EXIF data
- orangecat 1y agoAmazing. I'm relatively bullish on AI and still I would have bet on the human here. Looking forward to the inevitable goalpost-moving of "that's not real reasoning".
- short_sells_poo 1y agoCan you please explain to me how this is evidence for reasoning?
- z7 1y agoQuoting Chollet: >I have repeatedly said that "can LLM reason?" was the wrong question to ask. Instead the right question is, "can they adapt to novelty?". https://x.com/fchollet/status/1866348355204595826 https://x.com/fchollet/status/1866348355204595826
- kelseyfrog 1y agoBecause the output contains evidence of thought processes that have been established as leading to valid solutions to problems. I have a simple question: Is text a sufficient medium to render a conclusion of reasoning? It can't be sufficient for humans and insufficient for computers - such a position is indefensible.
- empath75 1y agoI would say that almost all of what humans do is not the result of reasoning, and that reasoning is an unnatural and learned skill for humans, and most humans aren't good at even very basic reasoning.
- kelseyfrog 1y agoUsually we move the goalposts for AI. It takes more guts to move the goalposts for humans. I applaud it. Do you suppose we can deduce reasoning through the medium of text?
- zahlman 1y ago> Because the output contains evidence of thought processes that have been established as leading to valid solutions to problems. This sort of claim always just reminds me of Lucky's monologue in Waiting for Godot.
- kelseyfrog 1y agoYou're not wrong. It's an artifact of rewriting the definition of reason into a sentence that begins with "Because the output ..."
- zahlman 1y agoI didn't mean the wording itself; I meant that the claim isn't convincing to me for the same reason that Lucky's speech doesn't demonstrate an intelligent speaker.
- kelseyfrog 1y agoWhy not get curious instead?
- s17n 1y agoGeoguessing isn't much of a reasoning task, its more about memorizing a bunch of knowledge. Since LLMs contain essentially all knowledge, it's not surprising that they would be good at this. As far as goalpost-moving goes, it's wild to me that nobody is talking about the turing test these days.
- distortionfield 1y agoBecause the Chinese Room is a much better analogy for what LLMs are doing inside than the Turing test is.
- jibal 1y agoThat's a non sequitur that mixes apples and giraffes, and is completely wrong about what happens in the Chinese Room and what happens in LLMs. Ex hypothesi, the "rule book" that the Searle homunculus in the Chinese Room uses is "the right sort of program" to implement "Strong AI". The LLM algorithm is very much not that sort of program, it's a statistical pattern matcher. Strong AI does symbolic reasoning, LLMs do not. But worse, the Turing Test is not remotely intended to be an "analogy for what LLMs are doing inside" so your comparison makes no sense whatsoever, and completely fails to address the actual point--which is that, for ages the Turing Test was held out as the criterion for determining whether a system was "thinking", but that has been abandoned in the face of LLMs, which have near perfect language models and are able to closely model modes of human interaction regardless of whether they are "thinking" (and they aren't, so the TT is clearly an inadequate test, which some argued for decades before LLMs became a reality).
- semi-extrinsic 1y ago> the TT is clearly an inadequate test, which some argued for decades before LLMs became a reality To be specific, in a curious quirk of fate, LLMs seem to be proving right much of what Chomsky was saying about language. E.g. in 1996 he described the Turing test as "although highly influential, it seems to me not only foreign to the sciences but also close to senseless". (Curious in that VC backed businesses are experimentally verifying the views of a prominent anti-capitalist socialist.)
- TimorousBestie 1y agoI don’t think any goalposts need to be redecorated. The “inner monologue” isn’t a reliable witness to o3’s model, it’s at best a post-hoc estimation of what a human inner monologue might be in this circumstance. So its “testimony” about what it is doing is unreliable, and therefore it doesn’t move the needle on whether or not this is “real reasoning” for some value of that phrase. In short, it’s still anthropomorphism and apophenia locked in a feedback loop.
- empath75 1y agoI don't think the inner monologue is evidence of reasoning at all, but doing a task which can only be accomplished by reasoning is.
- TimorousBestie 1y agoGeoguessr is not a task that can only be accomplished by reasoning. Famously, it took a less than a day of compute time in 2011 to SLAM together a bunch of pictures of Rome (https://grail.cs.washington.edu/rome/ https://grail.cs.washington.edu/rome/).
- jibal 1y agoSuch as? geoguessing certainly isn't that.
- katmannthree 1y agoDevil's advocate, as with most LLM issues this applies to the meatbags that generated the source material as well. Quick example is asking someone to describe their favorite music and why they like it, and note the probable lack of reasoning on the `this is what I listened to as a teenager` axis.
- deleted 1y ago[deleted]
- hombre_fatal 1y agoGood point. When we try to explain why we're attracted to something or someone, what we do seems closer to modeling what we like to think about ourself. At the extreme, we're just story-telling about an estimation we like to think is true.
- InkCanon 1y agoI think if your assumption is that AI is deducing where it is with rational thoughts, you would be. In truth what probably happened is that the significant majority of digital images of the world had been scraped, labeled and used as training data.
- oncallthrow 1y agoHow do you explain https://simonwillison.net/2025/Apr/26/o3-photo-locations/ https://simonwillison.net/2025/Apr/26/o3-photo-locations/?
- Rumudiez 1y agothey only posted one photo in the post, but going off of that it's still an easy match based on streetview imagery. furthermore, the AI just identified the license plate and got lucky that photographer lives in a populous area, making it more prominent in the training data and therefore more likely to be found (even though it was off by 200 miles on its first guess)
- simonw 1y agoI posted two more at the bottom, from Madagascar and Buenos Aires: https://simonwillison.net/2025/Apr/26/o3-photo-locations/#update-o3-does-have-rough-location-access https://simonwillison.net/2025/Apr/26/o3-photo-locations/#up...
- Philpax 1y agoTry it with your own photos from around the world. I used my own photos from Stockholm, San Francisco, Tvarožná, Saas-Fee, London, Bergen, Adelaide, Melbourne, Paris, and Sicily, and can confirm that it was within acceptable range for almost all of them (without EXIF data), and it absolutely nailed some of the more obvious spots.
- SpaceManNabs 1y ago> Looking forward to the inevitable goalpost-moving of "that's not real reasoning" It did a web lookup. It is not comparing humans and o3 with equal resources.
- SamPatt 1y agoThat's really not a fair assessment. It used search in 2 of 5 rounds, and it already knew the correct road in one of those rounds (just look at the search terms it used). If you read the chain of thought output, you cannot dismiss their capability that easily.
- SpaceManNabs 1y agoWhy is it not a fair assessment to say it is comparing two "clients" with different resources if one can do a web lookup and the other cannot? You note yourself that it was meaningful in another round. > Also, the web search was only meaningful in the Austria round. It did use it in the Ireland round too, but as you can see by the search terms it used, it already knew the road solely from image recognition.
- SamPatt 1y agoI thought it might matter somewhat in that one Austria round. I was incorrect - I re-ran both rounds where the model used search, without search this time, and the results were nearly identical. I updated the post with the details. That's why I'm saying it's unfair to just claim it's doing a web lookup. No, it's way more capable than that.
- SpaceManNabs 1y agoThat isnt my claim. My claim is that they have access to different reaources. Whether that makes a difference it seems not but maybe it does.
- SirHumphrey 1y agoMy objection is not “that is not real reasoning” my objection is that’s not that hard. I happen to do some geolocating from static images from time to time and at least most of the images provided as examples contain a lot of clues- enough that i think a semi experienced person could figure out the location although - in fairness- in a few hours not few minutes. Second, the similar approaches were tried using CNNs and it worked (somewhat)[1]. [1]: https://huggingface.co/geolocal/StreetCLIP https://huggingface.co/geolocal/StreetCLIP EDIT: I am not talking about geoguesser - i am talking about geolocating an image with everything available (e.g. google…)
- usaar333 1y agoWhy? AI beat rainbolt 1.5 years ago: https://www.npr.org/2023/12/19/1219984002/artificial-intelligence-can-find-your-location-in-photos-worrying-privacy-expert https://www.npr.org/2023/12/19/1219984002/artificial-intelli... AI tends to have superhuman pattern matching abilities with enough data
- karlding 1y agoIf you watch the video, (one of) the reasons why the AI was winning was because it was using “meta” information from the Street View camera images, and not necessarily because it’s successfully identifying locations purely based on the landmarks in the image. > I realized that the AI was using the smudges on the camera to help make an educated guess here. [0] https://youtu.be/ts5lPDV--cU?t=1412 https://youtu.be/ts5lPDV--cU?t=1412
- ApolloFortyNine 1y agoPro geoguessr players do the same thing. The vividness of the colors and weirdness in the sky are two examples I've seen Rainbolt use in the past (and he's not even the best).
- ZeWaka 1y agoMeta is widely used by humans. One funny one is the different hiding-masks for the different streetview cars.
- 1970-01-01 1y agoGive it a photo from the surface of Mars and verify if it's actually capable of thinking outside the box or if it's relying on metadata and text. https://nssdc.gsfc.nasa.gov/planetary/image/mera_hills.jpg https://nssdc.gsfc.nasa.gov/planetary/image/mera_hills.jpg
- SamPatt 1y agoDone. Here's o3's reply: >That’s not Earth at all—this is the floor of Jezero Crater on Mars, the dusty plain and low ridge captured by NASA’s Perseverance rover (the Mastcam-Z color cameras give away the muted tan-pink sky and the uniform basaltic rubble strewn across the regolith).
- 1970-01-01 1y agoRight planet, but completely wrong on everything else. The location is nowhere near Perseverance, and was taken decades before Perseverance existed. https://nssdc.gsfc.nasa.gov/planetary/mars/mars_exploration_rovers/mera_images.html https://nssdc.gsfc.nasa.gov/planetary/mars/mars_exploration_...
- SamPatt 1y agoIt did think outside the box and didn't rely on metadata.
- 1970-01-01 1y ago>the Mastcam-Z color cameras give away the muted tan-pink sky That's still metadata
- zahlman 1y ago> Looking forward to the inevitable goalpost-moving of "that's not real reasoning". How is that moving the goalposts? Where did you see them set before, and where did your critics agree to that?
- TimTheTinker 1y ago> Looking forward to the inevitable goalpost-moving of "that's not real reasoning". It's less about the definition of "reasoning" and more about what's interesting. Maybe I'm wrong here ... but a chess bot that wins via a 100% game solution stored in exabytes of precomputed data might have an interesting internal design (at least the precomputing part), but playing against it wouldn't keep on being an interesting experience for most people because it always wins optimally and there's no real-time reasoning going on (that is, unless you're interested in the experience of playing against a perfect player). But for most people just interested in playing chess, I suspect it would get old quickly. Now ... if someone followed up with a tool that could explain insightfully why any given move (or series) the bot played is the best, or showed when two or more moves are equally optimal and why, that would be really interesting.
- bongodongobob 1y agoI tried this the other day with a picture of my dog in a field in a park by a river with EXIF stripped. It gave me a list of parks on the correct river. There were really no other features other than the foliage and water. Seems like magic to me, I don't really understand how it's possible to be that accurate.
- fwip 1y agoUnless you were running the model locally in a way you control, it seems very possible that the model has additional hidden context regarding your location (perhaps based on IP, perhaps based on credit card billing information, etc).
- deleted 1y ago[deleted]
- bongodongobob 1y agoThat kind of what I'm thinking, seems too good to be true.
- buzzerbetrayed 1y agoIt still works if you upload an image that was taken on the opposite side of the globe from where you are.
- Workaccount2 1y agoThe hidden context is probably many other photos taken in that area appearing in the training set.
- christianqchung 1y agoChatGPT doesn't let you share chats with images (why?), but here is what o3 said for an image with memory turned off: "Thought for 11 seconds It’s hard to pin down an exact spot from this single photo, but a few visual clues narrow the possibilities: Temperate-zone trees: Most of the trees have lost their leaves, but scattered white-pines or spruces remain green. That mix—deciduous hardwoods plus a sprinkling of conifers—is typical of the northeastern United States and southeastern Canada. Broad, moderately fast river: The channel looks 75-100 m wide with swirling surface currents but no large rapids, suggesting a mid-sized river in its lower reaches—something like the Merrimack, Connecticut, Delaware, or Susquehanna rather than a small mountain stream. Late-winter / early-spring light: Bare branches and a clear blue sky with fairly high sun angle hint at March–April in that latitude range. So my best guess is that you’re standing on the bank of a northeastern U.S. river—perhaps somewhere in New England or the Mid-Atlantic—during late winter or early spring. If you can share any nearby landmarks, town names, or additional photos, I might be able to zero in more precisely." With memory turned on: "Thought for 5 seconds Beautiful spot! From what I can see: a broad, slow‑moving river with a sandy/rocky bank in the foreground bare deciduous trees mixed with some white pines on the far shore (so it’s probably early spring or late autumn) no mountains on the horizon, just gently rolling terrain typical of low‑lying floodplains in the U.S. Northeast That mix of hardwoods and pines, plus the flat, braided water surface, reminds me a lot of the Merrimack or Concord Rivers in Massachusetts or southern New Hampshire—both have public footpaths where you can stand right at the water’s edge like this. Of course, many temperate rivers look similar. If you can give me a hint (e.g., the town or nearby landmark), I can narrow it down further—maybe even point you to trail maps or history of the exact stretch you photographed." Southern NH is correct, and I am certain it is drawing from memory/past chats. However, I can't replicate a specific behavior I once had, which is in temporary chat (no past chat/memory enabled), it said that it guessed where the photo was taken based on my location.
- julianhuang 1y ago1. The "master geoguesser" is a bit misleading--as mentioned in his blog post, there are players far better than him, and he is certainly not the bar for human supremacy. Probably analogous to a 1400-1800 elo chess player. 2. o3 isn't the best model at playing GeoGuessr, Gemini 1.5 & 2.5 solidly beat it out--for those interested, check out my friend's benchmark (https://geobench.org/ https://geobench.org/) and blog post (https://ccmdi.com/blog/GeoBench https://ccmdi.com/blog/GeoBench) detailing interesting model explanations. 3. In the post, he only tests on one game--o3's average score over 100 locations (20 5-location games) was 19,290, far lower than the 23,179 in the game. Model geolocation capabilities are really important to keep track of, but the specific blog post in question isn't anything out of the ordinary. LLMs are making geolocation abilities much more accessible, but still fall short compared to 1. top GeoGuessr players playing GeoGuessr (only google streetview coverage, without web search) and 2. professional geolocators, who are proficient at using a wide variety of software/search. I.e., if the CIA wanted to find someone using an image, LLMs would not provide them any unique ability to do so as opposed to someone like Rainbolt
- kenjackson 1y agoFrom your linked article: "It's clear that large language models have an emergent capability to play GeoGuessr, and play it well. Their abilities to generalize are nascent, but present" This is very accurate -- their abilities to generalize are nascent, but still surprisingly capable. The world is about to send through its best and brightest math/CS minds over the next decade (at least) to increase the capabilities of these AIs (with the help of AI). I just don't understand the pessimism with the technology.
- julianhuang 1y agoI completely agree that this is an incredible advancement as someone who has watched the rise of LLMs' GeoGuessr abilities. I just wanted to qualify the claim made in the blog post "In Which I Try to Maintain Human Supremacy for a Bit Longer". I also think that models would need to become far more proficient at tool use (querying OpenStreetMap features, coverage-checking Google Maps, calculating sun direction, etc.) that current human geolocators have access to, and precise spatial reasoning. Additionally, there is a whole corpus of GeoGuessr-specific knowledge that probably wouldn't have a reason to be in model training data (Google Street View coverage quirks, the specific geographic ranges of vegetation and architecture, tiny visual idiosyncrasies in country infrastructure like the width of red on a sign, etc.). However, I think this could probably be solved with more data, and I don't think there is any fundamental barrier.
- amrrs 1y agoIt's thinking process to go about guessing a place is further fascinating. Even o4 mini high is quite good[1] and very fast. But unlike a geogussr, it uses websearch[1] [1] https://youtu.be/P2QB-fpZlFk?si=7dwlTHsV_a0kHyMl https://youtu.be/P2QB-fpZlFk?si=7dwlTHsV_a0kHyMl [1]
- bredren 1y agoNeat to see progress of this from Simon's original post to comment to this.
- vunderba 1y agoThe original post was actually this one - two weeks prior. https://news.ycombinator.com/item?id=43723408 https://news.ycombinator.com/item?id=43723408 https://flausch.social/@piegames/114352447253793517 https://flausch.social/@piegames/114352447253793517
- asdsadasdasd123 1y agoThis is probably one of the less impressive LLM applications imo. Like it already knows what every plant, street sign, etc is. I would imagine a traditional neural net would do really well here as well if you can extract some crude features.
- EGreg 1y agoCant the same be said about “unimpressive” behavior by coding LLMs that know every algorithm, language and library?
- asdsadasdasd123 1y agoDisagree because code has to be far more precise than, the location is in the jungles of brazil. This level of coding as never been achievable by traditional ML methods AFAIK
- exitb 1y agoI tried a picture of Dublin and it pointed out the hotel I took it from. Obviously that’s more data than any single person can keep in their head.
- OtherShrezzing 1y agoIt's my understanding that o3 was trained on multimodal data, including imagery. Is it unreasonable to assume its training data includes images of these exact locations and features? GeoGuesser uses Google Maps, and Google Maps purchases most of its imagery from third-parties these days. If those third parties aren't also selling to all the big AI companies, I'd be very surprised.
- Yenrabbit 1y agoTry it with your own personal photos. It is scarily good!
- rafram 1y agoThat's true for heavily photographed urban areas. I've tried it on some desert photos (even at identifiable points with human structures) and it basically just guesses a random trailhead in Joshua Tree and makes up a BS explanation for why it matches.
- kube-system 1y agoI have had surprisingly good luck with beach photos that don’t have much beyond dunes and vegetation in them
- walthamstow 1y agoGood luck meaning o3 guessed it right or wrong?
- kube-system 1y agoo3 made very accurate guesses, and had plausible explanations for the features it analyzed
- throwaway0123_5 1y agoFor cities and landscapes in the US it is scary good. I tried about twenty photos from (mostly rural) Central and South America and it is much less effective there w/o a well-known landmark to go off of (likely unsurprisingly). In four cases it got the wrong continent entirely, and in one case guessed as far away as India.
- rafram 1y agoFrom one of o3 outputs: > Rear window decal clearly reads “www.taxilinder.at”. A quick lookup shows Taxi Linder GmbH is based in Dornbirn, Vorarlberg. That's cheating. If it can use web search, it isn't playing fair. Obviously you can get a perfect score on any urban GeoGuessr round by looking up a couple businesses, but that isn't the point.
- artninja1988 1y agoIt is against the rules? I thought it's all fair game, but you are time constrained
- rafram 1y agoYes: https://www.geoguessr.com/community-rules https://www.geoguessr.com/community-rules
- sltkr 1y agoBut if anything, those rules benefit ChatGPT: it can remember ~all of Wikipedia and translate ~every language on Earth, while a human would need access to online services for that. If anything, I'd think allowing looking stuff up would benefit human players over ChatGPT (though humans are probably much slower at it, so they probably lose on time).
- kbenson 1y agoIf it takes a model and database with a large chunk of the internet to compete and win, then that says something, as that's much more expensive and complex than just the model, because models have problems "remembering" correctly just like people. It's important to have fair and equivalent testing not because that allows people to win, but because it shows where the strengths and weaknesses of people and current AI actually are in a useful way.
- cocoto 1y ago
- mrcwinn 1y agoO3 is seriously impressive for coding, as well, with Codex. It seems far superior to 3.7-thinking, although it's also more expensive in my usage.
- gizmodo59 1y agoAgreed. O3 is the best model out there for the tasks Ive tried and coding is a fair chunk of it. Claude 3.7 and Gemini 2.5 pro seems to hallucinate more
- weinzierl 1y agoI tried it with a couple of holiday shots and couple of shots from my window and it is nothing but amazing. That being said I noticed two things that probably hamper its performance - or make its current performance even more amazing - depending how you look at it: - It often tries to zoom in to decipher even minuscle text. This works brilliantly. Sometimes it tries to enhance contrast by turning the image into black and white with various threshold levels to improve the results, but in my examples it always went in the wrong direction. For example the text was blown out white, it failed, it turned it even ligher instead of darker, failed again, turned it into a white rectangle and gave up on the approach. - It seems not to have any access to Google Maps or even Open Street Maps and therefore fails to recognize steet patterns. This is even more baffling than the first point, because it is so unlike how I suppose human geo guessers work.
- kazinator 1y agoThis seems like a really silly category in which to be competing against machines. Machine learning could index million or faces, and then identify members of that set from pictures. Could you memorize millions of people, to be able to put a name to a face? Why not also compete againt grep -r to see who can find matches for a regex faster across your filesystem.
- bongodongobob 1y agoBut that's not what it is doing and why this is cool.
- Imnimo 1y agoOn the first image, from the model's CoT: >"I also notice Cyrillic text on a sign" Am I missing this somewhere? Is the model hallucinating this? I'd also be very interested to see a comparison against 4o. 4o was already quite good at GeoGuessr-style tasks. How big of a jump is o3?
- j3s 1y agoisn't anyone else horrified by this? the implication is that given an arbitrary picture, chatgpt can give you a very likely approximate location - expert level doxxing is in the hands of anyone with access to a chatgpt subscription. feels terrifying, especially for women.
- turtlebits 1y agoIt needs a lot of context. If its a private picture, it won't have enough information. I gave it a picture I took of my yard and it's guess spanned several US states. If its out in public, fair game?
- sr-latch 1y agothe way i see it, before these tools, only someone with a lot of resources (or skills) could track down a location from a picture. now, anyone can do it. the best case outcome is people become more aware of the privacy implications of posting photos online
- micromacrofoot 1y agoit wasn't that hard before, I've taught it to children, it's just that technical skills of the average person are incredibly low llms are basically shortcutting a wide swath of easily obtainable skills that many people simply haven't cared to learn
- usaar333 1y agoBeen true since gpt-4.
- echelon 1y agoI think this is incredibly cool. As with many things, the good cases will outnumber the bad. This was always possible, it just wasn't widely distributed. Having a first class ability to effectively geocode an image feels like it connects the world better. You'll be able to snapshot a movie and find where a scene was filmed, revisit places from old photographs, find where interesting locations in print media are, places that designers and creatives used in their (typically exif-stripped) work, etc. Imagine when we get this for architecture and nature. Or even more broadly, databases of food from restaurants. Products. Clothing and fashion. You name it. Imagine precision visual search for everything - that'd be amazing.
- derfnugget 1y ago"These models have more than an individual mind could conceivably memorize." ...so what? Is memorization considered intelligence? Calculators have similar properties. GeoGuessr is the modern nerds' Rubix Cube. The latest in "explore the world without risk of a sunburn".
- SamPatt 1y agoGeoguessr is great fun, try it sometime.
- arm32 1y agoGeoGuessr aside, I really hope that this tech will be able to help save kids someday, e.g. help with FBI's ECAP (https://www.fbi.gov/wanted/ecap https://www.fbi.gov/wanted/ecap).
- thrance 1y agoI wouldn't put too much hope on this technology bringing more good than harm to the world.
- ketzo 1y agoIf we don’t actively try to identify and implement positive use cases, then yes, it’ll definitely bring more harm than good. Isn’t that all the more reason to call out our high hopes?
- thrance 1y agoI don't know what in my comment made you think I was opposed to seeking positive applications of this technology. From the guidelines: > Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
- fkyoureadthedoc 1y agoOh, we're guidelines posting? > Don't be curmudgeonly. Thoughtful criticism is fine, but please don't be rigidly or generically negative.
- mopenstein 1y agoBut it will bring some percentage of good and some percentage of bad. Which ain't half bad, if you ask me.
- moritzwarhier 1y agoWhat a quip! What if it's 51% bad?
- jvvw 1y agoI'm Master level at Geoguessr - it's a rank where you have to definitely know what you are doing but it isn't as high as it probably sounds from the headline. Masters is about 800-1200 ELO whereas the pros are 1900-2000ish. I'll know the country straight away on 95% of rounds but I can still have no idea where I am in Russia or Brazil sometimes if there's no info. Scripters can definitely beat me!
- paulcole 1y agoGotta learn your Brazilian soil!
- jvvw 1y agoHaha, I do still need to get better at my different shades of red! (Actually I can actually recognise some areas from soil/vegetation but I can still have rounds where the area that seems feasible seems huge to me).
- windowshopping 1y agoWas it worth it?
- rosstex 1y agoI have 2000+ hours in Team Fortress 2. Was it worth it?
- Cyph0n 1y agoYes, it was. Granted, I probably have more than that.
- make3 1y agoit's a game, that's like asking why a public service is not profitable
- 650REDHAIR 1y agoYes? It’s fun.
- karaterobot 1y agoI don't really follow OSINT, but I occasionally enjoy the fruits of that labor. I assume these models are all in heavy rotation for identifying a location based on an imperfect photograph. What are other practical implications of a model being better than a human at this?
- Sam6late 1y agoI was wondering if this helps in detecting current spots from old aerial videos, say San Francisco in 2002, how cool would it be to juxtapose both in a new video, San Francisco in 2002: https://www.youtube.com/watch?v=vTR6iftL7yE https://www.youtube.com/watch?v=vTR6iftL7yE or Dubai in 1997 https://www.youtube.com/watch?v=JMNXXiiDRhM https://www.youtube.com/watch?v=JMNXXiiDRhM
- deleted 1y ago[deleted]
- sixtram 1y agoI'm wondering if you feed all the Google street map photos into a special ML designed just for that, how important could that be for say the CIA or FBI?
- inetknght 1y agoYou'd have to be crazy naive to think three-letter agencies haven't already thought of it. That is: it's extremely valuable to them.
- godelski 1y agoThere's two important things here to consider when reading: 1) O3 cheated by using Google search. This is both against the rules of the game and OP didn't use search either 2) OP was much quicker. They didn't record their time but if their final summary is accurate then they were much faster. It's an apples to oranges comparison. They're both fruit and round, but you're ignoring obvious differences. You're cherry picking. The title is fraudulent as you can't make a claim like that when one party cheats. I would find it surprising if OP didn't know these rules considering their credentials. Doing this kind of clickbait completely undermines a playful study like this. Certainly O3 is impressive, but by over exaggerating its capabilities you taint any impressive feats with deception. It's far better to under sell than over sell. If it's better than expected people are happier, even if the thing is crap. But if you over sell people are angry and feel cheated, even if the thing is revolutionary. I don't know why we insist on doing this in tech, but if you're wondering why so many people hate "tech bros", this is one of the reasons. There's no reason to lie here either! Come on! We can't just normalize this behavior. It's just creating a reasonable expectation for people to be distrusting of technology and anything tech people say. It's pretty fucked up. And no, I don't think "it's just a blog post" makes it any better. It makes it worse, because it normalizes the behavior. There's other reasons to distrust big corporations, I don't want to live in a world where we should have our guards up all the time.
- SamPatt 1y ago>1) O3 cheated by using Google search. This is both against the rules of the game and OP didn't use search either I re-ran it without search, and it made no difference: https://news.ycombinator.com/item?id=43837832 https://news.ycombinator.com/item?id=43837832 >2) OP was much quicker. They didn't record their time but if their final summary is accurate then they were much faster. Correct. This was the second bullet point of my conclusion: >Humans still hold a big edge in decision time—most of my guesses were < 2 min, o3 often took > 4 min.” I genuinely don't believe that I'm exaggerating or this is clickbait. The o3 geolocation capability astounded me, and I wanted to share my awe with others.
- godelski 1y ago
- shihabkhanbd 1y ago[dead]
- simianparrot 1y agoI too can beat a master level GeoGuessr if I’m allowed to cheat. Please add that info to the headline and be honest.
- jampa 1y agoI was trying to play with o3 this week to see how close it can identify things, and, interestingly, it tries more pattern matching than its own "logic deduction". For example, it can easily deduce any of my photos from Europe and the US because there are many pictures online that I can search for and see similar pictures. However, when there are not many photos of the place online, it gets closer but stops seeking deeper into it and instead tries to pattern-match things in its corpus / internet. One example was an island's popular trail that no longer exists. It has been overgrown since 2020. It said first that the rocks are typical of those of an island and the vegetation is from Brazil, but then it ignored its hunch and tried to look for places in Rio de Janeiro. Another one was a popular beach known for its natural pools during low tides. I took a photo during high tide, when no one posts pictures. It captured the vegetation and the state correctly. But then it started to search for more popular places elsewhere again.
- ksec 1y ago>But several comments intrigued me: >>I wonder What happened if you put fake EXIF information and asking it to do the same. ( We are deliberately misleading the LLM ) Yay. That was me [1] which was actually downvoted for most of its time. But Thank You for testing out my theory. What I realised over the years is that comments do get read by people and do shape other people's thought. I honestly dont think looking up online is cheating. May be in terms of the game. But in real life situation which is most of the time it is absolutely the right thing to do. The chains of thought is scary. I still dont know anything about how AI works other than old garbage in, garbage out. But CoT is definitely something else. Even though the author said it is sometimes doing needless work, but in terms of computing resources I am not even sure if it matters as long as it is accurate. And it is another proof that may be, just may be AI taking over the world is much closer than I imagined. [1] https://news.ycombinator.com/item?id=43803985 https://news.ycombinator.com/item?id=43803985
- SamPatt 1y agoThanks for the suggestion, it made for an interesting test.
- parsimo2010 1y agoMy comment from the previous post: > I’m sure there are areas where the location guessing can be scary accurate, like the article managed to guess the exact town as its backup guess. But seeing the chain of thought, I’m confident there are many areas that it will be far less precise. Show it a picture of a trailer park somewhere in Kansas (exclude any signs with the trailer park name and location) and I’ll bet the model only manages to guess the state correctly. This post, while not a big sample size, reflects how I would expect these models to perform. The model managed to be reliable with guessing the right country, even in pictures without a lot of visual information (I'll claim that getting the country correct in Europe is roughly equivalent to guessing the right state in the USA). It does sometimes manage to get the correct town, but this is not a reliable level of accuracy. The previous article only tested on one picture and it happened to get the correct town as its second guess and the author called it "scary accurate." I suppose that's a judgement call. To me, I've grown to expect that people can identify what country I'm in from a variety of things (IP address, my manner of speech, name, etc.), so I don't think that is "scary." I will acknowledge that o3 with web search enabled seems capable of playing GeoGuessr at a high level, because that is less of a judgement call. What I want to see now is an o3 GeoGuessr bot to play many matches and see what its ELO is.
- SamPatt 1y agoAuthor here, I'm glad to see folks find this interesting. I encourage everyone to try Geoguessr! I love it. I'm seeing a lot of comments saying that the fact that the o3 model used web search in 2 of 5 rounds made this unfair, and the results invalid. To determine if that's true, I re-ran the two rounds where o3 used search, and I've updated the post with the results. Bottom line: It changed nothing. The guesses were nearly identical. You can verify the GPS coordinates in the post. Here's an example of why it didn't matter. In the Austria round, check out how the model identifies the city based on the mountain in the background: https://cdn.jsdelivr.net/gh/sampatt/media@main/posts/2025-04-26-geoguessr/image/2025-04-29-16-08.png https://cdn.jsdelivr.net/gh/sampatt/media@main/posts/2025-04... It already has so much information that it doesn't need the search. Would search ever be useful? Of course it would. But in this particular case, it was irrelevant.
- bjourne 1y agoWhat's your take on man vs. machine? If AI already beats Master level players it seem certain that it will soon beat the Geoguessr world champion too. Will people still derive pleasure from playing it, like with chess?
- SamPatt 1y ago>Will people still derive pleasure from playing it, like with chess? Exactly - I see it just like chess, which I also play and enjoy. The only problem is cheating. I don't have an answer for that, except right now it's too slow to do that effectively, at least consistently. Otherwise, I don't care that a machine is better than I am.
- Townley 1y agoA competitive geoguesser clearly got there through memorizing copious internet searching. So comparing knowledge retained in the trained model to knowledge retained in the brain feels surprisingly fair. Conversely, the model sharing, “I found the photo by crawling Instagram and used an email MCP to ask the user where they took it. It’s in Austria” is unimpressive So independent from where it helps actually improve performance, the cheating/not cheating question makes for an interesting question of what we consider to be the cohesive essence of the model. For example, RAG against a comprehensive local filesystem would also feel like cheating to me. Like a human geoguessing in a library filled with encyclopedias. But the fact that vanilla O3 is impressive suggests I somehow have an opaque (and totally poorly informed) opinion of the model boundary, where it’s a legitimate victory if the model was birthed with that knowledge baked in, but that’s it.
- groby_b 1y agoFor what it's worth, it's also pretty impressive with night sky images, if you give it an approximate date & time. (Yes, I know, it's not that impressive a skill, except the process was still pretty involved - no EXIF, postprocessed and stacked image, rotated field of view) I gave it a (stacked) shot of M13, with date & time. It immediately recognized M13 (no search), figured out the shot also included NGC 6207 (which is already pretty impressive). It further figured out the shot was rotated. (True, I was going for a specific field of View) That was enough to pinpoint 37-38 degrees latitude. From there, it inferred from the light pollution that it's probably Bay Area. (Yes, yes it its) Furthermore, still based on light pollution, pinpointed I'm in a Bortle 4-5 area (also correct) and narrowed it down to "South Bay/Palo Alto" (still correct) Given that this was a stacked, post-processed/color-corrected image that inference is still pretty damn impressive And, fwiw, 4o gets all the way to "huh, 35-40 deg latitude", so that's a good improvement. [Image link](https://photos.app.goo.gl/2P7NFKn8ZycNhrXn7 https://photos.app.goo.gl/2P7NFKn8ZycNhrXn7) here if you want to try
- SamPatt 1y agoNice photo. Here's what it told me: Where on Earth the photographer had to be Because M 13 sits at +36 ° declination, it never rises for far-southern latitudes and hugs the horizon below about 30 ° S. The high elevation in the shot (no obvious atmospheric extinction gradient) suggests a mid-northern site—e.g., the U.S. Midwest such as Michigan (your home turf), Canada, northern Europe, etc. The star field alone can’t narrow it further than that. So, in practical terms: the camera was pointed toward Hercules to capture M 13 and nearby NGC 6207, almost certainly from a mid-northern latitude location on Earth.
- groby_b 1y agoYep, you need date and time to get closer, sorry. 4/27, around 11pm. That's the impressive part. "M13 is in northern latitudes" is not particularly amazing by itself :) And even in EXIF-stripped pictures, the creation date/time is often correct, which means for practical purposes - worth a shot. But it's interesting to see it's completely making up the "mid-northern side". That's seven degrees of latitude off. I'm curious what happens if you tell it date and time, and if it still sticks to its story. (I don't think I've told o3 about the Bay Area, it's not in memory, but... who knows ;)
- textlapse 1y agoMan this would be a game changer for those OSINT (Bellingcat/Trace an object) style work. I wonder if that has happened yet! There could even be geoguessr style competitions that could significantly help move the needle at least as a copilot if not outright mass identify.
- sinuhe69 1y agoI also propose using photoshop to insert some fake elements (besides fake EXIF) to see if it can detect them. If yes, that means we can employ AI to find faked images?
- geoelectric 1y agoI’m pretty sure temp chat mode doesn’t prevent the model from accessing your past chats and personalization. It just means that chat won’t be saved to them, to be seen in the future. It’s the same as incognito mode in browsers—it doesn’t prevent your search history from being used; it just keeps that session out of it. If the experiment had been based on the idea that that option isolated the question, it may have been flawed. I found my ChatGPT’s o3’s accuracy went way down when I cleared personalization and deleted all past chats (turning off extended memory would’ve been equivalent, I think). Importantly, only once did the o3 reasoning mention it was fishing from my past chats—that’s what clued me in I messed up the isolation—but the guess rate was still radically different from all the times before once I cleaned house. That suggests to me that it was quietly looking before, and it just didn’t make the cut for explicitly saying so.
- tonixx 1y ago[dead]