3 ms·
1. The "master geoguesser" is a bit misleading--as mentioned in his blog post, there are players far better than him, and he is certainly not the bar for human
by julianhuang 1y ago
1. The "master geoguesser" is a bit misleading--as mentioned in his blog post, there are players far better than him, and he is certainly not the bar for human supremacy. Probably analogous to a 1400-1800 elo chess player.
2. o3 isn't the best model at playing GeoGuessr, Gemini 1.5 & 2.5 solidly beat it out--for those interested, check out my friend's benchmark (https://geobench.org/ https://geobench.org/) and blog post (https://ccmdi.com/blog/GeoBench https://ccmdi.com/blog/GeoBench) detailing interesting model explanations.
3. In the post, he only tests on one game--o3's average score over 100 locations (20 5-location games) was 19,290, far lower than the 23,179 in the game.
Model geolocation capabilities are really important to keep track of, but the specific blog post in question isn't anything out of the ordinary. LLMs are making geolocation abilities much more accessible, but still fall short compared to 1. top GeoGuessr players playing GeoGuessr (only google streetview coverage, without web search) and 2. professional geolocators, who are proficient at using a wide variety of software/search. I.e., if the CIA wanted to find someone using an image, LLMs would not provide them any unique ability to do so as opposed to someone like Rainbolt
- kenjackson 1y agoFrom your linked article: "It's clear that large language models have an emergent capability to play GeoGuessr, and play it well. Their abilities to generalize are nascent, but present" This is very accurate -- their abilities to generalize are nascent, but still surprisingly capable. The world is about to send through its best and brightest math/CS minds over the next decade (at least) to increase the capabilities of these AIs (with the help of AI). I just don't understand the pessimism with the technology.
- julianhuang 1y agoI completely agree that this is an incredible advancement as someone who has watched the rise of LLMs' GeoGuessr abilities. I just wanted to qualify the claim made in the blog post "In Which I Try to Maintain Human Supremacy for a Bit Longer". I also think that models would need to become far more proficient at tool use (querying OpenStreetMap features, coverage-checking Google Maps, calculating sun direction, etc.) that current human geolocators have access to, and precise spatial reasoning. Additionally, there is a whole corpus of GeoGuessr-specific knowledge that probably wouldn't have a reason to be in model training data (Google Street View coverage quirks, the specific geographic ranges of vegetation and architecture, tiny visual idiosyncrasies in country infrastructure like the width of red on a sign, etc.). However, I think this could probably be solved with more data, and I don't think there is any fundamental barrier.
- fmbb 1y agoHow fast are they compared to human players?
- SamPatt 1y agoI'm the author - I tried to be as upfront as possible about my skill level in the post. The human supremacy line is just a joke, there are already models specifically trained for Geoguessr which are already beating the best players in the world, so that ship has sailed. That geobench work is really cool, thanks for sharing it.