4 ms·

Did you include location metadata with the photos by chance? I’m pretty surprised by these results.

by bobro 1y ago

SamPatt 1y ago
No, I took screenshots to ensure it. Your skepticism is warranted though - I was a part of an AI safety fellowship last year and our project was creating a benchmark for how good AI models are at geolocation from images. [This is where my Geoguessr obsession started!] Our first run showed results that seemed way too good; even the bad open source models were nailing some difficult locations, and at small resolutions too. It turned out that the pipeline we were using to get images was including location data in the filename, and the models were using that information. Oops. The models have improved very quickly since then. I assume the added reasoning is a major factor.
vessenes 1y ago
A) o3 is remarkably good, better than benchmarks seem to indicate in many circumstances B) it definitely cheats when it can — see this chat where it cheated by extracting EXIF data and wasn’t ashamed when I complained about it cheating: https://chatgpt.com/share/6802e229-c6a0-800f-898a-44171a0c7de4 https://chatgpt.com/share/6802e229-c6a0-800f-898a-44171a0c7d...
SamPatt 1y ago
As a further test, I dropped the street view marker on a random point in the US, near Wichita, Kansas, here's the image: https://cdn.jsdelivr.net/gh/sampatt/media@main/posts/2025-04-26-geoguessr/image/2025-04-26-13-05.png https://cdn.jsdelivr.net/gh/sampatt/media@main/posts/2025-04... I fed it o3, here's the response: https://cdn.jsdelivr.net/gh/sampatt/media@main/posts/2025-04-26-geoguessr/image/2025-04-26-13-04.png https://cdn.jsdelivr.net/gh/sampatt/media@main/posts/2025-04... Nailed it. There's no metadata there, and the reasoning it outputs makes perfect sense. I have no doubt it'll be tricky when it can be, but I can't see a way for it to cheat here.
- tylersmith 1y ago
  This is right by where I grew up and the broadcast tower and turnpike sign were the first two things I noticed too, but the ability to realize it was the East side instead of the West side because the tower platforms are lower is impressive.
  - SamPatt 1y ago
    Oh hey Tyler, nice to see you on HN :) Yeah it's an impressive result.

Did you include location metadata with the photos by chance? I’m pretty surprised by these results.

by bobro 1y ago

Did you include location metadata with the photos by chance? I’m pretty surprised by these results.

No, I took screenshots to ensure it. Your skepticism is warranted though - I was a part of an AI safety fellowship last year and our project was creating a benchmark for how good AI models are at geolocation from images. [This is where my Geoguessr obsession started!] Our first run showed results that seemed way too good; even the bad open source models were nailing some difficult locations, and at small resolutions too. It turned out that the pipeline we were using to get images was including location data in the filename, and the models were using that information. Oops. The models have improved very quickly since then. I assume the added reasoning is a major factor.

vessenes 1y ago

A) o3 is remarkably good, better than benchmarks seem to indicate in many circumstances B) it definitely cheats when it can — see this chat where it cheated by extracting EXIF data and wasn’t ashamed when I complained about it cheating: https://chatgpt.com/share/6802e229-c6a0-800f-898a-44171a0c7de4 https://chatgpt.com/share/6802e229-c6a0-800f-898a-44171a0c7d...

SamPatt 1y ago

As a further test, I dropped the street view marker on a random point in the US, near Wichita, Kansas, here's the image: https://cdn.jsdelivr.net/gh/sampatt/media@main/posts/2025-04-26-geoguessr/image/2025-04-26-13-05.png https://cdn.jsdelivr.net/gh/sampatt/media@main/posts/2025-04... I fed it o3, here's the response: https://cdn.jsdelivr.net/gh/sampatt/media@main/posts/2025-04-26-geoguessr/image/2025-04-26-13-04.png https://cdn.jsdelivr.net/gh/sampatt/media@main/posts/2025-04... Nailed it. There's no metadata there, and the reasoning it outputs makes perfect sense. I have no doubt it'll be tricky when it can be, but I can't see a way for it to cheat here.

tylersmith 1y ago
This is right by where I grew up and the broadcast tower and turnpike sign were the first two things I noticed too, but the ability to realize it was the East side instead of the West side because the tower platforms are lower is impressive.
- SamPatt 1y ago
  Oh hey Tyler, nice to see you on HN :) Yeah it's an impressive result.