8 ms·

From one of o3 outputs: > Rear window decal clearly reads “www.taxilinder.at”. A quick lookup shows Taxi Linder GmbH is based in Dornbirn, Vorarlberg. That's

by rafram 1y ago

From one of o3 outputs: > Rear window decal clearly reads “www.taxilinder.at”. A quick lookup shows Taxi Linder GmbH is based in Dornbirn, Vorarlberg. That's cheating. If it can use web search, it isn't playing fair. Obviously you can get a perfect score on any urban GeoGuessr round by looking up a couple businesses, but that isn't the point.

artninja1988 1y ago
It is against the rules? I thought it's all fair game, but you are time constrained
- rafram 1y ago
  Yes: https://www.geoguessr.com/community-rules https://www.geoguessr.com/community-rules
  - sltkr 1y ago
    But if anything, those rules benefit ChatGPT: it can remember ~all of Wikipedia and translate ~every language on Earth, while a human would need access to online services for that. If anything, I'd think allowing looking stuff up would benefit human players over ChatGPT (though humans are probably much slower at it, so they probably lose on time).
    - kbenson 1y ago
      If it takes a model and database with a large chunk of the internet to compete and win, then that says something, as that's much more expensive and complex than just the model, because models have problems "remembering" correctly just like people. It's important to have fair and equivalent testing not because that allows people to win, but because it shows where the strengths and weaknesses of people and current AI actually are in a useful way.
      - cocoto 1y ago
        Connecting an LLM to the web or database is something cheap, not something expensive.
        kbenson 1y ago
        I'm not sure how to make sense of this in the context of what we're discussing. Access to the web is exactly what's in question, and emulating the internet to a degree you don't actually need to access it to have the information is very expensive in resources because of how massive the dataset is, which is the point I was making.
        Gud 1y ago
        Same with a human.
- twojacobtwo 1y ago
  Why was this downvoted? It's a fair question and it wasn't stated as fact.
  - 3np 1y ago
    Because an accepted answer to that specific question is invariably a link/reference that the asker could have searched for (and posted if they think it's useful for the discussion) themselves directly, instead of putting that burden on the rest of us and amortizing everyone's attention. It's entitled and lazy. Alternative example: "I wondered what the rules actually say about web search and it is indeed not allowed: (link)"
silveraxe93 1y ago
Yeah, the author does note that in the article. He also points it out in the conclusion: > If it’s using other information to arrive at the guess, then it’s not metadata from the files, but instead web search. It seems likely that in the Austria round, the web search was meaningful, since it mentioned the website named the town itself. It appeared less meaningful in the Ireland round. It was still very capable in the rounds without search.
- rafram 1y ago
  Seems like they should've just repeated the test. But without the huge point lead from the rounds where it cheated, it wouldn't have looked very impressive at all.
  - silveraxe93 1y ago
    People found the original post so impressive they were saying that it had to be coming from cheating by looking at EXIF data. The point of this article was to show it doesn't. It got an unfair advantage in 1 (and say 0.5) out of 5. With the non-search rounds still doing great. If you think this is unimpressive, that's subjective so you're entitled to believe that. I think that's awesome.
    - godelski 1y ago
      Sorry, I think I misread you. I think you said People accused it of cheating by reading EXIF data. They were wrong, it cheated by using web search. That makes the people that accused it of cheating wrong and this post proves that. And is everyone forgetting that what OpenAI shows you during the CoT is not the full CoT? I don't think you can fully rely on that to make claims about when it did and didn't search
  - SamPatt 1y ago
    That's inaccurate. It beat me by 1,100 points, and given the chain of thought demonstrated that it knew the general region of both guesses before it employed search, it would likely have still beaten me in those rounds. Though probably by fewer points. I will try it again without web search and update the post though. Still, if you read the chain of thought, it demonstrates remarkable capabilities in all the rounds. It only used search in 2/5 rounds.
Ukv 1y ago
The author did specifically point out that > Using Google during rounds is technically cheating - I’m unsure about visiting domains you find during the rounds though. It certainly violates the spirit of the game, but it also shows the models are smart enough to use whatever information they can to win. and had noted in the methodology that > Browsing/tools — o3 had normal web access enabled. Still an interesting result - maybe more accurate to say O3+Search beats a human, but could also consider the search index/cache to just be a part of the system being tested.
- spookie 1y ago
  A human can also use the same tools if it wasn't for the rules or fair play. They should've simply redone the test.
  - ceph_ 1y ago
    The AI should be forced to use the same rules as the human. Not the other way around. The AI shouldn't be using outside resources.
    - bscphil 1y ago
      I think that's part of the point they're making, hence "They should've simply redone the test."
    - voxic11 1y ago
      Another rule bans "using third-party software or scripts in order to gain an unfair advantage over other players." So is it even possible for O3 to beat another player while complying with the rules?
      - ben_w 1y ago
        If a player uses such a model, the model is third-party and the player is cheating. But: when a specific model is itself under test, I would say that during the test it becomes "first" (or second?) party rather than "third".
layman51 1y ago
Using the decal as a clue is funny because what if there was a street scene where that happened to be misleading? For example, I had seen that a Sacramento County Sheriff car got to Europe and I guess it now belonged to a member of the public who is driving it with the original decals still attached. I wonder how the LLM would reason if it sees the car as “out of place”.
- victorbjorklund 1y ago
  Probabilities. That could happen with anything. Someone could build a classic japanese house with a japanese garden in Hawaii. But Japan is probably a better guess if you see a japanese house with japanese fauna.
  - qiqitori 1y ago
    Flora, not fauna
    - victorbjorklund 1y ago
      True
- yen223 1y ago
  Stands to reason a human might get fooled by this as well
  - SamPatt 1y ago
    Absolutely! It happens occasionally - the most common example I can think of it getting a license plate or other location from a tractor-trailer (semi) on the highway. Those are very unreliable. You also sometimes get flags in the wrong countries, immigrants showing their native pride or even embassies.
SamPatt 1y ago
Author here - it's a fair criticism, and I point it out in the article. However, I kept it in for a few reasons. I'm trying to show the model's full capabilities for image location generally, not just playing geoguessr specifically. The ability to combine web search with image recognition, iteratively, is powerful. Also, the web search was only meaningful in the Austria round. It did use it in the Ireland round too, but as you can see by the search terms it used, it already knew the road solely from image recognition. It beat me in the Colombia round without search at all. It's worthwhile to do a proper apples and apples comparison - I'll run it again and update the post. But the point was to show how incredibly capable the model is generally, and the lack of search won't change that. Just read the chain of thought, it's incredible!
- k4rli 1y ago
  It's still as much cheating as googling. Completely irrelevant. Even if it were to beat Blinky, it's not different from googlers/scripters.
  - SamPatt 1y ago
    I disagree. I ran those rounds again, without search this time, and the results were nearly identical: https://news.ycombinator.com/item?id=43837832 https://news.ycombinator.com/item?id=43837832
- IanCal 1y ago
  I tried the image without search and it talked about Dornbirn anyway but ended up choosing Bezau which is really quite close. edit - the models are also at a disadvantage in a way too, they don't have a map to look at while the pick the location.
  - SamPatt 1y ago
    Yes, I re-ran those rounds and it made the same guesses without search, within 1km I believe. You're right about not having a map - I cannot imagine trying to line up the Ireland coast round without referencing the map.
WhitneyLand 1y ago
As models continue to evolve it may not even need to cheat. Since web scale data is already part of pre-training this info is in principle available for most businesses without a web search. The exceptions would be if it’s recently added, or doesn’t appear often enough to generate a significant signal during training, as in this case with a really small business. It’s not hard to imagine base model knowledge improving to the point where it’s still performing at almost the same level without any web search needed.
ricardo81 1y ago
>isn't playing fair. the idea of having nth more dimensions of information, readable and ingestible within a short frame of time probably isn't either.
- rowanG077 1y ago
  Isn't it? We totally accept this fact for humans. Some people are just insane at memorization. We don't call unfair that they use this to their advantage compared to someone who doesn't have that talent.
ACS_Solver 1y ago
I just tried (o4-mini-high) and had it come to the wrong conclusion when I asked about the location and date, because it didn't search the web. I have a photo of a bench with a sign mentioning the cancellation of an event due to the Pope's death. It impressively figured out the location but then decided that Pope Francis is alive and the sign is likely a prank, so the photo is from April Fools day. Then after I explicitly instructed it to search the web to confirm whether the Pope is alive, it found news of his death and corrected its answer, but it was interesting to see how the LLM makes a mistake due to a major recent event being after its cutoff.
- xp84 1y ago
  That is truly a macabre and wild idea of an April Fools prank. o4, you’re a monster.
CamperBob2 1y ago
To be fair, my local copy of R1 isn't doing any searching at all, but it frequently says "A search suggests..." or something along the lines.
ycombinatrix 1y ago
the title really doesn't suit the content of the article
sReinwald 1y ago
While technically you're right about the rules of GeoGuessr as a game, I think this misses the more interesting and potentially concerning implications. IMO, whether or not it's "cheating" at GeoGuessr is less important than what this capability represents: an AI system that can extract specific details from images, determine what's significant, and use that information to precisely locate a photo. That's impressive regardless of method. The implications extend far beyond the game. Consider its potential use in automating OSINT tasks. An AI agent running o3 could analyze publicly posted photos (e.g., on Twitter, Instagram), identify locations from background details, and aggregate this data. Within an hour or two, this could build a detailed location profile (home, work, frequented places) for an individual - something that previously required considerable manual effort and expertise. While human analysis would likely still be needed, this significantly lowers the barrier for large-scale analysis. This ability to extract location clues from images - whether shop names, signs, or landscapes - has significant real-world implications, particularly for automating OSINT tasks. The potential for large-scale location analysis from public photos is the more profound point, regardless of whether one specific method breaks game rules. That capability is what deserves attention.
1212312123 1y ago
I think the main concern that anyone can upload your picture and locate your address, at a low cost.
mcv 1y ago
I use web search and still can't get a perfect score. Without it, I wouldn't be able to find anything. (I don't play competitively obviously; and just my free 3 tries per day.)

by rafram 1y ago

artninja1988 1y ago

It is against the rules? I thought it's all fair game, but you are time constrained

rafram 1y ago
Yes: https://www.geoguessr.com/community-rules https://www.geoguessr.com/community-rules
- sltkr 1y ago
  But if anything, those rules benefit ChatGPT: it can remember ~all of Wikipedia and translate ~every language on Earth, while a human would need access to online services for that. If anything, I'd think allowing looking stuff up would benefit human players over ChatGPT (though humans are probably much slower at it, so they probably lose on time).
  - kbenson 1y ago
    If it takes a model and database with a large chunk of the internet to compete and win, then that says something, as that's much more expensive and complex than just the model, because models have problems "remembering" correctly just like people. It's important to have fair and equivalent testing not because that allows people to win, but because it shows where the strengths and weaknesses of people and current AI actually are in a useful way.
    - cocoto 1y ago
      Connecting an LLM to the web or database is something cheap, not something expensive.
      - kbenson 1y ago
        I'm not sure how to make sense of this in the context of what we're discussing. Access to the web is exactly what's in question, and emulating the internet to a degree you don't actually need to access it to have the information is very expensive in resources because of how massive the dataset is, which is the point I was making.
      - Gud 1y ago
        Same with a human.
twojacobtwo 1y ago
Why was this downvoted? It's a fair question and it wasn't stated as fact.
- 3np 1y ago
  Because an accepted answer to that specific question is invariably a link/reference that the asker could have searched for (and posted if they think it's useful for the discussion) themselves directly, instead of putting that burden on the rest of us and amortizing everyone's attention. It's entitled and lazy. Alternative example: "I wondered what the rules actually say about web search and it is indeed not allowed: (link)"

silveraxe93 1y ago

Yeah, the author does note that in the article. He also points it out in the conclusion: > If it’s using other information to arrive at the guess, then it’s not metadata from the files, but instead web search. It seems likely that in the Austria round, the web search was meaningful, since it mentioned the website named the town itself. It appeared less meaningful in the Ireland round. It was still very capable in the rounds without search.

rafram 1y ago
Seems like they should've just repeated the test. But without the huge point lead from the rounds where it cheated, it wouldn't have looked very impressive at all.
- silveraxe93 1y ago
  People found the original post so impressive they were saying that it had to be coming from cheating by looking at EXIF data. The point of this article was to show it doesn't. It got an unfair advantage in 1 (and say 0.5) out of 5. With the non-search rounds still doing great. If you think this is unimpressive, that's subjective so you're entitled to believe that. I think that's awesome.
  - godelski 1y ago
    Sorry, I think I misread you. I think you said People accused it of cheating by reading EXIF data. They were wrong, it cheated by using web search. That makes the people that accused it of cheating wrong and this post proves that. And is everyone forgetting that what OpenAI shows you during the CoT is not the full CoT? I don't think you can fully rely on that to make claims about when it did and didn't search
- SamPatt 1y ago
  That's inaccurate. It beat me by 1,100 points, and given the chain of thought demonstrated that it knew the general region of both guesses before it employed search, it would likely have still beaten me in those rounds. Though probably by fewer points. I will try it again without web search and update the post though. Still, if you read the chain of thought, it demonstrates remarkable capabilities in all the rounds. It only used search in 2/5 rounds.

Ukv 1y ago

The author did specifically point out that > Using Google during rounds is technically cheating - I’m unsure about visiting domains you find during the rounds though. It certainly violates the spirit of the game, but it also shows the models are smart enough to use whatever information they can to win. and had noted in the methodology that > Browsing/tools — o3 had normal web access enabled. Still an interesting result - maybe more accurate to say O3+Search beats a human, but could also consider the search index/cache to just be a part of the system being tested.

spookie 1y ago
A human can also use the same tools if it wasn't for the rules or fair play. They should've simply redone the test.
- ceph_ 1y ago
  The AI should be forced to use the same rules as the human. Not the other way around. The AI shouldn't be using outside resources.
  - bscphil 1y ago
    I think that's part of the point they're making, hence "They should've simply redone the test."
  - voxic11 1y ago
    Another rule bans "using third-party software or scripts in order to gain an unfair advantage over other players." So is it even possible for O3 to beat another player while complying with the rules?
    - ben_w 1y ago
      If a player uses such a model, the model is third-party and the player is cheating. But: when a specific model is itself under test, I would say that during the test it becomes "first" (or second?) party rather than "third".

layman51 1y ago

Using the decal as a clue is funny because what if there was a street scene where that happened to be misleading? For example, I had seen that a Sacramento County Sheriff car got to Europe and I guess it now belonged to a member of the public who is driving it with the original decals still attached. I wonder how the LLM would reason if it sees the car as “out of place”.

victorbjorklund 1y ago
Probabilities. That could happen with anything. Someone could build a classic japanese house with a japanese garden in Hawaii. But Japan is probably a better guess if you see a japanese house with japanese fauna.
- qiqitori 1y ago
  Flora, not fauna
  - victorbjorklund 1y ago
    True
yen223 1y ago
Stands to reason a human might get fooled by this as well
- SamPatt 1y ago
  Absolutely! It happens occasionally - the most common example I can think of it getting a license plate or other location from a tractor-trailer (semi) on the highway. Those are very unreliable. You also sometimes get flags in the wrong countries, immigrants showing their native pride or even embassies.

SamPatt 1y ago

Author here - it's a fair criticism, and I point it out in the article. However, I kept it in for a few reasons. I'm trying to show the model's full capabilities for image location generally, not just playing geoguessr specifically. The ability to combine web search with image recognition, iteratively, is powerful. Also, the web search was only meaningful in the Austria round. It did use it in the Ireland round too, but as you can see by the search terms it used, it already knew the road solely from image recognition. It beat me in the Colombia round without search at all. It's worthwhile to do a proper apples and apples comparison - I'll run it again and update the post. But the point was to show how incredibly capable the model is generally, and the lack of search won't change that. Just read the chain of thought, it's incredible!

k4rli 1y ago
It's still as much cheating as googling. Completely irrelevant. Even if it were to beat Blinky, it's not different from googlers/scripters.
- SamPatt 1y ago
  I disagree. I ran those rounds again, without search this time, and the results were nearly identical: https://news.ycombinator.com/item?id=43837832 https://news.ycombinator.com/item?id=43837832
IanCal 1y ago
I tried the image without search and it talked about Dornbirn anyway but ended up choosing Bezau which is really quite close. edit - the models are also at a disadvantage in a way too, they don't have a map to look at while the pick the location.
- SamPatt 1y ago
  Yes, I re-ran those rounds and it made the same guesses without search, within 1km I believe. You're right about not having a map - I cannot imagine trying to line up the Ireland coast round without referencing the map.

WhitneyLand 1y ago

As models continue to evolve it may not even need to cheat. Since web scale data is already part of pre-training this info is in principle available for most businesses without a web search. The exceptions would be if it’s recently added, or doesn’t appear often enough to generate a significant signal during training, as in this case with a really small business. It’s not hard to imagine base model knowledge improving to the point where it’s still performing at almost the same level without any web search needed.

ricardo81 1y ago

>isn't playing fair. the idea of having nth more dimensions of information, readable and ingestible within a short frame of time probably isn't either.

rowanG077 1y ago
Isn't it? We totally accept this fact for humans. Some people are just insane at memorization. We don't call unfair that they use this to their advantage compared to someone who doesn't have that talent.

ACS_Solver 1y ago

I just tried (o4-mini-high) and had it come to the wrong conclusion when I asked about the location and date, because it didn't search the web. I have a photo of a bench with a sign mentioning the cancellation of an event due to the Pope's death. It impressively figured out the location but then decided that Pope Francis is alive and the sign is likely a prank, so the photo is from April Fools day. Then after I explicitly instructed it to search the web to confirm whether the Pope is alive, it found news of his death and corrected its answer, but it was interesting to see how the LLM makes a mistake due to a major recent event being after its cutoff.

xp84 1y ago
That is truly a macabre and wild idea of an April Fools prank. o4, you’re a monster.

CamperBob2 1y ago

To be fair, my local copy of R1 isn't doing any searching at all, but it frequently says "A search suggests..." or something along the lines.

ycombinatrix 1y ago

the title really doesn't suit the content of the article

sReinwald 1y ago

While technically you're right about the rules of GeoGuessr as a game, I think this misses the more interesting and potentially concerning implications. IMO, whether or not it's "cheating" at GeoGuessr is less important than what this capability represents: an AI system that can extract specific details from images, determine what's significant, and use that information to precisely locate a photo. That's impressive regardless of method. The implications extend far beyond the game. Consider its potential use in automating OSINT tasks. An AI agent running o3 could analyze publicly posted photos (e.g., on Twitter, Instagram), identify locations from background details, and aggregate this data. Within an hour or two, this could build a detailed location profile (home, work, frequented places) for an individual - something that previously required considerable manual effort and expertise. While human analysis would likely still be needed, this significantly lowers the barrier for large-scale analysis. This ability to extract location clues from images - whether shop names, signs, or landscapes - has significant real-world implications, particularly for automating OSINT tasks. The potential for large-scale location analysis from public photos is the more profound point, regardless of whether one specific method breaks game rules. That capability is what deserves attention.

1212312123 1y ago

I think the main concern that anyone can upload your picture and locate your address, at a low cost.

mcv 1y ago

I use web search and still can't get a perfect score. Without it, I wouldn't be able to find anything. (I don't play competitively obviously; and just my free 3 tries per day.)