6 ms·
The author did specifically point out that > Using Google during rounds is technically cheating - I’m unsure about visiting domains you find during the rounds
by Ukv 1y ago
The author did specifically point out that
> Using Google during rounds is technically cheating - I’m unsure about visiting domains you find during the rounds though. It certainly violates the spirit of the game, but it also shows the models are smart enough to use whatever information they can to win.
and had noted in the methodology that
> Browsing/tools — o3 had normal web access enabled.
Still an interesting result - maybe more accurate to say O3+Search beats a human, but could also consider the search index/cache to just be a part of the system being tested.
- spookie 1y agoA human can also use the same tools if it wasn't for the rules or fair play. They should've simply redone the test.
- ceph_ 1y agoThe AI should be forced to use the same rules as the human. Not the other way around. The AI shouldn't be using outside resources.
- bscphil 1y agoI think that's part of the point they're making, hence "They should've simply redone the test."
- voxic11 1y agoAnother rule bans "using third-party software or scripts in order to gain an unfair advantage over other players." So is it even possible for O3 to beat another player while complying with the rules?
- ben_w 1y agoIf a player uses such a model, the model is third-party and the player is cheating. But: when a specific model is itself under test, I would say that during the test it becomes "first" (or second?) party rather than "third".
- voxic11 1y agoI think third-party here means not produced by GeoGuesser rather than not produced by the player.
- krferriter 1y agoAn AI being better than a human at doing a google search and then skimming a bunch of pages to find location-related terms isn't as interesting of a result.
- inerte 1y agoHow the heck is not? Computers are looking into screenshots and searching the internet to support their "thinking", that's amazing! Have we become so used to AI and what was impossible 6 months ago is shruggable today? I've being doing this MIND-Dash diet lately and it's amazing I can just take a picture of whatever (nutritional info / ingredients are perfect for that) and just ask if it fits my plan and it tells me back into what bucket it falls, with detailed breakdown of macros in support of some additional goals I have (muscle building for powerlifting). It's amazing! And it does in 2 minutes passively what it would take me 5-10 active search.
- godelski 1y agoIn the same way a calculator performing arithmetic faster than humans isn't impressive. The same way running regex over a million lines and the computer beating a human in search isn't impressive
- ludwik 1y agoNeither is impressive solely because we've gotten used to them. Both were mind-blowing back in the day. When it comes to AI - and LLMs in particular - there’s a large cohort of people who seem determined to jump straight from "impossible and will never happen in our lifetime" to "obvious and not impressive", without leaving any time to actually be impressed by the technological achievement. I find that pretty baffling.
- godelski 1y agoI agree, but without removing search you cannot decouple. Has it embedded a regex method and is just leveraging that? Or is it doing something more? Yes, even the regex is still impressive but it is less impressive that doing something more complicated and understanding context and more depth.
- arandomhuman 1y agoBut then they couldn't make a click bait title for the article.
- _heimdall 1y agoThis seems like a great example of why some are so concerned with AI alignment. The game rules were ambiguous and the LLM did what it needed to (and was allowed to) to win. It probably is against the spirit of the game to look things up online at all but no one thought to define that rule beforehand.
- School-Cotton 1y agoNo, the game rules aren't ambiguous. This is 100% unambiguously cheating. From the list of things that are definitely considered cheating in the rules: > using Google or other external sources of information as assistance during play. The contents of URLs found during play is clearly an external source of information.
- GaggiX 1y agoI believed the rules were not explained to the model so it does use what it can.
- misnome 1y agoThen you can 100% not claim it is “Playing” the game
- deleted 1y ago[deleted]
- ben_w 1y agoThat right there *is the alignment problem*. If I task an AI with "peace on earth" and the solution the AI comes up with is ripped from The X-Files* and it kills everyone, it isn't good enough to say "that's cheating" or "that's not what I meant". * https://en.wikipedia.org/wiki/Je_Souhaite https://en.wikipedia.org/wiki/Je_Souhaite
- GaggiX 1y agoIt's playing a game in which the rules are a bit ambiguous if not explained.
- godelski 1y agoPointing out that it is cheating doesn't excuse the lie in the headline. That just makes it bait and switch, a form of fraud. OP knew they were doing a bait and switch. I remember when we were all pissed about clickbait headlines because they were deceptive. Did we just stop caring?
- bahmboo 1y agoThe headline said the AI beat him, it did not say it beat him in a GeoGuessr game. The article clearly states what he did and why.
- deleted 1y ago[deleted]
- SecretDreams 1y agoWhat's your suggestion for an alternative headline?
- godelski 1y agoCan O3 Beat a Master-Level GeoGuessr? How Good is O3 at GeoGuessr? EXIF Does Not Explain O3's GeoGuessr's Performance O3 Plays GeoGuessr (EXIF Removed) But honestly, OP had the foresight to remove EXIF data and memory from O3 to reduce contamination. The goal of the blog post was to show that O3 wasn't cheating. So by including search, they undermine the whole point of the post. The problem really stems from the lack of foresight. Lack of misunderstanding the critiques they sought to address in the first place. A good engineer understands that when their users/customers/<whatever> makes a critique, that what the gripe is about may not be properly expressed. You have to interpret your users complaints. Here, the complaint was "cheating", not "EXIF" per se. The EXIF complaints were just a guess at the mechanism in which it was cheating. But the complaint was still about cheating.
- SamPatt 1y ago>The goal of the blog post was to show that O3 wasn't cheating. No, the goal of the post was to show that o3 has incredible geolocation abilities. It's through the lens of a Geoguessr player who has experience doing geolocation, and my perspective on whether the chain of thought is genuine or nonsense. In Simon's original post, people were claiming that o3 doesn't have those capabilities, and we were fooled by a chain of thought that was just rationalizing the EXIF data. It only had the _appearance_ of capability. The ability to perform web search doesn't undermine the claim that o3 has incredible geolocation abilities, because it still needs to have an underlying capability in order to know what to search. That's not true for simply reading EXIF data. This is the best way I knew to show that the models are doing something really neat. Disagreements over the exact wording of my blog post title seem to be missing the point.