3 ms·

I do understand the skepticism, and I'll run it again without search to see what happens. But a serious question for you: what would you need to see in order t

I do understand the skepticism, and I'll run it again without search to see what happens. But a serious question for you: what would you need to see in order to be properly impressed? I ask because I made this post largely to push back on the idea that EXIF data matters and the models aren't that capable. Now the criticism moves to web search, even though it only mattered in one out of five rounds. What would impress you?

mattmanser 1y ago
You're kinda being your own worse enemy though. "Technically cheating"? Why even add the "technically". It just gives the impression that you're not really objectively looking for any smoke and mirrors by the AI.
- SamPatt 1y ago
  I hear you - but I had already read through the chain of thought which identified the right region before search, and had already seen the capabilities in many other rounds. It was self-evident to me that the search wasn't an essential part of the model's capabilities by that point. Which turned out to be true - I re-ran both of those rounds, without search this time, and the model's guesses were nearly identical. I updated the post with those details. I feel like I did enough to prove that o3's geolocation abilities aren't smoke and mirrors, and I tried to be very transparent about it all too. Do you disagree? What more could I do to show this objectively?
godelski 1y ago
> What would impress you? I want to be clear that you tainted the capacity to impress me by the clickbait title. I don't think it was through malice, but I hope you realize the title is deceptive.[0] (Even though I use strong language, I do want to clarify I don't think it is malice) To paraphrase from my comment: if you oversell and under deliver, people feel cheated, even if the deliverable is revolutionary. So I think you might have the wrong framing to achieve this goal. I am actually a bit impressed by O3's capabilities. But at the same time you set the bar high and didn't go over or meet it. So that's going to really hinder the ability to impress. On the other hand, you set the bar low, it usually becomes easy to. It i slike when you have low expectations for a movie and it's mediocre you still feel good, right? [0] https://news.ycombinator.com/item?id=43836791 https://news.ycombinator.com/item?id=43836791

mattmanser 1y ago

You're kinda being your own worse enemy though. "Technically cheating"? Why even add the "technically". It just gives the impression that you're not really objectively looking for any smoke and mirrors by the AI.

SamPatt 1y ago
I hear you - but I had already read through the chain of thought which identified the right region before search, and had already seen the capabilities in many other rounds. It was self-evident to me that the search wasn't an essential part of the model's capabilities by that point. Which turned out to be true - I re-ran both of those rounds, without search this time, and the model's guesses were nearly identical. I updated the post with those details. I feel like I did enough to prove that o3's geolocation abilities aren't smoke and mirrors, and I tried to be very transparent about it all too. Do you disagree? What more could I do to show this objectively?

godelski 1y ago

> What would impress you? I want to be clear that you tainted the capacity to impress me by the clickbait title. I don't think it was through malice, but I hope you realize the title is deceptive.[0] (Even though I use strong language, I do want to clarify I don't think it is malice) To paraphrase from my comment: if you oversell and under deliver, people feel cheated, even if the deliverable is revolutionary. So I think you might have the wrong framing to achieve this goal. I am actually a bit impressed by O3's capabilities. But at the same time you set the bar high and didn't go over or meet it. So that's going to really hinder the ability to impress. On the other hand, you set the bar low, it usually becomes easy to. It i slike when you have low expectations for a movie and it's mediocre you still feel good, right? [0] https://news.ycombinator.com/item?id=43836791 https://news.ycombinator.com/item?id=43836791